Big data-based file analysis, storage and division mode and distributed storage system

Abstract

The invention relates to the technical field of data processing, and discloses a big data-based file analysis storage division mode and a distributed storage system.The method comprises the following steps: S1, file analysis: analyzing an input big data file, and extracting a logic structure, field content and an association relationship of the big data file; s2, data block generation: dividing an analysis result into data blocks, and generating size and relevance information of the data blocks; s3, association weight generation: establishing a weight relationship between the similarity between the data blocks and the joint access frequency; and S4, storage division optimization: based on the association weight, optimizing the distribution of the data blocks among the storage nodes. According to the method, distribution of the data blocks is optimized through the synergistic effect of the association weight generation module and the dynamic decision-making module, so that the data blocks with high association degree are preferentially distributed to the same storage node, cross-node data transmission is greatly reduced, a storage strategy can be adjusted in real time, and the storage efficiency is improved. Therefore, communication delay and network resource consumption caused by cross-node access are reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a file parsing storage partitioning method and a distributed storage system based on big data. Background Art

[0002] In the era of big data, distributed storage systems have been widely used in data storage and management scenarios due to their high availability, scalability and reliability. Existing distributed storage systems improve storage efficiency and access performance by splitting large files into multiple data blocks and distributing them on different nodes. For structured and unstructured files, current technologies usually use static parsing to extract field content, divide data according to data size or simple rules, and distribute data in combination with certain storage node load balancing mechanisms. These methods can better meet basic storage needs in static scenarios, especially in environments with fixed access patterns and low dynamic changes, and have certain practicality and effectiveness.

[0003] However, with the continuous expansion of data scale and the increasing complexity of access patterns, the existing technologies have gradually exposed their shortcomings in practical applications. The main problem is that file parsing cannot fully explore the logical structure and field association relationship of complex files, resulting in unreasonable data block division; data block distribution only considers static load balancing, and does not combine the access pattern to dynamically optimize the distribution strategy, which easily leads to high cross-node communication overhead and reduced access performance; in addition, in dynamic scenarios, traditional storage systems lack the ability to respond to changes in hot data and node status adjustments, and cannot achieve global optimization and long-term benefit maximization. Therefore, the existing technology still has a lot of room for improvement in parsing complex files, optimizing data block division and dynamic storage adjustment. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a file parsing storage division method and a distributed storage system based on big data, which solves the problems of high communication overhead, unbalanced storage load and insufficient dynamic adaptability caused by unreasonable data block division of big data files in distributed storage systems.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a file parsing storage partitioning method based on big data includes the following steps: S1. File parsing: Parse the input big data file to extract the logical structure, field content and the relationship between fields of the file; S2, data block generation: according to the analysis results, the file is divided into several data blocks, and the size and correlation information of each data block is generated; S3, association weight generation: construct association weight information for describing the similarity and joint access frequency between data blocks; S4. Storage partition optimization: Based on the above-mentioned association weight information, an optimization algorithm is used to determine the storage partition scheme of the data blocks, so that the distribution of data blocks among storage nodes can minimize cross-node communication overhead, balance storage node loads, and meet storage capacity limitations.

[0006] Preferably, the step S3 further comprises the following steps: S31, analyzing the data block content and calculating the similarity between the data blocks; S32, analyzing the access log and calculating the joint access frequency of the data block; S33. Generate association weights between data blocks based on similarity and joint access frequency.

[0007] Preferably, the step S4 further comprises the following steps: S41, dynamically modeling the capacity of storage nodes and the distribution of data blocks; S42, establishing a multi-objective optimization model with the goal of optimizing communication overhead, load balancing and storage capacity; S43. Use dynamic game modeling to simulate the competition and cooperation relationship between storage nodes, and finally generate the globally optimal storage distribution plan.

[0008] Preferably, in the step S43, the dynamic game modeling is implemented in the following manner: S431, defining the global state of the storage system, including the current load and access mode of the storage nodes; S432. Design a set of storage node policies, including storing local data, migrating data to other nodes, or releasing cold data; S433. Dynamically adjust data block distribution according to storage node strategies to optimize overall communication overhead and load balancing.

[0009] Preferably, in the step S4, the optimization of the storage partitioning scheme is enhanced by a dynamic adjustment mechanism of long-term benefits, and the dynamic adjustment mechanism includes: S44, generating a new optimal storage strategy based on the current state of the system and historical decisions; S45, dynamically adjusting data block distribution in combination with changes in the load and access patterns of storage nodes; S46. Use reinforcement learning methods to optimize storage strategies to improve overall storage efficiency.

[0010] Preferably, in step S46, the reinforcement learning method is based on a deep neural network to model the state and strategy of the storage system, and the storage strategy is updated in real time through a proximal strategy optimization algorithm to adapt to dynamic changes in access patterns and node states.

[0011] The distributed storage system for file parsing based on big data includes: A file parsing module is used to parse the input big data file, divide the file into several data blocks, and extract the association relationship between the data blocks; A metadata management module, which is used to record the distribution location, size and associated weight information of data blocks; A dynamic decision module, which is used to generate the optimal storage partitioning strategy for data blocks based on the file parsing results and the current status of the storage nodes; The storage execution module is used to dynamically migrate data blocks and manage data in the distributed storage system in real time according to the optimal storage partitioning strategy.

[0012] Preferably, the dynamic decision module generates the optimal storage partitioning strategy based on the following technologies: (1) Dynamically update the storage distribution strategy by combining the association relationship of data blocks with the load status of storage nodes; (2) Design the long-term benefit optimization problem of storage nodes, taking into account communication overhead, load balancing, and storage capacity limitations; (3) Use reinforcement learning algorithms to optimize the storage distribution strategy and generate a continuously adjusted optimal strategy.

[0013] Preferably, the metadata management module includes the following functions: (1) Record the distribution information and associated weights of data blocks in storage nodes; (2) Update the distribution metadata of data blocks in real time when the storage node load or access pattern changes; (3) Provide metadata management capabilities to support data block migration and cleanup to ensure the system's dynamic adjustment capabilities.

[0014] Preferably, the storage execution module has fault tolerance capability, including the following functions: (1) When a storage node failure is detected, dynamic migration of data blocks is triggered; (2) When a node is overloaded, it automatically releases data blocks with low access frequency and adjusts the distribution plan; (3) Update the distribution information of data blocks through the metadata management module to ensure the high availability of the system.

[0015] The present invention provides a file parsing storage partitioning method and a distributed storage system based on big data. It has the following beneficial effects: 1. The present invention optimizes the distribution of data blocks through the synergy of association weight generation and dynamic decision modules, so that data blocks with high association are preferentially allocated to the same storage node, greatly reducing cross-node data transmission. Especially in scenarios where access hotspots change frequently, the dynamic decision module can adjust the storage strategy in real time, thereby reducing communication delays and network resource consumption caused by cross-node access.

[0016] 2. The present invention effectively balances the storage load of each node in the distributed storage system through dynamic modeling and multi-objective optimization, avoiding the phenomenon of overload or idleness of individual nodes. At the same time, combined with the dynamic adjustment mechanism of capacity limitations, it ensures that storage resources can be reasonably allocated and fully utilized, providing guarantee for the stability and scalability of the system.

[0017] 3. The present invention combines the Markov decision process with the reinforcement learning method, and the dynamic decision module can adapt to the changes in the storage node status and data block access mode in real time, so that when a node fails, the system can quickly reallocate data blocks, ensuring the efficient operation of the system and greatly improving the reliability and responsiveness in a dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of the steps of the method of the present invention; Figure 2 The step flow chart of expanding step S2 of the present invention; Figure 3 The step flow chart of expanding step S4 of the present invention; Figure 4 It is a flow chart of the steps of dynamic game modeling of the present invention; Figure 5 It is a flow chart of the steps of the dynamic adjustment mechanism of the present invention; Figure 6 It is a schematic diagram of the system module of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example

[0020] Please see attached Figure 1 -Attached Figure 5 The embodiment of the present invention provides a file parsing storage partitioning method based on big data, including the following steps: S1. File parsing: Parse the input big data file to extract the logical structure, field content and the relationship between fields of the file; S2, data block generation: according to the analysis results, the file is divided into several data blocks, and the size and correlation information of each data block is generated; S3, association weight generation: construct association weight information for describing the similarity and joint access frequency between data blocks; S4, storage partition optimization: Based on the above association weight information, an optimization algorithm is used to determine the storage partition scheme of the data blocks, so that the distribution of data blocks among storage nodes can minimize the cross-node communication overhead, balance the storage node load, and meet the storage capacity limit; The S3 step further includes the following steps: S31, analyzing the data block content and calculating the similarity between the data blocks; S32, analyzing the access log and calculating the joint access frequency of the data block; S33, comprehensively generating association weights between data blocks according to similarity and joint access frequency; The S4 step further comprises the following steps: S41, dynamically modeling the capacity of storage nodes and the distribution of data blocks; S42, establishing a multi-objective optimization model with the goal of optimizing communication overhead, load balancing and storage capacity; S43. Using dynamic game modeling to simulate the competition and cooperation relationship between storage nodes, and finally generate a globally optimal storage distribution solution; In step S43, dynamic game modeling is achieved by: S431, defining the global state of the storage system, including the current load and access mode of the storage nodes; S432. Design a set of storage node policies, including storing local data, migrating data to other nodes, or releasing cold data; S433, dynamically adjusting data block distribution according to the storage node strategy to optimize overall communication overhead and load balancing; In step S4, the optimization of the storage partitioning scheme is enhanced through a dynamic adjustment mechanism for long-term benefits. The dynamic adjustment mechanism includes: S44, generating a new optimal storage strategy based on the current state of the system and historical decisions; S45, dynamically adjusting data block distribution in combination with changes in the load and access patterns of storage nodes; S46. Use reinforcement learning methods to optimize storage strategies to improve overall storage efficiency; In step S46, the reinforcement learning method is based on a deep neural network to model the state and strategy of the storage system, and the storage strategy is updated in real time through the proximal strategy optimization algorithm to adapt to the dynamic changes in access patterns and node states.

[0021] Specifically, based on step S1, the present invention provides a file parsing storage partitioning method and a distributed storage system based on big data, which aims to optimize the storage distribution scheme of data by parsing big data files, so as to reduce the communication cost of distributed storage and achieve load balancing. Among them, step S1 is the starting point of the present invention, which is responsible for parsing the input big data file, extracting the logical structure of the file, the field content and the relationship between the fields. Step S1 provides basic data for subsequent data block generation and storage partition optimization, which plays a key role.

[0022] In this embodiment, file parsing is optimized for multiple file formats (such as JSON, XML, CSV, log files, etc.) to adapt to complex data structures in different data scenarios. Specifically, file parsing in this embodiment not only focuses on direct field extraction of structured data, but also performs specific semantic parsing and logical relationship construction on unstructured and semi-structured data.

[0023] As an option, in this embodiment, a recursive parsing method is used to process nested file structures (such as JSON or XML files) to extract their field relationships and contents layer by layer. It should be noted that the recursive parsing method can efficiently process deeply nested data levels, ensuring that field information of all nested levels is completely extracted.

[0024] Specifically, when processing a JSON file, this embodiment utilizes the nested relationship of key-value pairs to expand each level of fields one by one to generate a hierarchical tree structure.

[0025] The parsing process first extracts the root node field, then recursively enters the nested fields and extracts the sub-field content. It can be understood that this parsing method can effectively support complex hierarchical data extraction while keeping the logical relationship between fields intact.

[0026] In a possible implementation, for XML files, the parsing method is similar but needs to process tag attributes.

[0027] During the parsing process, based on the nested relationship of the tags, all node contents are extracted with their path information, generating a key-value pair format similar to the JSON parsing result.

[0028] As another option, in this embodiment, a direct parsing method of row and column segmentation is adopted for files in CSV format. Specifically, the first row of the CSV file is read as the field name, and each subsequent row of data is mapped to the field name in sequence to form a standardized key-value pair format.

[0029] In a possible implementation, this embodiment uses a named entity recognition model to extract key fields in the text, such as "user name", "language preference", etc.

[0030] It can be understood that completing file parsing in the above manner can not only adapt to a variety of data formats, but also provide high-quality parsing results for subsequent steps.

[0031] Based on the S2 step, its goal is to divide the parsed file content into multiple data blocks according to logical relationships and associated information, and generate necessary metadata information (such as size, location, and preliminary association) for each data block to provide basic data support for subsequent storage partitioning optimization.

[0032] The structured data generated by file parsing usually has a certain logical hierarchy and associated information. In order to efficiently manage this data, step S2 divides it into small-granularity data blocks. The data block division needs to take into account the integrity of the file content, the retention of logical relationships, and the performance requirements of distributed storage. Through a reasonable data block generation method, cross-node data interaction can be minimized while improving the overall efficiency of data access.

[0033] In this embodiment, the parsed data is divided into multiple data blocks according to its content, logical level, usage frequency, etc. Each data block carries specific metadata, including but not limited to size information, field information, hierarchical relationship, and association weight with other data blocks.

[0034] As an option, in this embodiment, a field-based logical segmentation method is used for structured data (such as JSON, XML or CSV files). It should be noted that this method can effectively retain the logical relationship between the fields within the file, ensure that each data block can be stored independently and support efficient query.

[0035] Specifically, for JSON files, the parsed fields are split in a root node-first manner.

[0036] It can be understood that the JSON file contains a multi-layer nested structure. During the data block generation process, the root node is regarded as a logical grouping, and the child nodes are further decomposed into child data blocks.

[0037] In a possible implementation, this embodiment records the logical position of each data block through a field path.

[0038] For XML files, the parsed nodes and attributes are divided into blocks according to the hierarchical relationship of the tags.

[0039] It should be noted that in this embodiment, the division of data blocks not only focuses on field logic, but also takes into account the usage pattern of the file. For example, fields with high access frequency may be preferentially stored as independent data blocks, thereby reducing access delay.

[0040] As another option, for a CSV file, this embodiment generates data blocks by dividing the file line by line.

[0041] It is understandable that after a CSV file is parsed, the size of the data block is mainly determined by the number of fields in the row content and the size of a single field.

[0042] When processing unstructured text files (such as logs), this embodiment adopts a segmentation method based on natural language processing (NLP). Specifically, through technologies such as word segmentation and entity recognition, a long text is decomposed into multiple short text units.

[0043] It should be noted that for the division of log files, the associated metadata includes the semantic label of the field (such as "user name" or "number of logins") and the location index of the content. This can enhance the correlation between fields in the subsequent weight generation step.

[0044] As an extension, this embodiment also limits the size of the data block to meet the block size requirements of the distributed storage system. For example, when the content of a single data block exceeds the preset size limit, this embodiment further divides the data block by field priority or logical independence. For the multi-block division results of a large file, the inter-block index is used to maintain its complete relationship.

[0045] Based on the S3 step, it performs association modeling on the data blocks generated after file parsing. Step S3 can effectively quantify the similarity and joint access frequency between data blocks, providing data support for subsequent storage partitioning.

[0046] In big data scenarios, data blocks in files often have complex logical associations and access pattern characteristics. The generation of association weights needs to fully consider the content similarity of data blocks, the frequency of joint access, and the priority weights in specific scenarios. By constructing a weight relationship matrix, an accurate mathematical model can be established for the storage partitioning of data blocks, thereby achieving reasonable distribution among distributed storage nodes.

[0047] In this embodiment, the generation of association weights is based on the file analysis results, and the weight values ​​between data blocks are calculated by multi-factor fusion. The specific process includes content similarity calculation, joint access frequency analysis and weight fusion formula design.

[0048] As an option, this embodiment uses a field-level content comparison method to quantify the content similarity between data blocks. It should be noted that content similarity not only considers the field name, but also includes the similarity of the field value. Specifically, for two data blocks and , its similarity can be calculated by the following formula: in, For data blocks and The content similarity score between them, the higher the value, the higher the similarity. For data blocks and A collection of common fields. For data blocks and The complete set of fields (including common fields and unique fields), is the weight of field k, which is used to describe the importance of the field in similarity calculation. The score of whether the value of field k matches. The match score is 1, and the mismatch score is 0.

[0049] It is understandable that the formula achieves an accurate assessment of the similarity of different data blocks by weighting the importance of the fields. In some embodiments, field matching not only uses direct value matching, but also includes matching based on edit distance or semantic similarity. For example, for two field values ​​"English" and "EN", it can be determined through semantic analysis that they are highly similar, thereby assigning a higher similarity score.

[0050] The calculation of the joint access frequency is another key step in generating the association weight. In this embodiment, the joint access frequency between data blocks is extracted by analyzing the historical access logs. Specifically, the data blocks and The joint access frequency of is defined as: ,in, For data blocks and The joint access frequency of , which indicates the probability of both being accessed at the same time, For data blocks and The number of simultaneous accesses in the historical access logs, It is the total number of times all data blocks in the system are accessed.

[0051] In a possible implementation, the calculation range of the joint access frequency is limited by a time window in this embodiment to capture the hotspot access relationship of data blocks in a specific time period. For example, the calculation within a 24-hour time window , which better reflects recent access patterns.

[0052] It should be noted that the calculation of the joint access frequency can reflect the coupling relationship of data blocks in actual usage scenarios. When it is higher, it indicates and In distributed storage, they should be assigned to the same storage node first to reduce cross-node communication overhead.

[0053] In this embodiment, the final calculation of the association weight is achieved by fusing content similarity and joint access frequency. The fusion formula is as follows: ,in, Represents a data block and The comprehensive correlation weight between them; and is a weight adjustment parameter used to adjust the relative importance of content similarity and joint access frequency.

[0054] It is understandable that this formula is adjusted by and The value of can be flexibly adapted to different scene requirements. For example, in the scene where the content integrity is important, Higher weight; in high-frequency access scenarios, it can be improved The value of to prioritize access modes.

[0055] In an exemplary scenario, consider a log file containing user information, where each record includes a user ID, access time, and access operation. After calculating the association weights between data blocks using the above method, the result may show that access time and access operation have a high correlation, indicating that these fields need to be prioritized in the same storage node in storage allocation.

[0056] Based on the S4 step, its goal is to achieve the optimal distribution of data blocks among storage nodes, thereby reducing cross-node communication overhead, balancing storage load, and meeting storage capacity constraints.

[0057] In big data scenarios, the data blocks after file parsing usually have complex associations and access patterns. Storage partition optimization requires modeling and solving the distribution of data blocks on a global scale. Based on the results of association weight generation, step S4 uses dynamic modeling and multi-objective optimization to comprehensively consider communication overhead, load balancing and storage capacity, and finally determines the global optimal storage distribution solution.

[0058] In this embodiment, the storage partition optimization is based on dynamic game modeling, combined with the Markov decision process, and the continuous optimization of storage partition is achieved through the dynamic adjustment mechanism of long-term benefits. The optimization goal is to minimize communication overhead while achieving load balancing and capacity constraints.

[0059] As an option, in this embodiment, the capacity of the storage node and the distribution of data blocks are dynamically modeled. It should be noted that dynamic modeling can reflect the state of the system in real time, including the current load of the storage node and the distribution of data blocks. Specifically, the global state of the system can be expressed as: ,in, Represents a storage node Storage load at time t; Indicates the current access pattern, such as the access frequency distribution of hot data blocks.

[0060] It can be understood that dynamic modeling provides real-time input data for the optimization algorithm, ensuring that the storage partitioning results can adapt to changes in system status.

[0061] In one possible implementation, storage partition optimization is solved based on the following three objective functions: Minimize cross-node communication overhead. The communication overhead is determined by the association weights between data blocks and the distribution of storage nodes. and The communication overhead calculation formula is: ,in, For data blocks and The communication overhead between is the association weight; express and Whether it is stored in the same node, if yes, it is 1, otherwise it is 0.

[0062] Achieve load balancing. The goal of load balancing is to make the storage load of each storage node as close to the average as possible. The load balancing cost can be expressed as: ,in, is the load imbalance cost of the storage node, k is the total number of storage nodes, For storage nodes The set of data blocks allocated on For data blocks The size of is the ratio of the total size of all data blocks to the number of nodes k, that is, the average expected storage load of each node. When the load of a storage node exceeds its capacity, storage partitioning will introduce a penalty cost. The penalty cost of capacity constraint is defined as: ,in, Represents a storage node The capacity, is the penalty cost introduced due to the storage node capacity limitation, Assigned to storage nodes The total size of all data blocks.

[0063] In this embodiment, the overall objective function of storage partition optimization is: ,in, and is the weight coefficient used to adjust the relative importance of load balancing and capacity limitation in the overall goal. The total optimized cost of the system.

[0064] In the optimization process, this embodiment uses dynamic game modeling to model the competition and cooperation relationship between storage nodes. As an option, each storage node is regarded as an intelligent agent, whose goal is to maximize its own storage efficiency while cooperating with other nodes to achieve global optimization.

[0065] Specifically, this embodiment models the storage partitioning strategy through a Markov decision process (MDP). The core of the MDP includes: State space S: describes the load status of storage nodes and the distribution of data blocks; Action space A: includes allocating data blocks to the current node, migrating to other nodes, or cleaning up low-priority data blocks; Reward function R: measures the quality of the storage partitioning strategy.

[0066] It can be understood that by introducing the Markov decision process, storage partition optimization can continuously adjust the storage strategy based on long-term benefits, thereby improving the overall performance of the system.

[0067] In a possible implementation, this embodiment further combines a reinforcement learning method to dynamically optimize the storage strategy. Reinforcement learning is based on a deep neural network and updates the storage partitioning strategy in real time through a proximal policy optimization (PPO) algorithm. Exemplarily, the optimization goal of the storage partitioning strategy is: ,in, Status and actions The value function is used to measure the Take action The total revenue obtained, For the current state and actions Instant rewards, is the discount factor, indicating the importance of future returns (values ​​range from 0 to 1), For the future state The expected maximum profit.

[0068] It should be noted that, through the dynamic adjustment mechanism of reinforcement learning, this embodiment can adapt to changes in access patterns and storage node states and achieve online optimization of storage partitioning. For example, in a scenario where hot data blocks are frequently switched, the storage partitioning strategy can respond quickly and prioritize the migration of hot data blocks to low-latency nodes, thereby reducing the communication overhead of the system.

[0069] Embodiment 2: Please see attached Figure 6 , a file parsing distributed storage system based on big data, including: A file parsing module is used to parse the input big data file, divide the file into several data blocks, and extract the association relationship between the data blocks; A metadata management module, which is used to record the distribution location, size and associated weight information of data blocks; A dynamic decision module, which is used to generate the optimal storage partitioning strategy for data blocks based on the file parsing results and the current status of the storage nodes; A storage execution module, which is used to dynamically migrate data blocks and manage data in a distributed storage system in real time according to the optimal storage partitioning strategy; The dynamic decision module generates the optimal storage partitioning strategy based on the following techniques: (1) Dynamically update the storage distribution strategy by combining the association relationship of data blocks with the load status of storage nodes; (2) Design the long-term benefit optimization problem of storage nodes, taking into account communication overhead, load balancing, and storage capacity limitations; (3) Use reinforcement learning algorithms to optimize storage distribution strategies and generate continuously adjusted optimal strategies; The metadata management module includes the following functions: (1) Record the distribution information and associated weights of data blocks in storage nodes; (2) Update the distribution metadata of data blocks in real time when the storage node load or access pattern changes; (3) Provide metadata management capabilities to support data block migration and cleanup to ensure the system's dynamic adjustment capabilities; The storage execution module is fault-tolerant and includes the following features: (1) When a storage node failure is detected, dynamic migration of data blocks is triggered; (2) When a node is overloaded, it automatically releases data blocks with low access frequency and adjusts the distribution plan; (3) Update the distribution information of data blocks through the metadata management module to ensure the high availability of the system.

[0070] Specifically, the file parsing module of the present invention is one of the core components of the entire solution, responsible for structurally parsing the input big data files, converting them into standardized, operable data blocks, and providing basic data support for subsequent association weight generation and storage partition optimization. The design of the file parsing module can adapt to a variety of file formats, ensuring that the content and logical structure in the big data files are completely extracted and expressed.

[0071] The file parsing module supports efficient parsing of structured files (such as JSON, XML, CSV) and unstructured files (such as log text, document content) by combining recursive parsing, hierarchical modeling, field extraction, and semantic analysis. Its main goal is to divide complex file content into logically clear fields or blocks, while recording the size, location, and logical relationship of each field.

[0072] In this embodiment, the file parsing module includes functions such as parsing process design, field content extraction, logical hierarchy maintenance and field metadata generation, and can cover a variety of data scenarios.

[0073] As an option, this embodiment uses a recursive parsing method to extract the nested structure of a JSON file. It should be noted that a JSON file is a structured data format widely used in big data scenarios, and the parsing of its nested fields directly affects the accuracy of subsequent data block generation. Specifically, the basic process of JSON file parsing includes: First, expand the key-value pair structure of the JSON file layer by layer, and for each key, record its path and corresponding value.

[0074] Second, for keys whose values ​​are nested objects or arrays, the resolve function is called recursively until all levels of nesting are expanded.

[0075] Finally, the parsed key paths and corresponding values ​​are constructed into a standardized data representation.

[0076] For CSV files, this embodiment uses a line-by-line parsing method to extract field content. Specifically, by reading the first line of the CSV file as the field name and each subsequent line as the field value, a key-value pair representation is generated.

[0077] It should be noted that files in CSV format are usually used to store flat data structures. The parsing process is relatively simple, but the fields generated after parsing need to be further mapped to the logical hierarchy to support more complex association modeling.

[0078] As an extension, in this embodiment, the file parsing module supports segmented parsing of large files. When the size of a single file exceeds the memory limit of the parsing system, this embodiment divides the file into blocks and parses each block separately. The block division method can be based on logical fields, number of lines or byte size to ensure that the parsed results maintain logical integrity.

[0079] It should be noted that the structured representation generated by the file parsing module includes not only the field content, but also metadata information such as field size and logical path.

[0080] The metadata management module of the present invention is a key module in the file parsing storage partitioning method and the distributed storage system, responsible for maintaining the distribution of data blocks, logical associations and dynamic adjustment information during system operation. The metadata management module plays a core hub role in the overall system architecture, and its design directly affects the data query efficiency, dynamic adjustment capability and fault tolerance mechanism of the storage system.

[0081] By performing logical mapping and metadata maintenance on the parsed file data blocks, the module realizes the centralized management and real-time update of data block information, and provides support for the dynamic decision module and the storage execution module. The metadata management module can effectively process the distribution location, size, associated weight and other related information of data blocks in the system to meet the requirements of consistency, high availability and flexible adjustment in a distributed storage environment.

[0082] In this embodiment, the metadata management module includes metadata structure design, dynamic update mechanism and efficient query function, and its implementation method fully considers the complexity and efficiency requirements of large-scale data block management.

[0083] As an option, in this embodiment, the core data structure of the metadata management module adopts a distributed hash table (DHT) or a tree structure to support fast query and dynamic adjustment. Specifically, the metadata information of each data block includes the following: Data block identifier (ID): uniquely identifies each data block, usually a value generated by a hash function.

[0084] Data block path: records the logical path of the data block after file parsing.

[0085] Data block size: indicates the physical storage space size of the data block, in bytes.

[0086] Data block association weight: used to describe the logical relationship between a data block and other data blocks, and comes from the association weight generation module.

[0087] Storage node location: records the current storage node ID or physical location of the data block.

[0088] It is understandable that the design of the metadata management module should not only support efficient storage and access of metadata, but also provide real-time update capabilities for dynamic adjustment of the system.

[0089] In one possible implementation, the metadata management module adopts a distributed storage design to improve the fault tolerance and scalability of the system. For example, for a large-scale distributed storage system, the metadata management module can store metadata in multiple nodes and implement distributed query and fault tolerance management through a consistent hashing algorithm.

[0090] Specifically, the dynamic update mechanism of metadata can adjust the data block distribution in real time according to the status changes of storage nodes. For example, when a storage node is overloaded or fails, the metadata management module will trigger the migration of data blocks and update the corresponding metadata records to ensure the consistency and high availability of the system.

[0091] As an extension, in this embodiment, the metadata management module supports incremental updates to record changes in data block association weights. It should be noted that data block association weights may be dynamically adjusted as access patterns change. For example, the weight of a hot data block may increase as access frequency increases. To this end, the metadata management module implements incremental updates through the following mechanisms: The associated weight matrix is ​​scanned periodically to compare the current weights with the historical records.

[0092] When the weight change exceeds the preset threshold, the metadata update operation is triggered and the update log is recorded.

[0093] The dynamic decision module in the present invention is a key component of the file parsing storage partitioning method and the distributed storage system. Its function is to dynamically generate the optimal storage partitioning strategy for data blocks based on the data provided by the file parsing and metadata management modules. The dynamic decision module not only needs to meet the storage requirements of the data blocks, but also needs to adapt to the dynamic changes of the system, including node load, changes in access modes, and storage capacity limitations. Therefore, the module can achieve real-time optimization of storage partitioning, thereby improving the overall performance of the system.

[0094] The design of the dynamic decision module mainly includes three core functions: dynamic modeling, decision optimization, and strategy generation. In the big data scenario, the access mode of data blocks and the status of storage nodes usually change over time. Therefore, the dynamic decision module needs to establish a dynamic mapping relationship between system status and storage strategy through dynamic game theory and Markov decision process (MDP), and combine reinforcement learning methods to generate the storage strategy with the best long-term benefits.

[0095] In this embodiment, the dynamic decision module calculates the optimal solution of the storage partitioning scheme by modeling the status of the storage nodes and analyzing the association of the data blocks, and adjusts the storage strategy in real time to adapt to the dynamic changes in the system status.

[0096] As an option, in this embodiment, the dynamic decision module first models the global state of the system. It should be noted that the global state of the system includes the load information of the storage nodes, the distribution of the data blocks, and the current access mode. Specifically, the main components of the state model include: Storage node load: indicates the current storage usage and remaining capacity of each node.

[0097] Data block distribution: describes the storage location and size of each data block.

[0098] Access pattern: records hot data blocks and their access frequency distribution.

[0099] In a possible implementation, the dynamic decision module generates a storage partitioning scheme based on the above state information through a multi-objective optimization model. The optimization objectives mainly include the following three aspects: Minimize cross-node communication overhead: Reduce the cost of cross-node data transmission by allocating highly associated data blocks to the same node first.

[0100] Achieve load balancing: By evenly distributing data blocks, the storage nodes can be prevented from being overloaded or underloaded.

[0101] Meeting storage capacity constraints: Ensure that the storage usage of each node does not exceed its available capacity by adjusting the data block distribution.

[0102] It is understandable that the dynamic decision module needs to make a trade-off between the above optimization objectives to generate a globally optimal storage partitioning solution.

[0103] Specifically, the optimization objective function in this embodiment is defined as: The objective function of communication cost is expressed as the sum of products of association weight and distribution across nodes.

[0104] The objective function of load balancing is expressed as the sum of the deviations between the storage load of each node and the average load.

[0105] The objective function of capacity limitation is expressed as the penalty cost of node overload.

[0106] The final expression of the optimization objective function is the weighted sum of the objective functions, where the weight coefficients are used to adjust the importance of different objectives.

[0107] In an exemplary implementation, the dynamic decision module uses dynamic game theory to model the competition and cooperation relationship between storage nodes. Each storage node is regarded as a participant in the game, and its strategy space includes operations such as retaining data blocks, migrating data blocks, and releasing data blocks. Through dynamic game modeling, the dynamic decision module can simulate the interaction process between storage nodes and ultimately generate a globally optimal storage partitioning strategy.

[0108] As an extension, this embodiment also introduces a Markov decision process (MDP) to model the storage partitioning process. It should be noted that the MDP model transforms the storage partitioning problem into a long-term benefit optimization problem of state transfer.

[0109] The dynamic decision module can adapt to changes in storage node status and access patterns in real time through the dynamic adjustment mechanism of reinforcement learning. For example, when the access frequency of a hot data block suddenly increases, the dynamic decision module will prioritize migrating it to a high-performance node to reduce access latency and communication overhead.

[0110] In some embodiments, the dynamic decision module also supports an incremental adjustment strategy, that is, quickly adjusting the data block distribution in a small range through local optimization without recalculating the global optimal solution. The main advantage of the incremental adjustment mechanism is its fast response speed, which is suitable for scenarios where access patterns change rapidly in a short period of time.

[0111] It can be understood that the dynamic decision module implemented by the above design in this embodiment can not only generate the global optimal storage partitioning strategy in real time, but also continuously optimize the storage distribution in a dynamic environment, thereby significantly improving the performance and stability of the distributed storage system. The efficiency and robustness of the dynamic decision module provide important guarantees for the practical application of the solution of the present invention. The dynamic decision module in the present invention is one of the core modules of the file parsing storage partitioning method and the distributed storage system, which is responsible for dynamically generating the optimal storage partitioning strategy according to the current state of the system and the distribution of data blocks. This module uses dynamic game theory, Markov decision process (MDP) and reinforcement learning methods, comprehensively considers factors such as communication overhead, storage load balancing and capacity limitations, and realizes dynamic scheduling and efficient distribution of data blocks in the distributed storage system.

[0112] During operation, the dynamic decision module continuously adjusts the data block distribution strategy based on the data provided by the file parsing module and the metadata management module through real-time modeling and optimization to adapt to the dynamic changes in system operation. Its core goal is to maximize the overall performance of the system and achieve optimal storage division based on long-term benefits.

[0113] In this embodiment, the dynamic decision module includes four key parts: state modeling, decision optimization, strategy generation and dynamic adjustment.

[0114] As an option, in this embodiment, the dynamic decision module first models the global state of the distributed storage system. It should be noted that the global state is composed of the load information of the storage node, the distribution of the data block and the current access mode, and specifically includes the following contents: Storage node load: used to indicate the current storage usage and remaining capacity of each node.

[0115] Data block distribution: records the storage node and size of each data block.

[0116] Access pattern: used to describe hot data blocks and their access frequency distribution, which can reflect the current access hotspot areas.

[0117] In one possible implementation, the dynamic decision module uses a multi-objective optimization model to generate the optimal storage partitioning strategy. The optimization objectives mainly include the following aspects: Minimize cross-node communication overhead: Reduce cross-node communication between data blocks by allocating data blocks with higher association weights to the same node first.

[0118] Achieve load balancing: By adjusting the distribution of data blocks, the load of storage nodes is made as close to the average level as possible.

[0119] Meeting storage capacity limits: By dynamically allocating data blocks, ensure that the storage usage of each storage node does not exceed its capacity limit.

[0120] It should be noted that there may be certain conflicts between these optimization objectives. For example, minimizing communication overhead may cause excessive load on individual nodes. Therefore, the dynamic decision module weighs each objective through a weighted combination of objective functions.

[0121] In this embodiment, the dynamic decision module models the interaction relationship between storage nodes through dynamic game modeling. The storage nodes are regarded as game participants, and the goal of each participant is to maximize its own benefits while cooperating with other nodes to achieve global optimization.

[0122] Specifically, the dynamic game model is implemented through the following steps: Define the state, policy set, and revenue function of each storage node. The policy set includes operations such as local storage, migrating data blocks to other nodes, and deleting low-priority data blocks.

[0123] Through the game process, the benefits of each node under different strategy combinations are calculated, and finally the Nash equilibrium solution is obtained.

[0124] In a possible implementation, this embodiment introduces a Markov decision process to model the storage partitioning problem.

[0125] It can be understood that the Markov decision process provides a mathematical modeling and solution basis for the dynamic decision-making module through the definition of state transition and reward function.

[0126] In this embodiment, the dynamic decision module solves the MDP model by using a reinforcement learning method. Specifically, the dynamic decision module uses a deep reinforcement learning algorithm to approximate the optimal storage strategy through proximal policy optimization or deep Q learning. The goal of reinforcement learning is to maximize long-term benefits.

[0127] It should be noted that the dynamic decision module can adapt to changes in access patterns and storage node status in real time through the dynamic adjustment mechanism of reinforcement learning. For example, when the access frequency of certain data blocks increases significantly, the dynamic decision module can quickly adjust the storage location of these data blocks and migrate them to high-performance nodes, thereby reducing access latency and communication overhead.

[0128] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A file parsing storage partitioning method based on big data, characterized in that: The following steps are involved: S1. File parsing: Parse the input big data file to extract the logical structure, field content and the relationship between fields of the file; S2, data block generation: according to the analysis results, the file is divided into several data blocks, and the size and correlation information of each data block is generated; S3, association weight generation: construct association weight information for describing the similarity and joint access frequency between data blocks; S4. Storage partition optimization: Based on the above-mentioned association weight information, an optimization algorithm is used to determine the storage partition scheme of the data blocks, so that the distribution of data blocks among storage nodes can minimize cross-node communication overhead, balance storage node loads, and meet storage capacity limitations.

2. The file parsing storage partitioning method based on big data according to claim 1 is characterized in that: The S3 step further includes the following steps: S31, analyzing the data block content and calculating the similarity between the data blocks; S32, analyzing the access log and calculating the joint access frequency of the data block; S33. Generate association weights between data blocks based on similarity and joint access frequency.

3. The file parsing storage partitioning method based on big data according to claim 1 is characterized in that: The S4 step further comprises the following steps: S41, dynamically modeling the capacity of storage nodes and the distribution of data blocks; S42, establishing a multi-objective optimization model with the goal of optimizing communication overhead, load balancing and storage capacity; S43. Use dynamic game modeling to simulate the competition and cooperation relationship between storage nodes, and finally generate the globally optimal storage distribution plan.

4. The file parsing storage partitioning method based on big data according to claim 3 is characterized in that: In the step S43, dynamic game modeling is implemented in the following manner: S431, defining the global state of the storage system, including the current load and access mode of the storage nodes; S432. Design a set of storage node policies, including storing local data, migrating data to other nodes, or releasing cold data; S433. Dynamically adjust data block distribution according to storage node strategies to optimize overall communication overhead and load balancing.

5. The file parsing storage partitioning method based on big data according to claim 1 is characterized in that: In the step S4, the optimization of the storage partitioning scheme is enhanced by a dynamic adjustment mechanism of long-term benefits, and the dynamic adjustment mechanism includes: S44, generating a new optimal storage strategy based on the current state of the system and historical decisions; S45, dynamically adjusting data block distribution in combination with changes in the load and access patterns of storage nodes; S46. Use reinforcement learning methods to optimize storage strategies to improve overall storage efficiency.

6. The file parsing storage partitioning method based on big data according to claim 5 is characterized in that: In the step S46, the reinforcement learning method is based on a deep neural network to model the state and strategy of the storage system, and updates the storage strategy in real time through a proximal strategy optimization algorithm to adapt to the dynamic changes in access patterns and node states.

7. A file parsing distributed storage system based on big data, according to the file parsing storage partitioning method based on big data according to any one of claims 1 to 6, characterized in that: include: A file parsing module is used to parse the input big data file, divide the file into several data blocks, and extract the association relationship between the data blocks; A metadata management module, which is used to record the distribution location, size and associated weight information of data blocks; A dynamic decision module, which is used to generate the optimal storage partitioning strategy for data blocks based on the file parsing results and the current status of the storage nodes; The storage execution module is used to dynamically migrate data blocks and manage data in the distributed storage system in real time according to the optimal storage partitioning strategy.

8. The file parsing distributed storage system based on big data according to claim 7 is characterized in that: The dynamic decision module generates the optimal storage partitioning strategy based on the following technologies: (1) Dynamically update the storage distribution strategy by combining the association relationship of data blocks with the load status of storage nodes; (2) Design the long-term benefit optimization problem of storage nodes, taking into account communication overhead, load balancing, and storage capacity limitations; (3) Use reinforcement learning algorithms to optimize the storage distribution strategy and generate a continuously adjusted optimal strategy.

9. The file parsing distributed storage system based on big data according to claim 7 is characterized in that: The metadata management module includes the following functions: (1) Record the distribution information and associated weights of data blocks in storage nodes; (2) Update the distribution metadata of data blocks in real time when the storage node load or access pattern changes; (3) Provide metadata management capabilities to support data block migration and cleanup to ensure the system's dynamic adjustment capabilities.

10. The file parsing distributed storage system based on big data according to claim 7, characterized in that: The storage execution module has fault tolerance capability and includes the following functions: (1) When a storage node failure is detected, dynamic migration of data blocks is triggered; (2) When a node is overloaded, it automatically releases data blocks with low access frequency and adjusts the distribution plan; (3) Update the distribution information of data blocks through the metadata management module to ensure the high availability of the system.

Images