Finance and accounting data storage management optimization method and system based on big data

Abstract

The invention discloses a big data-based finance and accounting data storage management optimization method and system, and relates to the technical field of financial information management, and the method comprises the steps: collecting enterprise financial original data, carrying out the multi-dimensional cleaning and structured preprocessing, recognizing the data type based on a classification model, and generating a metadata label; executing a cold and hot hierarchical storage and load balancing strategy in combination with the access heat and the node resource state; encryption, copying and compression operations are completed before data storage, and a multi-level index is constructed to improve the access efficiency; and finally realizing long-term archiving and system operation and maintenance management. The system comprises a data acquisition module, a preprocessing module, a classification module, a storage optimization module, a data storage module, a data access module and a system management module. According to the invention, intelligent management and efficient storage of massive heterogeneous financial and accounting data are realized, and the data processing capability, security and availability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of financial information management, and in particular to a method and system for optimizing financial accounting data storage and management based on big data. Background Art

[0002] With the advancement of enterprise informatization, financial data sources are becoming increasingly diverse, encompassing both structured transaction data generated by ERP systems and large amounts of unstructured data such as scanned documents and invoice images. This data is characterized by large volumes, heterogeneous structures, and complex access patterns. Traditional processing methods that rely on manual archiving or standalone databases face significant challenges and are unable to meet the demands of modern digital financial management.

[0003] Although financial data processing methods based on big data platforms have emerged, there is still a lack of pre-reconstruction and noise processing of the original data, which leads to a large amount of redundant information being stored in the system, affecting storage efficiency. In addition, the encryption mechanism is usually relatively simple, lacking a flexible key hierarchy and access control strategy, and insufficient security. The hot and cold data storage strategy is rough and does not fully consider changes in data access trends. The query and indexing mechanism is not deeply integrated with the characteristics of financial data, resulting in poor retrieval performance.

[0004] In response to the above problems, the present invention proposes a financial accounting data storage management optimization method and system based on big data, which integrates data preprocessing, intelligent classification, encrypted storage, access optimization and system archiving, and realizes the intelligence and security improvement of the entire life cycle of financial data. Summary of the Invention

[0005] In response to the above problems, the present invention provides a financial accounting data storage management optimization method and system based on big data to solve the problems of data redundancy, low access efficiency, storage imbalance and weak security in the storage management process of the prior art.

[0006] To solve the above technical problems, the present invention provides the following technical solution: a method for optimizing accounting data storage management based on big data, comprising the following steps: Step S1: Calling the raw data acquisition unit in the data acquisition module to collect financial accounting raw data files from the enterprise ERP system, financial software, and manual bill scanning system, and classifying them into structured data and unstructured data based on file characteristics; Step S2: Through the data preprocessing module, the processed structured data and unstructured data are comprehensively cleaned, verified, and integrated to construct an initial financial data set; Step S3: Through the data classification module, intelligent classification recognition and structured metadata generation are performed on each data record output by the data preprocessing module; Step S4: Using the storage optimization module, a storage strategy is developed for data classification based on current data access trends and resource status of each node. Step S5, by calling the data storage module, complete the physical writing of data, encryption protection before storage, copy synchronization and compression optimization; Step S6: Using the data access module, an index structure is constructed for the newly written data, and a query execution plan is generated based on the data distribution. Step S7: Through the system management module, long-term financial data archiving, continuous monitoring of system operation status, data backup and system maintenance tasks are performed.

[0007] In step S1, the following sub-steps are also included: S1-1, for structured data, call the data format conversion unit to perform format parsing. For spreadsheet files in .xls and .xlsx formats, their contents are extracted into standard database tables according to the table field structure; for text documents in .doc and .pdf formats, the text content is extracted through the parser and converted into standard text records; S1-2: For unstructured data, the image preprocessing unit is called to perform image denoising, image segmentation, and OCR text recognition operations, converting the financial voucher content in the image file into character information and generating the corresponding account field structure; S1-3, unify the structured data and unstructured data that have completed format conversion into intermediate data and temporarily cache them in the system's data buffer unit.

[0008] In step S2, the following sub-steps are also included: S2-1, through the data cleaning unit, based on business rule verification and missing value processing, records that violate accounting rules are marked as abnormal and eliminated; data items with null values ​​and missing fields are filled using interpolation algorithms based on mean, median, and historical trends; duplicate records are deduplicated; S2-2, through the data verification unit, performs semantic and structural consistency verification on various key financial fields, performs debit and credit balance verification, account code legitimacy verification, and field format verification; S2-3, through the data integration unit, match and aggregate multi-source data according to the accounting voucher number, serial number and department ID. For vouchers, reports and approval records from different systems, establish a one-to-many relationship with the voucher number as the primary key; automatically fill in missing primary key and foreign key fields to build a unified logical table structure; write the integrated data into the integration result temporary storage unit.

[0009] In step S3, the following sub-steps are also included: S3-1, using the data type identification unit in the data classification module, extracts features and classifies each data record in the initial financial data set, constructing a feature vector based on multiple feature dimensions; the multiple feature dimensions include financial field features, text keywords, file structure features, and source identification features; Combined with the above feature vectors, the logistic regression model is used to classify the feature vectors, as shown in the formula:

[0010] in, Indicates the probability that the input data is "voucher data", is the eigenvector corresponding to the financial data, is the training weight parameter of the model, b is the bias term, and y is the output label representing the type category; S3-2, based on the classification result, a metadata generation unit generates a structured metadata label for each classified data record, the label consisting of multiple predefined metadata fields; Each structured metadata tag is bound to its corresponding data record and written uniformly into the metadata storage unit in the data classification module as the logical identifier of the data record.

[0011] In step S4, the following sub-steps are also included: S4-1, through the trend analysis unit, based on historical access logs and metadata records, statistics the access frequency and change rate of each type of data in multiple time periods, and calculate the popularity score of each type of data, as shown in the formula:

[0012] in, For the heat score, is the data access frequency of the current cycle, is the visit growth rate compared with the previous period, Contribution weight coefficient for heat, Configure parameters for the system; Setting the heat threshold ,like > , then this type of data is marked as "hot data", otherwise it is "cold data"; S4-2, through the storage status monitoring unit, obtains the status indicators of each storage node in real time, including available storage capacity, current I / O pressure and node write rate; Based on the preset load threshold , determine the status of each node; S4-3, through the hot and cold data management unit, based on the heat determination and node status, executes the hot and cold data hierarchical processing logic, as follows: If the data is hot data and the current period forecast shows a continued high popularity trend, it is written to high-speed media; if the data is cold data, it is written to low-cost large-capacity storage; if the data is hot data but all high-performance nodes are under high load, it is also written to low-cost large-capacity storage; During the node selection phase, the write sensitivity of each candidate node is calculated, and the node with the lowest sensitivity is selected first for writing. If all nodes are close to saturation, the load balancing unit is activated to dynamically allocate data according to the remaining capacity ratio; S4-4, after determining the target node, uses the storage policy unit to generate a structured storage instruction for each data. The storage instruction consists of multiple fields, including: target node number, data replication factor, data compression configuration and write priority tag.

[0013] In step S5, the following sub-steps are also included: S5-1, through the storage node management unit, uses the determined target storage node as the write entry to write the classified and marked business data in batches into the distributed file system; After writing is completed, the write log, timestamp, and data writing status are automatically recorded, and the local storage view of each node is updated; S5-2: Data involving financial information, audit records, and sensitive identity fields are encrypted using the data encryption unit. The default encryption algorithm is SM4, a national standard symmetric encryption algorithm. The encryption model is defined as follows:

[0014] Where M is the plaintext data to be encrypted, K is the symmetric key dynamically generated by the system through the key management module, and C is the encrypted ciphertext result; each encrypted data will record the encryption algorithm identifier and key index number in its metadata tag; S5-3, through the data replication unit, according to the system preset number of copies, after the primary write node is completed, the data copy is automatically synchronized to the backup node, and the node location information, replication status and version mark of each copy are recorded; S5-4: For large-scale unstructured files, including invoice images, PDF contracts, and bill scans, the data compression unit performs compression processing before storage. The compression process is based on the compression algorithm. The compression configuration is used to specifically control the compression behavior parameters, which are dynamically set according to the data type and access frequency. After compression is complete, each compression record will have the following information appended to the corresponding metadata tag: original file size, compressed file size, type of compression algorithm used, and whether subsequent decompression caching is enabled.

[0015] In step S6, the following sub-steps are also included: S6-1, using the index unit, establishes a structured search index for the written data records, specifically including: primary key index, composite index, full-text index, and time series index; After the construction is completed, the system will automatically update the distributed index directory and uniformly record the logical distribution, node, update time and expiration strategy of each index structure; S6-2, through the query processing unit, schedules and optimizes various query requests initiated by users: using the internal integrated query optimizer to generate execution plans based on the currently available indexes and metadata statistics, including selectivity, data distribution, and interval skew; S6-3, using the cache management unit, dynamically caches hot data blocks and hot index pages in memory during runtime: hot data is identified based on historical access frequency, access burstiness, and service priority; the cache uses LRU and LFU strategies for automatic replacement; The system sets the cache granularity according to data type, source and user role. The status of all cache blocks will be recorded in the system's cache view table in real time, and the cache hit rate and invalidation strategy will be updated synchronously.

[0016] In step S7, the following sub-steps are also included: S7-1, through the data archiving unit, identifies data records that have not been accessed for a long time and have exceeded the retention period according to the configured data lifecycle policy and performs archiving operations, specifically: For historical data that exceeds the age limit, sensitive fields are first desensitized; the desensitized data is then migrated to offline archiving media to relieve pressure on primary storage. All archiving actions will add fields to the system metadata tag and record archive logs to support subsequent recovery; S7-2, through the system monitoring unit, continuously collects and analyzes the operating performance indicators and resource usage of each subsystem, including: CPU utilization, memory utilization, network throughput, available capacity of each storage node, I / O throughput, write latency, and response delays for system maintenance tasks and data services; When the system detects an abnormal state, it immediately triggers the preset alarm mechanism and automatically calls the backup resources to complete the fault switch; S7-3, through the maintenance scheduling unit, the system regularly performs system-level maintenance and disaster recovery operations according to scheduling rules, mainly including: log cleaning, system upgrades and data backup; All operation and maintenance operations are recorded in maintenance logs, and administrators can view execution status, historical task records and current load information in the system management background.

[0017] A financial accounting data storage and management system based on big data, including the following modules: Data acquisition module, data preprocessing module, data classification module, storage optimization module, data storage module, data access module and system management module; The data acquisition module includes a raw data acquisition unit, a data format conversion unit, an image pre-processing unit and a data buffer unit, and is used to collect raw accounting data from multiple sources such as the enterprise ERP system, the financial software platform and the bill scanning device; The data preprocessing module includes a data cleaning unit, a data verification unit, a data integration unit and an integration result temporary storage unit, which are responsible for cleaning, verifying and integrating the collected intermediate data; The data classification module includes a data type identification unit, a metadata generation unit and a metadata storage unit, which are used to automatically classify and identify the pre-processed standard data set and generate metadata tags; The storage optimization module includes a trend analysis unit, a storage status monitoring unit, a hot and cold data management unit, a load balancing unit, and a storage strategy unit, which formulates storage strategies based on the access popularity of data and the status of system resources; The data storage module includes a storage node management unit, a data encryption unit, a data replication unit and a data compression unit, which are used to perform physical writing, encryption protection, backup replication and compression optimization of data; The data access module includes an index unit, a query processing unit and a cache management unit, which supports fast query, index construction and cache optimization of stored data; The system management module includes a data archiving unit, a system monitoring unit, and a maintenance scheduling unit, which are responsible for archiving old data, monitoring system status, and scheduling periodic maintenance tasks.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention introduces a "reconstruction and noise reduction" step before data encryption and storage, uniformly processes data formats from multi-source heterogeneous systems, removes invalid fields and noise interference, and significantly reduces the amount of useless information stored; by integrating key indexing, arithmetic coding and chaotic encryption, a multi-layer variable key structure is constructed to enhance the security protection capabilities of sensitive data, and support dynamic key replacement strategies to meet multi-role access control requirements.

[0019] The present invention uses a rule engine and classification model to extract features and determine the type of each piece of data, automatically generating metadata tags containing data type, time tag, source system, etc., providing structural support for subsequent indexing and auditing; and proposes a dynamic heat scoring model based on data access frequency and node status, combined with node write sensitivity and remaining capacity, to implement a joint strategy of hot and cold data stratification and node load balancing, significantly improving the overall storage efficiency of the system.

[0020] The present invention performs SM4 encryption and compression algorithm optimization before data is written, and generates data copies distributed on multiple nodes according to the strategy, thereby improving data security and high availability; by constructing multiple index structures such as primary key, full text, time, etc. for newly written data, and loading hot data or index pages into the cache area, the access speed in key scenarios such as report generation and audit query is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It is understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but is merely for selected embodiments of the present invention.

[0024] Please refer to Figure 1 , Figure 1 This is a flow chart of a method for optimizing financial accounting data storage management based on big data provided by an embodiment of the present invention, comprising the following steps: Step S1: Call the raw data collection unit in the data collection module to batch collect financial accounting raw data files from the enterprise ERP system, financial software, and manual bill scanning system, and classify them into structured data and unstructured data according to file characteristics; S1-1: For structured data, the data format conversion unit is called to perform format parsing. For spreadsheet files in .xls and .xlsx formats, their contents are extracted into standard database tables according to the table field structure. For text documents in .doc and .pdf formats, the text content is extracted by the parser and converted into standard text records for subsequent field mapping and classification. S1-2: For unstructured data, i.e., image files, the image preprocessing unit is called to perform image denoising, image segmentation, and OCR text recognition operations. The financial voucher content in the image file is converted into character information and the corresponding account field structure is generated, completing the initial text processing. S1-3 unifies the structured data and unstructured data that have completed format conversion into intermediate data and temporarily caches them in the system's data buffer unit. Intermediate data specifically refers to a data set that has completed basic structure unification and has a standard field structure, but has not yet undergone data cleaning, field verification, and consistency verification. It will be further processed by the data preprocessing module.

[0025] It should be noted that the sources of financial data files are diverse, and the file formats include spreadsheets (.xls, .xlsx), text documents (.doc, .pdf) and image files (such as .png, .jpg, etc.); after the data collection is completed, the system calls the data format conversion unit to uniformly convert and parse the above multi-source heterogeneous data. The specific processing is: automatic decompression and splitting of compressed and encapsulated data packets; text conversion of scanned image data through OCR technology; and uniform mapping of data file structures with format differences into standard field formats.

[0026] The purpose of the processing is to convert raw accounting data of different formats and sources into a unified intermediate data set with recognizable structure and parsable content, so that subsequent modules of the system can efficiently call and process it.

[0027] Step S2: Call the data cleaning unit, data verification unit and data integration unit in the data preprocessing module to perform comprehensive cleaning, verification and integration processing on the intermediate data formed in S1 to construct an initial financial data set.

[0028] S2-1, through the data cleaning unit, based on business rule verification and missing value processing, specifically including: abnormally marking and removing records that violate accounting rules; filling in data items with null values ​​and missing fields using interpolation algorithms based on mean, median, and historical trends; and performing deduplication operations on duplicate records to ensure that each account is unique and valid; S2-2, through the data verification unit, performs semantic and structural consistency verification on various key financial fields, performs debit and credit balance verification, account code legitimacy verification, and field format verification; The debit / credit balance verification automatically calculates the debit / credit amount in each account and checks whether the total meets the debit / credit calculation formula. If it is not balanced, it will be marked as a logical error. Account code validity verification involves calling the account dictionary to check whether the account code used in the account exists in the legal code set. Field format verification includes verifying whether the amount field is a valid number, whether the date field format is standardized, and whether the personnel code complies with unified rules. S2-3, through the data integration unit, matches and aggregates multi-source data according to the accounting voucher number, serial number and department ID. The specific operations are as follows: for vouchers, reports and approval records from different systems, establish a one-to-many or many-to-one mapping relationship with the voucher number as the primary key; automatically fill in missing primary key and foreign key fields to build a unified logical table structure; write the integrated data into the integration result temporary storage unit to achieve unified temporary storage of structured financial data.

[0029] It should be noted that the data cleaning unit is responsible for systematic detection and processing of duplicate records, format errors and missing fields in the original intermediate data. The system has a built-in set of verification expressions based on industry rules and supports custom extended rule sets; common exceptions such as "voucher number is empty", "amount field contains non-numeric characters", "reimbursement time is greater than accounting time", etc. will be automatically marked and eliminated; for missing items such as "invoice remarks are empty", fields can be completed through mean interpolation of the same type of data, median estimation, etc.

[0030] The data verification unit has built-in structural consistency verification logic, which ensures the semantics and structure of each record are consistent by traversing the key logical fields in the financial entries; the debit and credit balance relationship of each voucher is verified, and if "debit and credit are not equal", it will be directly eliminated; the accounting subject code is compared with the system dictionary library through an external interface. If it is an illegal code, it will be marked as an exception and moved to the pending area.

[0031] In step S3, the data classification module performs intelligent classification recognition and structured metadata generation on each data record in the initial financial data set output by the data preprocessing module. Based on the built-in rule engine and classification model, the financial feature information is automatically extracted and the data type determination is completed. Then, a standardized metadata structure is generated based on the classification results to provide necessary label support for subsequent data hierarchical storage, access scheduling and index retrieval.

[0032] S3-1, using the data type identification unit in the data classification module, extracts features and classifies each data record in the initial financial data set, constructing a feature vector based on multiple feature dimensions; the multiple feature dimensions include financial field features, text keywords, file structure features, and source identification features; Among them, the financial field characteristics are specifically the amount, accounting subject code and settlement method; The text keywords are specifically typical words in titles such as "report" and "voucher"; the file structure characteristics are specifically table density, number of fields, and field positions; the source identification characteristics are specifically the data source system ID and interface code; Combined with the above feature vectors, the logistic regression model is used to classify the feature vectors, as shown in the formula:

[0033] in, Indicates the probability that the input data is "voucher data", is the eigenvector corresponding to the financial data, is the training weight parameter of the model, b is the bias term, and y is the output label representing the type category; S3-2, based on the results of the classification completed in S3-1, generates a structured metadata tag for each classified data record through the metadata generation unit. The tag consists of multiple predefined metadata fields, among which the metadata fields include: data type identifier, time interval, source system identifier, accounting subject code, amount importance level, system creation time and storage timestamp.

[0034] Each structured metadata tag is bound to its corresponding data record and written uniformly into the metadata storage unit in the data classification module as the logical identifier of the data record.

[0035] It should be noted that each piece of data is specifically each independent record in the "initial financial data set after structural standardization" output by the data preprocessing module, which may come from different files and systems; model training is completed using labeled sample sets and cross-validation mechanisms. After deployment and online, it supports automatic classification and processing of real-time data and outputs standard data type labels. In order to take into account both recognition accuracy and computational efficiency, the system uniformly selects the logistic regression model as the main classifier in actual deployment, and does not use multi-model hybrid methods. Relevant model training is completed at one time before deployment.

[0036] Step S4: The storage optimization module formulates a storage strategy for data classification based on the current data access trend and the resource status of each node. The storage strategy includes a hot and cold data tiering strategy and a node load balancing strategy. S4-1, through the trend analysis unit, based on historical access logs and metadata records, statistics the access frequency and change rate of each type of data in multiple time periods, and calculate the popularity score of each type of data, as shown in the formula:

[0037] in, For the heat score, is the data access frequency of the current cycle, is the visit growth rate compared with the previous period, Contribution weight coefficient for heat, Configure system parameters; set heat thresholds ,like > , then this type of data is marked as "hot data", otherwise it is "cold data"; S4-2, through the storage status monitoring unit, obtains the status indicators of each storage node in real time, including available storage capacity, current I / O pressure and node write rate; Based on the preset load threshold , the system determines the status of each node as: the first storage state (high load), the available capacity is less than ; Second storage state (low load), available capacity is higher than ; S4-3, through the hot and cold data management unit, executes the hot and cold data tiered processing logic based on the heat determination of S4-1 and the node status of S4-2, as follows: If the data is hot data and the current cycle forecast shows a continued high popularity trend, it will be written to high-speed media first. If the data is cold data, or the data is hot data but all high-performance nodes are under high load, it will be written to low-cost, large-capacity storage. During the node selection phase, the write sensitivity of each candidate node, i.e., the amount of writes per unit time, is calculated. Nodes with the lowest sensitivity are prioritized for writing. If all nodes are close to saturation, the load balancing unit is activated to dynamically allocate data based on the remaining capacity to ensure system stability. S4-4, after determining the target node, using the storage policy unit, generates a structured storage instruction for each data. The storage instruction consists of multiple fields, including: target node number, data replication factor, data compression configuration, and write priority tag; The generated instructions are uniformly encapsulated into a standard format and transmitted by the system to the storage execution engine to complete the actual writing of subsequent data and policy execution.

[0038] It should be noted that the heat score is defined as a measure of whether a certain type of data is likely to be frequently accessed in future cycles, affecting whether it should be given priority in high-performance storage resources; the write sensitivity value is used to describe the write load intensity of a node per unit time and is an important indicator for dynamic load scheduling; the hot and cold data management unit and the load balancing unit are both functional units under the storage optimization module. The former is responsible for heat decision-making and media stratification, and the latter is responsible for performing cross-node balanced write scheduling under resource pressure.

[0039] Step S5: Complete the final writing, encryption, replication, and compression of the data by calling the data storage module to ensure the security, high availability, and storage efficiency of the financial data in a distributed environment; S5-1, using the storage node management unit to write the classified and marked business data in batches to the distributed file system using the target storage node determined in step S4 as a write entry; After writing is completed, the system automatically records the write log, timestamp, and data writing status, and updates the local storage view of each node to support subsequent data synchronization and status monitoring; S5-2: Data involving financial information, audit records, and sensitive identity fields is encrypted using the data encryption unit to ensure that the data is encrypted before being stored on disk. The system defaults to using the national standard symmetric encryption algorithm SM4. The encryption model is defined as follows:

[0040] Where M is the plaintext data to be encrypted, K is the symmetric key dynamically generated by the system through the key management module, and C is the encrypted ciphertext result. Each piece of encrypted data will record the encryption algorithm identifier and key index number in its metadata tag to facilitate subsequent authorized decryption and audit recovery. S5-3, through the data replication unit, according to the system preset number of replicas, after the primary write node completes, the data replicas are automatically synchronized to the backup node, and the node location information, replication status and version mark of each replica are recorded to prevent data unavailability caused by single point failure; S5-4: For large-scale unstructured files, including invoice images, PDF contracts, and bill scans, the data compression unit performs compression processing before storage. The compression process is based on a compression algorithm and supports common compression standards such as ZIP, LZO, and Snappy. The compression configuration is used to control the compression behavior parameters and can be dynamically set based on data type and access frequency. After compression is complete, each compression record will have the following information appended to the corresponding metadata tag: original file size, compressed file size, type of compression algorithm used, and whether subsequent decompression caching is enabled.

[0041] Step S6: Using the data access module, an efficient index structure is constructed for the newly written data to improve data access speed and optimize the query plan and execution path. S6-1, using the index unit, establishes a structured search index for the written data records, specifically including: primary key index, composite index, full-text index, and time series index; The primary key index uses unique identification fields such as voucher number and invoice number as index keys to support precise positioning. The composite index combines multiple fields such as account code and time interval to form a composite query path. The full-text index constructs an inverted index for text data such as contracts and audit records to improve fuzzy matching efficiency. The time series index partitions the entry time and invoice time fields to support time range search and rolling window statistics. After the construction is completed, the system will automatically update the distributed index directory and uniformly record the logical distribution, node, update time and expiration strategy of each index structure; S6-2, through the query processing unit, schedules and optimizes various user-initiated query requests, including audit evidence collection and financial statement extraction. The internally integrated query optimizer generates the optimal execution plan based on currently available indexes and metadata statistics, including selectivity, data distribution, and interval skew. The query optimizer automatically selects the shortest path index combination, the lowest-cost join sequence, and the appropriate data scanning method. The query processing unit can handle complex SQL query requests, fuzzy query conditions and multi-table join logic, and supports query link tracking, execution status feedback and interruption recovery; S6-3, using the cache management unit, dynamically caches hot data blocks and hot index pages in memory or SSD during runtime, improving system response speed in high-frequency access scenarios: Hotspot data is identified based on historical access frequency, access burstiness, and business priority; The cache uses LRU (least recently used) and LFU (least frequently used) strategies for automatic replacement; The system supports setting cache granularity by data type / source / user role. The status of all cache blocks will be recorded in the system's cache view table in real time, and the cache hit rate and invalidation strategy will be updated synchronously to improve overall query throughput performance.

[0042] Step S7, through the system management module, performs archiving of long-term financial data, continuous monitoring of system operation status, as well as data backup and system maintenance tasks to ensure the sustainability, maintainability and security of the entire data management system.

[0043] S7-1, through the data archiving unit, the system identifies data records that have not been accessed for a long time or have exceeded the retention period according to the configured data lifecycle policy and performs archiving operations, specifically: For historical data that exceeds the age limit, sensitive fields are first desensitized. The desensitized data is then migrated to offline archival media, including low-frequency access object storage or cold standby tape libraries, to relieve pressure on primary storage. All archiving actions will add fields such as "archiving status identifier", "archiving timestamp", and "archiving path" to the system metadata tag, and record archiving logs to support subsequent recovery or audit queries; S7-2, through the system monitoring unit, continuously collects and analyzes the operating performance indicators and resource usage of each subsystem, including: CPU utilization, memory utilization, network throughput, available capacity of each storage node, I / O throughput, write latency, and response delays of system maintenance tasks and data services; Once the system detects an abnormal state, it immediately triggers a preset alarm mechanism and automatically calls on backup resources to complete the failover, ensuring uninterrupted system operation. Abnormal states include continuous packet loss on a node, storage exceeding 90%, and long-term CPU saturation. S7-3, through the maintenance scheduling unit, the system regularly performs system-level maintenance and disaster recovery operations according to scheduling rules, mainly including: log cleaning, system upgrades and data backup; Log cleaning automatically cleans up historical access logs and redundant intermediate processing files that have not been accessed for a long time; system upgrades push component patches, rule updates, and index reconstruction tasks according to set windows; data backup supports both full and incremental backup strategies, regularly replicating master data and metadata to off-site disaster recovery storage nodes and recording backup image indexes to enhance data recovery capabilities. All operation and maintenance operations are recorded in maintenance logs, and administrators can view execution status, historical task records and current load information in the system management background.

[0044] Please refer to Figure 2 , Figure 2 This is an architecture diagram of a financial accounting data storage and management system based on big data provided by an embodiment of the present invention, which includes the following modules: Data acquisition module, data preprocessing module, data classification module, storage optimization module, data storage module, data access module and system management module.

[0045] The data acquisition module is used to collect raw accounting data from multiple sources such as the enterprise ERP system, financial software platform, and bill scanning equipment, covering structured files (such as spreadsheets and documents) and unstructured image data; the module includes a raw data acquisition unit, a data format conversion unit, an image preprocessing unit, and a data buffer unit; the format conversion unit realizes the conversion of various files into a standard structured format, the image preprocessing unit extracts image text information through OCR recognition, and the data buffer unit temporarily stores data files that have not yet been cleaned, providing input cache for subsequent processing.

[0046] The data preprocessing module is responsible for cleaning, verifying and integrating the collected intermediate data to improve data consistency and reliability; this module includes a data cleaning unit, a data verification unit, a data integration unit and an integration result temporary storage unit; the cleaning unit is used to eliminate duplicate records and fill in missing fields; the verification unit verifies the legality of fields, debit and credit balance and other accounting rules; the integration unit merges data from different sources according to key fields (such as voucher number, account code), unifies the storage format, and temporarily stores it locally for subsequent analysis.

[0047] The data classification module is used to automatically classify and identify preprocessed standard data sets and generate metadata labels; this module includes a data type identification unit, a metadata generation unit, and a metadata storage unit; by constructing feature vectors, the classification model can identify data types (such as vouchers, reports, contracts, etc.) and generate a metadata structure for each record that contains fields such as type, amount level, time interval, system source, etc., and store them uniformly in the metadata storage unit as logical index support.

[0048] The storage optimization module formulates the optimal storage strategy based on the data access popularity and system resource status; this module includes a trend analysis unit, a storage status monitoring unit, a hot and cold data management unit, a load balancing unit and a storage policy unit; the trend analysis unit evaluates data popularity, the monitoring unit perceives the resource load of each node, the hot and cold management unit allocates hot data to high-performance media and writes cold data to low-cost storage, the load balancing unit dynamically allocates write paths when resources are tight, and finally the policy unit outputs structured instructions to control the data writing process.

[0049] The data storage module is used to perform physical writing, encryption protection, backup replication and compression optimization of data to ensure storage security and high availability; the module includes a storage node management unit, a data encryption unit, a data replication unit and a data compression unit; the encryption unit uses the national secret SM4 algorithm by default to perform group encryption on sensitive data, the replication unit synchronizes the master node data to the backup node, and the compression unit performs standard compression on large-volume images or PDF files and annotates relevant metadata. All write operations are coordinated and logged by the management unit.

[0050] The data access module supports fast querying, index building, and cache optimization of stored data, improving user access efficiency. This module includes an indexing unit, a query processing unit, and a cache management unit. The indexing unit generates primary key, full-text, and time-partitioned indexes based on metadata and fields. The query processing unit optimizes execution plans based on data distribution. The cache unit loads hot data or index pages into memory or SSD cache to improve responsiveness for high-frequency access.

[0051] The system management module is responsible for archiving old data, monitoring system status and scheduling periodic maintenance tasks to ensure the long-term stable operation of the system; this module includes a data archiving unit, a system monitoring unit and a maintenance scheduling unit; the archiving unit desensitizes data that has not been accessed for a long time and migrates it to low-frequency storage media according to life cycle rules, the monitoring unit continuously detects system load and automatically triggers alarms or resource switching, and the scheduling unit regularly performs data backup, log cleaning and system upgrade tasks to ensure data security and platform maintainability.

[0052] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for optimizing accounting data storage management based on big data, characterized in that: The following steps are involved: Step S1: Calling the raw data acquisition unit in the data acquisition module to collect financial accounting raw data files from the enterprise ERP system, financial software, and manual bill scanning system, and classifying them into structured data and unstructured data based on file characteristics; Step S2: Through the data preprocessing module, the processed structured data and unstructured data are comprehensively cleaned, verified, and integrated to construct an initial financial data set; Step S3: Through the data classification module, intelligent classification recognition and structured metadata generation are performed on each data record output by the data preprocessing module; Step S4: Using the storage optimization module, a storage strategy is developed for data classification based on current data access trends and resource status of each node. Step S5, by calling the data storage module, complete the physical writing of data, encryption protection before storage, copy synchronization and compression optimization; Step S6: Using the data access module, an index structure is constructed for the newly written data, and a query execution plan is generated based on the data distribution. Step S7: Through the system management module, long-term financial data archiving, continuous monitoring of system operation status, data backup and system maintenance tasks are performed.

2. The method for optimizing financial accounting data storage management based on big data according to claim 1, characterized in that: In step S1, the following sub-steps are also included: S1-1, for structured data, call the data format conversion unit to perform format parsing. For spreadsheet files in .xls and .xlsx formats, their contents are extracted into standard database tables according to the table field structure; for text documents in .doc and .pdf formats, the text content is extracted through the parser and converted into standard text records; S1-2: For unstructured data, the image preprocessing unit is called to perform image denoising, image segmentation, and OCR text recognition operations, converting the financial voucher content in the image file into character information and generating the corresponding account field structure; S1-3, unify the structured data and unstructured data that have completed format conversion into intermediate data and temporarily cache them in the system's data buffer unit.

3. The method for optimizing financial accounting data storage management based on big data according to claim 1, characterized in that: In step S2, the following sub-steps are also included: S2-1, through the data cleaning unit, based on business rule verification and missing value processing, records that violate accounting rules are marked as abnormal and eliminated; data items with null values ​​and missing fields are filled using interpolation algorithms based on mean, median, and historical trends; duplicate records are deduplicated; S2-2, through the data verification unit, performs semantic and structural consistency verification on various key financial fields, performs debit and credit balance verification, account code legitimacy verification, and field format verification; S2-3, through the data integration unit, match and aggregate multi-source data according to the accounting voucher number, serial number and department ID. For vouchers, reports and approval records from different systems, establish a one-to-many relationship with the voucher number as the primary key; automatically fill in missing primary key and foreign key fields to build a unified logical table structure; write the integrated data into the integration result temporary storage unit.

4. The method for optimizing financial accounting data storage management based on big data according to claim 1, characterized in that: In step S3, the following sub-steps are also included: S3-1, using the data type identification unit in the data classification module, extracts features and classifies each data record in the initial financial data set, constructing a feature vector based on multiple feature dimensions; the multiple feature dimensions include financial field features, text keywords, file structure features, and source identification features; Combined with the above feature vectors, the logistic regression model is used to classify the feature vectors, as shown in the formula: in, Indicates the probability that the input data is "voucher data", is the eigenvector corresponding to the financial data, is the training weight parameter of the model, b is the bias term, and y is the output label representing the type category; S3-2, based on the classification result, a metadata generation unit generates a structured metadata label for each classified data record, the label consisting of multiple predefined metadata fields; Each structured metadata tag is bound to its corresponding data record and written uniformly into the metadata storage unit in the data classification module as the logical identifier of the data record.

5. The method for optimizing financial accounting data storage management based on big data according to claim 1, characterized in that: In step S4, the following sub-steps are also included: S4-1, through the trend analysis unit, based on historical access logs and metadata records, statistics the access frequency and change rate of each type of data in multiple time periods, and calculate the popularity score of each type of data, as shown in the formula: in, For the heat score, is the data access frequency of the current cycle, is the visit growth rate compared with the previous period, Contribution weight coefficient for heat, Configure parameters for the system; Setting the heat threshold ,like > , then this type of data is marked as "hot data", otherwise it is "cold data"; S4-2, through the storage status monitoring unit, obtains the status indicators of each storage node in real time, including available storage capacity, current I / O pressure and node write rate; Based on the preset load threshold , determine the status of each node; S4-3, through the hot and cold data management unit, based on the heat determination and node status, executes the hot and cold data hierarchical processing logic, as follows: If the data is hot data and the current period forecast shows a continued high popularity trend, it is written to high-speed media; if the data is cold data, it is written to low-cost large-capacity storage; if the data is hot data but all high-performance nodes are under high load, it is also written to low-cost large-capacity storage; During the node selection phase, the write sensitivity of each candidate node is calculated, and the node with the lowest sensitivity is selected first for writing. If all nodes are close to saturation, the load balancing unit is activated to dynamically allocate data according to the remaining capacity ratio; S4-4, after determining the target node, uses the storage policy unit to generate a structured storage instruction for each data. The storage instruction consists of multiple fields, including: target node number, data replication factor, data compression configuration and write priority tag.

6. The method for optimizing financial accounting data storage management based on big data according to claim 1, characterized in that: In step S5, the following sub-steps are also included: S5-1, through the storage node management unit, uses the determined target storage node as the write entry to write the classified and marked business data in batches into the distributed file system; After writing is completed, the write log, timestamp, and data writing status are automatically recorded, and the local storage view of each node is updated; S5-2: Data involving financial information, audit records, and sensitive identity fields are encrypted using the data encryption unit. The default encryption algorithm is SM4, a national standard symmetric encryption algorithm. The encryption model is defined as follows: Where M is the plaintext data to be encrypted, K is the symmetric key dynamically generated by the system through the key management module, and C is the encrypted ciphertext result; each encrypted data will record the encryption algorithm identifier and key index number in its metadata tag; S5-3, through the data replication unit, according to the system preset number of copies, after the primary write node is completed, the data copy is automatically synchronized to the backup node, and the node location information, replication status and version mark of each copy are recorded; S5-4: For large-scale unstructured files, including invoice images, PDF contracts, and bill scans, the data compression unit performs compression processing before storage. The compression process is based on the compression algorithm. The compression configuration is used to specifically control the compression behavior parameters, which are dynamically set according to the data type and access frequency. After compression is complete, each compression record will have the following information appended to the corresponding metadata tag: original file size, compressed file size, type of compression algorithm used, and whether subsequent decompression caching is enabled.

7. The method for optimizing financial accounting data storage management based on big data according to claim 1, characterized in that: In step S6, the following sub-steps are also included: S6-1, using the index unit, establishes a structured search index for the written data records, specifically including: primary key index, composite index, full-text index, and time series index; After the construction is completed, the system will automatically update the distributed index directory and uniformly record the logical distribution, node, update time and expiration strategy of each index structure; S6-2, through the query processing unit, schedules and optimizes various query requests initiated by users: using the internal integrated query optimizer to generate execution plans based on the currently available indexes and metadata statistics, including selectivity, data distribution, and interval skew; S6-3, using the cache management unit, dynamically caches hot data blocks and hot index pages in memory during runtime: hot data is identified based on historical access frequency, access burstiness, and service priority; the cache uses LRU and LFU strategies for automatic replacement; The system sets the cache granularity according to data type, source and user role. The status of all cache blocks will be recorded in the system's cache view table in real time, and the cache hit rate and invalidation strategy will be updated synchronously.

8. The method for optimizing financial accounting data storage management based on big data according to claim 1, characterized in that: In step S7, the following sub-steps are also included: S7-1, through the data archiving unit, identifies data records that have not been accessed for a long time and have exceeded the retention period according to the configured data lifecycle policy and performs archiving operations, specifically: For historical data that exceeds the age limit, sensitive fields are first desensitized; the desensitized data is then migrated to offline archiving media to relieve pressure on primary storage. All archiving actions will add fields to the system metadata tag and record archive logs to support subsequent recovery; S7-2, through the system monitoring unit, continuously collects and analyzes the operating performance indicators and resource usage of each subsystem, including: CPU utilization, memory utilization, network throughput, available capacity of each storage node, I / O throughput, write latency, and response delays for system maintenance tasks and data services; When the system detects an abnormal state, it immediately triggers the preset alarm mechanism and automatically calls the backup resources to complete the fault switch; S7-3, through the maintenance scheduling unit, the system regularly performs system-level maintenance and disaster recovery operations according to scheduling rules, mainly including: log cleaning, system upgrades and data backup; All operation and maintenance operations are recorded in maintenance logs, and administrators can view execution status, historical task records and current load information in the system management background.

9. A financial accounting data storage management system based on big data, applied to the financial accounting data storage management optimization method based on big data according to any one of claims 1 to 8, characterized in that: Includes the following modules: Data acquisition module, data preprocessing module, data classification module, storage optimization module, data storage module, data access module and system management module; The data acquisition module includes a raw data acquisition unit, a data format conversion unit, an image pre-processing unit and a data buffer unit, and is used to collect raw accounting data from multiple sources such as the enterprise ERP system, the financial software platform and the bill scanning device; The data preprocessing module includes a data cleaning unit, a data verification unit, a data integration unit and an integration result temporary storage unit, which are responsible for cleaning, verifying and integrating the collected intermediate data; The data classification module includes a data type identification unit, a metadata generation unit and a metadata storage unit, which are used to automatically classify and identify the pre-processed standard data set and generate metadata tags; The storage optimization module includes a trend analysis unit, a storage status monitoring unit, a hot and cold data management unit, a load balancing unit, and a storage strategy unit, which formulates storage strategies based on the access popularity of data and the status of system resources; The data storage module includes a storage node management unit, a data encryption unit, a data replication unit and a data compression unit, which are used to perform physical writing, encryption protection, backup replication and compression optimization of data; The data access module includes an index unit, a query processing unit and a cache management unit, which supports fast query, index construction and cache optimization of stored data; The system management module includes a data archiving unit, a system monitoring unit, and a maintenance scheduling unit, which are responsible for archiving old data, monitoring system status, and scheduling periodic maintenance tasks.

Images