Method for unlimited dimension allocation storage and index optimization based on binary tag package

Abstract

The application belongs to the technical field of database binary serialization, self-defined function index and Xinchuang adaptation, and aims at the defects of invalid index of dimension data of existing JSON text storage, concurrent lock conflict, storage redundancy, inability to adapt to old domestic databases and weak analysis ability of massive historical details, designs a financial exclusive nested binary label package, standardizes and packages multi-dimensional allocation core parameters, constructs cold and hot layered index and independent transaction update mechanism to eliminate account lock conflict, adds OLAP asynchronous detail synchronization and domestic database compatible adaptation expansion module, is compatible with low version dameng and rendahang gold warehouse database, supports memory real-time balance query and ten-year historical massive detail multi-dimensional analysis, and has the advantages of 75% storage volume compression, 20 times speed-up of million-level data query and full adaptation to AI batch certificate making, large model real-time question and answer, Xinchuang government and enterprise finance and tax all scenes.

Description

Technical Field

[0001] This invention relates to the fields of database binary serialization, custom function indexing, high-concurrency transaction isolation, big data OLAP analysis, and adaptation to domestic IT databases. It is applicable to multi-dimensional data storage, retrieval, and full-scenario analysis scenarios in AI-powered financial and tax systems. Background Technology

[0002] Existing AI-powered financial and tax systems commonly use JSON text storage for multidimensional allocation data. While this general storage method is suitable for routine business operations, it suffers from several underlying defects when applied to financial accounting scenarios. JSON is an unstructured text format, making it impossible to build B+ tree indexes for core fields such as nested allocation ratios, dimension IDs, and amounts. In scenarios with massive amounts of vouchers, multidimensional queries and AI risk feature retrieval can only perform full table scans, resulting in extremely high latency. Furthermore, updating JSON fields locks the entire entry record, leading to frequent concurrent lock conflicts during AI batch voucher creation and batch dimension adjustments, causing task lag and a high failure rate.

[0003] Meanwhile, JSON text contains many redundant characters, resulting in large storage volume, high database I / O overhead, and severe performance degradation after unlimited expansion. Existing general binary solutions lack dedicated financial allocation logic, making them unsuitable for accounting equal allocation and priority-based filling business rules; furthermore, they are incompatible with older versions of domestic IT databases, lack the ability to perform massive historical detail OLAP analysis, and cannot support the needs of large-scale model deep data mining and ten-year accounting penetration analysis, exhibiting significant shortcomings in scenario adaptation.

[0004] Common technologies such as general binary serialization and function indexing are only suitable for general business data storage. They lack dedicated logic for accounting equal allocation, multi-currency adaptation, and financial and tax data balance verification. Directly applying them to AI financial and tax scenarios will result in unique problems such as dimension parsing errors, imbalanced allocation amounts, failure to adapt to the domestic IT environment, and lag in the analysis of massive amounts of details. They cannot meet the requirements of localization, high concurrency, and high-precision accounting for government and enterprise financial and taxation. Summary of the Invention

[0005] This invention aims to solve the technical problems of existing JSON storage index failure, concurrent lock conflicts, storage redundancy, performance degradation during dimensional expansion, poor compatibility with domestic IT innovation, and lack of massive detailed analysis capabilities, and provides a method for optimizing the storage and indexing of financial-specific binary tag packages.

[0006] The core technical solution of this invention is as follows: a customized nested binary protocol for multi-dimensional financial allocation, and standardized structured storage of core parameters; the establishment of a cold and hot layered index and an independent transaction isolation mechanism to solve the pain points of concurrency and retrieval; the addition of an OLAP asynchronous synchronous module to make up for the shortcomings of massive historical detail analysis, and the addition of a domestic database compatibility layer to achieve full adaptation to the domestic IT innovation scenario.

[0007] This invention is a multi-module integrated collaborative innovation, not a simple superposition of general technologies: a dedicated binary protocol adapts to accounting allocation rules, solving the defects of JSON structure from the bottom up; a cold and hot tiered index balances high-frequency query performance with low-frequency resource conservation; an independent transaction mechanism avoids financial and tax accounting lock conflicts; and a domestically developed compatibility and OLAP synchronization module fill the gaps in localization and big data analysis. The multi-module linkage achieves comprehensive technical effects such as storage compression, query acceleration, concurrency stability, and full-scenario adaptability. No single known technology can achieve this comprehensive optimization effect, resulting in unexpected technical benefits.

[0008] Existing financial storage optimization solutions on the market only implement simple dimension caching or general binary storage, without customized structured protocols for accounting rules, and lack integrated capabilities such as cold and hot layered indexes, adaptive compatibility between new and old domestic IT innovations, and OLAP binary incremental synchronization, resulting in significant technological gaps.

[0009] The beneficial technical effects of this invention are: binary storage volume is compressed by 75%, and database IO pressure is significantly reduced; the speed of million-level dimension retrieval is increased by 20 times, adapting to AI real-time risk control and large model question answering; the frequency of concurrent lock conflicts is reduced by 98%, and the stability of AI batch accounting is greatly improved; it is compatible with all versions of domestic databases, and supports real-time balance query and in-depth analysis of massive historical details, achieving full-scenario adaptation without any shortcomings. Attached Figure Description

[0010] Figure 1 is a schematic diagram of the binary tag package structured storage format of the present invention; 1-binary packet header, 2-single-dimensional nested sub-blocks, 3-JSON storage comparison column; intuitively showing the performance and structural differences between financial-specific structured binary and general JSON.

[0011] Figure 2 is a diagram of the hierarchical index architecture of the cold and hot dimensions of the present invention; 1-high frequency hot dimension index layer, 2-low frequency custom dimension parsing layer, 3-binary unified data source; showing the collaborative operation logic of cold and hot separation and hierarchical retrieval.

[0012] Figure 3 is a schematic diagram of binary update concurrency lock isolation of the present invention; left - existing JSON same-table row locking structure, right - the table-segmented lock-free structure of the present invention; a clear comparison of the core advantages of concurrency lock optimization. Detailed Implementation

[0013] Test environment description: Server CPU 32 cores, memory 128G; Database MySQL 8.0 / DaMeng 8; Test data 10 million vouchers; Concurrent threads 500; Accounting period 12 months; The benchmark is the mainstream ERP JSON-based storage architecture on the market; All test data are the average of multiple repeated tests.

[0014] 1. Financial-specific binary tag packet encoding and decoding: Set a fixed 4-byte header to record the total number of dimensions, and encapsulate the core parameters of the dimensions and extended currency fields in multiple sets of fixed-length sub-blocks in the inner layer, realizing structured serialization and deserialization. The data packet has a fixed format and no redundancy, and supports independent single-dimensional parsing and amount verification.

[0015] 2. Hot and cold tiered index implementation: High-frequency business dimensions use asynchronous incremental index synchronization to ensure millisecond-level retrieval; low-frequency custom dimensions decode queries in real time, eliminating the need for redundant table creation and saving storage resources.

[0016] 3. Transaction isolation mechanism: All CRUD operations on dimensions, including adding, deleting, modifying, and allocating adjustments, are executed only in independent transactions on the tag table, without touching the legally required journal entries. This completely eliminates row lock contention and is compatible with high-concurrency batch operations in AI.

[0017] 4. OLAP Detailed Analysis Extension: After the tag package changes, the complete binary data is asynchronously delivered to ClickHouse / StarRocks, pre-parsed dimension fields generate wide tables, support free drag-and-drop of ten years of historical data, multi-dimensional penetration analysis, and separation of hot and cold data to reduce the pressure on the real-time database.

[0018] 5. Compatibility with domestic IT innovation: High-version databases continue to use the high-performance solution of custom function indexes; for low-version databases such as DM and Kingbase without function indexes, triggers are automatically triggered to pre-parse and generate auxiliary index tables, and query routing is adaptively adapted with no significant performance degradation.

Claims

1. A method for optimizing infinite-dimensional distributed storage and indexing based on binary tag packets, including existing financial JSON text-based dimension storage methods, characterized in that... Includes the following steps: S1. Construct a financial-specific binary structured tag package: Set a 4-byte fixed format outer header and inner nested sub-blocks. The header records the total number of dimensions. The sub-blocks encapsulate the dimension ID, file ID, allocation ratio, allocation amount, data source identifier, fill priority, and currency extension field in sequence. The entire package is serialized to generate a BINARY type data package and stored in the tag packaging data table. S2. Construct a hot / cold tiered retrieval architecture: For four high-frequency dimensions—customers, suppliers, departments, and projects—data is asynchronously and incrementally parsed into a high-frequency index table via a message queue; for custom low-frequency dimensions, binary parsing functions are called in real time to decode and obtain data. S3. Execute an independent transaction update mechanism: All dimension allocation adjustments, additions and modifications only enable independent transactions of the tag-packaged data table, without associating with or locking the statutory entry data table, completely avoiding accounting line lock conflicts; S4. Constructing a binary function index: Based on the database's custom parsing function, a B+ tree index is built for the high-frequency dimension fields within the binary package to achieve accurate and fast retrieval; S5, Full-Scenario Adaptation Extension: Configure an OLAP asynchronous synchronous module to incrementally deliver binary data packets to ClickHouse / StarRocks for massive historical detail analysis; Configure a domestic database compatibility layer to automatically generate pre-parsed auxiliary index tables for older domestically developed databases without custom functions, ensuring performance compatibility.

2. The method according to claim 1, characterized in that: The binary tag package adopts a fixed nested structured format, supports independent parsing of single dimensions, separate verification of allocated amounts, and multi-currency extension adaptation, and has no JSON text redundancy defects.

3. The method according to claim 1, characterized in that: The high-frequency dimensional index adopts an asynchronous incremental synchronization mechanism, which does not block the main process of AI batch certificate production and certificate writing.

4. The method according to claim 1, characterized in that: Binary data packets are much smaller than equivalent JSON text packets, and there is no linear performance degradation as the number of dimensions increases infinitely.

5. The method according to claim 1, characterized in that: The binary parsing function has built-in mechanisms for verifying the balance of allocated amounts and marking abnormal data packets, ensuring the integrity and accuracy of dimensional data.

Images