Entity library table generation method and system based on dynamic data model

Abstract

The invention discloses an entity library table generation method and system based on a dynamic data model, and the system comprises a dynamic materialization engine module which configures a server cluster, deploys a three-node ZooKeeper cluster for a metadata server, and is responsible for storing model version metadata; a difference calculation module of an FPGA acceleration card is configured for a calculation node, and the Levenshtein editing distance is calculated; the elastic storage distribution module is used for setting a storage distribution strategy of a mixed layout of column storage and row storage and recombining a data space; and the model storage consistency guarantee module is used for setting a transaction log capture layer at a cloud end to monitor a change event of the model definition library, generating a CDC change data capture log with a timestamp, and configuring a two-stage materialization protocol at the same time. According to the scheme, the problem of service interruption caused by the fact that a traditional relational database needs to be shut down to execute DDL operation is solved, and real-time synchronization from model change to table structure adjustment is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database architecture and data model engineering, and in particular to an entity library table generation method and system based on a dynamic data model. BACKGROUND

[0002] In the daily operation of an enterprise, a large amount of business data is generated, which is stored in different business systems, and the data structure is complex and dynamically changing. For example, in an e-commerce enterprise, commodity information, order information, user information and other data will change constantly with factors such as market strategy adjustment and new business expansion. The traditional database table design method is difficult to quickly adapt to such dynamic changes, resulting in many problems in system data update and expansion, such as data inconsistency and system performance degradation.

[0003] The current mainstream database architecture mainly adopts two schemes: 1. Static table structure design (such as MySQL, Oracle, etc. relational database): the physical table structure of fixed fields needs to be defined in advance, and the change needs to perform ALTER TABLE and other DDL operations. This mode has the defects of high schema change cost and high service downtime risk when the business rules are frequently iterated.

[0004] 2. ORM dynamic mapping scheme (such as Hibernate, Entity Framework): through object-relational mapping to realize the decoupling of the logical layer and the physical layer, it still relies on pre-compiled entity classes and fixed mapping rules, and cannot realize runtime table structure adjustment.

[0005] In the above existing technical solutions, the database architecture has systematic defects due to the mismatch between the static table structure and the dynamic business requirements, such as: lack of physical table structure dynamic, resource allocation rigidity, and model-driven fault; in the financial industry, the regulatory policies change frequently, and financial institutions need to quickly adjust the business system and data storage structure according to the new policy requirements. Therefore, a technical solution and architecture is needed to help financial institutions quickly adapt to these changes and ensure the compliance of data and the stability of the system. SUMMARY

[0006] In view of the above technical problems, the present application provides an entity library table generation method and system based on a dynamic data model, which realizes the real-time materialization of an abstract data model into a dynamically adjustable physical database table structure, solving the adaptability problem of traditional ORM framework and static table structure in the scene of frequent business changes.

[0007] The present application is implemented by using the following technical solutions: In a first aspect, an entity library table generation system based on a dynamic data model includes the following modules: Dynamic materialization engine module: Configure server cluster, deploy a 3-node ZooKeeper cluster on the metadata server to store model version metadata; equip the computing nodes with FPGA acceleration card difference calculation modules to calculate Levenshtein edit distance; Elastic storage allocation module: Sets a storage allocation strategy that combines columnar and row-based storage layouts, and reorganizes the data space; Model storage consistency guarantee module: Set up a transaction log capture layer in the cloud to listen for change events in the model definition library, generate CDC change data capture logs with timestamps, and configure a two-phase materialization protocol.

[0008] Specifically, the components of the dynamic materialization engine module include: Model parser: Parses abstract data model definitions in real time and generates a metadata tree carrying version identifiers; Difference calculation module: Based on the Levenshtein distance algorithm, it compares the structural changes between model versions and outputs the minimum change set; Execution scheduler: Automatically selects the switching strategy based on changeset priority.

[0009] Specifically, the elastic storage allocation module employs a hybrid layout strategy of columnar and row-based storage, automatically selecting the physical storage format based on the field access frequency defined in the model; and through pre-allocated blank storage pages and dynamic pointer reorganization, it enables real-time adjustment of storage space when fields are added or deleted.

[0010] Specifically, the automatic selection of the physical storage format is as follows: Query fields with a frequency > 1000 times / minute are designated as hot fields and stored in columnar format, while query fields with a frequency < 10 times / day are designated as cold fields and dynamically compressed into row-based storage. The reorganization of the data space specifically involves: when a field is deleted, marking the corresponding block of the storage page as recyclable, shifting subsequent field data forward through pointer redirection, and triggering the garbage collection thread.

[0011] Specifically, the model storage consistency guarantee module also includes: for edge devices, creating a virtual table in the embedded database to map the latest model version, and synchronizing structural changes between the base table and the virtual table through triggers.

[0012] Specifically, the two-phase materialization protocol in the model storage consistency guarantee module includes: During the preparation phase, the compatibility of the new model with existing data is verified in the verification node cluster. During the commit phase, distributed locks are used to ensure that all data nodes synchronously switch the table structure version.

[0013] On the other hand, the entity database table generation method based on dynamic data models includes the following steps: Step S1: Data model definition. Based on business requirements, use the corresponding data modeling tools to define a dynamic data model. Step S2: Entity database table generation. When a data model change notification is received or a timed trigger is initiated, the latest data model is read from the data model repository, and the corresponding database table structure is automatically generated according to the definition in the data model. Step S3: During operation, when new business data is generated, data is stored and operated according to the generated database table structure and data processing logic.

[0014] Specifically, step S1 further includes: defining a dynamic data model to describe business, entities and their relationships and attribute information; after the data model is defined, storing it in a data model repository for use in subsequent database table generation and system operation.

[0015] Specifically, step S2 further includes: generating data processing logic related to the database table structure, including code templates for data insertion, update and deletion operations, and view definitions for data queries.

[0016] Specifically, step S3 further includes: calling external business modules through a unified interface provided by the system to realize data query and update operations; during system operation, if the data model changes, the entity database table generation process is automatically triggered to regenerate the database table structure and data processing logic, and to ensure the smooth migration of existing data and seamless switching of the system.

[0017] The beneficial effects of this invention are as follows: This invention solves the business interruption problem caused by the need for system downtime to execute DDL operations in traditional relational databases, achieves real-time synchronization from model changes to table structure adjustments, eliminates storage waste caused by pre-designing redundant fields to cope with business changes, realizes elastic allocation of storage resources through an on-demand materialization mechanism, breaks through the limitation of ORM frameworks relying on pre-compiled entity classes, establishes an automated transmission path from data model changes to the physical storage layer, and ensures strong consistency between the logical model and physical storage. Specifically, it includes the following advantages: Improved architecture responsiveness: By using the version difference calculation mechanism of the dynamic materialization engine, the response time for database table structure adjustments is reduced from minutes to milliseconds in traditional DDL operations, and no service interruption is required throughout the process, significantly supporting high-frequency business iteration needs; Storage resource utilization optimization: The elastic storage allocation system is based on a column / row hybrid layout strategy, which can reduce the storage space waste caused by reserved redundant fields. Theoretical calculations show that it can reduce storage usage in typical business scenarios by 35%-60% (depending on field reuse rate). Enhanced system reliability: The model-storage consistency guarantee mechanism adopts a two-phase materialization protocol to ensure data integrity during structural changes, reducing the model breakdown incident rate, which is common in traditional ORM frameworks, from the order of 10^-4 to below the order of 10^-7. Reduced operation and maintenance costs: The automated model transfer mechanism eliminates the need for manual migration script writing, reducing the workload of database architecture maintenance by more than 70%, and is particularly suitable for multi-database table collaborative management in microservice scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the entity database table generation method based on a dynamic data model according to the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0022] The following is in conjunction with the appendix Figure 1 The following describes some embodiments of the present invention in detail. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] This invention proposes a method and system for generating entity database tables based on a dynamic data model. In a preferred embodiment, the system includes the following modules: Dynamic materialization engine module: Configure server cluster, deploy a 3-node ZooKeeper cluster on the metadata server to store model version metadata; equip the computing nodes with FPGA acceleration card difference calculation modules to calculate Levenshtein edit distance; Elastic storage allocation module: Sets a storage allocation strategy that combines columnar and row-based storage layouts, and reorganizes the data space; Model storage consistency guarantee module: Set up a transaction log capture layer in the cloud to listen for change events in the model definition library, generate CDC change data capture logs with timestamps, and configure a two-phase materialization protocol.

[0024] The technical principles and implementation of each module are explained below: I. Dynamic Materialization Engine Module Hardware implementation includes: (1) Server cluster configuration: Metadata server: Deploy a 3-node ZooKeeper cluster to store model version metadata (JSON Schema format), with each node configured with ≥16 CPU cores / 64GB RAM to support high-concurrency parsing; Computing node: Equipped with a difference calculation module using an FPGA acceleration card (model Xilinx Alveo U250), with a latency of ≤2ms when executing the Levenshtein distance algorithm.

[0025] (2) Action flow: Model change trigger: Receive Git repository schema commit events via Webhook; Difference calculation: Parallel comparison of the AST syntax tree structure differences between the old and new models using FPGA; DDL generation: Automatically converts to ALTERTABLE statements that match the database type (supports MySQL / PostgreSQL syntax variants).

[0026] The components include: (1) Model parser: Real-time parsing of abstract data model definitions such as JSONSchema or GraphQL to generate a metadata tree carrying version identifier (Technical function: to establish a unified semantic mapping between the logical layer and the physical layer). (2) Difference Calculation Module: Based on the Levenshtein distance algorithm, the structural changes between model versions are compared and the minimum change set is output (edit distance theory is applicable to the application of data structure difference detection). (3) Execution scheduler: Automatically select online ALTER or shadow table switching strategy according to change set priority (technical effect: realize zero-downtime deployment of DDL operations).

[0027] II. Flexible Storage Allocation Module Implementation methods include: (1) By adopting a hybrid layout strategy of columnar storage and row storage, the physical storage format can be automatically selected according to the access frequency of the fields defined in the model, which can solve the problem of resource waste. (2) By using pre-allocated blank storage pages and dynamic pointer reorganization technology, the storage space can be adjusted in real time when fields are added or deleted, breaking through the fixed page size limitation of the traditional B+ tree structure.

[0028] Storage allocation strategies include: Hot fields: Query fields with a frequency > 1000 times / minute are stored in columnar format (Parquet format is used in this embodiment); Cold fields: Fields with a frequency of less than 10 times / day are dynamically compressed into row storage (in this embodiment, a B+ tree index is used).

[0029] Example of spatial reorganization: When a field is deleted: (1) mark the corresponding block of the storage page as "reclaimable"; (2) move the subsequent field data forward by pointer redirection; (3) trigger the garbage collection thread (GC interval can be configured to 1-60 seconds).

[0030] Cloud implementation includes: (1) Use AWSDMS service to capture source database change logs; (2) Verify model compatibility using the Lambda function (verification rules include: field type compatibility matrix, transitivity of non-null constraints).

[0031] Edge device implementation includes: In an embedded SQLite database: Create a virtual table that maps to the latest model version, and synchronize structural changes between the base table and the virtual table using triggers.

[0032] III. Model-Storage Consistency Guarantee Mechanism The workflow includes: (1) Transaction log capture layer: listens for change events in the model definition library and generates CDC logs with timestamps; (2) Two-stage materialization agreement: Preparation phase: Verify the compatibility of the new model with existing data on the verification node cluster. Commit phase: Ensure all data nodes synchronously switch table structure versions using distributed locks (technical effect: meets the atomicity and consistency requirements of ACID properties).

[0033] Based on the module composition and implementation of the above system, this invention also proposes a method for generating entity database tables based on a dynamic data model. In this embodiment, as shown... Figure 1 As shown, it includes the following steps: Step S1: Data model definition. Based on business requirements, use the corresponding data modeling tools to define a dynamic data model. Step S2: Entity database table generation. When a data model change notification is received or a timed trigger is initiated, the latest data model is read from the data model repository, and the corresponding database table structure is automatically generated according to the definition in the data model. Step S3: During operation, when new business data is generated, data is stored and operated according to the generated database table structure and data processing logic.

[0034] In this embodiment, during the data model definition phase: business analysts define a dynamic data model based on business needs using specialized data modeling tools. This model describes business entities and the relationships, attributes, and other information between them. For example, in an e-commerce scenario, product entities, order entities, and user entities are defined, along with their relationships, such as the purchase relationship between an order and a product, and the ownership relationship between a user and an order. After the data model is defined, it is stored in a data model repository for subsequent database table generation and system operation.

[0035] Entity table generation phase: Upon receiving a data model change notification or a scheduled trigger, the system reads the latest data model from the data model repository. Based on the definitions in the data model, the system automatically generates the corresponding database table structure. For example, based on the attribute definitions of the product entity, a product table containing fields such as product ID, name, price, and inventory is generated. Simultaneously, the system also generates data processing logic related to the table structure, such as code templates for data insertion, update, and deletion operations, as well as view definitions for data queries.

[0036] System Operation Phase: During system operation, when new business data is generated, it is stored and manipulated according to the generated database table structure and data processing logic. For example, when a user places an order and new order data is generated, the system stores the order information according to the order table structure and updates related data, such as product inventory information. The system provides a unified interface for external business modules to call, enabling data querying, updating, and other operations. For example, the e-commerce front-end page can query product lists, user order information, etc., through the interface. During system operation, if the data model changes, the system automatically triggers the entity database table generation process, regenerates the database table structure and data processing logic, and ensures smooth migration of existing data and seamless system switching, guaranteeing business continuity.

[0037] The entity database table generation method and system proposed in this invention, based on a dynamic data model, can dynamically generate and adjust the entity database table structure according to changes in business needs and data models. Specifically, when a company's business rules change, such as adding a new product type, the system can automatically generate the corresponding database table structure and update the relevant data processing logic based on preset data models and rules, without requiring manual modification of the database tables and code, greatly improving the system's flexibility and response speed.

[0038] In the financial industry, regulatory policies change frequently, requiring financial institutions to quickly adjust their business systems and data storage structures to meet new requirements. This solution can help financial institutions adapt quickly to these changes, ensuring data compliance and system stability.

[0039] For the foregoing embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0040] The above embodiments describe the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the invention should be within the protection scope of the appended claims.

Claims

1. An entity database table generation system based on a dynamic data model, characterized in that, Includes the following modules: Dynamic materialization engine module: Configure server cluster, deploy a 3-node ZooKeeper cluster on the metadata server to store model version metadata; equip the computing nodes with FPGA acceleration card difference calculation modules to calculate Levenshtein edit distance; Elastic storage allocation module: Sets a storage allocation strategy that combines columnar and row-based storage layouts, and reorganizes the data space; Model storage consistency guarantee module: Set up a transaction log capture layer in the cloud to listen for change events in the model definition library, generate CDC change data capture logs with timestamps, and configure a two-phase materialization protocol.

2. The entity database table generation system based on a dynamic data model as described in claim 1, characterized in that, The components of the dynamic materialization engine module specifically include: Model parser: Parses abstract data model definitions in real time and generates a metadata tree carrying version identifiers; Difference calculation module: Based on the Levenshtein distance algorithm, it compares the structural changes between model versions and outputs the minimum change set; Execution scheduler: Automatically selects the switching strategy based on changeset priority.

3. The entity database table generation system based on a dynamic data model as described in claim 1, characterized in that, The elastic storage allocation module specifically employs a hybrid layout strategy of columnar and row-based storage, automatically selecting the physical storage format based on the field access frequency defined in the model; and through pre-allocated blank storage pages and dynamic pointer reorganization, it enables real-time adjustment of storage space when fields are added or deleted.

4. The entity database table generation system based on a dynamic data model as described in claim 3, characterized in that, The automatic selection of the physical storage format specifically refers to: Query fields with a frequency > 1000 times / minute are designated as hot fields and stored in columnar format, while query fields with a frequency < 10 times / day are designated as cold fields and dynamically compressed into row-based storage. The reorganization of the data space specifically involves: when a field is deleted, marking the corresponding block of the storage page as recyclable, shifting subsequent field data forward through pointer redirection, and triggering the garbage collection thread.

5. The entity database table generation system based on a dynamic data model as described in claim 1, characterized in that, The model storage consistency guarantee module also includes: for edge devices, creating a virtual table in the embedded database to map the latest model version, and synchronizing the structural changes of the base table and the virtual table through triggers.

6. The entity database table generation system based on a dynamic data model as described in claim 5, characterized in that, The two-phase materialization protocol in the model storage consistency guarantee module specifically includes: During the preparation phase, the compatibility of the new model with existing data is verified in the verification node cluster. During the commit phase, distributed locks are used to ensure that all data nodes synchronously switch the table structure version.

7. A method for generating entity database tables based on a dynamic data model, implemented based on the entity database table generation system based on a dynamic data model as described in any one of claims 1 to 6, characterized in that, Includes the following steps: Step S1: Data model definition. Based on business requirements, use the corresponding data modeling tools to define a dynamic data model. Step S2: Entity database table generation. When a data model change notification is received or a timed trigger is initiated, the latest data model is read from the data model repository, and the corresponding database table structure is automatically generated according to the definition in the data model. Step S3: During operation, when new business data is generated, data is stored and operated according to the generated database table structure and data processing logic.

8. The entity database table generation method based on a dynamic data model as described in claim 7, characterized in that, Step S1 further includes: defining a dynamic data model to describe business, entities and their relationships and attribute information. After the data model is defined, it is stored in the data model repository for use in subsequent database table generation and system operation.

9. The entity database table generation method based on a dynamic data model as described in claim 7, characterized in that, Step S2 further includes: generating data processing logic related to the database table structure, including code templates for data insertion, update and deletion operations, as well as view definitions for data queries.

10. The entity database table generation method based on a dynamic data model as described in claim 7, characterized in that, Step S3 further includes: calling external business modules through a unified interface provided by the system to realize data query and update operations; during system operation, if the data model changes, the entity database table generation process is automatically triggered to regenerate the database table structure and data processing logic, and to ensure the smooth migration of existing data and seamless switching of the system.

Images