A multi-tenant transaction data unified storage method based on dynamic schema metadata, a storage medium and an apparatus

Abstract

The present application relates to the technical field of data storage, and particularly relates to a multi-tenant transaction data unified storage method based on dynamic schema metadata, a storage medium and equipment, comprising: receiving a schema definition request, performing legality verification and generating target schema metadata, the legality verification comprising unique partition key verification, primary key binding verification and nested field recursive verification; according to the target schema metadata, dynamically creating a target physical table and a search index in the table storage according to the partition key priority rule; receiving a transaction data write request, converting the data according to the target schema metadata, arranging the structure and filtering the fields, generating standardized write data and writing into the target physical table; performing primary key stability verification when the schema is changed, and only allowing non-primary key field adjustment. The present application supports online schema evolution, realizes zero-code access and unified retrieval of multi-tenant heterogeneous transaction data, and reduces operation and maintenance complexity.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a method, storage medium, and device for unified storage of multi-tenant transaction data based on dynamic schema metadata. Background Technology

[0002] As the digital transformation of the retail industry accelerates, enterprises' transaction data processing needs are showing a significant trend towards large-scale and multi-source development. Against this backdrop, in order to support the unified management of transaction data across multiple business systems, traditional technologies typically adopt a solution of independently deploying relational databases. Each business system maintains its own database instance and table structure, and independently completes data storage and retrieval within each system.

[0003] The independent database and table creation solution requires pre-defined data table structures, with each business system creating its own corresponding database tables according to its data structure. When business requirements change and fields need to be added or modified, a data definition language (DDL) change operation must be performed. In high-concurrency transaction scenarios, such operations often require table locking or even service shutdown, severely impacting business continuity and real-time responsiveness. For new business systems requiring integration, this solution necessitates creating independent databases and tables for each, causing the number of database instances to grow linearly with the number of business systems, leading to a simultaneous increase in operational complexity and resource overhead. Furthermore, the field type definitions of each business system are independent, and fields with the same business meaning may use different storage formats and precisions in different systems, making unified data retrieval and analysis across systems difficult.

[0004] In real-world applications, transaction data typically contains nested one-to-many relationships, such as order and product details, payment and refund records. Traditional relational databases usually require splitting these nested objects into multiple tables and using foreign keys for cross-table queries, which not only increases query complexity but also significantly impacts system read and write performance. However, once the tables are created using this traditional approach, it's difficult to adjust the relationships and field structures between them; all data is processed using a fixed relational model. This forces the system to design separate table structures and maintain complex relationship logic for each type of business, resulting in a significant increase in system development and maintenance costs, and demonstrating clear shortcomings in terms of economy and practicality. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a unified storage method for multi-tenant transaction data based on dynamic schema metadata, comprising: S1: Receive a schema definition request submitted by the business side. The schema definition request includes at least a tenant identifier, a logical table identifier, and a set of field definitions. Each field definition in the set of field definitions includes at least one or more of the following: field name, field type, primary key flag, and partition key flag. S2: Perform a validity check on the pattern definition request to generate target pattern metadata. The validity check includes at least: checking whether there is one and only one partition key field in the field definition set, and checking whether the partition key field is a primary key field. S3: Persist the target schema metadata that has passed the legality verification, and establish a mapping relationship between the logical table and the target physical table based on the tenant identifier and the logical table identifier; S4: Based on the target schema metadata, dynamically create the target physical table in Table Storage according to the rule of priority of partition key field; S5: Dynamically create a target search index for the target physical table based on the preset mapping relationship between field type and index type, wherein a nested index structure is generated for nested type fields; S6: When a schema change request for an existing schema is received, perform a primary key stability check on the schema metadata before and after the change. The primary key stability check includes at least the following: the number of primary key fields is immutable, the existence of primary key fields is immutable, and the type of primary key fields is immutable. When the primary key stability check passes, update the persisted schema metadata. When the primary key stability check fails, reject the schema change request. S7: Receive transaction data write requests submitted by the business side, perform type conversion and field filtering on the transaction data according to the corresponding target schema metadata, generate standardized write data and write it to the target physical table; S8: Attach preset hidden attributes during the writing process to support event routing, retrieval processing, and isolation management of multi-tenant transaction data.

[0006] Preferably, in step S2, performing the validity check on the pattern definition request includes: S211: Parse the set of field definitions and extract all primary key fields and all partition key fields; S212: Count the number of partition key fields, and if the number of partition key fields is not 1, determine that the pattern definition request does not meet the partition constraints and return a verification failure result; S213: When the number of partition key fields is 1, determine whether the partition key field belongs to the primary key field, and if the determination result is no, determine that the pattern definition request does not meet the primary key binding constraint and return a verification failure result; S214: Read the data type of the partition key field, determine whether the data type belongs to the preset partitionable type, and if the determination result is negative, determine that the mode definition request does not meet the partition type constraint and return the verification failure result; S215: When the partition constraint, primary key binding constraint and partition type constraint are all satisfied, output the target schema metadata.

[0007] Preferably, in step S2, recursive validation is performed on nested type fields, including: S221: Traverse the set of field definitions to identify target nested fields whose field type is either object type or array type; S222: Read the subfield definition set corresponding to the target nested field, and verify whether the subfield definition set is empty, whether the subfield name is duplicated, and whether the subfield type belongs to the preset field type set; S223: When the set of subfield definitions contains object type fields or array type fields, recursive validation continues to be performed on deeper-level subfield definitions until all leaf fields at each level are basic type fields; S224: If there is a structural missing, duplicate field name, or illegal field type at any level, the pattern definition request verification is deemed to have failed. S225: When all levels pass the verification, generate nested schema metadata containing hierarchical path relationships.

[0008] Preferably, in step S4, dynamically creating the target physical table based on the physical primary key definition includes: S41: Extract all primary key fields from the target schema metadata and identify the partition key fields therein; S42: Place the partition key field at the beginning of the physical primary key definition, and arrange the remaining primary key fields sequentially according to the declaration order in the field definition set or the preset sorting rule; S43: Perform primary key type mapping based on the data type of each primary key field to generate physical primary key column definitions compatible with the underlying table storage; S44: Create a target physical table based on the physical primary key column definition, and configure the lifecycle parameters and version control parameters corresponding to the target physical table; S45: After the target physical table is created, save the mapping information between the logical table identifier and the target physical table identifier.

[0009] Preferably, in step S5, generating the target index definition and dynamically creating the target search index includes: S51: Read the set of field definitions in the target schema metadata and extract the indexable fields therein; S52: Based on the mapping rules between field types and index types, generate corresponding index field definitions for each basic type field. Character fields are mapped to character index types, numeric fields are mapped to numeric index types, boolean fields are mapped to boolean index types, and date fields are mapped to time numeric index types. S53: Identify nested type fields in the field definition set, and read the corresponding sub-field definition set for each nested type field; S54: Recursively perform index field generation processing on the subfield definition set to form a nested index field tree, and define the nested type field as a nested index type; S55: Merge the index field definition corresponding to the basic type field with the nested index field tree to generate a complete target index definition, and create a target search index based on the complete target index definition.

[0010] Preferably, in step S6, a primary key stability check is performed, including: S61: Extract the primary key field sets from the schema metadata before and after the change, respectively; S62: Compare the number of primary key fields in the schema metadata before and after the change, and if the number is inconsistent, determine that the primary key structure has changed and reject the schema change request; S63: When the number of primary key fields is the same, compare each primary key field in the original schema metadata with the original schema metadata to see if it still exists in the changed schema metadata. If any primary key field is missing, determine that the primary key is deleted and reject the schema change request. S64: When all primary key fields in the schema metadata before the change exist in the schema metadata after the change, compare the data types of primary key fields with the same name, and if the data types of any primary key field with the same name are inconsistent, determine that the semantics of the primary key have changed, and reject the schema change request. S65: When the number of primary key fields is the same, all primary key fields exist, and the data types of primary key fields with the same name are the same, it is allowed to write the addition, deletion, or attribute adjustment of non-primary key fields to the metadata storage area to complete the schema extension update.

[0011] Preferably, in step S7, generating standardized write data includes: S71: Receive one or more business records from the transaction data write request, and load the corresponding target mode metadata according to the tenant identifier and logical table identifier; S72: Extract the corresponding field values ​​from the business records according to the order of the primary key fields defined in the target schema metadata, and assemble them to form a data row primary key that is consistent with the target physical table; S73: Traverse the non-primary key fields in the business records, and perform type conversion and structure organization on each non-primary key field according to the field type definition in the target schema metadata, including at least numeric type conversion, boolean type conversion, date format parsing, and nested field structure standardization; S74: Identify redundant fields in the business record that are not defined in the target schema metadata, and perform filtering processing on the redundant fields; S75: Outputs standardized write data containing the primary key of the data rows and the set of standardized attribute columns.

[0012] Preferably, in step S8, the preset hidden attribute is added, including: S81: After the standardized write data is generated, an event routing identifier is generated based on the tenant identifier and logical table identifier; S82: Write the event routing identifier into the preset hidden attribute, wherein the preset hidden attribute is not used as a business field exposed externally; S83: Write the complete data row containing the business attribute column and the preset hidden attribute into the target physical table; S84: In subsequent event subscription, data distribution or retrieval processing, the tenant and logical table to which the data belongs are identified based on the preset hidden attributes.

[0013] Based on the same concept, this embodiment also provides a computer-readable storage medium, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors perform the steps of a unified storage method for multi-tenant transaction data based on dynamic schema metadata as described in the embodiment.

[0014] Based on the same concept, this embodiment also provides a computer device, including a memory and one or more processors, wherein the memory stores computer code, and when the computer code is executed by the one or more processors, the one or more processors cause the one or more processors to perform the steps of a unified storage method for multi-tenant transaction data based on dynamic schema metadata as described in the embodiment.

[0015] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention separates the data table structure from the code and database, and stores and manages it uniformly in the form of schema metadata, realizing the design concept of schema as configuration. When a new business system is connected, it only needs to submit a schema definition request containing tenant identifier, logical table identifier and field definition set. The system automatically completes the legality verification and generates target schema metadata. There is no need to manually create tables, which greatly reduces the operation and maintenance complexity and time cost of multi-tenant access, and enables the system to flexibly support the unified access needs of dozens or even hundreds of heterogeneous business systems in a zero-code manner.

[0016] (2) This invention achieves standardized processing of heterogeneous data by defining a unified field type system and performing type conversion, structure organization, and field filtering on non-primary key fields in transaction data according to field type during the data writing stage. At the same time, it dynamically creates search indexes based on the preset mapping relationship between field types and index types, and generates nested index structures for nested type fields in a recursive manner, so that transaction data across systems and tables can be retrieved and analyzed in a unified manner, effectively solving the data silo problem caused by different field formats and unsupported nested structures in traditional solutions.

[0017] (3) This invention uses a primary key stability verification mechanism to strictly verify the number of primary key fields, the existence of primary key fields, and the consistency of primary key field types when a schema change request is received. Only the addition, deletion, or attribute adjustment of non-primary key fields is allowed, thereby enabling online schema evolution while ensuring data structure compatibility. Table structure changes do not require service interruption or table locking, avoiding the business interruption caused by changes in the execution data definition language of traditional relational databases to high-concurrency transaction data, and significantly improving the availability and flexibility of the system.

[0018] (4) This invention adds a preset hidden attribute generated based on the tenant identifier and logical table identifier during the writing process, and implants a source identifier into each transaction data. This supports subsequent event routing and isolation management of data based on the hidden attribute in a multi-tenant environment. While achieving unified data storage, it ensures the logical independence of data in each business system and facilitates data subscription, distribution and incremental processing. Attached Figure Description

[0019] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0020] Figure 1 This is a flowchart of the dynamic schema storage process of the present invention; Figure 2 This is a flowchart of a method for unified storage of multi-tenant transaction data based on dynamic schema metadata according to the present invention; Figure 3 This is a schematic diagram of the field type mapping relationship of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Obviously, the described embodiments are only some, not all, of the embodiments described in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application.

[0022] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a” and “an” used herein, and “the”, may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] First Embodiment Please see Figure 1 As shown, Figure 1 This is a flowchart of the dynamic schema storage process in this embodiment. In this embodiment, when the caller initiates a schema definition save request, it submits schema definition data to the system through a preset interface, such as sending a schema definition request containing tenant identifier, logical table identifier, and field definition set through the "POST / v1 / schema / define / save" interface. After receiving the request, the system receives it and forwards it to TradeSchemaServiceImpl for processing. TradeSchemaServiceImpl first performs a validity check on the submitted schema; after the check passes, it checks whether the target physical table corresponding to the schema exists. If it does not exist, it creates the corresponding table storage physical table; then it checks whether the target search index exists. If it does not exist, it creates the corresponding search index; further, it checks whether the trigger associated with the logical table exists. If it does not exist, it creates the corresponding function computation trigger; after completing the above resource checks and creation, it saves the schema metadata corresponding to the schema to the metadata table trade_schema and returns a successful save result to the caller.

[0024] Please see Figure 2 As shown, this embodiment provides a unified storage method for multi-tenant transaction data based on dynamic schema metadata, including the following steps: S1: Receive the schema definition request submitted by the business side. The schema definition request includes at least the tenant identifier, logical table identifier, and field definition set. Each field definition in the field definition set includes at least one or more of the following: field name, field type, primary key flag, and partition key flag.

[0025] In this embodiment, the tenant identifier is used to distinguish different tenants, the logical table identifier is used to identify the business logical table under a certain tenant, and the field definition set is used to describe the data structure corresponding to the logical table. For example, a business terminal can submit a schema definition request for a transaction log logical table, where the tenant identifier is TenantA, the logical table identifier is TradeRecord, and the field definition set includes fields such as biz_date, trade_id, amount, status, and ext_info; where biz_date is of type Date, with a primary key and a partition key marked as yes; trade_id is of type Keyword, with a primary key marked as yes; amount is of type Decimal; status is of type Keyword; and ext_info is of type Object, and may further contain nested subfields.

[0026] S2: Perform a validity check on the schema definition request to generate target schema metadata. The validity check includes at least: checking whether the tenant identifier and logical table identifier meet the preset naming constraints, checking whether there is one and only one partition key field in the field definition set, checking whether the partition key field belongs to the primary key field, checking whether the data type of the partition key field belongs to the preset partitionable type, checking for conflicts in reserved fields, and performing recursive checks on the subfield definitions when the field type is a nested type.

[0027] Preferably, upon receiving a pattern definition request, the system first verifies whether the tenant identifier and logical table identifier conform to the system's preset naming rules, such as allowing only combinations of letters, numbers, and underscores, and that the length does not exceed a preset threshold.

[0028] Preferably, in step S2, a validity check is performed on the pattern definition request, including: S211: Parse the set of field definitions and extract all primary key fields and all partition key fields.

[0029] S212: Count the number of partition key fields, and if the number of partition key fields is not 1, determine that the schema definition request does not meet the partition constraints and return a validation failure result.

[0030] S213: When the number of partition key fields is 1, determine whether the partition key field belongs to the primary key field, and if the result is negative, determine that the schema definition request does not meet the primary key binding constraint and return a validation failure result.

[0031] S214: Read the data type of the partition key field and determine whether the data type belongs to the preset partitionable type. If the determination result is negative, determine that the mode definition request does not meet the partition type constraint and return a verification failure result. Specifically, in this embodiment, the preset partitionable type may include Keyword type, Integer type, Long type, or Date type. If it does not belong to the preset partitionable type, it is determined that the partition type constraint is not met and a failure result is returned. In addition, it also checks whether the field name defined on the business side conflicts with the reserved fields internally in the system. Reserved fields may include hidden fields used for internal event routing, search filtering, or tenant isolation. If there is a field name conflict, the verification is determined to fail.

[0032] S215: Output target schema metadata when partition constraints, primary key binding constraints, and partition type constraints are all satisfied.

[0033] Preferably, in step S2, recursive validation is performed on nested type fields, including: S221: Traverse the field definition set to identify target nested fields whose field type is either object type or array type; S222: Read the set of subfield definitions corresponding to the target nested field, and check whether the set of subfield definitions is empty, whether the subfield name is duplicated, and whether the subfield type belongs to the preset field type set; S223: When the set of subfield definitions contains object type fields or array type fields, continue to perform recursive validation on deeper-level subfield definitions until all leaf fields at each level are basic type fields; S224: If there is a missing structure, duplicate field name, or illegal field type at any level, the schema definition request verification fails. S225: When all levels pass the verification, generate nested schema metadata containing hierarchical path relationships.

[0034] S3: Persist the target schema metadata that has passed the validity check (that is, write the target schema metadata that has passed the validity check to the metadata storage area), and establish a mapping relationship between the logical table and the target physical table based on the tenant identifier and the logical table identifier.

[0035] In this embodiment, the target schema metadata may include tenant identifiers, logical table identifiers, field definition sets, primary key field sets, partition key field identifiers, field hierarchy path relationships, field type information, and schema version information. The metadata storage area can be implemented using a relational database or other suitable storage medium for storing structured metadata. After the system persistently saves the target schema metadata, it generates the corresponding target physical table identifier based on the tenant identifier and logical table identifier, and establishes a mapping relationship between the logical table and the target physical table for subsequent table creation, index creation, and data writing.

[0036] S4: Read the target schema metadata and check if the target physical table exists. If the target physical table does not exist, extract all primary key fields from the target schema metadata and generate a physical primary key definition according to the rule of priority of partition key fields and subsequent arrangement of other primary key fields. Dynamically create the target physical table in Table Storage based on the physical primary key definition.

[0037] Preferably, in step S4, the target physical table is dynamically created based on the physical primary key definition, including: S41: Extract all primary key fields from the target schema metadata and identify the partition key fields among them; S42: Place the partition key field at the beginning of the physical primary key definition, and arrange the remaining primary key fields in the order of declaration in the field definition set or the preset sorting rules. S43: Perform primary key type mapping based on the data type of each primary key field to generate physical primary key column definitions compatible with the underlying table storage; S44: Create a target physical table based on the physical primary key column definition, and configure the corresponding lifecycle parameters and version control parameters for the target physical table; S45: After the target physical table is created, save the mapping information between the logical table identifier and the target physical table identifier.

[0038] Please see Figure 3As shown in this embodiment, there is a preset mapping relationship between the field types in the field definition set and the underlying table storage type and search index type. For the primary key field of the target physical table, the Keyword type is mapped to the STRING type, the Integer type to the INTEGER type, and the Date type to the INTEGER type; wherein, the Date type field can be converted to an integer date value or a timestamp value when generating the physical primary key column definition. For the index fields in the target search index, the Keyword type is mapped to the KEYWORD type, the Integer type to the LONG type, the Date type to the LONG type, the Text type to the TEXT type, the Decimal type to the DOUBLE type, the Boolean type to the BOOLEAN type, and the Array and Object types to the NESTED type. Based on the mapping relationship, the schema management service can automatically generate physical primary key definitions compatible with table storage and target index definitions compatible with the search index service according to the target schema metadata, thereby achieving consistent conversion between schema definitions, physical storage, and retrieval structures.

[0039] S5: Read the set of field definitions in the target schema metadata, generate the target index definition based on the preset mapping relationship between field type and index type, and check whether the target search index corresponding to the target physical table exists. If the target search index does not exist, dynamically create the target search index based on the target index definition. For nested type fields, generate the corresponding nested index structure recursively.

[0040] Preferably, in step S5, generating the target index definition and dynamically creating the target search index includes: S51: Read the set of field definitions in the target schema metadata and extract the indexable fields from it; S52: Based on the mapping rules between field types and index types, generate corresponding index field definitions for each basic type field. Specifically, Keyword type is mapped to KEYWORD type, Integer type is mapped to LONG type, Date type is mapped to LONG type, Text type is mapped to TEXT type, Decimal type is mapped to DOUBLE type, and Boolean type is mapped to BOOLEAN type. S53: Identify nested type fields in the field definition set, and read the corresponding sub-field definition set for each nested type field. Specifically, in this embodiment, identify Array type fields and Object type fields in the field definition set. S54: Recursively perform index field generation processing on the subfield definition set to form a nested index field tree, and define the nested type field as a nested index type. Specifically, in this embodiment, the Array type field and the Object type field are defined as NESTED type to support the retrieval of nested structure data. S55: Merge the index field definition corresponding to the basic type field with the nested index field tree to generate a complete target index definition, and create a target search index based on the complete target index definition.

[0041] In some implementations, after the target search index is created, the system can further detect whether the target trigger associated with the logical table exists; if the target trigger does not exist, a corresponding function computation trigger is created to support event routing, retrieval synchronization, or extended processing after subsequent transaction data is written.

[0042] S6: When a schema change request for an existing schema is received, read the schema metadata before the change and the schema metadata after the change, and perform primary key stability checks on both. The primary key stability check includes at least the primary key field quantity check, the primary key field existence check, and the primary key field type consistency check. When the primary key stability check passes, update the schema metadata in the metadata storage area. When the primary key stability check fails, reject the schema change request.

[0043] Preferably, in step S6, a primary key stability check is performed, including: S61: Extract the primary key field sets from the schema metadata before and after the change, respectively; S62: Compare the number of primary key fields in the schema metadata before and after the change, and if the number is inconsistent, determine that the primary key structure has changed and reject the schema change request; S63: When the number of primary key fields is the same, compare each primary key field in the schema metadata before the change to see if it still exists in the schema metadata after the change. If any primary key field is missing, determine that the primary key has been deleted and reject the schema change request. S64: When all primary key fields in the schema metadata before the change exist in the schema metadata after the change, compare the data types of primary key fields with the same name, and if the data types of any primary key field with the same name are inconsistent, determine that the semantics of the primary key have changed and reject the schema change request. S65: When the number of primary key fields is the same, all primary key fields exist, and the data types of primary key fields with the same name are the same, it is allowed to write the addition, deletion, or attribute adjustment of non-primary key fields to the metadata storage area to complete the schema extension update.

[0044] S7: Receives transaction data write requests submitted by the business side, locates the corresponding target schema metadata based on the tenant identifier and logical table identifier in the transaction data write request, constructs the data row primary key according to the primary key fields defined in the target schema metadata, performs type conversion, structure organization and field filtering on the non-primary key fields in the transaction data according to the field type, and generates standardized write data.

[0045] Preferably, in step S7, generating standardized write data includes: S71: Receive one or more business records from the transaction data write request, and load the corresponding target schema metadata based on the tenant identifier and logical table identifier; S72: Extract the corresponding field values ​​from the business records according to the order of the primary key fields defined in the target schema metadata, and assemble them to form a data row primary key that is consistent with the target physical table; S73: Traverse the non-primary key fields in the business records and perform type conversion and structure organization on each non-primary key field according to the field type definition in the target schema metadata, including at least numeric type conversion, boolean type conversion, date format parsing, and nested field structure standardization; S74: Identify redundant fields in business records that are not defined in the target schema metadata and perform filtering on the redundant fields; S75: Outputs standardized write data containing the primary key of the data rows and the set of standardized attribute columns.

[0046] S8: Writes standardized data to the target physical table and attaches preset hidden attributes during the writing process to support event routing, retrieval processing, and isolation management of multi-tenant transaction data.

[0047] Preferably, in step S8, a preset hidden attribute is added, including: S81: After the standardized write data is generated, an event routing identifier is generated based on the tenant identifier and logical table identifier; S82: Write the event routing identifier into a preset hidden attribute, where the preset hidden attribute is not exposed as a business field. S83: Write the complete data row containing business attribute columns and preset hidden attributes to the target physical table; S84: In subsequent event subscription, data distribution or retrieval processing, the tenant and logical table to which the data belongs are identified based on preset hidden attributes, thereby realizing event routing, retrieval filtering and isolation management of multi-tenant data.

[0048] For example, in this embodiment, if the business submits a transaction record containing biz_date="2025-01-01", trade_id="T001", amount="100.50", status="SUCCESS", and nested fields ext_info={merchant_code:"M001", channel_info:{channel_id:"C01"}}, the system first constructs the primary key of the data row according to the target schema metadata in the order of biz_date and trade_id. Then, it converts amount from a string to a numeric type, standardizes the structure of ext_info, and filters out undefined redundant fields. After that, it generates an event routing identifier based on the tenant identifier and logical table identifier, writes it as a hidden attribute into the same data row, and finally writes the complete data row into the target physical table, which can be used synchronously for subsequent retrieval processing.

[0049] Through the above technical solution, this embodiment can realize the unified execution of transaction data pattern definition, metadata management, physical storage creation, search index creation and standardized write processing in multi-tenant scenarios, while taking into account dynamic expansion capabilities, retrieval capabilities and tenant isolation capabilities.

[0050] Second Embodiment Based on the same concept, this embodiment also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the steps of a unified storage method for multi-tenant transaction data based on dynamic schema metadata as described in the embodiment.

[0051] Based on the same concept, the present invention also provides a storage medium storing computer-readable instructions, characterized in that, when the computer-readable instructions are executed by one or more processors, the one or more processors cause the one or more processors to perform the steps of a unified storage method for multi-tenant transaction data based on dynamic schema metadata as described in any one of the embodiments.

[0052] It is understood that, regarding the aforementioned unified storage method for multi-tenant transaction data based on dynamic schema metadata, if all components are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer server or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0053] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0054] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for unified storage of multi-tenant transaction data based on dynamic schema metadata, characterized in that, Includes the following steps: S1: Receive a schema definition request submitted by the business side. The schema definition request includes at least a tenant identifier, a logical table identifier, and a set of field definitions. Each field definition in the set of field definitions includes at least one or more of the following: field name, field type, primary key flag, and partition key flag. S2: Perform a validity check on the pattern definition request to generate target pattern metadata. The validity check includes at least: checking whether there is one and only one partition key field in the field definition set, and checking whether the partition key field is a primary key field. S3: Persist the target schema metadata that has passed the legality verification, and establish a mapping relationship between the logical table and the target physical table based on the tenant identifier and the logical table identifier; S4: Based on the target schema metadata, dynamically create the target physical table in Table Storage according to the rule of priority of partition key field; S5: Dynamically create a target search index for the target physical table based on the preset mapping relationship between field type and index type, wherein a nested index structure is generated for nested type fields; S6: When a schema change request for an existing schema is received, perform a primary key stability check on the schema metadata before and after the change. The primary key stability check includes at least the following: the number of primary key fields is immutable, the existence of primary key fields is immutable, and the type of primary key fields is immutable. When the primary key stability check passes, update the persisted schema metadata. When the primary key stability check fails, reject the schema change request. S7: Receive transaction data write requests submitted by the business side, perform type conversion and field filtering on the transaction data according to the corresponding target schema metadata, generate standardized write data and write it to the target physical table; S8: Attach preset hidden attributes during the writing process to support event routing, retrieval processing, and isolation management of multi-tenant transaction data.

2. The method for unified storage of multi-tenant transaction data based on dynamic schema metadata according to claim 1, characterized in that, In step S2, the validity check of the pattern definition request is performed, including: S211: Verify whether the tenant identifier and logical table identifier meet the preset naming constraints; S212: Validate for conflicts in reserved fields; S213: Parse the set of field definitions and extract all primary key fields and all partition key fields; S214: Count the number of partition key fields, and if the number of partition key fields is not 1, determine that the pattern definition request does not meet the partition constraints and return a verification failure result; S215: When the number of partition key fields is 1, determine whether the partition key field belongs to the primary key field, and if the determination result is no, determine that the pattern definition request does not meet the primary key binding constraint and return a verification failure result; S216: Read the data type of the partition key field, determine whether the data type belongs to the preset partitionable type, and if the determination result is negative, determine that the mode definition request does not meet the partition type constraint and return the verification failure result; S217: When the partition constraint, primary key binding constraint and partition type constraint are all satisfied, output the target schema metadata.

3. The method for unified storage of multi-tenant transaction data based on dynamic schema metadata according to claim 1, characterized in that, In step S2, recursive validation is performed on nested type fields, including: S221: Traverse the set of field definitions to identify target nested fields whose field type is either object type or array type; S222: Read the subfield definition set corresponding to the target nested field, and verify whether the subfield definition set is empty, whether the subfield name is duplicated, and whether the subfield type belongs to the preset field type set; S223: When the set of subfield definitions contains object type fields or array type fields, recursive validation continues to be performed on deeper-level subfield definitions until all leaf fields at each level are basic type fields; S224: If there is a structural missing, duplicate field name, or illegal field type at any level, the pattern definition request verification is deemed to have failed. S225: When all levels pass the verification, generate nested schema metadata containing hierarchical path relationships.

4. The method for unified storage of multi-tenant transaction data based on dynamic schema metadata according to claim 1, characterized in that, In step S4, the target physical table is dynamically created based on the physical primary key definition, including: S41: Extract all primary key fields from the target schema metadata and identify the partition key fields therein; S42: Place the partition key field at the beginning of the physical primary key definition, and arrange the remaining primary key fields sequentially according to the declaration order in the field definition set or the preset sorting rule; S43: Perform primary key type mapping based on the data type of each primary key field to generate physical primary key column definitions compatible with the underlying table storage; S44: Create a target physical table based on the physical primary key column definition, and configure the lifecycle parameters and version control parameters corresponding to the target physical table; S45: After the target physical table is created, save the mapping information between the logical table identifier and the target physical table identifier.

5. The method for unified storage of multi-tenant transaction data based on dynamic schema metadata according to claim 1, characterized in that, In step S5, the target index definition is generated and the target search index is dynamically created, including: S51: Read the set of field definitions in the target schema metadata and extract the indexable fields therein; S52: Based on the mapping rules between field types and index types, generate corresponding index field definitions for each basic type field. Character fields are mapped to character index types, numeric fields are mapped to numeric index types, boolean fields are mapped to boolean index types, and date fields are mapped to time numeric index types. S53: Identify nested type fields in the field definition set, and read the corresponding sub-field definition set for each nested type field; S54: Recursively perform index field generation processing on the subfield definition set to form a nested index field tree, and define the nested type field as a nested index type; S55: Merge the index field definition corresponding to the basic type field with the nested index field tree to generate a complete target index definition, and create a target search index based on the complete target index definition.

6. The method for unified storage of multi-tenant transaction data based on dynamic schema metadata according to claim 1, characterized in that, In step S6, a primary key stability check is performed, including: S61: Extract the primary key field sets from the schema metadata before and after the change, respectively; S62: Compare the number of primary key fields in the schema metadata before and after the change, and if the number is inconsistent, determine that the primary key structure has changed and reject the schema change request; S63: When the number of primary key fields is the same, compare each primary key field in the original schema metadata with the original schema metadata to see if it still exists in the changed schema metadata. If any primary key field is missing, determine that the primary key is deleted and reject the schema change request. S64: When all primary key fields in the schema metadata before the change exist in the schema metadata after the change, compare the data types of primary key fields with the same name, and if the data types of any primary key field with the same name are inconsistent, determine that the semantics of the primary key have changed, and reject the schema change request. S65: When the number of primary key fields is the same, all primary key fields exist, and the data types of primary key fields with the same name are the same, it is allowed to write the addition, deletion, or attribute adjustment of non-primary key fields to the metadata storage area to complete the schema extension update.

7. The method for unified storage of multi-tenant transaction data based on dynamic schema metadata according to claim 1, characterized in that, In step S7, standardized write data is generated, including: S71: Receive one or more business records from the transaction data write request, and load the corresponding target mode metadata according to the tenant identifier and logical table identifier; S72: Extract the corresponding field values ​​from the business records according to the order of the primary key fields defined in the target schema metadata, and assemble them to form a data row primary key that is consistent with the target physical table; S73: Traverse the non-primary key fields in the business records, and perform type conversion and structure organization on each non-primary key field according to the field type definition in the target schema metadata, including at least numeric type conversion, boolean type conversion, date format parsing, and nested field structure standardization; S74: Identify redundant fields in the business record that are not defined in the target schema metadata, and perform filtering processing on the redundant fields; S75: Outputs standardized write data containing the primary key of the data rows and the set of standardized attribute columns.

8. The method for unified storage of multi-tenant transaction data based on dynamic schema metadata according to claim 1, characterized in that, In step S8, the preset hidden attribute is attached, including: S81: After the standardized write data is generated, an event routing identifier is generated based on the tenant identifier and logical table identifier; S82: Write the event routing identifier into the preset hidden attribute, wherein the preset hidden attribute is not used as a business field exposed externally; S83: Write the complete data row containing the business attribute column and the preset hidden attribute into the target physical table; S84: In subsequent event subscription, data distribution or retrieval processing, the tenant and logical table to which the data belongs are identified based on the preset hidden attributes.

9. A computer-readable storage medium, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors perform the steps of a unified storage method for multi-tenant transaction data based on dynamic schema metadata as described in any one of claims 1 to 8.

10. A computer device, characterized in that, The device includes a memory and one or more processors, wherein the memory stores computer code that, when executed by the one or more processors, causes the one or more processors to perform the steps of a unified storage method for multi-tenant transaction data based on dynamic schema metadata as described in any one of claims 1-8.

Images