An automatic data transaction processing method and system applied to a big data platform

Abstract

The application discloses an automatic data transaction processing method and system applied to a big data platform, and the method comprises the following steps: acquiring data change information, analyzing the data change information to obtain data change content after analysis; classifying the data change content according to table types and change types to obtain a two-dimensional data change operation list; querying table attribute tables and field tables of the big data platform system level to obtain all table level attribute and field attribute information; traversing and matching the change information in the two-dimensional data change operation list in all table level attribute and field attribute information; classifying and processing according to the change types to obtain subsequent series operations required to be completed by each change content; and adopting a multi-thread high-concurrency operation method to perform multi-thread concurrent processing on the subsequent series operations. The application is suitable for automatic data transaction processing of the star ring science and technology big data platform TDH.

Description

Technical Field

[0001] This invention relates to the field of data transaction processing technology in the big data field, specifically to an automated data transaction processing method and system applied to a big data platform. Background Technology

[0002] The Transwarp Technology Big Data Platform (TDH) stores a lot of data on the Hadoop Distributed File System (HDFS) through the distributed SQL engine Inceptor (a commercial component developed based on Hive, which is a data warehouse tool based on Hadoop). When making changes to tables in the distributed SQL engine Inceptor (adding / deleting fields, modifying table bucketing fields / bucketing field types / number of buckets, etc.), there are many restrictions, and it is not possible to directly operate through Data Manipulation Language (DML) / Data Definition Language (DDL) statements.

[0003] A common approach is to modify the Hive configuration file or set numerous transaction parameters to enable Hive's transactional features, and then operate on the transactional tables. However, non-transactional tables cannot undergo data transaction operations. This method is cumbersome and has strict requirements on the table type and storage method. The storage method must be a columnar storage format table (ORC table) from the Hadoop ecosystem, and there are risks of newly added fields containing null values ​​and field misalignment.

[0004] The existing technologies described above, which modify Hive parameter files or initiate transactions by setting transaction parameters in the Hive command line, work on the principle that transactions are managed through the Hive Metastore. The specific modification operations are executed via the ORC API on the Hadoop Distributed File System (HDFS), bypassing the Metastore. When a read operation begins, the Metastore provides the table to be read and a set of currently committed transaction IDs (a list of active transaction IDs at the moment of execution). These sets include the highest committed transaction ID and the IDs of ongoing transactions. When a query operation is performed, an MapReduce task (a parallel computing model and method for large-scale data processing) is started. The same set of transaction IDs is provided to the MapReduce task through the jobconf configuration. Therefore, read operations see a continuous snapshot view of the tables at the moment the command begins. For write operations, the Metastore provides a list of tables being read and written, obtains a set of valid transaction IDs for reading, and write transaction IDs. All written data is tagged with the write transaction ID.

[0005] In both of the above scenarios, when the command completes, it should notify the metadata service (metastore). The metadata service (metastore) must acknowledge or abort the abandoned transaction. The Hive client must notify the metadata service (metastore) every ten minutes to indicate that the command is still running; otherwise, the transaction will be aborted. The metadata service (metastore) maintains the set of tables read and their corresponding transaction IDs, allowing it to schedule the deletion of merged data after the last read operation is completed.

[0006] Therefore, existing data transaction processing methods applied to big data platforms suffer from low data processing efficiency, lack of support for direct modification of table attribute information, inconvenience in data use when newly added fields contain null values, and limitations on tables only under specific conditions (i.e., only ORC-bucketed transaction tables with Hive transactions enabled), and do not support operations on other tables. Summary of the Invention

[0007] The technical problem this invention aims to solve is that existing data transaction processing methods applied to big data platforms suffer from low data processing efficiency, lack of support for direct modification of table attribute information, and difficulties in data usage when newly added fields contain null values. Furthermore, they can only modify tables under specific conditions, namely, ORC-bucketed transaction tables with Hive transactions enabled, and do not support operations on other tables. The purpose of this invention is to provide an automated data transaction processing method and system for big data platforms, thereby solving the above problems.

[0008] This invention is achieved through the following technical solution:

[0009] In a first aspect, the present invention provides an automated data transaction processing method applied to a big data platform, the method comprising:

[0010] Obtain data change information, parse the data change information to obtain the parsed data change content; classify the data change content according to table type and change type to obtain a two-dimensional data change operation list;

[0011] Query the table attribute tables and field tables at the system level of the big data platform to obtain all table-level attribute and field attribute information;

[0012] The change information in the two-dimensional data change operation list is traversed and matched in all table-level attributes and field attribute information; the change is classified and processed according to the change type to obtain the subsequent series of operations required for each change.

[0013] A multi-threaded high-concurrency operation method is adopted to process the subsequent series of operations concurrently using multiple threads.

[0014] The technical solutions of this invention, after data transaction processing, store different data contents in the fields. This invention allows users to define the position and default value of fields according to their needs. If no custom value is defined, standardized data processing is automatically performed, and null values ​​will not appear, facilitating subsequent data use and analysis. This invention can also quickly change relevant attribute information according to user settings and automatically complete the relevant changes. When processing tables with multiple partitions and large amounts of data, this invention adopts high concurrency and multi-threading for data processing, which can handle large amounts of data with high processing efficiency.

[0015] Furthermore, data change information is obtained and parsed to obtain the parsed data change content. This data change content is then categorized according to table type and change type to obtain a two-dimensional data change operation list, including:

[0016] Enter a data file containing data change information (such as an Excel file or other data file);

[0017] Use a Python program to parse the data file: read the contents of the data file, traverse each line of data in the data file, and obtain the data changes involved in this data transaction.

[0018] The data changes are categorized and grouped according to the change type and table type to obtain a two-dimensional data change operation list.

[0019] Furthermore, the data changes include the database name, table name, change type, newly added field information, new bucketing information, primary key flag, and distribution key flag; the change type includes adding fields, modifying table-level attributes, and / or modifying distribution key information, etc.

[0020] Change types include table structure changes, distributed key information changes, table transaction attribute changes, and table storage type changes;

[0021] Table types include ORC transactional tables, ORC non-transactional tables, TEXT transactional tables, TEXT non-transactional tables, Holodesk tables, HBase tables, partitioned tables, and non-partitioned tables. Partitioned tables include range partitioned tables and single-value partitioned tables.

[0022] Furthermore, queries are performed on the system-level table attribute tables and field tables of the big data platform to obtain information on all table-level attributes and field attributes, including:

[0023] Data queries are performed on the system-level table attribute tables of the big data platform to obtain attribute information of various tables at the system level;

[0024] Data queries are performed on the system-level field tables of the big data platform to obtain field attribute information for various system-level tables; and

[0025] Partition information is obtained from the system tables at the big data platform system level. The partition information includes the partition field, partition value, and partition type.

[0026] Furthermore, the change information in the two-dimensional data change operation list is traversed and matched across all table-level attributes and field attributes; it is then categorized and processed according to change type to obtain the subsequent series of operations required for each change, including:

[0027] Step A: Loop through the change information in the two-dimensional data change operation list: Based on the database name and table name, traverse all table-level attribute information and field attribute information, compare the database name and table name, and if the database name and table name are the same, obtain the old table-level attribute and field attribute.

[0028] Determine the type of change information. If it is a table structure change, iterate through the table structure change information and compare it with the old field attributes until the corresponding field is found. Insert the changed field content into the old field information to generate the new field attribute. If it is a distribution key information change and / or a table transaction attribute change and / or a table storage type change, the field information remains unchanged and is the old field information. Iterate through the old table-level attributes and update the corresponding attribute information, such as transaction attributes, distribution key field / type, table storage type, partition field information, etc.

[0029] Step B: Loop through the change information in the two-dimensional data change operation list, obtain the partition information of the corresponding table in the system partition table according to the database name and table name, store it in the configuration table, and set the running status recorded at preset time intervals (e.g., daily) to the ready state.

[0030] Step C: Based on the information from Step A, generate the corresponding table creation statements for the new tables according to the transaction attributes. Generate backup table creation statements for transaction tables and temporary table creation statements for non-transaction tables. Based on the information from Step A and Step B, generate data insertion statements. If the table structure has changed, the generated data insertion statements will convert the changed fields according to the field type and standardize the data, replacing null values ​​and special characters (such as carriage return, newline, etc.).

[0031] Step D: Based on the table creation statement and data insertion statement generated in Step C, group them according to the database name and table name in the change information. Each group list contains information such as database name, table name, new table creation statement, data insertion statement, and table switching statement, and serves as the basis for subsequent series of operations.

[0032] Furthermore, a multi-threaded high-concurrency operation method is adopted to process the subsequent series of operations concurrently using multiple threads, including:

[0033] The subsequent series of operations are stored in a two-dimensional list. A multi-threaded program is then called to execute the operations in the two-dimensional list with high concurrency, thus completing the automated data transaction change.

[0034] Secondly, the present invention provides an automated data transaction processing system applied to a big data platform, which supports an automated data transaction processing method applied to a big data platform; the system includes:

[0035] The acquisition unit is used to acquire data change information;

[0036] The parsing unit is used to parse the data change information to obtain the parsed data change content;

[0037] The classification processing unit is used to classify the data changes according to table type and change type, and obtain a two-dimensional data change operation list;

[0038] The big data platform system-level query unit is used to query the table attribute tables and field tables at the big data platform system level to obtain all table-level attribute and field attribute information;

[0039] The traversal matching unit is used to traverse and match the change information in the two-dimensional data change operation list, and to process the change information in all table-level attributes and field attribute information; it is then processed according to the change type to obtain the subsequent series of operations that need to be completed for each change content;

[0040] The multi-threaded high-concurrency unit is used to perform multi-threaded concurrent processing on a series of subsequent operations using a multi-threaded high-concurrency operation method.

[0041] Furthermore, the system is adapted to the automated data transaction processing of the Transwarp Technology Big Data Platform (TDH).

[0042] Thirdly, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned automated data transaction processing method applied to a big data platform.

[0043] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned automated data transaction processing method applied to a big data platform.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] This invention discloses an automated data transaction processing method and system applied to big data platforms. After data transaction processing, the stored data content of fields differs. This invention allows users to define the position and default value of fields according to their needs. If no custom value is defined, standardized data processing is automatically performed, preventing null values ​​and facilitating subsequent data use and analysis. This invention can also quickly change relevant attribute information based on user settings, automating the process. When processing multi-partition, large-volume tables, this invention employs high-concurrency multi-threading for data processing, achieving high efficiency despite large data volumes. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a flowchart of an automated data transaction processing method applied to a big data platform according to the present invention.

[0048] Figure 2 This is a schematic diagram of the data content of each row of the Excel file in step 1 of embodiment 2 of the present invention.

[0049] Figure 3 This is a schematic diagram of an automated data transaction processing system applied to a big data platform according to the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0051] Example 1

[0052] Existing data transaction processing methods applied to big data platforms suffer from low data processing efficiency, lack of support for direct modification of table attribute information, inconvenience when newly added fields contain null values, and limitations on tables only under specific conditions (ORC bucketed transaction tables with Hive transactions enabled), and do not support operations on other tables.

[0053] This invention designs an automated data transaction processing method for big data platforms. After data transaction processing, the stored data content of fields differs. This invention allows users to define the position and default value of fields according to their needs. If no custom value is defined, standardized data processing is automatically performed, preventing null values ​​and facilitating subsequent data use and analysis. This invention can also quickly change relevant attribute information based on user settings, automating the process. When processing multi-partition, large-volume tables, this invention employs high-concurrency multi-threading for data processing, achieving high efficiency despite large data volumes.

[0054] like Figure 1 As shown, the present invention provides an automated data transaction processing method applied to a big data platform, the method comprising:

[0055] Step 1: Obtain data change information, parse the data change information to obtain the parsed data change content; classify the data change content according to table type and change type to obtain a two-dimensional data change operation list;

[0056] Step 1 specifically includes the following steps:

[0057] Step 1.1: Input a data file containing data change information (such as an Excel file or other data file); use a Python program to parse the data file: read the contents of the data file, traverse each row of data in the data file, and obtain the data change content involved in this data transaction; the data change content includes the database name, table name (such as English table name), change type, newly added field information, new bucket information, primary key flag, and distribution key flag; change type includes adding fields, modifying table-level attributes, and / or modifying distribution key information, etc.

[0058] Step 1.2: Categorize and group the data changes according to change type and table type to obtain a two-dimensional data change operation list. Change types include table structure changes, distribution key information changes, table transaction attribute changes, and table storage type changes; table types include ORC transactional tables, ORC non-transactional tables, TEXT transactional tables, TEXT non-transactional tables, Holodesk tables, HBase tables, partitioned tables, and non-partitioned tables. Partitioned tables include range partitioned tables and single-value partitioned tables.

[0059] Step 2: Query the table attribute tables and field tables of the Transwarp Technology Big Data Platform TDH system to obtain all table-level attribute and field attribute information;

[0060] Step 2 specifically includes the following steps:

[0061] Step 2.1: Perform a data query on the tables_v table at the system level of the Transwarp Technology Big Data Platform TDH to obtain the attribute information of various tables at the system level;

[0062] Step 2.2: Perform a data query on the columns_v field table of the TDH system level of the StarRing Technology Big Data Platform to obtain the field attribute information of various tables at the system level;

[0063] Step 2.3: Obtain partition information from the system table system.partitions_v at the TDH system level of the StarRing Technology Big Data Platform. The partition information includes partition fields, partition values, and partition types.

[0064] Step 3: For the change information in the two-dimensional data change operation list, iterate and match all table-level attributes and field attribute information; classify and process according to change type to obtain the subsequent series of operations required for each change content;

[0065] Step 3 specifically includes the following steps:

[0066] Step A: Loop through the change information in the two-dimensional data change operation list: Based on the database name and table name, traverse all table-level attribute information and field attribute information obtained in Step 2, compare them with the database name and table name, and if the database name and table name are the same, obtain the old table-level attributes and field attributes.

[0067] Determine the type of change information. If it is a table structure change, iterate through the table structure change information and compare it with the old field attributes until the corresponding field is found. Insert the changed field content into the old field information to generate the new field attribute. If it is a distribution key information change and / or a table transaction attribute change and / or a table storage type change, the field information remains unchanged and is the old field information. Iterate through the old table-level attributes and update the corresponding attribute information, such as transaction attributes, distribution key field / type, table storage type, partition field information, etc.

[0068] Step B: Loop through the change information in the two-dimensional data change operation list, obtain the partition information of the corresponding table in the system partition table according to the database name and table name, store it in the configuration table, and set the running status recorded at preset time intervals (e.g., daily) to the ready state.

[0069] Step C: Based on the information from Step A, generate the corresponding table creation statements for the new tables according to the transaction attributes. Generate backup table creation statements for transaction tables and temporary table creation statements for non-transaction tables. Based on the information from Step A and Step B, generate data insertion statements. If the table structure has changed, the generated data insertion statements will convert the changed fields according to the field type and standardize the data, replacing null values ​​and special characters (such as carriage return, newline, etc.).

[0070] Step D: Based on the table creation statement and data insertion statement generated in Step C, group them according to the database name and table name in the change information. Each group list contains information such as database name, table name, new table creation statement, data insertion statement, and table switching statement, and serves as the basis for subsequent series of operations.

[0071] Step 4: Use a multi-threaded high-concurrency operation method to process the subsequent series of operations concurrently using multiple threads.

[0072] Step 4 specifically includes the following steps:

[0073] The subsequent series of operations are stored in a two-dimensional list. A multi-threaded program is then called to execute the operations in the two-dimensional list with high concurrency, thus completing the automated data transaction change.

[0074] The difference between this invention and existing technical solutions is that:

[0075] 1. After data transaction processing is completed, the stored data content of the fields is different. In the existing technology, all newly added fields after data processing are null values, and other fields cannot be added in the middle of existing fields. If fields are forcibly added, it will cause field misalignment, problems with the underlying data, and thus affect the data query of the table. However, the present invention allows users to define the position of fields and the default value of fields according to their needs. If no custom value is defined, the data will be automatically standardized and there will be no null values, which will facilitate the use and analysis of subsequent data.

[0076] 2. Existing technologies cannot directly modify the transaction attributes, bucketing fields, number of buckets, partition information, and table storage type of big data platform tables; while this invention can quickly change the relevant attribute information according to the user's settings and automatically complete the relevant changes.

[0077] 3. The data processing efficiency is different. When processing tables with multiple partitions and large data volumes, the existing technology uses single-threaded processing, which has low data processing efficiency and is limited in the amount of data that can be processed. In contrast, the present invention adopts high-concurrency multi-threaded processing, which can process large amounts of data and has high processing efficiency.

[0078] Example 2

[0079] like Figure 1 and Figure 2 As shown, the difference between this embodiment and Embodiment 1 is that this embodiment takes the following data transaction processing as an example:

[0080] Modify the table structure of the single-value partitioned non-transactional table cls_loan in the edwitl database. Add a new field ccy, with a string data type and a default value of 1. This field needs to be added in the middle of the existing fields and after the field gl_code. The table currently has 1200 partitions, with partition key etl_dt and distribution key id.

[0081] Step 1: Organize the change information according to the requirements. The change operation is a table structure change, and the table is an ORC single-value partitioned non-transactional table. Enter the above information into an Excel file. Use the Python program app.py to parse the completed Excel file, obtain the data content of each row, and combine them to obtain a two-dimensional data change list: [['table struct change','edwitl','cla_loan',",",",",'ccy','string','currency','1','gl_code','orc','false','etl_dt']]. Specifically, as shown below... Figure 2 As shown.

[0082] Step 2: Based on the database and table name, obtain the table's attribute information (table storage format, transaction attributes, table name comments) from the system table system.tables_v;

[0083] Database name, table name, table comment, table type, whether it is a transactional table, distribution key, distribution key size

[0084] edwitl,cls_loan,'loan table',orc,false,id,7

[0085] Retrieve the field's attribute information (field name, field data type, field comment, whether the field can be null, etc.) from the system table system.column_v;

[0086] Database name, table name, serial number, field name, field data type, field comment, whether nullable

[0087] edwitl,cls_loan,1,id,'auto-incrementing sequence number',string,no

[0088] edwitl,cls_loan,2,loan_no,'loan number',string,no

[0089] edwitl,cls_loan,3,gl_code,'Organization Number',string,no ......

[0091] Retrieve partition information (partition field, partition value, partition type) from the system table system.partitions_v;

[0092] Database name, table name, partition field, type, partition value

[0093] edwitl,cls_loan,etl_dt,string,etl_dt=2022-09-01

[0094] edwitl,cls_loan,etl_dt,string,etl_dt=2022-09-02 ...

[0096] Step 3: Based on the table type and operation type of the change information, determine that a table structure change operation is required this time. Traverse the list [['table struct change','edwitl','cla_loan',",",",",'ccy','string','currency','1','gl_code','orc','false','etl_dt']]. From the information obtained in Step 2, compare the database and table name of the change information to obtain the table-level information and field information of the database name edwitl and the table name cls_loan. Since the new field needs to be added in the middle of all fields and a preceding field has been specified, it is necessary to traverse the old field list, find the preceding field, and insert the new field after the field to form a new field list.

[0097] Based on the new field list and table-level attribute information, generate the table creation statement for the temporary table:

[0098]

[0099] Based on the partition information obtained in step 3, the data of the newly added field is standardized, and the corresponding data back-insertion statement is generated:

[0100] Alter table edwitl.temp_cls_loan drop partition(etl_dt='2022-09-01');

[0101] insert into edwitl.temp_cls_loan partition(etl_dt)

[0102] select

[0103] id

[0104] ,loan_no

[0105] gl_code

[0106] ,'1'as ccy ......

[0108] From edwitl.cls_loan

[0109] Where etl_dt = '2022-09-01' ; ......

[0112] Statement for switching between temporary and permanent tables:

[0113] Use edwitl;

[0114] Alter table cls_loan rename to cls_loan_bak;

[0115] Alter table temp_cls_loan rename to cls_loan;

[0116] Store the above operations in a two-dimensional list:

[0117] ['create table...',['insert into...','insert into...'],['usedwitl;','alter table...','alter table...']]

[0118] Step 4: Based on the two-dimensional list result passed in Step 3, enable multi-threading to perform relevant operations and complete automated data transaction changes.

[0119] This invention has the following features: 1. This invention uses custom transaction data change content, intelligently parses the change content, and generates standardized data processing steps; 2. This invention supports agile modification of table attribute information; 3. This invention supports automated data transaction changes for bucketed transaction tables and other types of tables, and realizes intelligent verification of transaction data processing results; 4. This invention uses multi-threading and high concurrency to efficiently process data transaction operations.

[0120] Example 3

[0121] like Figure 3As shown, the difference between this embodiment and Embodiment 1 is that this embodiment provides an automated data transaction processing system applied to a big data platform. This system supports the automated data transaction processing method for a big data platform as described in Embodiment 1; the system is adapted to the automated data transaction processing of the Transwarp Technology Big Data Platform (TDH). The system includes:

[0122] The acquisition unit is used to acquire data change information;

[0123] The parsing unit is used to parse the data change information to obtain the parsed data change content;

[0124] The classification processing unit is used to classify the data changes according to table type and change type, and obtain a two-dimensional data change operation list;

[0125] The big data platform system-level query unit is used to query the table attribute tables and field tables at the big data platform system level to obtain all table-level attribute and field attribute information;

[0126] The traversal matching unit is used to traverse and match the change information in the two-dimensional data change operation list, and to process the change information in all table-level attributes and field attribute information; it is then processed according to the change type to obtain the subsequent series of operations that need to be completed for each change content;

[0127] The multi-threaded high-concurrency unit is used to perform multi-threaded concurrent processing on a series of subsequent operations using a multi-threaded high-concurrency operation method.

[0128] The execution process of each unit can be carried out according to the steps of the automated data transaction processing method applied to a big data platform as described in Example 1, and will not be described in detail in this example.

[0129] Meanwhile, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned automated data transaction processing method applied to a big data platform.

[0130] Meanwhile, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned automated data transaction processing method applied to a big data platform.

[0131] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An automated data transaction processing method applied to a big data platform, characterized in that, The method includes: Obtain data change information, parse the data change information to obtain the parsed data change content; classify the data change content according to table type and change type to obtain a two-dimensional data change operation list; Query the table attribute tables and field tables at the system level of the big data platform to obtain all table-level attribute and field attribute information; The change information in the two-dimensional data change operation list is traversed and matched in all table-level attributes and field attribute information; the change is classified and processed according to the change type to obtain the subsequent series of operations required for each change. A multi-threaded high-concurrency operation method is adopted to perform multi-threaded concurrent processing on the subsequent series of operations; The change information in the two-dimensional data change operation list is traversed and matched in all table-level attributes and field attribute information; it is then processed according to the change type to obtain the subsequent series of operations required for each change, including: Step A: Loop through the change information in the two-dimensional data change operation list: based on the database name and table name, traverse through all table-level attribute information and field attribute information, compare the database name and table name, and if the database name and table name are the same, obtain the old table-level attribute and field attribute. Determine the type of change information. If it is a table structure change, iterate through the table structure change information and compare it with the old field attributes until the corresponding field is found. Insert the changed field content into the old field information to generate the new field attribute. If it is a distribution key information change and / or a table transaction attribute change and / or a table storage type change, the field information remains unchanged and is the old field information. Iterate through the old table-level attributes and update the corresponding attribute information. Step B: Iterate through the change information in the two-dimensional data change operation list, obtain the partition information of the corresponding table in the system partition table according to the database name and table name, store it in the configuration table, and set the running status recorded at the preset time interval to the ready state. Step C: Based on Step A, generate the corresponding table creation statements for the new tables according to the transaction attributes. Generate backup table creation statements for transaction tables and temporary table creation statements for non-transaction tables. Based on Step A and Step B, generate data insertion statements. If the table structure has changed, the generated data insertion statements will convert the changed fields according to the field type and standardize the data, replacing null values ​​and special characters. Step D: Based on the table creation statement and data insertion statement generated in Step C, group them according to the database name and table name in the change information. Each group list contains information such as database name, table name, new table creation statement, data insertion statement, and table switching statement, and serves as the basis for subsequent series of operations. The method of employing multi-threaded high-concurrency operation to perform multi-threaded concurrent processing on the subsequent series of operations includes: The subsequent series of operations are stored in a two-dimensional list, and a multi-threaded program is called to start high-concurrency execution of the operations in the two-dimensional list to complete automated data transaction changes.

2. The automated data transaction processing method applied to a big data platform according to claim 1, characterized in that, The process involves acquiring data change information, parsing the data change information, and obtaining the parsed data change content. The data changes are categorized according to table type and change type, resulting in a two-dimensional data change operation list, including: Input a data file containing data change information; The data file is parsed using a Python program: the contents of the data file are read, each line of data in the data file is traversed, and the data change content involved in this data transaction is obtained; The data changes are categorized and grouped according to change type and table type to obtain a two-dimensional data change operation list.

3. The automated data transaction processing method applied to a big data platform according to claim 2, characterized in that, The data changes include the database name, table name, change type, newly added field information, new bucketing information, primary key flag, and distribution key flag; the change type includes adding fields, modifying table-level attributes, and / or modifying distribution key information. The change types include table structure changes, distributed key information changes, table transaction attribute changes, and table storage type changes; The table types include ORC transactional tables, ORC non-transactional tables, TEXT transactional tables, TEXT non-transactional tables, Holodesk tables, HBase tables, partitioned tables, and non-partitioned tables. The partitioned tables include range partitioned tables and single-value partitioned tables.

4. The automated data transaction processing method applied to a big data platform according to claim 1, characterized in that, The query process involves querying the table attribute tables and field tables at the system level of the big data platform to obtain all table-level attribute and field attribute information, including: Data queries are performed on the system-level table attribute tables of the big data platform to obtain attribute information of various tables at the system level; Data queries are performed on the system-level field tables of the big data platform to obtain field attribute information for various system-level tables; and Partition information is obtained from the system tables at the big data platform system level. The partition information includes partition fields, partition values, and partition types.

5. An automated data transaction processing system applied to a big data platform, characterized in that, The system supports an automated data transaction processing method for big data platforms as described in any one of claims 1 to 4; The system includes: The acquisition unit is used to acquire data change information; The parsing unit is used to parse the data change information to obtain the parsed data change content; The classification processing unit is used to classify the data change content according to table type and change type to obtain a two-dimensional data change operation list; The big data platform system-level query unit is used to query the table attribute tables and field tables at the big data platform system level to obtain all table-level attribute and field attribute information; The traversal matching unit is used to traverse and match the change information in the two-dimensional data change operation list in all table-level attributes and field attribute information; classify and process according to change type to obtain the subsequent series of operations that need to be completed for each change content; A multi-threaded high-concurrency unit is used to perform multi-threaded concurrent processing on the subsequent series of operations using a multi-threaded high-concurrency operation method.

6. The automated data transaction processing system applied to a big data platform according to claim 5, characterized in that, This system is adapted to the automated data transaction processing of the Transwarp Technology Big Data Platform (TDH).

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements an automated data transaction processing method for a big data platform as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements an automated data transaction processing method for a big data platform as described in any one of claims 1 to 4.

Images