Wide table data synchronization method and device and computer equipment

Abstract

The invention relates to a wide table data synchronization method and device and computer equipment, and relates to the technical field of data processing. The method comprises the following steps: acquiring task information and a configuration information set of a wide table data synchronization task; wherein the configuration information set is used for representing configuration information of a wide table and configuration information of an original table associated with the wide table; the configuration information set is analyzed according to the task information, a synchronization operation instruction of the wide table data synchronization task is generated, and the synchronization operation instruction is used for standardly describing synchronization operation of the wide table on at least part of source data in the original table; and executing the synchronous operation instruction. By adopting the method, the synchronous operation instruction can be generated, and the wide table data can be automatically synchronized by executing the synchronous operation instruction, so that the data synchronization efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a wide table data synchronization method, device and computer equipment. Background Art

[0002] With the development of big data technology, data warehouse technology has emerged. Data warehouses can process and analyze large-scale data in real time and meet rapidly changing business needs.

[0003] As a data storage structure, wide tables are widely used in data warehouses. Wide tables are suitable for scenarios that require fast query and analysis of a large number of fixed attributes, especially when multiple attributes need to be accessed simultaneously. However, in the current process of synchronizing from the original table to the wide table, the synchronization operation needs to be created manually, and in the case of demand updates, it needs to be updated, tested, or even recreated, resulting in low efficiency of wide table data synchronization and failure to meet the real-time needs of data processing. Summary of the invention

[0004] Based on this, it is necessary to provide a wide table data synchronization method, device and computer equipment to address the above technical problems, which can realize automatic synchronization of wide table data.

[0005] In a first aspect, the present application provides a wide table data synchronization method, the method comprising:

[0006] Obtaining task information and a configuration information set of a wide table data synchronization task; wherein the configuration information set is used to represent the configuration information of the wide table and the configuration information of the original table associated with the wide table;

[0007] Parsing the configuration information set according to the task information to generate a synchronization operation instruction for the wide table data synchronization task, wherein the synchronization operation instruction is used to standardize the description of the operation of synchronizing the wide table with at least part of the source data in the original table;

[0008] Execute the synchronization operation instruction.

[0009] In one embodiment, parsing the configuration information set according to the task information to generate a synchronization operation instruction for the wide table data synchronization task includes:

[0010] Acquire, from the configuration information set according to the task information, stream grouping information corresponding to each stream group of the wide table data synchronization task;

[0011] According to the information of each stream grouping, the configuration information set is parsed according to the stream grouping to generate a group synchronization operation instruction for each stream grouping;

[0012] A synchronization operation instruction of the wide table data synchronization task is generated according to the stream grouping information and the group synchronization operation instruction of each of the stream groups.

[0013] In one embodiment, the step of parsing the configuration information set according to the stream grouping information and generating the group synchronization operation instruction for each stream grouping includes:

[0014] According to the information of each of the stream groups, performing wide table interpolation and lookup field parsing on the configuration information set according to the stream grouping, and generating an interpolation and lookup instruction for each of the stream groups;

[0015] According to the information of each of the stream groups, the configuration information set is associated and parsed according to the stream grouping to generate an associated instruction for each of the stream groups;

[0016] According to the information of each of the flow groups, the configuration information set is parsed for filtering conditions according to the flow groups, and filtering instructions for each of the flow groups are generated;

[0017] According to the insertion and checking instructions, the association instructions and the filtering instructions of each of the stream groups, a group synchronization operation instruction of each of the stream groups is generated accordingly.

[0018] In one embodiment, the step of performing association analysis on the configuration information set according to the flow grouping information to generate association instructions for each flow grouping includes:

[0019] According to the information of each stream grouping, respectively acquiring the task wide table association information and the metadata association information of each stream grouping from the configuration information set;

[0020] According to the task wide table association information and metadata association information of each of the stream groups, table association parsing is performed respectively according to the stream groups to generate table association instructions for each of the stream groups;

[0021] The association instructions for each of the flow groups are generated according to the table association instructions for each of the flow groups.

[0022] In one embodiment, generating the association instruction of each flow grouping according to the table association instruction of each flow grouping includes:

[0023] According to the information of each stream group, correspondingly obtaining metadata prompt information of each stream group from the configuration information set;

[0024] According to the metadata prompt information of each of the stream groups, respectively perform prompt association analysis according to the stream groups, and generate prompt association instructions for each of the stream groups;

[0025] According to the table association instructions and prompt association instructions of each of the flow groups, the association instructions of each of the flow groups are generated accordingly.

[0026] In one embodiment, the filtering condition parsing of the configuration information set according to the flow grouping information to generate filtering instructions for each flow grouping includes:

[0027] According to the information of each stream grouping, correspondingly obtaining the filtering condition information of each stream grouping from the configuration information set;

[0028] Convert the filtering condition information of each of the flow groups into a tree structure;

[0029] For each of the flow groups, the filter condition is parsed according to the tree structure and the corresponding filter instruction is generated.

[0030] In one embodiment, the tree structure includes at least one level, and the filtering condition parsing according to the tree structure and generating the corresponding filtering instruction includes:

[0031] For the target level of the tree structure, obtaining the node type and operation mode of each node in the target level;

[0032] According to the node type and operation mode of each node in the target level, a filtering sub-instruction of the target level is generated, and the next level of the target level is determined as a new target level, and the step of obtaining the node type and operation mode of each node in the target level is returned to be executed until all levels in the tree structure are traversed;

[0033] According to the filtering sub-instructions of all levels of the tree structure, a filtering instruction corresponding to the tree structure is generated.

[0034] In one embodiment, performing wide table interpolation and lookup field parsing on the configuration information set according to the flow grouping information to generate an interpolation and lookup instruction for each flow grouping includes:

[0035] According to the information of each stream grouping, metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each stream grouping are correspondingly obtained from the configuration information set;

[0036] According to the metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each of the stream groups, the wide table fields are parsed respectively according to the stream groups to generate the insertion and query instructions of each of the stream groups.

[0037] In a second aspect, the present application provides a wide table data synchronization device, the device comprising:

[0038] An acquisition module, used to acquire task information and a configuration information set of a wide table data synchronization task; wherein the configuration information set is used to represent the configuration information of the wide table and the configuration information of the original table associated with the wide table;

[0039] A generation module, used to parse the configuration information set according to the task information, and generate a synchronization operation instruction of the wide table data synchronization task, wherein the synchronization operation instruction is used to standardize the operation of synchronizing the wide table with at least part of the source data in the original table;

[0040] A synchronization module is used to execute the synchronization operation instruction.

[0041] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores computer instructions, and the processor implements the steps of the above method when executing the computer instructions.

[0042] The above-mentioned wide table data synchronization method, device and computer equipment obtain the task information and configuration information set of the wide table data synchronization task, parse the configuration information set according to the task information, generate the synchronization operation instruction of the wide table data synchronization task, and execute the synchronization operation instruction, so as to synchronize the source data in the original table to the wide table, and realize the automatic synchronization of the wide table data. Since the synchronization operation instruction standardizes the operation of the wide table to synchronize at least part of the source data in the original table, the synchronization operation instruction can be directly executed based on the existing data synchronization tool to synchronize the source data in the original table to the wide table, thereby realizing the automatic synchronization of the wide table data, saving the time of manual creation, update and testing, reducing the test process, improving the efficiency of wide table data synchronization, reducing operation errors and risks, ensuring data consistency, and improving data synchronization accuracy. In addition, the method is applicable to wide tables of multiple storage bodies, with high flexibility and applicability, and the automatically generated synchronization operation instructions comply with the standardized data query rules, avoiding the use and learning costs of wide tables and / or original tables for wide table data synchronization tasks with different requirements, and further improving the efficiency of wide table data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is an application environment diagram of a wide table data synchronization method in an embodiment;

[0044] Figure 2 This is one of the flowcharts of a wide table data synchronization method in one embodiment;

[0045] Figure 3 This is a second flow chart of a wide table data synchronization method in an embodiment;

[0046] Figure 4 This is a third flow chart of a wide table data synchronization method in an embodiment;

[0047] Figure 5 This is a fourth flow chart of a wide table data synchronization method according to an embodiment;

[0048] Figure 6 This is a fifth flow chart of a wide table data synchronization method in one embodiment;

[0049] Figure 7 This is a sixth flow chart of a wide table data synchronization method according to an embodiment;

[0050] Figure 8 FIG7 is a flowchart of a wide table data synchronization method according to an embodiment;

[0051] Fig. 9 The eighth flowchart of a wide table data synchronization method in one embodiment;

[0052] Fig.10 FIG9 is a ninth flowchart of a wide table data synchronization method according to an embodiment;

[0053] Fig.11 FIG10 is a flowchart of a wide table data synchronization method according to an embodiment;

[0054] Fig.12 FIG11 is a flow chart of a wide table data synchronization method according to an embodiment;

[0055] Fig.13 FIG. 12 is a flowchart of a wide table data synchronization method according to an embodiment;

[0056] Fig.14 FIG. 13 is a flowchart of a method for synchronizing wide table data in an embodiment;

[0057] Fig.15 is a structural block diagram of a wide table data synchronization device in an embodiment;

[0058] Fig.16 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The wide table data synchronization method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The data storage system is used to store the wide table to be synchronized and / or the original table including metadata. The terminal 102 automatically synchronizes the operation instructions by obtaining the task information and configuration information set of the wide table data synchronization task, and realizes the automatic synchronization of the wide table data by executing the synchronization operation instructions. Among them, the terminal 102 can be but is not limited to various personal computers, laptops and smart phones, tablets and Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0061] In one embodiment, Figure 2 As shown, a wide table data synchronization method is provided, which is applied to Figure 1 Taking the terminal in as an example, the method includes the following steps S202 to S206.

[0062] S202: Obtain task information and a configuration information set of a wide table data synchronization task; wherein the configuration information set is used to represent the configuration information of the wide table and the configuration information of the original table associated with the wide table.

[0063] The wide table data synchronization task refers to synchronizing or storing at least part of the source data in at least one original table to a wide table. Among them, the wide table refers to the wide table to be synchronized. The original table includes the source data to be synchronized to the wide table. It should be noted that the storage body of the wide table and the original table can be any type of storage body, for example, flink, hudi, doris and other storage bodies, which can be determined by the wide table data synchronization task and is not limited here.

[0064] The task information is used to represent information related to the wide table data synchronization task. Exemplarily, the task information includes at least one of a task name, a task identifier, and a task description information. It should be noted that the task information may also be any other information that can represent the wide table data synchronization task, which is only an exemplary description.

[0065] The configuration information set is pre-built. The configuration information set includes multiple information. Exemplarily, the configuration information set is used to indicate the configuration information of the wide table and the configuration information of the original table associated with the wide table. Exemplarily, the configuration information set includes metadata information, metadata association information, task construction information, task wide table association information, stream grouping information and filtering condition information, and may also include at least one of metadata prompt information and task wide table field increase and decrease information. Among them, the metadata information is used to indicate the required relevant information of the table (including the wide table and the original table) in the data warehouse. Metadata prompt information, such as schema metadata prompt information, is used to indicate the prompt information of the wide table data synchronization task. In the application, metadata prompt information can be selectively configured. Metadata association information is used to indicate the association relationship information between tables. Task construction information is used to indicate information related to the wide table data synchronization task. Task wide table association information is used to indicate the corresponding relationship between the task and the wide table. Task wide table field increase and decrease information is used to indicate the relevant information of the operation on the field based on the general rules (pre-set), wherein the operation includes at least one of deletion, increase, decrease and renaming, for example, the same-name fields existing in multiple original tables associated with the wide table are deleted and renamed. The flow grouping information is used to indicate the information of the combination grouping between tables. The filtering condition information is used to indicate the filtering condition configuration information of the flow or table.

[0066] Exemplarily, the various information in the configuration information set is represented in the form of a table. Accordingly, the configuration information set may include a metadata information table, a metadata prompt information table, a metadata association information table, a task construction information table, a task wide table association information table, a task wide table field increase and decrease information table, a stream grouping information table, and a filtering condition information table. Exemplarily, the various information tables in the configuration information set are configured in a table structure. Exemplarily, the table structures of the various information tables are as follows:

[0067] a) [Metadata information] table: [Primary key], [Original table name], [Data warehouse layer type, 1: ODS table, 2: DWD wide table], [Add mode, 0: append, 1: upsert], [Column whose value is immutable].

[0068] b) [schema metadata prompt information] table: [primary key], [identifier of the [task wide table association information] table], [identifier of the [metadata information] table], [prompt information corresponding to the table], [system time field name].

[0069] c) [Metadata related information] table: [primary key], [foreign key], [related data field], [relational model], [identifier of the [metadata information] table], [identifier of the related [metadata information] table].

[0070] d) [Task construction information] table: [Primary key], [Task name], [Wide table identifier (corresponding to the [Metadata information] table identifier)].

[0071] e) [Task wide table association information] table: [primary key], [ID of the involved [stream grouping information] table], [ID of the involved [metadata information] table], [whether it is a temporary time association], [sort number].

[0072] f) [Task wide table field increase and decrease information] table: [Primary key], [Identifier of the [Task wide table related information] table], [Identifier of the involved [metadata related information] table], [Set of fields to be excluded], [Set of fields to be retained], [Set of fields to be renamed].

[0073] g) [Flow grouping information] table: [Primary key], [Identifier of the [task wide table association information] table], [Different groups are combined using a joint statement (such as a union statement), and the same group is placed in the same union statement: 0: default public group, other values: custom group].

[0074] h) [Filter condition information] table: [Primary key], [ID of the [stream grouping information] table], [ID of the table to be filtered], [Operation mode: 0: and (and), 1: or (or)], [Field to be filtered], [Filter condition], [ID of the parent filter condition], [ID of the previous filter condition], [Sorting number], [Is it a leaf node, 0: no, 1: yes].

[0075] Among them, ODS (Operational Data Store) table, the ODS layer is located between the source system and the data warehouse, and is mainly used to store the original data extracted from various business systems. DWD (Data Warehouse Detail) wide table, the DWD layer is the detailed data layer in the data warehouse, it cleans, transforms and integrates the data of the ODS layer through the ETL (Extract, Transform, Load) process to form a wide table structure suitable for analysis. The add mode append represents the supplementary mode, which means adding new data rows to the end of the existing data set without changing the existing data rows. The add mode upsert represents the update or insert mode, which combines the functions of update (Update) and insert (Insert). When performing an upsert operation, if the specified record already exists, the record is updated; if it does not exist, a new record is inserted. Identity (Identity, ID) is used to uniquely identify the corresponding information.

[0076] It should be noted that the configuration information of the wide table and the configuration information of the original table can be configured in different information tables respectively. For details, please refer to the table structure of each information table mentioned above, which will not be described in detail here.

[0077] S204: Parse the configuration information set according to the task information to generate a synchronization operation instruction for the wide table data synchronization task, where the synchronization operation instruction is used to standardize the description of the operation of the wide table synchronizing at least part of the source data in the original table.

[0078] Synchronous operation instructions refer to operation instructions for automatically executing wide table data synchronization tasks. Synchronous operation instructions comply with standardized data query rules. Exemplarily, data query rules include SQL (Structured Query Language), and correspondingly, synchronous operation instructions include SQL statements. Among them, the synchronous operation instructions include but are not limited to query fields, associated tables, prompt words, stream groupings, multi-layer nested complex filtering conditions, etc. involved in the automated synchronization of wide tables. This is only an exemplary explanation. The specific statements or fields included in the synchronous operation instructions can be determined based on the actual wide table data synchronization task, and are not overly limited here.

[0079] In the application, the terminal can parse various information in the configuration information set according to the task information, and generate synchronization operation instructions that comply with the standardized data query rules. It should be noted that the method for parsing and generating synchronization operation instructions can be set accordingly according to factors such as the configuration method of the configuration information set and the data query rules that the synchronization operation instructions comply with, and no excessive restrictions are made here.

[0080] S206: Execute the synchronization operation instruction.

[0081] In the application, the terminal can execute synchronization operation instructions based on the data synchronization tool to synchronize the source data in the original table to the corresponding position of the wide table, thereby realizing automatic synchronization of the wide table data. Among them, the data synchronization tool is a tool that supports synchronization operation instructions. For example, a command line tool, a graphical user interface tool, or an integrated development environment, etc. For example, the synchronization operation instruction includes an SQL statement, and correspondingly, the data synchronization tool can be any tool that supports SQL statements.

[0082] The above-mentioned wide table data synchronization method obtains the task information and configuration information set of the wide table data synchronization task, and parses the configuration information set according to the task information, generates the synchronization operation instruction of the wide table data synchronization task, and executes the synchronization operation instruction, thereby synchronizing the source data in the original table to the wide table, and realizing the automatic synchronization of the wide table data. Since the synchronization operation instruction standardizes the operation of the wide table to synchronize at least part of the source data in the original table, the synchronization operation instruction can be directly executed based on the existing data synchronization tool to synchronize the source data in the original table to the wide table, thereby realizing the automatic synchronization of the wide table data, eliminating the time of manual creation and update and testing, reducing the test process, improving the efficiency of wide table data synchronization, and reducing operational errors and risks, ensuring data consistency, and improving data synchronization accuracy. In addition, the method is applicable to wide tables of multiple storage bodies (for example, Flink, Hudi, Doris, etc.), with high flexibility and applicability, and the automatically generated synchronization operation instructions conform to the standardized data query rules, avoiding the use and learning costs of wide tables and / or original tables for wide table data synchronization tasks with different requirements, and further improving the efficiency of wide table data synchronization.

[0083] In one embodiment, Figure 3 As shown, step S204, parsing the configuration information set according to the task information, generating a synchronization operation instruction for the wide table data synchronization task, includes the following steps S302 to S306.

[0084] S302: Obtaining the stream grouping information corresponding to each stream group of the wide table data synchronization task from the configuration information set according to the task information.

[0085] The stream grouping information is used to represent the information of the combination grouping between tables. Exemplarily, the task construction information corresponding to the task information is obtained from the configuration information table according to the task information, and the stream grouping information of each corresponding stream grouping is obtained according to the task construction information. Among them, the task construction information is used to represent the information related to the wide table data synchronization task.

[0086] Exemplarily, the task construction information and the stream grouping information are respectively represented by table information, that is, the configuration information set includes a task construction information table and a stream grouping information table, wherein the data structures corresponding to the task construction information table and the stream grouping information table are as follows:

[0087] [Task construction information] table: [primary key], [task name], [wide table identifier (corresponding to the identifier of the [metadata information] table)];

[0088] [Stream grouping information] table: [primary key], [identifier of [task wide table association information] table], [different groups are combined using a joint statement (such as a union statement), and the same group is placed in the same union statement: 0: default public group, other values: custom group].

[0089] The wide table data synchronization task includes at least one stream group. Each stream group corresponds to an independent stream group information. Exemplarily, the stream group information includes a stream group identifier (ID). Wherein, one stream group corresponds to one joint instruction.

[0090] Exemplarily, the task information includes a task name, and the stream grouping information includes a stream grouping identifier. Correspondingly, based on the task name of the wide table data synchronization task, the task identifier of the wide table data synchronization task is obtained from the configuration information set, and based on the task identifier, the stream grouping identifier corresponding to each stream group of the wide table data synchronization task is obtained.

[0091] S304: According to the information of each stream grouping, the configuration information set is parsed according to the stream grouping to generate a group synchronization operation instruction for each stream grouping.

[0092] The group synchronization operation instruction is used to standardize the data synchronization operation corresponding to the stream group. The group synchronization operation instruction can be understood as an operation instruction to synchronize at least part of the source data in the original table to the wide table according to the stream group. In the application, the terminal parses the configuration information set according to the stream group information of each stream group, and generates a group synchronization operation instruction for each stream group accordingly.

[0093] S306: Generate a synchronization operation instruction for the wide table data synchronization task according to the flow grouping information and the group synchronization operation instruction of each flow group.

[0094] Exemplarily, a joint instruction is generated according to the stream grouping information of each stream grouping, and a synchronization operation instruction of the wide table data synchronization task is generated according to the joint instruction and the group synchronization instruction of each stream grouping.

[0095] It can be understood that one stream group corresponds to one join instruction, and the join instruction is used to merge two or more group synchronization operation instructions. Exemplarily, the join instruction includes a union statement, and the union statement is used to merge the result sets of two or more select statements into one result set.

[0096] The above-mentioned wide table data synchronization method obtains the stream grouping information corresponding to each stream group of the wide table data synchronization task from the configuration information set according to the task information, parses the configuration information set according to the stream grouping information, generates the group synchronization operation instructions for each stream group, and generates the synchronization operation instructions for the wide table data synchronization task according to the stream grouping information and the group synchronization operation instructions of each stream group. This method can parse the configuration information set according to the stream grouping, so that the stream groups based on different conditions can independently associate with other tables and association relationships according to their own characteristics, and can meet various wide table data synchronization needs, with a wide range of applications and high flexibility.

[0097] In one embodiment, Figure 4 As shown, step S304, according to the information of each stream grouping, the configuration information set is parsed according to the stream grouping, and the group synchronization operation instruction of each stream grouping is generated, including the following steps S402 to S408.

[0098] S402: According to the information of each flow group, perform wide table insertion and lookup field parsing on the configuration information set according to the flow group, and generate insertion and lookup instructions for each flow group.

[0099] Wide table insert and query field parsing refers to wide table insert field parsing and wide table query field parsing for the configuration information set. The insert and query instruction is used to normalize the description of the operation of inserting the source data queried from the original table according to the filter condition into the wide table. Exemplarily, the synchronous operation instruction includes an SQL statement, and correspondingly, the insert and query instruction includes an SQL insert and query statement, for example, an SQL insert and query statement: insert into x1_table(a1,b1,…)select aa….

[0100] In the application, the terminal can perform wide table insertion and lookup field parsing on the configuration information set according to the stream grouping identifier of each stream grouping, and generate an insertion and lookup instruction corresponding to each stream grouping.

[0101] S404: According to the information of each flow group, the configuration information set is associated and parsed according to the flow group, and an associated instruction of each flow group is generated.

[0102] Association parsing refers to parsing the association relationship between the association tables and tables of the configuration information set. Association instructions are used to standardize the description of operations for associating two or more tables and / or related information. Exemplarily, according to the flow grouping identifier of each flow grouping, the configuration information set is subjected to table association parsing according to the flow grouping, or the configuration information set is subjected to table association parsing and prompt association parsing to generate association instructions for each flow grouping. Exemplarily, the synchronization operation instruction includes an SQL statement, and correspondingly, the association instruction includes an SQL association statement, for example, an SQL association statement: from y1_table / *+OPTIONS('c1'='xx','c2'='xx',...)* / as y1 join y2_table as y2 on xx=xx.

[0103] S406: According to the information of each flow group, the configuration information set is parsed according to the flow group to generate a filtering instruction for each flow group.

[0104] Filter condition parsing refers to parsing the table filter condition or table screening condition of the configuration information set. The filter instruction is used to normalize the description of the operation of filtering the data in the table. Exemplarily, the synchronization operation instruction includes an SQL statement, and correspondingly, the filter instruction includes an SQL filter statement, for example, an SQL filter statement: where...

[0105] S408: Generate corresponding group synchronization operation instructions for each flow group according to the insertion and checking instructions, association instructions and filtering instructions of each flow group.

[0106] Exemplarily, according to a preset combination rule, the insert and check instructions, the association instructions, and the filtering instructions of each stream group are combined to generate a corresponding group synchronization operation instruction for each stream group. For example, the synchronization operation instruction includes an SQL statement, and accordingly, the group synchronization operation instruction includes an SQL group synchronization operation statement, for example, an SQL group synchronization operation statement: insert in to ... select ... join ... union select ... join ... where ....

[0107] The above-mentioned wide table data synchronization method performs wide table insertion and query field analysis on the configuration information set and generates insertion and query instructions for each stream grouping according to the stream grouping information of each stream grouping, and performs association analysis on the configuration information set and association instructions for each stream grouping, and performs filtering condition analysis on the configuration information set and filtering instructions for each stream grouping. This method uses the stream grouping information to independently perform wide table field insertion and query analysis, association analysis, and filtering condition analysis according to the stream grouping’s own characteristics, which can ensure the accuracy of wide table data synchronization, avoid unnecessary data transmission and processing, and improve data synchronization efficiency; moreover, the stream groups are decoupled from each other, and can be flexibly configured for different data streams, with high flexibility, simpler maintenance and management, and easy expansion. This method achieves high flexibility, high performance, high data quality and consistency of wide table data synchronization, while simplifying maintenance and management, and improving data security and controllability.

[0108] In one embodiment, Figure 5 and Figure 6 As shown, step S404, according to the information of each flow grouping, the configuration information set is associated and parsed according to the flow grouping, and the association instruction of each flow grouping is generated, including the following steps S602 to S606.

[0109] S602: According to the information of each stream group, respectively obtain the task wide table association information and metadata association information of each stream group from the configuration information set.

[0110] Task wide table association information is used to indicate the corresponding relationship between tasks and wide tables. Metadata association information is used to indicate the association relationship information between tables.

[0111] Exemplarily, the task wide table association information and the metadata association information are respectively represented by table information, that is, the configuration information set includes a task wide table association information table and a metadata association information table, wherein the data structures corresponding to the task wide table association information table and the metadata association information table are as follows:

[0112] [Metadata related information] table: [Primary key], [Foreign key], [Related data field], [Relationship model], [Identifier of [Metadata information] table], [Identifier of the related [Metadata information] table];

[0113] [Task wide table association information] table: [primary key], [ID of the involved [stream grouping information] table], [ID of the involved [metadata information] table], [whether it is a temporary time association], [sort number].

[0114] In the application, the terminal can obtain the task wide table association information table and the metadata association information table of each stream group from the configuration information set according to the stream group identifier of each stream group.

[0115] S604: According to the task wide table association information and metadata association information of each stream group, table association analysis is performed respectively according to the stream group, and a table association instruction of each stream group is generated.

[0116] Table association instructions are used to standardize the description of operations for associating two or more tables based on conditions. Exemplarily, based on the task-wide table association information and metadata association information of each stream grouping, the [associated data field], [[identifier of metadata information table]] and [identifier of associated [metadata information table]] corresponding to each stream grouping are obtained respectively, and based on the [associated data field], [[identifier of metadata information table]] and [identifier of associated [metadata information table]] corresponding to each stream grouping, the table association instructions of each stream grouping are generated. Exemplarily, the synchronization operation instructions include SQL statements, and correspondingly, the table association instructions include SQL table association instructions. For example, the SQL table association instruction: from y1_table as y1 join y2_tableas y2 on xx=xx.

[0117] S606: Generate association instructions for each flow group according to the table association instructions of each flow group.

[0118] The above-mentioned wide table data synchronization method obtains the task wide table association information and metadata association information of each stream group from the configuration information set according to the information of each stream group, performs table association analysis according to the task wide table association information and metadata association information of each stream group, generates table association instructions for each stream group, and generates association instructions for each stream group according to the table association instructions of each stream group. This method uses the stream group information to independently perform table association analysis and generate association instructions according to the characteristics of the stream group, so that the table can be automatically associated according to the association instructions to achieve automatic synchronization of wide table data, thereby improving data synchronization efficiency; since the stream groups are decoupled from each other, the association table analysis process for different data streams can be flexibly configured, with high flexibility, simpler maintenance and management, and easy expansion. This method achieves high flexibility, high performance, high data quality and consistency in wide table data synchronization, while simplifying maintenance and management, and improving data security and controllability.

[0119] In one embodiment, Figure 5 and Figure 7 As shown, step S606, generating an association instruction for each flow grouping according to the table association instruction of each flow grouping, includes the following steps S702 to S706.

[0120] S702: According to the information of each stream group, corresponding metadata prompt information of each stream group is obtained from the configuration information set.

[0121] Metadata prompt information, such as schema metadata prompt information, is used to indicate prompt information for wide table data synchronization tasks. In an application, metadata prompt information can be selectively configured. Exemplarily, metadata prompt information is represented by table information, that is, the configuration information set includes a metadata prompt information table, such as a schema metadata prompt information table. The data structure of the metadata prompt information table is as follows:

[0122] [schema metadata prompt information] table: [primary key], [identifier of the [task wide table association information] table], [identifier of the [metadata information] table], [prompt information corresponding to the table], [system time field name].

[0123] In the application, the terminal can obtain the metadata prompt information table of each stream group from the configuration information set according to the stream group identifier of each stream group.

[0124] S704: According to the metadata prompt information of each stream group, perform prompt association analysis according to the stream group respectively, and generate prompt association instructions for each stream group.

[0125] The prompt association instruction is used to standardize the description of the operation of associating a table with prompt information, wherein the prompt information refers to information related to the table and serving as a prompt for the table. Exemplarily, according to the metadata prompt information table of each stream group, the [primary key], [identifier of the [task-wide table association information] table], [identifier of the [metadata information] table], [prompt information corresponding to the table], and [system time field name] of each stream group are obtained respectively, and according to the [primary key], [identifier of the [task-wide table association information] table], [identifier of the [metadata information] table], [prompt information corresponding to the table], and [system time field name] of each stream group, the table to which the prompt needs to be added and the prompt information corresponding to the table are determined, and the prompt association instruction of each stream group is generated.

[0126] Exemplarily, the synchronization operation instruction includes an SQL statement, and correspondingly, the prompt association instruction includes an SQL prompt association instruction, for example, the prompt association instruction: y1_table / *+OPTIONS('c1'='xx', 'c2'='xx', ...)* / .

[0127] S706: Generate corresponding association instructions for each flow group according to the table association instructions and the prompt association instructions for each flow group.

[0128] Exemplarily, the table association instructions and the prompt association instructions of each stream grouping are combined according to the stream grouping, and the association instructions of each stream grouping are generated accordingly. Exemplarily, the synchronization operation instruction includes an SQL statement, and correspondingly, the table association instruction includes an SQL table association statement, and the prompt association instruction includes an SQL prompt association statement. For example, the SQL table association instruction: from y1_table as y1 join y2_table as y2 on xx=xx, the SQL prompt association instruction: y1_table / *+OPTIONS('c1'='xx','c2'='xx',...)* / , correspondingly, the SQL association instruction: from y1_table / *+OPTIONS('c1'='xx','c2'='xx',...)* / as y1 join y2_table as y2 on xx=xx.

[0129] The above-mentioned wide table data synchronization method obtains metadata prompt information of each stream grouping from the configuration information set according to the information of each stream grouping, performs prompt association analysis according to the metadata prompt information of each stream grouping, generates prompt association instructions for each stream grouping, and generates association instructions for each stream grouping according to the table association instructions and prompt association instructions of each stream grouping. This method uses the stream grouping information to independently perform prompt association analysis according to the stream grouping according to its own characteristics and generates prompt association instructions accordingly. It can generate association instructions in combination with table association instructions, so that the table can be automatically associated according to the association instructions. Automated wide table data synchronization can be achieved, which improves data synchronization efficiency, has high flexibility, is simpler to maintain and manage, and is easy to expand. In addition, the readability of the synchronization operation instructions can be increased through prompt association instructions, making it easier to maintain and manage.

[0130] In one embodiment, Figure 8 and Fig. 9 As shown, step S406, according to the information of each flow group, the configuration information set is parsed according to the flow group, and the filtering instruction of each flow group is generated, including the following steps S902 to S906.

[0131] S902: According to the information of each flow group, corresponding filtering condition information of each flow group is obtained from the configuration information set.

[0132] The filtering condition information is used to represent the filtering condition configuration information of a flow or table. Exemplarily, the filtering condition information is represented in the form of a table. Accordingly, the configuration information set may include a filtering condition information table. The data structure of the filtering condition information table is as follows:

[0133] [Filter condition information] table: [Primary key], [ID of the [stream grouping information] table], [ID of the table to be filtered], [Operation mode: 0: and (and), 1: or (or)], [Field to be filtered], [Filter condition], [ID of the parent filter condition], [ID of the previous filter condition], [Sorting number], [Is it a leaf node, 0: No, 1: Yes].

[0134] In the application, the terminal can obtain the filtering condition information table of each flow group from the configuration information set according to the flow group identifier of each flow group.

[0135] S904: Convert the filtering condition information of each flow group into a tree structure.

[0136] Exemplarily, the filtering condition information is represented in the form of a table, and the filtering condition information table of each flow group is converted into a tree structure. The tree structure includes at least one level, each level includes multiple nodes, the parent node of the first level is identified as 0, and the next level of the parent node is a leaf node. The parent node of the first level is identified as 0 by default, and the sort number is effective for the filtering condition of the current level.

[0137] S906: Analyze the filter conditions according to the tree structure and generate corresponding filter instructions.

[0138] The above-mentioned wide table data synchronization method obtains the filtering condition information of each flow grouping from the configuration information set according to the information of each flow grouping, converts the filtering condition information of each flow grouping into a tree structure, and parses the filtering condition according to the tree structure and generates corresponding filtering instructions. This method uses the flow grouping information to independently parse the table filtering condition according to the flow grouping's own characteristics and generates corresponding filtering condition instructions, so that the source data in the original table can be automatically filtered according to the filtering condition instructions, and then the source data obtained by the filtered query can be synchronized to the wide table, realizing automatic wide table data synchronization and improving data synchronization efficiency. Since the flow groups are decoupled from each other, the associated table parsing process of different data streams can be flexibly configured, which is highly flexible, simpler to maintain and manage, and easy to expand.

[0139] In one embodiment, Figure 8 and Fig.10 As shown, step S906, parsing the filter conditions according to the tree structure and generating corresponding filter instructions, includes the following steps S1002 to S1006.

[0140] S1002: For a target level of a tree structure, obtain a node type and an operation mode of each node in the target level.

[0141] S1004: Generate a filter sub-instruction for the target level according to the node type and operation mode of each node in the target level, determine the next level of the target level as the new target level, and return to execute the step of obtaining the node type and operation mode of each node in the target level until all levels in the tree structure are traversed.

[0142] S1006: Generate a filter instruction corresponding to the tree structure according to the filter sub-instructions of all levels of the tree structure.

[0143] The tree structure includes at least one level, and the target level is one of the at least one level. In the application, corresponding filter sub-instructions can be generated layer by layer starting from the first level. Exemplarily, for a target level of a tree structure, a node type and an operation mode of a target node in the target level are obtained, wherein the target node is any node in the target level. When the node type of the target node is a leaf node, a filtering sub-instruction of the target node is generated according to the [field to be filtered] and [filtering condition] in the filtering condition information, and the remaining node in the target level is used as the new target node, and the step of obtaining the node type and operation mode of the target node in the target level is returned to the step of executing until all nodes in the target level are traversed; when the node type of the target node is a parent node, a filtering sub-instruction of the parent node is generated, that is, the filtering sub-instruction of the next level is placed in brackets, and the next level of the target level is determined as the new target level, and the step of obtaining the node type and operation mode of the target node in the target level is returned to the step of executing until all levels in the tree structure are traversed, and filtering sub-instructions of all levels in the tree structure are obtained, thereby generating filtering instructions corresponding to the tree structure according to the filtering sub-instructions of all levels of the tree structure.

[0144] The above-mentioned wide table data synchronization method, for the target level of the tree structure, obtains the node type and operation mode of each node in the target level, generates the filtering sub-instructions of the target level according to the node type and operation mode of each node in the target level, and determines the next level of the target level as the new target level, and returns to execute the step of obtaining the node type and operation mode of each node in the target level until all levels in the tree structure are traversed, and generates the filtering instructions corresponding to the tree structure according to the filtering sub-instructions of all levels of the tree structure. This method converts the filtering condition information into a tree structure, parses the filtering conditions layer by layer according to the tree structure, and generates the filtering condition instructions accordingly, so that the source data in the original table can be automatically filtered according to the filtering condition instructions, thereby realizing the synchronization of the source data obtained by the filtering query to the wide table, realizing automatic wide table data synchronization, improving data synchronization efficiency, high flexibility, simpler maintenance and management, and easy expansion.

[0145] In one embodiment, Fig.11and Fig.12 As shown, step S402, according to the information of each flow grouping, the configuration information set is parsed by wide table insertion and lookup fields according to the flow grouping, and the insertion and lookup instructions of each flow grouping are generated, including the following steps S1202 to S1204.

[0146] S1202: According to the information of each stream group, metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each stream group are correspondingly obtained from the configuration information set.

[0147] Metadata information is used to represent the required relevant information of tables (including wide tables and original tables) in the data warehouse. Metadata association information is used to represent the association relationship information between tables. Task wide table association information is used to represent the corresponding relationship between tasks and wide tables. Task wide table field increase and decrease information is used to represent the relevant information of operating fields based on general rules (pre-set), where the operation includes at least one of deletion, increase and decrease, and renaming, for example, deleting and renaming fields with the same name that exist in multiple original tables associated with a wide table.

[0148] Exemplarily, metadata information, metadata association information, task wide table association information, and task wide table field increase and decrease information are respectively represented in the form of tables, that is, the configuration information set may include a metadata information table, a metadata association information table, a task wide table association information table, and a task wide table field increase and decrease information table, wherein the table structures of various information tables are as follows:

[0149] [Metadata information] table: [primary key], [original table name], [data warehouse layer type, 1: ODS table, 2: DWD wide table], [addition mode, 0: append, 1: upsert], [columns with immutable values];

[0150] [Metadata related information] table: [Primary key], [Foreign key], [Related data field], [Relationship model], [Identifier of [Metadata information] table], [Identifier of the related [Metadata information] table];

[0151] [Task wide table association information] table: [primary key], [ID of the involved [stream grouping information] table], [ID of the involved [metadata information] table], [whether it is a temporary association], [sequencing number];

[0152] [Task wide table field addition and deletion information] table: [primary key], [[Task wide table related information] table identifier], [Identifier of the involved [metadata related information] table], [Set of fields to be excluded], [Set of fields to be retained], [Set of fields to be renamed].

[0153] S1204: According to the metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each stream group, the wide table fields are parsed according to the stream groups respectively to generate the insertion and query instructions of each stream group.

[0154] Wide table field parsing includes wide table insert field parsing and wide table query field parsing. Correspondingly, insert query instructions include insert instructions and query instructions.

[0155] For example, Fig.11 As shown, the wide table insertion field parsing process may include: obtaining the [metadata information] table of each stream grouping from the configuration information set according to the stream grouping information, and obtaining the [metadata information] table, [schema metadata prompt information] table, [metadata association information] table, [task construction information] table, [task wide table association information] table, [task wide table field increase and decrease information] table, [stream grouping information] table and [filter condition information] table according to the [metadata information] table, so as to find out the association relationship between the association tables involved in the wide table and the information of the corresponding association tables, and query the storage body to which the wide table corresponding to the wide table data synchronization task needs to be synchronized, and generate the corresponding insertion instruction.

[0156] For example, Fig.11 As shown, wide table query field parsing may include:

[0157] (1) Based on the wide table and its associated relationships and the information of the corresponding associated tables obtained in the aforementioned wide table insertion field parsing process, the [field name required for the wide table] information is obtained.

[0158] Exemplarily, determine whether it is a key value, if so, retain it, and determine the [add mode] of the [associated table] involved in the wide table. If not, directly determine the [add mode] of the associated table involved in the wide table; if the [add mode] of the [associated table] involved in the wide table is the append mode, then the [field name required for the wide table] is all the fields of the [associated table]; if the [add mode] of the [associated table] involved in the wide table is the upsert mode, then the [field name required for the wide table] is the combination of the [column with immutable value], [column of the primary key of the table] and [key of the table partition] of the [associated table], that is, the [column with immutable value] of the [associated table], and the corresponding [key field] set of the [metadata storage body information] table record, after deduplication.

[0159] (2) Unify the format of fact table fields.

[0160] The fact table field format unification process is used to store the fields with non-standardized field names in a standardized manner in the table, making it easier to find repeated fields. For example, the fields in the fact table are converted from camel case to underscores.

[0161] (3) Addition and subtraction of specific fields.

[0162] For example, after obtaining the [Task Wide Table Field Increase / Decrease Information] table (or called [Task Wide Table Specific Field Increase / Decrease] table) corresponding to the task according to the task name, obtain the record of the field to be deleted according to the [Task Wide Table Field Increase / Decrease Information] table, and perform increase / deletion and renaming (can process fields with the same name) processing on [Field Name Required by Wide Table] according to the record, thereby generating the corresponding query instruction. Among them, the renaming processing is generally used to process fields with the same name.

[0163] The insert instruction is used to normalize the description of inserting the queried data into the wide table. The query instruction is used to normalize the description of querying the data to be synchronized from the table according to the filter conditions. Exemplarily, the synchronization operation instruction includes an SQL statement, and correspondingly, the insert instruction includes an SQL insert statement and an SQL query statement, for example, an SQL insert statement: INSERT INTOx1_table(a1,b1,…), and an SQL query statement: SELECT aa….

[0164] The above-mentioned wide table data synchronization method obtains the metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each stream group from the configuration information set according to the information of each stream grouping, and performs wide table field parsing according to the metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each stream grouping, and generates insertion and query instructions for each stream grouping. This method uses the stream grouping information to independently parse the wide table fields according to the characteristics of the stream grouping and generates corresponding insertion and query instructions, so that it can realize automatic query of the data to be synchronized according to the insertion and query instructions, and automatically insert the queried data into the wide table, thereby realizing automatic wide table data synchronization, improving data synchronization efficiency, high flexibility, simpler maintenance and management, and easy expansion.

[0165] In one embodiment, Fig.13 and Fig.14 As shown, a wide table data synchronization method is provided, and the wide table data synchronization method includes the following steps S1402 to S1416.

[0166] S1402: Obtain the task name and configuration information set of the wide table data synchronization task.

[0167] The configuration information set includes a metadata information table, a metadata prompt information table, a metadata association information table, a task construction information table, a task wide table association information table, a task wide table field increase and decrease information table, a stream grouping information table, and a filtering condition information table. The table structure of each information table can be found in the previous introduction and will not be repeated here.

[0168] S1404: Search the stream grouping information table from the configuration information table set according to the task name, and obtain the stream grouping information corresponding to each stream group of the wide table data synchronization task from the stream grouping information table.

[0169] Among them, one stream group corresponds to one union statement.

[0170] S1406: According to the information of each flow group, perform wide table insertion and lookup field parsing on the configuration information set according to the flow group, and generate insertion and lookup instructions for each flow group.

[0171] The insert query instruction is generated by parsing the wide table insert query field, for example, insert into...select... The process can be specifically referred to in the above Fig.11 and Fig.12 And related introduction, I will not go into details here.

[0172] S1408: According to the information of each flow group, the configuration information set is associated and parsed according to the flow group, and an associated instruction of each flow group is generated.

[0173] Among them, the association instruction is generated through association analysis, for example, from y1_table / *+OPTIONS('c1'='xx','c2'='xx',...)* / as y1 join y2_table as y2 on xx=xx. The process can be specifically referred to in the above Figures 5 to 7 And related introduction, I will not go into details here.

[0174] S1410: According to the information of each flow group, the configuration information set is parsed according to the flow group, and a filtering instruction for each flow group is generated.

[0175] The filtering instruction is generated by parsing the filtering condition, for example, where xx and (xx or (xx…)). The process can be specifically referred to in the previous Figures 8 to 10 And related introduction, I will not go into details here.

[0176] S1412: Generate corresponding group synchronization operation instructions for each stream group according to the insertion and checking instructions, association instructions and filtering instructions of each stream group.

[0177] According to the flow grouping, the insertion and query instructions, association instructions and filtering instructions of each flow grouping are combined according to preset rules, and the group synchronization operation instructions of each flow grouping are generated accordingly, for example, insert into xx(xx)select xx join xx.

[0178] S1414: Generate a synchronization operation instruction for the wide table data synchronization task according to the flow grouping information and the group synchronization operation instruction of each flow group.

[0179] One stream group corresponds to one union statement. According to the stream group, the group synchronization operation instructions of each stream group are combined according to the preset rules, for example, insert into xx(xx)select xx join xx union select xxjoin xx...where...

[0180] S1416: Execute synchronization operation instructions to automatically synchronize wide table data.

[0181] The above wide table data synchronization method can efficiently and automatically synchronize data to wide tables of multiple storage bodies (such as flink, hudi, doris, etc.) by configuring the schema metadata of the table configuration and its associated information configuration table in the database, and automatically generate the query fields, associated tables, prompt words, stream groups, and multi-layer nested complex filtering conditions involved in the required wide tables according to the best practice rules; in addition, the standardized wide table field setting rules reduce the use and learning costs of associating basic tables with different wide tables each time; and different stream groups can independently associate the required tables and associated relationships, and cooperate with the filtering condition parsing steps to achieve different conditions-based stream groups. After the data meets different conditions, they can be associated with different tables and associated relationships according to their own characteristics, which saves the time of manual creation, update and testing, reduces operational errors and risks, reduces the testing process, ensures data field consistency, and ensures correct and fast operation processes.

[0182] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0183] Based on the same inventive concept, the embodiment of the present application also provides a wide table data synchronization device for implementing the wide table data synchronization method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more wide table data synchronization device embodiments provided below can refer to the limitations of the wide table data synchronization method above, and will not be repeated here.

[0184] In one embodiment, Fig.15 As shown, a wide table data synchronization device 1500 is provided, including: an acquisition module 1501, a generation module 1502 and a synchronization module 1503. The acquisition module is used to obtain task information and a configuration information set of a wide table data synchronization task. The configuration information set is used to represent the configuration information of the wide table and the configuration information of the original table associated with the wide table. The generation module is used to parse the configuration information set according to the task information and generate a synchronization operation instruction for the wide table data synchronization task. The synchronization operation instruction is used to standardize the description of the operation of the wide table to synchronize at least part of the source data in the original table. The synchronization module is used to execute the synchronization operation instruction.

[0185] The above-mentioned wide table data synchronization device 1500 obtains the task information and configuration information set of the wide table data synchronization task through the acquisition module 1501, and parses the configuration information set according to the task information through the generation module 1502 to generate the synchronization operation instruction of the wide table data synchronization task, and executes the synchronization operation instruction through the synchronization module 1503, so as to synchronize the source data in the original table to the wide table, and realize the automatic synchronization of the wide table data. Since the synchronization operation instruction standardizes the operation of the wide table to synchronize at least part of the source data in the original table, the synchronization operation instruction can be directly executed based on the existing data synchronization tool to synchronize the source data in the original table to the wide table, thereby realizing the automatic synchronization of the wide table data, eliminating the time of manual creation, update and testing, reducing the test process, improving the efficiency of wide table data synchronization, and reducing operation errors and risks, ensuring data consistency, and improving data synchronization accuracy. In addition, this method is applicable to wide tables of multiple storage bodies, with high flexibility and applicability. Moreover, the automatically generated synchronization operation instructions comply with the standardized data query rules, thus avoiding the use and learning costs of wide tables and / or original tables for wide table data synchronization tasks with different requirements, and further improving the efficiency of wide table data synchronization.

[0186] In one embodiment, the generation module is also used to obtain the stream grouping information corresponding to each stream group of the wide table data synchronization task from the configuration information set according to the task information; according to each stream grouping information, the configuration information set is parsed according to the stream grouping to generate group synchronization operation instructions for each stream group; according to the stream grouping information and the group synchronization operation instructions of each stream group, the synchronization operation instructions of the wide table data synchronization task are generated.

[0187] In one embodiment, the generation module is also used to perform wide-table interpolation and field analysis on the configuration information set according to the stream grouping according to the stream grouping, and generate interpolation instructions for each of the stream groupings; perform association analysis on the configuration information set according to the stream grouping according to the stream grouping according to the stream grouping information, and generate association instructions for each of the stream groupings; perform filtering condition analysis on the configuration information set according to the stream grouping according to the stream grouping information, and generate filtering instructions for each of the stream groupings; and generate corresponding group synchronization operation instructions for each of the stream groupings according to the interpolation instructions, association instructions and filtering instructions of each of the stream groupings.

[0188] In one embodiment, the generation module is also used to obtain the task wide table association information and metadata association information of each stream group from the configuration information set according to the information of each stream group; perform table association analysis according to the stream groupings according to the task wide table association information and metadata association information of each stream group, and generate table association instructions for each stream group; and generate association instructions for each stream group according to the table association instructions of each stream group.

[0189] In one embodiment, the generation module is also used to obtain metadata prompt information of each stream grouping from the configuration information set according to the information of each stream grouping; perform prompt association analysis according to the stream groupings according to the metadata prompt information of each stream grouping, and generate prompt association instructions for each stream grouping; and generate association instructions for each stream grouping according to the table association instructions and prompt association instructions of each stream grouping.

[0190] In one embodiment, the generation module is also used to obtain the filtering condition information of each stream grouping from the configuration information set according to the information of each stream grouping; convert the filtering condition information of each stream grouping into a tree structure; and for each stream grouping, parse the filtering conditions according to the tree structure and generate the corresponding filtering instructions.

[0191] In one embodiment, the generation module is also used to obtain the node type and operation mode of each node in the target level of the tree structure; generate a filtering sub-instruction for the target level based on the node type and operation mode of each node in the target level, and determine the next level of the target level as the new target level, and return to execute the step of obtaining the node type and operation mode of each node in the target level until all levels in the tree structure are traversed; generate a filtering instruction corresponding to the tree structure based on the filtering sub-instructions of all levels of the tree structure.

[0192] In one embodiment, the generation module is also used to obtain the metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each stream grouping from the configuration information set according to the information of each stream grouping; according to the metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each stream grouping, perform wide table field parsing respectively according to the stream grouping, and generate insertion and query instructions for each stream grouping.

[0193] Each module in the above wide table data synchronization device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0194] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.16 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer instructions. The internal memory provides an environment for the operation of the operating system and computer instructions in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer instructions are executed by the processor, a wide table data synchronization method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0195] Those skilled in the art will understand that Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0196] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores computer instructions, and the processor implements the steps of the aforementioned wide table data synchronization method when executing the computer instructions.

[0197] In one embodiment, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of the aforementioned wide table data synchronization method are implemented.

[0198] In one embodiment, a computer instruction product is provided, including computer instructions, which implement the steps of the aforementioned wide table data synchronization method when executed by a processor.

[0199] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through computer instructions, and the computer instructions can be stored in a non-volatile computer-readable storage medium. When the computer instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0201] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A wide table data synchronization method, characterized in that: The method comprises: Obtaining task information and a configuration information set of a wide table data synchronization task; wherein the configuration information set is used to represent the configuration information of the wide table and the configuration information of the original table associated with the wide table; Parsing the configuration information set according to the task information to generate a synchronization operation instruction for the wide table data synchronization task, wherein the synchronization operation instruction is used to standardize and describe an operation of synchronizing the wide table with at least part of the source data in the original table; Execute the synchronization operation instruction.

2. The method according to claim 1, characterized in that The step of parsing the configuration information set according to the task information to generate a synchronization operation instruction for the wide table data synchronization task includes: Acquire, from the configuration information set according to the task information, stream grouping information corresponding to each stream group of the wide table data synchronization task; According to the information of each stream grouping, the configuration information set is parsed according to the stream grouping to generate a group synchronization operation instruction for each stream grouping; A synchronization operation instruction of the wide table data synchronization task is generated according to the stream grouping information and the group synchronization operation instruction of each of the stream groups.

3. The method according to claim 2, characterized in that The step of parsing the configuration information set according to the stream grouping information and generating a group synchronization operation instruction for each stream grouping includes: According to the information of each of the stream groups, performing wide table interpolation and lookup field parsing on the configuration information set according to the stream grouping, and generating an interpolation and lookup instruction for each of the stream groups; According to the information of each of the stream groups, the configuration information set is associated and parsed according to the stream grouping to generate an associated instruction for each of the stream groups; According to the information of each of the flow groups, the configuration information set is parsed for filtering conditions according to the flow groups, and filtering instructions for each of the flow groups are generated; According to the insertion and checking instructions, the association instructions and the filtering instructions of each of the stream groups, a group synchronization operation instruction of each of the stream groups is generated accordingly.

4. The method according to claim 3, characterized in that The step of performing association analysis on the configuration information set according to the stream grouping information to generate association instructions for each stream grouping includes: According to the information of each stream grouping, respectively acquiring the task wide table association information and the metadata association information of each stream grouping from the configuration information set; According to the task wide table association information and metadata association information of each of the stream groups, table association parsing is performed respectively according to the stream groups to generate table association instructions for each of the stream groups; The association instructions for each of the flow groups are generated according to the table association instructions for each of the flow groups.

5. The method according to claim 4, characterized in that The generating of the association instruction of each flow grouping according to the table association instruction of each flow grouping includes: According to the information of each stream group, correspondingly obtaining metadata prompt information of each stream group from the configuration information set; According to the metadata prompt information of each of the stream groups, respectively perform prompt association analysis according to the stream groups, and generate prompt association instructions for each of the stream groups; According to the table association instructions and prompt association instructions of each of the flow groups, the association instructions of each of the flow groups are generated accordingly.

6. The method according to claim 3, characterized in that The step of performing filtering condition parsing on the configuration information set according to the flow grouping information to generate filtering instructions for each flow grouping includes: According to the information of each stream grouping, correspondingly obtaining the filtering condition information of each stream grouping from the configuration information set; Convert the filtering condition information of each of the flow groups into a tree structure; For each of the flow groups, the filter condition is parsed according to the tree structure and the corresponding filter instruction is generated.

7. The method according to claim 6, characterized in that The tree structure includes at least one level, and the filtering condition parsing according to the tree structure and generating the corresponding filtering instruction includes: For the target level of the tree structure, obtaining the node type and operation mode of each node in the target level; According to the node type and operation mode of each node in the target level, a filtering sub-instruction of the target level is generated, and the next level of the target level is determined as a new target level, and the step of obtaining the node type and operation mode of each node in the target level is returned to be executed until all levels in the tree structure are traversed; According to the filtering sub-instructions of all levels of the tree structure, a filtering instruction corresponding to the tree structure is generated.

8. The method according to any one of claims 3 to 7, characterized in that: The step of performing wide table interpolation and lookup field parsing on the configuration information set according to the flow grouping information to generate an interpolation and lookup instruction for each flow grouping includes: According to the information of each stream grouping, metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each stream grouping are correspondingly obtained from the configuration information set; According to the metadata information, metadata association information, task wide table association information and task wide table field increase and decrease information of each of the stream groups, the wide table fields are parsed respectively according to the stream groups to generate the insertion and query instructions of each of the stream groups.

9. A wide table data synchronization device, characterized in that: The device comprises: An acquisition module, used to acquire task information and a configuration information set of a wide table data synchronization task; wherein the configuration information set is used to represent the configuration information of the wide table and the configuration information of the original table associated with the wide table; A generation module, used to parse the configuration information set according to the task information, and generate a synchronization operation instruction of the wide table data synchronization task, wherein the synchronization operation instruction is used to standardize the operation of synchronizing the wide table with at least part of the source data in the original table; A synchronization module is used to execute the synchronization operation instruction.

10. A computer device comprising a memory and a processor, wherein the memory stores computer instructions, characterized in that: When the processor executes the computer instructions, the steps of the method according to any one of claims 1 to 8 are implemented.

Images