A data processing method, device and computer equipment

Abstract

The application discloses a data processing method and device and computer equipment, the method comprises the following steps: analyzing obtained task information, extracting source data table information and target data table information from the task information, the target data table information comprises a target data table type and target data table parameters; according to the target data table type, obtaining a table building strategy corresponding to the target data table type from a preset storage location, the table building strategy comprises a data extraction strategy and a data persistence strategy; after determining a source data table corresponding to the source data table information according to the source data table information, extracting metadata of the source data table by using the data extraction strategy; and constructing a target data table according to the target data table type, the target data table parameters, the metadata and the data persistence strategy.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a data processing method, apparatus, and computer equipment. Background Technology

[0002] One of the most critical aspects of data processing is metadata management. Flink, as a mainstream computing engine, possesses extremely powerful data processing capabilities. However, during Flink's data processing, metadata is temporary, and the mapping relationship between Flink and metadata is not persistent. This means that each task needs to carry or rewrite the corresponding DDL (Data Definition Language) information, resulting in cumbersome task execution and difficulties in managing metadata. Summary of the Invention

[0003] Therefore, to address the shortcomings of the prior art, embodiments of the present invention provide a data processing method, apparatus, and computer device.

[0004] According to a first aspect, embodiments of the present invention disclose a data processing method, comprising:

[0005] The obtained task information is parsed, and the source data table information and target data table information are extracted from the task information. The target data table information includes the target data table type and target data table parameters.

[0006] Based on the target data table type, obtain the table creation strategy corresponding to the target data table type from the preset storage location. The table creation strategy includes data extraction strategy and data persistence strategy.

[0007] After determining the source data table corresponding to the source data table information based on the source data table information, the metadata of the source data table is extracted using data extraction strategies;

[0008] The target data table is constructed based on the target data table type, target data table parameters, metadata, and data persistence strategy.

[0009] Optionally, the table creation strategy consists of a data definition language corresponding to the target database type.

[0010] Optionally, metadata from the source data table can be extracted using a data extraction strategy, specifically including:

[0011] Based on the data extraction strategy, determine the field information in the source data table that corresponds to the target data table type;

[0012] Extract metadata from the source data table based on field information.

[0013] Optionally, the target data table is constructed based on the target data table type, target data table parameters, metadata, and table creation strategy, specifically including:

[0014] Determine the plugin path based on the target data table type and the mapping relationship between the target data table type and the plugin path;

[0015] Determine the target file in the folder corresponding to the plugin path;

[0016] Construct the target data table based on the target file, target data table parameters, metadata, and table creation strategy.

[0017] Optionally, the target data table is constructed based on the target file, target data table parameters, metadata, and table creation strategy, specifically including:

[0018] Based on the target file, target data table parameters, and table creation strategy, generate instructions to build the target data table;

[0019] When the target data table instruction is verified to be correct, the target data table is constructed based on the target data table construction instruction and metadata.

[0020] Optionally, if the instructions for building the target data table are incorrect, the method further includes:

[0021] An alarm message is generated to alert staff of data processing errors.

[0022] According to a second aspect, embodiments of the present invention also disclose a data processing apparatus, comprising:

[0023] The parsing module is used to parse the acquired task information and extract source data table information and target data table information from the task information. The target data table information includes the target data table type and target data table parameters.

[0024] The acquisition module is used to obtain the table creation strategy corresponding to the target data table type from a preset storage location based on the target data table type. The table creation strategy includes data extraction strategy and data persistence strategy.

[0025] The extraction module is used to extract metadata from the source data table after determining the source data table corresponding to the source data table information based on the source data table information, and then using data extraction strategies.

[0026] The build module is used to construct the target data table based on the target data table type, target data table parameters, metadata, and data persistence strategy.

[0027] According to a third aspect, embodiments of the present invention also disclose a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform steps of a data processing method as described in the first aspect or any optional embodiment of the first aspect.

[0028] According to a fourth aspect, embodiments of the present invention also disclose a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the data processing method as described in the first aspect or any optional embodiment of the first aspect.

[0029] The technical solution of this invention has the following advantages:

[0030] The data processing method, apparatus, and computer equipment provided by this invention include: parsing acquired task information, extracting source data table information and target data table information from the task information, wherein the target data table information includes target data table type and target data table parameters; further, based on the target data table type, obtaining a table building strategy corresponding to the target data table type from a preset storage location, where the table building strategy is persisted to the preset storage location, therefore, once the target data table type is determined, the corresponding table building strategy can be uniquely determined to complete the construction of the target data table and subsequent other operations. The table building strategy includes a data extraction strategy and data persistence. The metadata creation strategy retrieves corresponding metadata from the source data table based on the data extraction strategy, and the source data table can be uniquely identified based on the information in the source data table. Finally, the target data table is constructed based on the target data table type, target data table parameters, metadata, and data persistence strategy, thereby completing the persistence of metadata. Since the table creation strategy has already been stored in the preset storage location, when a target data table of the same type needs to be constructed, the table creation strategy can be directly retrieved. After the metadata is persisted to the target data table, if subsequent operations are needed on the metadata, the relevant operations can be performed directly according to the corresponding operation methods of the target data table, without the need for further persistence. Attached Figure Description

[0031] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating a specific example of the data processing method in an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating a specific example of the data processing method in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram illustrating a specific example of the data processing method in an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram illustrating a specific example of the data processing method in an embodiment of the present invention;

[0036] Figure 5 This is a schematic block diagram of a specific example of a data processing device in an embodiment of the present invention;

[0037] Figure 6 This is a specific example diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0038] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0040] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can also refer to the internal connection of two components; and they can refer to a wireless connection or a wired connection. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0041] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0042] To address the technical problems mentioned in the background section, this application provides a data processing method, as detailed below. Figure 1 .

[0043] Before executing this method, a custom Catalog, or target database directory, needs to be pre-built. This custom Catalog is implemented based on MySQL or other RDB databases to achieve metadata persistence in Flink. Of course, the table creation strategy can also be stored in different locations depending on the application scenario, such as Hive or MySQL. Different tasks can share a single table creation strategy, as long as the type of the target data table is the same.

[0044] Specifically, the process of building a custom Catalog can be as follows:

[0045] Implement the Flink AbstractCatalog abstract class interface in Java, declare the parameters required to construct a Catalog (catalog name, URL, password, etc.), and implement CRUD operations on the Catalog.

[0046] Inherit the CatalogFactory interface and implement the createCatalog method. Create the corresponding Catalog implementation class based on the passed parameters to handle subsequent metadata operations in the database.

[0047] Set the full path to the AbstractCatalog implementation class in META_INF / services / org.apache.flink.table.factories.Factory for SPI calls.

[0048] like Figure 1 As shown, the method includes the following steps:

[0049] Step 101: Parse the obtained task information and extract the source data table information and target data table information from the task information.

[0050] The target data table information includes the target data table type and target data table parameters.

[0051] For example, task information can be instructions for Flink to process data, including source data table information and target data table information. The source data table is the actual data source table that Flink reads or writes during data processing; it can also be called the Flink mapping table. The target data table is the data table that needs to be built. Metadata from the source data table needs to be persisted to the target data table. Metadata cannot be manipulated in the source data table, so it needs to be persisted to the target data table, which can be called a Catalog table.

[0052] The task information is parsed using Apache Calcite, which analyzes the task script. This parsing combines the source and target table information retrieved from the user platform. Specifically, the source table information includes the table name, IP address, and port number—anything that uniquely identifies the source table. The target table information includes the table type (e.g., MySQL or Hive) and its parameters. These parameters are uniquely assigned to each table type and can be names or passwords. For example, a Hive table might contain Hive's configuration information, while a MySQL table might contain the URL, username, and password.

[0053] The task information can be configured and generated through a user-friendly visual interface. This information includes source data (specifically, selecting the actual data source, target table type, fields, etc.), as shown below. Figure 2 , Figure 3 and Figure 4 As shown, select the Catalog type (target data table type), upload the information required for the Catalog (target data table parameters), and then generate task information based on the target data table type and target data table parameters.

[0054] Step 102: Based on the target data table type, obtain the table creation strategy corresponding to the target data table type from the preset storage location. The table creation strategy includes a data extraction strategy and a data persistence strategy.

[0055] The table creation strategy consists of a data definition language corresponding to the target database type.

[0056] For example, after determining the target data table type, the table creation strategy corresponding to the target data table type is obtained from a preset storage location. The preset storage location can be the business library of the system platform, which stores the mapping relationship between the jar path of the custom Catalog that has been constructed as mentioned above and the corresponding table creation strategy. The corresponding Catalog can be determined according to the target data table type to obtain the table creation strategy.

[0057] Step 103: After determining the source data table corresponding to the source data table information based on the source data table information, extract the metadata of the source data table using the data extraction strategy.

[0058] For example, the table building strategy is to construct the target data table based on the parameters of the target data table and persist the metadata. Before persisting, it is necessary to obtain the metadata that needs to be persisted.

[0059] In a preferred embodiment, metadata from the source data table is extracted using a data extraction strategy, specifically including:

[0060] To extract metadata from the source data table according to the data extraction strategy, it is first necessary to obtain the field names, types, field mappings, primary keys, and custom parameters configured in the source data table, and then generate metadata based on this information.

[0061] Step 104: Construct the target data table based on the target data table type, target data table parameters, metadata, and data persistence strategy.

[0062] For example, after obtaining the metadata, the target data table is constructed based on the obtained data persistence strategy, target data table type, target data table parameters, and metadata. Specifically, firstly, a Catalog creation instruction can be generated based on the data persistence strategy, target data table type, and target data table parameters, and the corresponding Catalog can be obtained according to the Catalog type; secondly, the metadata is persisted to the Catalog according to the preset operation instructions (CRUD operations) of the Catalog to complete the persistence of the metadata.

[0063] If subsequent operations such as querying or deleting metadata are required, simply determine the Catalog based on the corresponding task information and complete the query or deletion of metadata according to the preset operation instructions of the target data. These preset operation instructions are available for each type and can be either default or user-defined.

[0064] When multiple target data tables need to be built simultaneously, class loading conflicts may occur during Catalog construction due to different versions of the data source driver. To solve this problem, each type of Catalog has a corresponding plugin. In a preferred embodiment, the target data table is constructed based on the target data table type, target data table parameters, metadata, and table creation strategy, specifically including:

[0065] Determine the plugin path based on the target data table type and the mapping relationship between the target data table type and the plugin path;

[0066] Determine the target file in the folder corresponding to the plugin path;

[0067] Construct the target data table based on the target file, target data table parameters, metadata, and table creation strategy.

[0068] For example, there is a one-to-one mapping relationship between Catalog type and plugin path. The corresponding plugin path can be determined according to the Catalog type. The corresponding jar package exists in the folder corresponding to the plugin path. Loading the jar package loads the class file (target file) under the corresponding path. The target data table is constructed based on the class file.

[0069] You can also determine whether the target data table has been built by its name. If it has not been built, you can re-register it with the CatalogManager of StreamTableEnvironment to complete the construction of the target data table. Subsequently, you can persist the metadata information to the Catalog by calling the API of the custom Catalog implementation, which is the Catalog operation method.

[0070] When customizing a Catalog, multiple Catalog types can be created, supporting storage for various database types. The catalog is managed and can interface with external platforms. By selecting the corresponding Catalog type and entering Catalog parameters through the platform's interface, task information is generated. Users do not need to rewrite DDL each time, and multiple tables pre-defined in the system's Catalogs can be used within a task. This greatly simplifies the steps required to use FlinkSQL in existing technologies.

[0071] This method parses the acquired task information and extracts source and target data table information. The target data table information includes the target data table type and parameters. Further, based on the target data table type, a table creation strategy corresponding to that type is retrieved from a preset storage location. Therefore, once the target data table type is determined, the corresponding table creation strategy can be uniquely identified to complete the construction of the target data table and subsequent operations. This table creation strategy includes a data extraction strategy and a data persistence strategy. The corresponding metadata can be retrieved from the source data table based on the data extraction strategy, and the source data table can be uniquely identified based on the source data table information. Finally, the target data table is constructed based on the target data table type, parameters, metadata, and persistence strategy, thus completing the persistence of the metadata. Since the table creation strategy has already been stored in the preset storage location, when another target data table of the same type needs to be constructed, the table creation strategy can be directly retrieved. After persisting the metadata to the target data table, if subsequent operations on the metadata are required, the relevant operations can be performed directly according to the corresponding operation methods of the target data table, without needing to persist it again. The above are embodiments of the data processing method provided in this application. Other embodiments of the data processing method provided in this application will be described below. Please refer to the following for details.

[0072] This invention also discloses a data processing apparatus, such as... Figure 5 As shown, the device includes:

[0073] The parsing module 501 is used to parse the acquired task information and extract source data table information and target data table information from the task information. The target data table information includes the target data table type and target data table parameters.

[0074] The acquisition module 502 is used to acquire the table creation strategy corresponding to the target data table type from a preset storage location according to the target data table type. The table creation strategy includes a data extraction strategy and a data persistence strategy.

[0075] The extraction module 503 is used to extract the metadata of the source data table after determining the source data table corresponding to the source data table information based on the source data table information and then using a data extraction strategy.

[0076] Module 504 is used to construct the target data table based on the target data table type, target data table parameters, metadata, and data persistence strategy.

[0077] As an optional implementation of the present invention, the table creation strategy is constituted by a data definition language corresponding to the Catalog type.

[0078] In an optional embodiment of the present invention, the extraction module is specifically used for:

[0079] Based on the data extraction strategy, determine the field information in the source data table that corresponds to the target data table type;

[0080] Extract metadata from the source data table based on field information.

[0081] As an optional embodiment of the present invention, the construction module is specifically used for:

[0082] Determine the plugin path based on the target data table type and the mapping relationship between the target data table type and the plugin path;

[0083] Determine the target file in the folder corresponding to the plugin path;

[0084] Construct the target data table based on the target file, target data table parameters, metadata, and table creation strategy.

[0085] As an optional embodiment of the present invention, the construction module is further configured to:

[0086] Based on the target file, target data table parameters, and table creation strategy, generate instructions to build the target data table;

[0087] When the target data table instruction is verified to be correct, the target data table is constructed based on the target data table construction instruction and metadata.

[0088] As an optional embodiment of the present invention, if the instruction to construct the target data table is incorrect, the device is further configured to: generate alarm information to alert staff of data processing errors.

[0089] The functions performed by each component in the data processing device provided in the embodiments of the present invention have been described in detail in any of the above method embodiments, and therefore will not be repeated here.

[0090] By executing this device, the acquired task information is parsed, and source data table information and target data table information are extracted from the task information. The target data table information includes the target data table type and target data table parameters. Further, based on the target data table type, a table creation strategy corresponding to the target data table type can be retrieved from a preset storage location. This table creation strategy is persisted to the preset storage location; therefore, once the target data table type is determined, the corresponding table creation strategy can be uniquely determined to complete the construction of the target data table and subsequent operations. This table creation strategy includes a data extraction strategy and a data persistence strategy, which can be determined based on... The data extraction strategy retrieves the corresponding metadata from the source data table, which can be uniquely identified based on the source data table information. Finally, the target data table is constructed based on the target data table type, target data table parameters, metadata, and data persistence strategy, thereby completing the persistence of the metadata. Since the table construction strategy has already been stored in the preset storage location, when a target data table of the same type needs to be constructed, the table construction strategy can be directly retrieved. After the metadata is persisted to the target data table, if subsequent operations on the metadata are required, the relevant operations can be performed directly according to the corresponding operation methods of the target data table, without the need for further persistence.

[0091] This invention also provides a computer device, such as... Figure 6 As shown, the computer device may include a processor 601 and a memory 602, wherein the processor 601 and the memory 602 may be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0092] Processor 601 may be a central processing unit (CPU). Processor 601 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0093] The memory 602, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the data processing method in the embodiments of the present invention. The processor 601 executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory 602, thereby implementing the data processing method in the above-described method embodiments.

[0094] The memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor 601, etc. Furthermore, the memory 602 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 602 may optionally include memory remotely located relative to the processor 601, and these remote memories may be connected to the processor 601 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] One or more modules are stored in memory 602, and when executed by processor 601, they perform actions such as... Figure 1 The data processing method in the illustrated embodiment.

[0096] For specific details regarding the aforementioned computer equipment, please refer to the relevant documentation. Figure 1 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.

[0097] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0098] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A data processing method, characterized by, The method comprises: parsing the obtained task information, extracting source data table information and target data table information from the task information, the target data table information comprising a target data table type and target data table parameters; acquiring a table building strategy corresponding to the target data table type from a preset storage location according to the target data table type, the table building strategy comprising a data extraction strategy and a data persistence strategy; the table building strategy is constituted by a data definition language corresponding to the target data table type; after determining a source data table corresponding to the source data table information according to the source data table information, extracting metadata of the source data table by using the data extraction strategy; constructing a target data table according to the target data table type, the target data table parameters, the metadata, and the data persistence strategy, comprising: determining a plug-in path according to the target data table type and a mapping relationship between the target data table type and the plug-in path; determining a target file under a folder corresponding to the plug-in path according to the plug-in path; and constructing the target data table according to the target file, the target data table parameters, the metadata, and the data persistence strategy.

2. The method of claim 1, wherein, The method of extracting the metadata of the source data table by using the data extraction strategy specifically comprises: determining field information corresponding to the target data table type in the source data table according to the data extraction strategy; extracting the metadata in the source data table according to the field information.

3. The method of claim 1, wherein, The method of constructing the target data table according to the target file, the target data table parameters, the metadata, and the data persistence strategy specifically comprises: generating a target data table construction instruction according to the target file, the target data table parameters, and the data persistence strategy; when the target data table construction instruction is verified to be correct, constructing the target data table according to the target data table construction instruction and the metadata.

4. The method of claim 3, wherein, If the target data table construction instruction is incorrect, the method further comprises: generating an alarm information, the alarm information being used to prompt a staff of a data processing error.

5. A data processing apparatus, characterized by, The device comprises: a parsing module configured to parse obtained task information, and extract source data table information and target data table information from the task information, the target data table information comprising a target data table type and target data table parameters; an acquisition module configured to acquire a table building strategy corresponding to the target data table type from a preset storage location according to the target data table type, the table building strategy comprising a data extraction strategy and a data persistence strategy; the table building strategy is constituted by a data definition language corresponding to the target data table type; an extraction module configured to extract metadata of a source data table by using the data extraction strategy after determining a source data table corresponding to the source data table information according to the source data table information; a construction module configured to construct a target data table according to the target data table type, the target data table parameters, the metadata, and the data persistence strategy. The construction module is specifically configured to determine the plug-in path according to the target data table type and a mapping relationship between the target data table type and the plug-in path; determine a target file under a folder corresponding to the plug-in path according to the plug-in path; and construct the target data table according to the target file, the target data table parameter, the metadata and the data persistence strategy.

6. A computer device, comprising: Comprise: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the data processing method of any one of claims 1-4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a server, implements the data processing method of any one of claims 1-4.

Images