An Apache Flink-based data transmission method, device, equipment and medium

Abstract

The application discloses an Apache Flink-based data transmission method and device, equipment and medium, and relates to the technical field of data transmission. The method comprises the following steps: connecting a data warehouse tool based on a tool address and configuration file information to obtain structure information corresponding to all to-be-determined data tables; obtaining to-be-transmitted data collected by Apache Flink, analyzing the to-be-transmitted data, determining a target data table name and a data partition corresponding to the to-be-transmitted data based on an analysis result and the structure information of each to-be-determined data table; and writing each to-be-transmitted data and the corresponding data partition into a file system corresponding to the data warehouse tool by using a preset data writing component, so that the file system stores each to-be-transmitted data into the corresponding data partition and transmits the structure information of the target data table corresponding to each to-be-transmitted data to the data warehouse tool. In this way, the efficiency of data transmission can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data transmission technology, and in particular to a data transmission method, apparatus, device, and medium based on Apache Flink. Background Art

[0002] With the advent of the big data era, the amount of data on the internet is growing exponentially. To cope with this ever-increasing volume, stream data processing is currently the preferred approach. Among the numerous stream data processing frameworks, Apache Flink (an open-source stream processing framework) has become widely recognized in the industry for its powerful stream processing capabilities. Furthermore, the emergence of Hive (a data warehouse tool) has simplified data management and querying.

[0003] However, Apache Flink currently only supports outputting data to a single Hive table through SQLConnectors (a database connection tool). It does not directly support DataStream Connectors (components used to connect external data sources and data sinks). This means that each Hive table requires developing a separate Apache Flink task to write real-time data streams to the Hive table. When there are hundreds or thousands of data tables in Hive, this approach not only increases development costs but also makes maintenance more difficult.

[0004] As can be seen from the above, how to improve the efficiency of data transmission in the data transmission process based on Apache Flink is an urgent problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a data transmission method, apparatus, device, and medium based on Apache Flink, which can improve the efficiency of data transmission in the data transmission process based on Apache Flink, thereby enhancing the flexibility of data processing. The specific solution is as follows:

[0006] In a first aspect, the present application provides a data transmission method based on Apache Flink, which is applied to a target operator deployed in Apache Flink. The target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool. The method includes:

[0007] Establishing a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table names of the data tables to be determined and the field information of the data tables to be determined;

[0008] Obtain the data to be transmitted collected by the Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained and the structural information of each of the data tables to be determined;

[0009] A preset data writing component is used to write each of the data to be transmitted and the corresponding data partitions into the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

[0010] Optionally, establishing a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all data tables to be determined from the data warehouse tool includes:

[0011] Determining the data warehouse tool based on the tool address, and establishing a communication connection line with the data warehouse tool using the configuration file information;

[0012] The table names of the to-be-determined data tables and the field information of the to-be-determined data tables corresponding to all the to-be-determined data tables in the data warehouse tool are obtained from the data warehouse tool through the communication connection line, and the table names of the to-be-determined data tables and the field information of the to-be-determined data tables are stored in the local cache.

[0013] Optionally, obtaining the data to be transmitted collected by Apache Flink and parsing the data to be transmitted using the data parsing configuration information, so as to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results and the structural information of each of the data tables to be determined, includes:

[0014] Obtaining data to be transmitted collected from data sources corresponding to respective data source connectors in the Apache Flink using the data source connectors;

[0015] Parsing any field value in the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result and the structural information of each of the data tables to be determined;

[0016] Alternatively, using the data parsing configuration information to extract preset field information from the data to be transmitted, and parsing the preset field information using preset parsing rules, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained and the structural information of each of the data tables to be determined;

[0017] Or, using the data parsing configuration information to parse the preset regular expression corresponding to the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained and the structural information of each of the data tables to be determined;

[0018] Alternatively, the code corresponding to the data to be transmitted is parsed using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result and the structural information of each data table to be determined.

[0019] Optionally, after parsing the data to be transmitted using the data parsing configuration information to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results and the structural information of each of the data tables to be determined, the method further includes:

[0020] The preset format adjustment rules are used to adjust the information format of the data to be transmitted based on the field information corresponding to the table name of each target data table, so that the data to be transmitted after the information format adjustment is matched with the field information corresponding to the table name of each target data table.

[0021] Optionally, after using a preset data writing component to write each of the data to be transferred and the corresponding data partitions to a file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transferred in the corresponding data partition and transmits the structural information of the target data table corresponding to each of the data to be transferred to the data warehouse tool, the method further includes:

[0022] The output state corresponding to each of the target data tables is saved to the state backend of the Apache Flink, so that when an abnormal situation occurs and a restart is performed, the file system continues to output the data to be transmitted to the target data tables corresponding to the table names of the target data tables based on the current output state obtained from the state backend of the Apache Flink.

[0023] Optionally, after using a preset data writing component to write each of the data to be transferred and the corresponding data partitions to a file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transferred in the corresponding data partition and transmits the structural information of the target data table corresponding to each of the data to be transferred to the data warehouse tool, the method further includes:

[0024] The data warehouse tool client is used to perform partition synchronization operations on the data corresponding to each target data table and the data corresponding to the file system to keep the data corresponding to each target data table consistent with the data corresponding to the file system, and query the data in the file system based on the data query instructions issued by the data warehouse tool.

[0025] Optionally, the target operator is an operator that inherits the AbstractStreamOperator in the Apache Flink; the data warehouse tool is Hive; and the data to be transmitted is data in JSON format.

[0026] In a second aspect, the present application provides an Apache Flink-based data transmission device, which is applied to a target operator deployed in Apache Flink. The target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool. The device includes:

[0027] a connection establishment module, configured to establish a connection with the data warehouse tool based on the tool address and the configuration file information, so as to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table names of the data tables to be determined and the field information of the data tables to be determined;

[0028] a data parsing module, configured to obtain the data to be transmitted collected by Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result and the structural information of each of the data tables to be determined;

[0029] A data writing module is used to use a preset data writing component to write each of the data to be transmitted and the corresponding data partitions into the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

[0030] In a third aspect, the present application provides an electronic device, comprising:

[0031] Memory, used to store computer programs;

[0032] A processor is configured to execute the computer program to implement the aforementioned data transmission method based on Apache Flink.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned Apache Flink-based data transmission method.

[0034] As can be seen from the above, before performing data transmission based on Apache Flink, this application needs to establish a connection with the data warehouse tool based on the tool address and the configuration file information, so as to obtain the structural information of the data tables to be determined corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table name of the data table to be determined and the field information of the data table to be determined; obtain the data to be transmitted collected by Apache Flink, and use the data parsing configuration information to parse the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing results obtained and the structural information of each data table to be determined; use the preset data writing component to write each data to be transmitted and the corresponding data partition to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool.

[0035] It can be seen that the present application obtains the structural information of the data tables to be determined that correspond to all the data tables to be determined from the data warehouse tool based on the tool address and the configuration file information. Then, the data to be transmitted collected by Apache Flink is obtained, and the data parsing configuration information is used to parse the data to be transmitted, so as to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each data table to be determined. Finally, the preset data writing component is used to write each data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool. In this way, the efficiency of data transmission is improved, thereby improving the flexibility of data processing and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0037] Figure 1 This is a flow chart of a data transmission method based on Apache Flink disclosed in this application;

[0038] Figure 2 This is a schematic diagram of a data transmission process based on Apache Flink disclosed in this application;

[0039] Figure 3 This is a schematic diagram of the use of data transmission based on Apache Flink disclosed in this application;

[0040] Figure 4 This is a structural diagram of a data transmission device based on Apache Flink disclosed in this application;

[0041] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] With the advent of the big data era, the amount of data on the Internet is growing exponentially. In order to cope with the ever-increasing amount of data, the method currently adopted is the streaming data processing method. Among the many streaming data processing frameworks, Apache Flink has become a widely recognized processing framework in the industry due to its powerful stream processing capabilities. In addition, the emergence of Hive has simplified the management and query of data. To this end, this application provides a data transmission method based on Apache Flink, which can improve the efficiency of data transmission in the data transmission process based on Apache Flink, thereby enhancing the flexibility of data processing.

[0044] See also Figure 1 As shown, an embodiment of the present invention discloses a data transmission method based on Apache Flink, which is applied to a target operator deployed in Apache Flink. The target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool. The method includes:

[0045] Step S11: establish a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table name of the data table to be determined and the field information of the data table to be determined.

[0046] In this embodiment, the target operator is an operator that inherits Flink's AbstractStreamOperator (a core component in the Apache Flink framework), enabling it to process multi-dimensional, multi-link data. The target operator is named HiveRoutConnector. It is worth mentioning that after the target operator is created, it must be initialized. In one specific implementation, the process of initializing the target operator is as follows: the Hive address, Hive configuration file information, and data parsing configuration information are passed to the target operator, thereby initializing HiveRoutConnector.

[0047] After initializing HiveRoutConnector, the embodiment of the present application needs to determine the target Hive based on the Hive address, and then use the Hive configuration file information to obtain the Hive table information from the target Hive, and cache the obtained Hive table information in the local memory. Specifically, establishing a connection with the data warehouse tool based on the tool address and configuration file information to obtain the structural information corresponding to all the data tables to be determined from the data warehouse tool can include: determining the data warehouse tool based on the tool address, and establishing a communication connection line with the data warehouse tool using the configuration file information; obtaining the table names of the data tables to be determined and the field information of the data tables to be determined corresponding to all the data tables to be determined in the data warehouse tool through the communication connection line, and storing the table names of the data tables to be determined and the field information of the data tables to be determined in the local cache.

[0048] Step S12: Obtain the data to be transmitted collected by the Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result and the structural information of each of the data tables to be determined.

[0049] In this embodiment, after obtaining the data to be transmitted collected by Apache Flink, the data to be transmitted needs to be parsed based on the data parsing configuration information to obtain the Hive table in which the data to be transmitted is to be stored. Specifically, obtaining the data to be transmitted collected by Apache Flink and parsing the data to be transmitted using the data parsing configuration information to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsed results and the structural information of each data table to be determined may include: obtaining the data to be transmitted collected from the data sources corresponding to each data source connector using multiple data source connectors in Apache Flink.

[0050] It is worth mentioning that there are four parsing methods for parsing the data to be transmitted. Specifically, it can include using data parsing configuration information to parse the value of any field in the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result obtained by parsing and the structural information of each data table to be determined; or, using data parsing configuration information to extract preset field information in the data to be transmitted, and using preset parsing rules to parse the preset field information, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result obtained by parsing and the structural information of each data table to be determined; or, using data parsing configuration information to parse the preset regular expression corresponding to the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result obtained by parsing and the structural information of each data table to be determined; or, using data parsing configuration information to parse the code corresponding to the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result obtained by parsing and the structural information of each data table to be determined.

[0051] In a first specific implementation, a value is obtained from any field in JSON (a lightweight data exchange format) data, and the obtained information is used as the name of a Hive table to which the JSON data is to be output.

[0052] In a second specific implementation, based on the @type information in the JSON data, the @type information in the JSON data is then parsed into a corresponding Hive table name according to built-in rules.

[0053] In a third specific implementation, a custom regular expression matching the JSON data is parsed into a corresponding Hive table name.

[0054] In a fourth specific implementation, the Java code matching the JSON data is parsed to obtain the corresponding Hive table name.

[0055] In this embodiment, after obtaining the Hive table name corresponding to the JSON data, the embodiment of the present application needs to perform format conversion on the data to be transmitted based on the Hive table name corresponding to the JSON data and the field information in the Hive table, so as to match the data to be transmitted after the information format adjustment with the field information corresponding to the target data table name. Specifically, after using the data parsing configuration information to parse the data to be transmitted, and determining the target data table name and data partition corresponding to the data to be transmitted based on the parsing results obtained and the structural information of each data table to be determined, it can also include: using the preset format adjustment rules, and adjusting the information format of the data to be transmitted based on the field information corresponding to the table name of each target data table, so as to match the data to be transmitted after the information format adjustment with the field information corresponding to the table name of each target data table.

[0056] In a specific implementation, the data to be transmitted is in JSON format, and the preset format adjustment rule is to convert the data to be transmitted into camel case and underline formats, and package the converted JSON data into a RowData.

[0057] Step S13: Use the preset data writing component to write each of the data to be transmitted and the corresponding data partitions into the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

[0058] In a specific embodiment, after packaging the JSON data into a RowData. The embodiment of the present application needs to pass the RowData into a custom-implemented BucketWriter, so as to use BucketWriter to output the RowData to the distributed file system corresponding to the specific Hive table, such as HDFS (Hadoop Distributed File System, i.e., a distributed file system). It is worth mentioning that in the process of outputting RowData to the distributed file system corresponding to the specific Hive table, the embodiment of the present application finds or creates the corresponding partition directory according to the different partitions of each data and outputs the corresponding data to the corresponding data partition. Among them, the data stored in the data partition in the distributed file system supports multiple data formats and compression formats, including but not limited to textFile (i.e., a text file), Parquet (i.e., a columnar storage format), CSV (i.e., a text file format), Avro (i.e., a data serialization system), and the compression format includes but is not limited to Snappy (i.e., a fast compression algorithm), LZO (i.e., LZO compression algorithm), Zstandard (i.e., Z standard compression algorithm), Gzip (i.e., GNU zip compression algorithm).

[0059] It is worth mentioning that after storing the data to be transmitted in the corresponding data partitions and transmitting the structural information of the target data tables corresponding to each data to be transmitted to the data warehouse tool, the output status of all target data tables needs to be saved in the state backend of Flink, so that when the process fails and is automatically or manually restarted, the data will continue to be output according to the status saved in the state backend of Flink to avoid the problem of repeated data output. Specifically, the preset data writing component is used to write each data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partitions, and transmits the structural information of the target data tables corresponding to each data to be transmitted to the data warehouse tool, it can also include: saving the output status corresponding to each target data table to the state backend of Apache Flink, so that when restarting in the event of an abnormal situation, the file system continues to output the data to be transmitted to the target data tables corresponding to the table names of each target data table based on the current output status obtained from the state backend of Apache Flink.

[0060] In this embodiment, after the data to be transmitted is stored in the corresponding data partition, the embodiment of the present application needs to perform a partition synchronization on the Hive table output by the data to be transmitted through the Hive client, so that the HQL query can be executed after the partition synchronization is completed, so as to query the Hive table data output to the distributed file system based on the query information. Specifically, the preset data writing component is used to write each data to be transmitted and the corresponding data partition to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool. It can also include: using the data warehouse tool client to perform a partition synchronization operation on the data corresponding to each target data table and the data corresponding to the file system, so as to keep the data corresponding to each target data table consistent with the data corresponding to the file system, and query the data in the file system based on the data query instruction issued by the data warehouse tool.

[0061] It is worth mentioning that after all the data to be transmitted have been output, the embodiment of the present application will save the output status of all the data to be transmitted and synchronize all partition information to the Hive client so that when data is transmitted again based on the status, the data can continue to be output based on the last output status, thereby ensuring the consistency and integrity of the data.

[0062] As can be seen from the above, before performing data transmission based on Apache Flink, the embodiment of the present application first needs to obtain the structural information of the data tables to be determined corresponding to all the data tables to be determined from the data warehouse tool based on the tool address and configuration file information. Then, the data to be transmitted collected by Apache Flink is obtained, and the data parsing configuration information is used to parse the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each data table to be determined. Finally, the preset data writing component is used to write each data to be transmitted and the corresponding data partition to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool. In this way, the efficiency of data transmission is improved, thereby improving the flexibility of data processing and improving the user experience.

[0063] For further information, see Figure 2 As shown, an embodiment of the present invention discloses a data transmission method based on Apache Flink, which is applied to a target operator deployed in Apache Flink. The target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool. The method includes:

[0064] In this embodiment, the data transmission process diagram based on Apache Flink is as follows: Figure 2 As shown:

[0065] First, inherit Flink's AbstractStreamOperator to create a Connector operator that can process multi-dimensional, multi-link data, and name the Connector operator HiveRoutConnector.

[0066] Secondly, initialize HiveRoutConnector, that is, pass the Hive address, Hive configuration file information and data parsing configuration into HiveRoutConnector to initialize the entire Connector.

[0067] Subsequently, the target Hive is determined according to the input Hive address, and a connection is established with the target Hive to obtain Hive table information from the target Hive based on the configuration file information through the connection, and cache the obtained Hive table information in the memory.

[0068] Furthermore, the JSON data flowing into the HiveRoutConnector is parsed according to the data parsing configuration to obtain the Hive table to which the JSON data is to be output. At the same time, the JSON data is converted to camel case and underline format according to the field information in the Hive table so that the data in each field in the JSON matches the field information in the Hive table. The data in each field in the converted JSON is then packaged into a RowData.

[0069] The RowData is then passed to a custom-implemented BucketWriter, which then outputs the RowData to the HDFS distributed file system corresponding to the specific Hive table. Based on the different partitions of each data entry, the partition directory corresponding to the RowData is found or created, and the data is output. HDFS supports multiple data storage and compression formats.

[0070] Subsequently, when outputting data, the embodiment of the present application needs to perform a checkpoint, that is, save the output status of all tables to Flink's state backend so that when the process fails and is automatically or manually restarted, data can continue to be output according to the saved status, avoiding the problem of repeated data output.

[0071] Further, after the checkpoint is completed, the embodiment of the present application needs to synchronize the partition of the output Hive table through the Hive client, so that the HQL (Hive Query Language, a query language based on Hive) query can be performed on the Hive table data output to the distributed file system after the synchronization is completed.

[0072] Finally, when the data output is completed and the data output process is stopped, the embodiment of the present application saves the output state of all data and synchronizes all partition information to the Hive client, so that the data can be continuously output based on the last output state when the data output is performed again based on the state, ensuring the consistency and integrity of the data.

[0073] The use schematic diagram of the Apache Flink-based data transmission is as shown in Figure 3 Firstly, the embodiment of the present application needs to use the DataStream Connectors (a kind of connector for connecting external data sources to the data stream processing framework) provided by Flink to collect the data to be transmitted from various data sources, wherein the data sources include but are not limited to Kafka (a kind of distributed stream processing platform), RabbitMQ (a kind of message queue service software), files, databases, and perform corresponding business logic processing on the collected data, to ensure the integrity and accuracy of the data.

[0074] Subsequently, the HiveRoutConnector node is configured, wherein the contents configured in the HiveRoutConnector node are as follows: Hive service information and data parsing mode. The Hive service information includes Hive address and Hive configuration file information.

[0075] The data parsing mode includes taking the value of any field in the JSON data as the Hive table name to be output, parsing the @type information in the JSON data by using the built-in rules to obtain the Hive table name corresponding to the JSON data, parsing the JSON data by using the custom regular expression matched with the JSON data to obtain the corresponding Hive table name, and parsing the custom Java code corresponding to the JSON data to obtain the corresponding Hive table name.

[0076] Subsequently, the saving path of the abnormal data is configured, so that the data can be processed and debugged based on the saving path of the abnormal data in the future, and the configured HiveRoutConnector node, the saving path of the abnormal data and the data parsing mode are saved as a Flink task.

[0077] Finally, the entire Flink task is packaged and compiled to generate a corresponding JAR package, and the JAR package is submitted to a big data cluster for running.

[0078] As can be seen from the above, in the process of data transmission based on Apache Flink, the embodiment of the application needs to obtain the structure information of all to-be-determined data tables respectively corresponding to the to-be-determined data tables from the data warehouse tool based on the tool address and the configuration file information. Then, the to-be-transmitted data collected by Apache Flink is obtained, and the to-be-transmitted data is parsed using the data parsing configuration information, to determine the target data table table name and data partition respectively corresponding to the to-be-transmitted data based on the parsed result obtained by parsing and the structure information of each to-be-determined data table. Finally, the preset data writing component is used to write each to-be-transmitted data and the respective data partition to the file system corresponding to the data warehouse tool, so that the file system stores each to-be-transmitted data in the corresponding data partition, and transmits the structure information of the target data table respectively corresponding to each to-be-transmitted data to the data warehouse tool. In this way, the multi-dimensional and multi-link data is configured and automatically parsed, and one Flink task is used to output data to a large number of hive tables at the same time, and the system can flexibly cope with changes in different data sources and data structures, improving the adaptability and scalability of the system and the consistency of the data, and improving the efficiency of data transmission.

[0079] Correspondingly, referring to Figure 4 The application also provides an Apache Flink-based data transmission device, which is applied to a target operator deployed in Apache Flink. The target operator includes a tool address corresponding to a data warehouse tool, configuration file information, and data parsing configuration information. The device includes:

[0080] A connection establishment module 11 is configured to establish a connection with the data warehouse tool based on the tool address and the configuration file information, to obtain the structure information of all to-be-determined data tables respectively corresponding to the to-be-determined data tables from the data warehouse tool. The structure information of the to-be-determined data tables includes a to-be-determined data table table name and field information of a to-be-determined data table.

[0081] A data parsing module 12 is configured to obtain to-be-transmitted data collected by the Apache Flink, and parse the to-be-transmitted data using the data parsing configuration information, to determine a target data table table name and a data partition respectively corresponding to the to-be-transmitted data based on a parsed result obtained by parsing and the structure information of each to-be-determined data table.

[0082] The data writing module 13 is used to use the preset data writing component to write each of the data to be transmitted and the corresponding data partitions into the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

[0083] As can be seen from the above, before performing data transmission based on Apache Flink, the embodiment of the present application needs to establish a connection with the data warehouse tool based on the tool address and the configuration file information to obtain the structural information of the data tables to be determined corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table name of the data table to be determined and the field information of the data table to be determined; obtain the data to be transmitted collected by Apache Flink, and use the data parsing configuration information to parse the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing results obtained and the structural information of each data table to be determined; use the preset data writing component to write each data to be transmitted and the corresponding data partition to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool.

[0084] It can be seen that the present application obtains the structural information of the data tables to be determined that correspond to all the data tables to be determined from the data warehouse tool based on the tool address and the configuration file information. Then, the data to be transmitted collected by Apache Flink is obtained, and the data parsing configuration information is used to parse the data to be transmitted, so as to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each data table to be determined. Finally, the preset data writing component is used to write each data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool. In this way, the efficiency of data transmission is improved, thereby improving the flexibility of data processing.

[0085] In some specific implementations, the connection establishing module 11 may specifically include:

[0086] a connection establishing subunit, configured to determine the data warehouse tool based on the tool address and establish a communication connection line with the data warehouse tool using the configuration file information;

[0087] The data table field information acquisition unit is configured to acquire, from the data warehouse tool via the communication connection line, field information of each of the to-be-determined data tables in the data warehouse tool and a table name of each of the to-be-determined data tables, and store the field information of each of the to-be-determined data tables and the table name of each of the to-be-determined data tables in a local cache.

[0088] In some embodiments, the data analysis module 12 can specifically include:

[0089] The data acquisition unit is configured to acquire to-be-transmitted data collected from data sources corresponding to each of the data source connectors by using the data source connectors in the Apache Flink.

[0090] The first data analysis unit is configured to analyze a value of any field in the to-be-transmitted data by using the data analysis configuration information, and determine a target data table name and a data partition corresponding to the to-be-transmitted data based on an analysis result obtained by the analysis and structure information of each of the to-be-determined data tables.

[0091] The second data analysis unit is configured to extract preset field information in the to-be-transmitted data by using the data analysis configuration information, analyze the preset field information by using a preset analysis rule, and determine a target data table name and a data partition corresponding to the to-be-transmitted data based on an analysis result obtained by the analysis and structure information of each of the to-be-determined data tables.

[0092] The third data analysis unit is configured to analyze a preset regular expression corresponding to the to-be-transmitted data by using the data analysis configuration information, and determine a target data table name and a data partition corresponding to the to-be-transmitted data based on an analysis result obtained by the analysis and structure information of each of the to-be-determined data tables.

[0093] The fourth data analysis unit is configured to analyze a code corresponding to the to-be-transmitted data by using the data analysis configuration information, and determine a target data table name and a data partition corresponding to the to-be-transmitted data based on an analysis result obtained by the analysis and structure information of each of the to-be-determined data tables.

[0094] In some embodiments, the Apache Flink-based data transmission apparatus can further include:

[0095] An information format adjustment unit is used to use preset format adjustment rules and adjust the information format of the data to be transmitted based on the field information corresponding to the table name of each target data table, so as to match the data to be transmitted after the information format adjustment with the field information corresponding to the table name of each target data table.

[0096] In some specific implementations, the Apache Flink-based data transmission device may further include:

[0097] An output state saving unit is used to save the output state corresponding to each of the target data tables to the state backend of the Apache Flink, so that when an abnormal situation occurs and the system is restarted, the file system continues to output the data to be transmitted to the target data tables corresponding to the table names of the target data tables based on the current output state obtained from the state backend of the Apache Flink.

[0098] In some specific implementations, the Apache Flink-based data transmission device may further include:

[0099] A synchronization operation execution unit is used to use the data warehouse tool client to perform partition synchronization operations on the data corresponding to each target data table and the data corresponding to the file system, so as to keep the data corresponding to each target data table consistent with the data corresponding to the file system, and query the data in the file system based on the data query instructions issued by the data warehouse tool.

[0100] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data transmission method based on Apache Flink disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0101] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0102] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0103] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including a computer program capable of implementing the Apache Flink-based data transmission method disclosed in any of the aforementioned embodiments and executed by the electronic device 20, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0104] Furthermore, this application discloses a computer-readable storage medium for storing a computer program. When executed by a processor, the computer program implements the aforementioned Apache Flink-based data transmission method. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be further described here.

[0105] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0106] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0107] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0108] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0109] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data transmission method based on Apache Flink, characterized in that: Applied to a target operator deployed in Apache Flink, the target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool. The method includes: Establishing a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table names of the data tables to be determined and the field information of the data tables to be determined; Obtain the data to be transmitted collected by the Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained and the structural information of each of the data tables to be determined; A preset data writing component is used to write each of the data to be transmitted and the corresponding data partitions into the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

2. The data transmission method based on Apache Flink according to claim 1, characterized in that: The establishing a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all the to-be-determined data tables from the data warehouse tool includes: Determining the data warehouse tool based on the tool address, and establishing a communication connection line with the data warehouse tool using the configuration file information; The table names of the to-be-determined data tables and the field information of the to-be-determined data tables corresponding to all the to-be-determined data tables in the data warehouse tool are obtained from the data warehouse tool through the communication connection line, and the table names of the to-be-determined data tables and the field information of the to-be-determined data tables are stored in the local cache.

3. The data transmission method based on Apache Flink according to claim 1, characterized in that: The step of obtaining the data to be transmitted collected by Apache Flink and parsing the data to be transmitted using the data parsing configuration information, so as to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results and the structural information of each of the data tables to be determined, includes: Obtaining data to be transmitted collected from data sources corresponding to respective data source connectors in the Apache Flink using the data source connectors; Parsing any field value in the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result and the structural information of each of the data tables to be determined; Alternatively, using the data parsing configuration information to extract preset field information from the data to be transmitted, and parsing the preset field information using preset parsing rules, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained and the structural information of each of the data tables to be determined; Or, using the data parsing configuration information to parse the preset regular expression corresponding to the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained and the structural information of each of the data tables to be determined; Alternatively, the code corresponding to the data to be transmitted is parsed using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result and the structural information of each data table to be determined.

4. The data transmission method based on Apache Flink according to claim 1, characterized in that: After parsing the data to be transmitted using the data parsing configuration information to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results and the structural information of each of the data tables to be determined, the method further includes: The preset format adjustment rules are used to adjust the information format of the data to be transmitted based on the field information corresponding to the table name of each target data table, so that the data to be transmitted after the information format adjustment is matched with the field information corresponding to the table name of each target data table.

5. The data transmission method based on Apache Flink according to claim 1, characterized in that: After using the preset data writing component to write each of the data to be transferred and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transferred in the corresponding data partition and transmits the structure information of the target data table corresponding to each of the data to be transferred to the data warehouse tool, the method further includes: The output state corresponding to each of the target data tables is saved to the state backend of the Apache Flink, so that when an abnormal situation occurs and a restart is performed, the file system continues to output the data to be transmitted to the target data tables corresponding to the table names of the target data tables based on the current output state obtained from the state backend of the Apache Flink.

6. The data transmission method based on Apache Flink according to claim 1, characterized in that: After using the preset data writing component to write each of the data to be transferred and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transferred in the corresponding data partition and transmits the structure information of the target data table corresponding to each of the data to be transferred to the data warehouse tool, the method further includes: The data warehouse tool client is used to perform partition synchronization operations on the data corresponding to each target data table and the data corresponding to the file system to keep the data corresponding to each target data table consistent with the data corresponding to the file system, and query the data in the file system based on the data query instructions issued by the data warehouse tool.

7. The data transmission method based on Apache Flink according to any one of claims 1 to 6, characterized in that: The target operator is an operator that inherits the AbstractStreamOperator in Apache Flink; the data warehouse tool is Hive; and the data to be transmitted is data in JSON format.

8. A data transmission device based on Apache Flink, characterized in that: Applied to a target operator deployed in Apache Flink, the target operator includes the tool address, configuration file information, and data parsing configuration information corresponding to the data warehouse tool. The device includes: a connection establishment module, configured to establish a connection with the data warehouse tool based on the tool address and the configuration file information, so as to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table names of the data tables to be determined and the field information of the data tables to be determined; a data parsing module, configured to obtain the data to be transmitted collected by Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result and the structural information of each of the data tables to be determined; A data writing module is used to use a preset data writing component to write each of the data to be transmitted and the corresponding data partitions into the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the data transmission method based on Apache Flink according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the steps of the data transmission method based on Apache Flink are implemented.

Images