Data transmission method, device and equipment based on Apache Flink and medium

Abstract

The invention discloses a data transmission method and device based on Apache Flink, equipment and a medium, and relates to the technical field of data transmission, and the method comprises the steps: building a connection with a data warehouse tool based on a tool address and configuration file information, so as to obtain structure information corresponding to all to-be-determined data tables; obtaining to-be-transmitted data collected by the Apache Flink, analyzing the to-be-transmitted data, and determining a target data table name and a data partition corresponding to the to-be-transmitted data based on an analysis result and the structure information of each to-be-determined data table; and writing each piece of to-be-transmitted data and the corresponding data partition into a file system corresponding to the data warehouse tool by using a preset data writing component, so that the file system stores each piece of to-be-transmitted data to the corresponding data partition, and transmits the structure information of the target data table corresponding to each piece of to-be-transmitted data to the data warehouse tool. Therefore, the data transmission efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a data transmission method, device, equipment and medium based on Apache Flink. Background Art

[0002] With the advent of the big data era, the amount of data on the Internet is growing exponentially. In order to deal with the growing amount of data, the current method used is the streaming data processing method. Among the many streaming data processing frameworks, Apache Apache Flink (an open source stream processing framework) has become a processing framework widely recognized by the industry with its powerful stream processing capabilities. In addition, the emergence of Hive (a data warehouse tool) simplifies the management and query of data.

[0003] However, currently Apache Flink only supports outputting data to a single Hive table through SQLConnectors (a database connection tool), and does not yet provide direct support for DataStream Connectors (a component used to connect external data sources and data receivers). In other words, each Hive table needs to develop an independent Apache Flink task to write real-time data streams into the Hive table. When there are hundreds or thousands of data tables in Hive, the above method not only increases development costs, but also increases maintenance difficulties.

[0004] As can be seen from the above, how to improve the efficiency of data transmission in the data transmission process based on Apache Flink is an urgent problem to be solved. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a data transmission method, device, equipment and medium based on Apache Flink, which can improve the efficiency of data transmission in the data transmission process based on Apache Flink, thereby enhancing the flexibility of data processing. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a data transmission method based on Apache Flink, which is applied to a target operator deployed in Apache Flink, wherein the target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool, and the method includes:

[0007] Establishing a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data table to be determined includes the table name of the data table to be determined and the field information of the data table to be determined;

[0008] Acquire the data to be transmitted collected by the Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table names and data partitions respectively corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each of the data tables to be determined;

[0009] A preset data writing component is used to write each of the data to be transmitted and the corresponding data partitions into a file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

[0010] Optionally, establishing a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all data tables to be determined from the data warehouse tool includes:

[0011] Determining the data warehouse tool based on the tool address, and establishing a communication connection line with the data warehouse tool using the configuration file information;

[0012] The table names of the to-be-determined data tables and the field information of the to-be-determined data tables corresponding to all the to-be-determined data tables in the data warehouse tool are obtained from the data warehouse tool through the communication connection line, and the table names of the to-be-determined data tables and the field information of the to-be-determined data tables are stored in the local cache.

[0013] Optionally, the acquiring the data to be transmitted collected by the Apache Flink, and parsing the data to be transmitted using the data parsing configuration information, so as to determine the target data table names and data partitions respectively corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each of the data tables to be determined, includes:

[0014] Acquire data to be transmitted collected from data sources respectively corresponding to each of the data source connectors using several data source connectors in the Apache Flink;

[0015] Parsing any field value in the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition respectively corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined;

[0016] Or, using the data parsing configuration information to extract preset field information in the data to be transmitted, and using preset parsing rules to parse the preset field information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained by parsing and the structural information of each of the data tables to be determined;

[0017] Or, using the data parsing configuration information to parse the preset regular expression corresponding to the data to be transmitted, so as to determine the target data table name and data partition respectively corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined;

[0018] Or, the code corresponding to the data to be transmitted is parsed using the data parsing configuration information, so as to determine the target data table name and data partition respectively corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each data table to be determined.

[0019] Optionally, after parsing the data to be transmitted by using the data parsing configuration information to determine the target data table names and data partitions respectively corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each of the data tables to be determined, the method further includes:

[0020] The information format of the data to be transmitted is adjusted by using preset format adjustment rules and based on the field information corresponding to the table names of the target data tables, so as to match the data to be transmitted after the information format adjustment with the field information corresponding to the table names of the target data tables.

[0021] Optionally, after the preset data writing component is used to write each of the data to be transmitted and the corresponding data partitions to a file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool, the method further includes:

[0022] The output status corresponding to each of the target data tables is saved to the state backend of the Apache Flink, so that when an abnormal situation occurs and restart is performed, the file system continues to output the data to be transmitted to the target data tables corresponding to the table names of the target data tables based on the current output status obtained from the state backend of the Apache Flink.

[0023] Optionally, after the preset data writing component is used to write each of the data to be transmitted and the corresponding data partitions to a file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool, the method further includes:

[0024] The data warehouse tool client is used to perform partition synchronization operations on the data corresponding to each target data table and the data corresponding to the file system to keep the data corresponding to each target data table consistent with the data corresponding to the file system, and the data in the file system is queried based on the data query instructions issued by the data warehouse tool.

[0025] Optionally, the target operator is an operator that inherits the AbstractStreamOperator in the Apache Flink; the data warehouse tool is Hive; and the data to be transmitted is data in JSON format.

[0026] In a second aspect, the present application provides a data transmission device based on Apache Flink, which is applied to a target operator deployed in Apache Flink, wherein the target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool, and the device includes:

[0027] A connection establishment module, used to establish a connection with the data warehouse tool based on the tool address and the configuration file information, so as to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data table to be determined includes the table name of the data table to be determined and the field information of the data table to be determined;

[0028] A data parsing module, used to obtain the data to be transmitted collected by the Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined;

[0029] A data writing module is used to use a preset data writing component to write each of the data to be transmitted and the corresponding data partitions into a file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

[0030] In a third aspect, the present application provides an electronic device, including:

[0031] Memory, used to store computer programs;

[0032] A processor is used to execute the computer program to implement the aforementioned data transmission method based on Apache Flink.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned data transmission method based on Apache Flink.

[0034] As can be seen from the above, before performing data transmission based on Apache Flink, the present application needs to establish a connection with the data warehouse tool based on the tool address and the configuration file information, so as to obtain the structural information of the data tables to be determined corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table names of the data tables to be determined and the field information of the data tables to be determined; the data to be transmitted collected by the Apache Flink is obtained, and the data to be transmitted is parsed using the data parsing configuration information, so as to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each data table to be determined; and the preset data writing component is used to write each data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool.

[0035] It can be seen that the present application obtains the structural information of the to-be-determined data tables corresponding to all the to-be-determined data tables from the data warehouse tool based on the tool address and the configuration file information. Then, the data to be transmitted collected by Apache Flink is obtained, and the data to be transmitted is parsed using the data parsing configuration information, so as to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each to-be-determined data table, and finally, the preset data writing component is used to write each data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool. In this way, the efficiency of data transmission is improved, thereby improving the flexibility of data processing and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 This is a flow chart of a data transmission method based on Apache Flink disclosed in this application;

[0038] Figure 2 A schematic diagram of a data transmission process based on Apache Flink disclosed in this application;

[0039] Figure 3 This is a schematic diagram of the use of data transmission based on Apache Flink disclosed in this application;

[0040] Figure 4 This is a schematic diagram of the structure of a data transmission device based on Apache Flink disclosed in this application;

[0041] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] With the advent of the big data era, the amount of data on the Internet is growing exponentially. In order to solve the growing amount of data, the method currently used is the streaming data processing method. Among them, among the many streaming data processing frameworks, Apache Apache Flink has become a processing framework widely recognized by the industry with its powerful stream processing capabilities. In addition, the emergence of Hive simplifies the management and query of data. To this end, the present application provides a data transmission method based on Apache Flink, which can improve the efficiency of data transmission in the data transmission process based on Apache Flink, thereby improving the flexibility of data processing.

[0044] See also Figure 1 As shown, an embodiment of the present invention discloses a data transmission method based on Apache Flink, which is applied to a target operator deployed in Apache Flink, wherein the target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool, and the method includes:

[0045] Step S11, establishing a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table name of the data table to be determined and the field information of the data table to be determined.

[0046] In this embodiment, the target operator is an operator that inherits Flink's AbstractStreamOperator (i.e., a core component in the Apache Flink framework), so that the target operator can process multi-dimensional, multi-link data, and the target operator is named HiveRoutConnector. It is worth mentioning that after the target operator is created, the target operator needs to be initialized. In a specific implementation, the process of initializing the target operator is as follows: the Hive address, Hive configuration file information, and data parsing configuration information are passed into the target operator to initialize HiveRoutConnector.

[0047] After initializing HiveRoutConnector, the embodiment of the present application needs to determine the target Hive based on the Hive address, and then use the configuration file information of Hive to obtain the Hive table information from the target Hive, and cache the obtained Hive table information in the local memory. Specifically, establishing a connection with the data warehouse tool based on the tool address and configuration file information to obtain the structural information corresponding to all the data tables to be determined from the data warehouse tool can include: determining the data warehouse tool based on the tool address, and establishing a communication connection line with the data warehouse tool using the configuration file information; obtaining the table names of the data tables to be determined and the field information of the data tables to be determined corresponding to all the data tables to be determined in the data warehouse tool through the communication connection line, and storing the table names of each data table to be determined and the field information of the data tables to be determined in the local cache.

[0048] Step S12: Obtain the data to be transmitted collected by the Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined.

[0049] In this embodiment, after obtaining the data to be transmitted collected by Apache Flink, the data to be transmitted needs to be parsed based on the data parsing configuration information to obtain the Hive table where the data to be transmitted is to be stored. Specifically, obtaining the data to be transmitted collected by Apache Flink, and parsing the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each data table to be determined, may include: obtaining the data to be transmitted collected from the data sources corresponding to each data source connector using several data source connectors in Apache Flink.

[0050] It is worth mentioning that there are four parsing methods for parsing the data to be transmitted, specifically, it can include using data parsing configuration information to parse any field value in the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result obtained by parsing and the structural information of each data table to be determined; or, using data parsing configuration information to extract preset field information in the data to be transmitted, and using preset parsing rules to parse the preset field information, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result obtained by parsing and the structural information of each data table to be determined; or, using data parsing configuration information to parse the preset regular expression corresponding to the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result obtained by parsing and the structural information of each data table to be determined; or, using data parsing configuration information to parse the code corresponding to the data to be transmitted, so as to determine the target data table name and data partition corresponding to the data to be transmitted based on the parsing result obtained by parsing and the structural information of each data table to be determined.

[0051] In a first specific implementation, a value is taken from any field in JSON (ie, a lightweight data exchange format) data, and the obtained information is used as the name of a Hive table to be outputted by the JSON data.

[0052] In a second specific implementation, based on the @type information in the JSON data, the @type information in the JSON data is then parsed into a corresponding Hive table name according to a built-in rule.

[0053] In a third specific implementation, a custom regular expression matching the JSON data is parsed into a corresponding Hive table name.

[0054] In a fourth specific implementation, the Java code matching the JSON data is parsed to obtain the corresponding Hive table name.

[0055] In this embodiment, after obtaining the Hive table name corresponding to the JSON data, the embodiment of the present application needs to perform format conversion on the data to be transmitted based on the Hive table name corresponding to the JSON data and the field information in the Hive table, so as to match the data to be transmitted after the information format adjustment with the field information corresponding to the table name of the target data table. Specifically, after the data to be transmitted is parsed using the data parsing configuration information, and the table names and data partitions of the target data tables corresponding to the data to be transmitted are determined based on the parsing results obtained by the parsing and the structural information of each data table to be determined, it can also include: using the preset format adjustment rules, and adjusting the information format of the data to be transmitted based on the field information corresponding to the table names of each target data table, so as to match the data to be transmitted after the information format adjustment with the field information corresponding to the table names of each target data table.

[0056] In a specific implementation, the data to be transmitted is in JSON format, and the preset format adjustment rule is to convert the data to be transmitted into camel case and underline formats, and to package the JSON data for information conversion into a RowData.

[0057] Step S13, use the preset data writing component to write each of the data to be transmitted and the corresponding data partitions into the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

[0058] In a specific implementation, after packaging the JSON data into a RowData. The embodiment of the present application needs to pass RowData into a custom-implemented BucketWriter so that BucketWriter can be used to output RowData to a distributed file system corresponding to a specific Hive table, such as HDFS (Hadoop Distributed File System, i.e., a distributed file system). It is worth mentioning that in the process of outputting RowData to a distributed file system corresponding to a specific Hive table, the embodiment of the present application finds or creates a corresponding partition directory according to different partitions of each data and outputs the corresponding data to the corresponding data partition. Among them, the data stored in the data partition in the distributed file system supports a variety of data formats and compression formats, including but not limited to textFile (i.e., a text file), Parquet (i.e., a column storage format), CSV (i.e., a text file format), Avro (i.e., a data serialization system), and the compression format includes but is not limited to Snappy (i.e., a fast compression algorithm), LZO (i.e., LZO compression algorithm), Zstandard (i.e., Z standard compression algorithm), Gzip (i.e., GNU zip compression algorithm).

[0059] It is worth mentioning that after storing the data to be transmitted in the corresponding data partitions and transmitting the structural information of the target data tables corresponding to each data to be transmitted to the data warehouse tool, the output status of all target data tables needs to be saved in the state backend of Flink, so that when the process fails and is automatically or manually restarted, the data will continue to be output according to the state saved in the state backend of Flink to avoid the problem of repeated data output. Specifically, the preset data writing component is used to write each data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partitions, and transmits the structural information of the target data tables corresponding to each data to be transmitted to the data warehouse tool, it can also include: saving the output status corresponding to each target data table to the state backend of Apache Flink, so that when an abnormal situation occurs and restarts, the file system continues to output the data to be transmitted to the target data tables corresponding to the table names of each target data table based on the current output status obtained from the state backend of Apache Flink.

[0060] In this embodiment, after the data to be transmitted is stored in the corresponding data partition, the embodiment of the present application needs to perform a partition synchronization on the Hive table output of the data to be transmitted through the Hive client, so that HQL query can be executed after the partition synchronization is completed, so as to query the Hive table data output to the distributed file system based on the query information. Specifically, the preset data writing component is used to write each data to be transmitted and the corresponding data partition to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool, and it can also include: using the data warehouse tool client to perform partition synchronization operations on the data corresponding to each target data table and the data corresponding to the file system, so as to keep the data corresponding to each target data table consistent with the data corresponding to the file system, and query the data in the file system based on the data query instructions issued by the data warehouse tool.

[0061] It is worth mentioning that when all the data to be transmitted are output, the embodiment of the present application will save the output status of all the data to be transmitted, and synchronize all partition information to the Hive client so that when data is transmitted again based on the status, the data can continue to be output based on the last output status, thereby ensuring the consistency and integrity of the data.

[0062] As can be seen from the above, before the data transmission based on Apache Flink is performed, the embodiment of the present application first needs to obtain the structural information of the to-be-determined data tables corresponding to all the to-be-determined data tables from the data warehouse tool based on the tool address and the configuration file information. Then, the data to be transmitted collected by Apache Flink is obtained, and the data to be transmitted is parsed using the data parsing configuration information, so as to determine the table name and data partition of the target data table corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each to-be-determined data table, and finally, the preset data writing component is used to write each data to be transmitted and the corresponding data partition to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool. In this way, the efficiency of data transmission is improved, thereby improving the flexibility of data processing and improving the user experience.

[0063] For further information, see Figure 2 As shown, an embodiment of the present invention discloses a data transmission method based on Apache Flink, which is applied to a target operator deployed in Apache Flink, wherein the target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool, and the method includes:

[0064] In this embodiment, the data transmission process diagram based on Apache Flink is as follows: Figure 2 As shown:

[0065] First, inherit Flink's AbstractStreamOperator to create a Connector operator that can process multi-dimensional and multi-link data, and name the Connector operator HiveRoutConnector.

[0066] Secondly, initialize HiveRoutConnector, that is, pass the Hive address, Hive configuration file information and data parsing configuration into HiveRoutConnector to initialize the entire Connector.

[0067] Subsequently, the target Hive is determined according to the Hive address passed in, and a connection is established with the target Hive to obtain the Hive table information from the target Hive through the connection and based on the configuration file information, and cache the obtained Hive table information in the memory.

[0068] Furthermore, the JSON data flowing into HiveRoutConnector is parsed according to the data parsing configuration to obtain the Hive table to be output by the JSON data. At the same time, according to the field information in the Hive table, the JSON data is converted into camel case and underline formats so that the data of each field in the JSON matches the field information in the Hive table, and the data of each field in the converted JSON is packaged into a RowData.

[0069] Then, the RowData is passed to the custom-implemented BucketWriter, which uses BucketWriter to output the RowData to the distributed file system HDFS corresponding to the specific Hive table, and find or create the partition directory corresponding to the RowData and output the data according to the different partitions of each data. HDFS supports multiple data storage formats and compression formats.

[0070] Subsequently, when outputting the data, the embodiment of the present application needs to perform a checkpoint, that is, save the output status of all tables to the state backend of Flink, so that when the process fails and is automatically or manually restarted, the data can continue to be output according to the saved status, avoiding the problem of repeated data output.

[0071] Furthermore, after the Checkpoint is completed, the embodiment of the present application needs to perform a partition synchronization on the output Hive table through the Hive client, so that after the synchronization is completed, HQL (Hive Query Language, a Hive-based query language) can be executed to query the Hive table data output to the distributed file system.

[0072] Finally, when the data output is completed and the data output process is stopped, the embodiment of the present application will save the output status of all data and synchronize all partition information to the Hive client, so that when the data is output again based on the status, the data can continue to be output based on the last output status, ensuring the consistency and integrity of the data.

[0073] The following is a schematic diagram of data transmission based on Apache Flink: Figure 3 As shown: First, the embodiment of the present application needs to use the DataStream Connectors provided by Flink (i.e., a method for connecting external data sources to a data stream processing framework) to collect data to be transmitted from multiple data sources, where the data sources include but are not limited to Kafka (i.e., a distributed stream processing platform), RabbitMQ (i.e., a message queue service software), files, and databases, and perform corresponding business logic processing on the collected data to ensure the integrity and accuracy of the data.

[0074] Then, the HiveRoutConnector node is configured, where the content configured in the HiveRoutConnector node is as follows: Hive service information and data parsing method. The Hive service information includes the Hive address and Hive configuration file information.

[0075] Among them, the data parsing method includes taking the value of any field in the JSON data as the Hive table name to be output; using the built-in rules to parse the @type information in the JSON data to obtain the Hive table name corresponding to the JSON data; using a custom regular expression that matches the JSON data to parse the JSON data to obtain the corresponding Hive table name; parsing the custom Java code corresponding to the JSON data to obtain the corresponding Hive table name.

[0076] Then, configure the storage path of the abnormal data so that the data can be processed and debugged based on the storage path of the abnormal data in the future, and save the configured HiveRoutConnector node, the storage path of the abnormal data, and the data parsing method as a Flink task.

[0077] Finally, the entire Flink task is packaged and compiled to generate the corresponding JAR package, and the JAR package is submitted to the big data cluster for execution.

[0078] As can be seen from the above, in the process of data transmission based on Apache Flink, the embodiment of the present application needs to obtain the structural information of the data tables to be determined that correspond to all the data tables to be determined from the data warehouse tool based on the tool address and the configuration file information. Then, the data to be transmitted collected by Apache Flink is obtained, and the data to be transmitted is parsed using the data parsing configuration information, so as to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each data table to be determined. Finally, the preset data writing component is used to write each data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool. In this way, configurable and automatic parsing of multi-dimensional and multi-link data is realized, and data is output to a large number of hive tables at the same time using a Flink task, and the system can flexibly respond to changes in different data sources and data structures, improve the adaptability and scalability of the system and the consistency of data, and improve the efficiency of data transmission.

[0079] Accordingly, see Figure 4 As shown, the present application also provides a data transmission device based on Apache Flink, which is applied to a target operator deployed in Apache Flink, wherein the target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool, and the device includes:

[0080] A connection establishment module 11 is used to establish a connection with the data warehouse tool based on the tool address and the configuration file information, so as to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data table to be determined includes the table name of the data table to be determined and the field information of the data table to be determined;

[0081] A data parsing module 12 is used to obtain the data to be transmitted collected by the Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined;

[0082] The data writing module 13 is used to use a preset data writing component to write each of the data to be transmitted and the corresponding data partitions into a file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

[0083] As can be seen from the above, before performing data transmission based on Apache Flink, the embodiment of the present application needs to establish a connection with the data warehouse tool based on the tool address and the configuration file information, so as to obtain the structural information of the data tables to be determined corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data tables to be determined includes the table names of the data tables to be determined and the field information of the data tables to be determined; the data to be transmitted collected by the Apache Flink is obtained, and the data to be transmitted is parsed using the data parsing configuration information, so as to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each of the data tables to be determined; and the preset data writing component is used to write each of the data to be transmitted and the corresponding data partitions into the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

[0084] It can be seen that the present application obtains the structural information of the to-be-determined data tables corresponding to all the to-be-determined data tables from the data warehouse tool based on the tool address and the configuration file information. Then, the data to be transmitted collected by Apache Flink is obtained, and the data to be transmitted is parsed using the data parsing configuration information, so as to determine the target data table names and data partitions corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each to-be-determined data table, and finally, the preset data writing component is used to write each data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each data to be transmitted to the data warehouse tool. In this way, the efficiency of data transmission is improved, thereby improving the flexibility of data processing.

[0085] In some specific implementations, the connection establishing module 11 may specifically include:

[0086] a connection establishing subunit, configured to determine the data warehouse tool based on the tool address, and to establish a communication connection line with the data warehouse tool using the configuration file information;

[0087] A data table field information acquisition unit is used to obtain the table names of the data tables to be determined and the field information of the data tables to be determined corresponding to all the data tables to be determined in the data warehouse tool through the communication connection line, and store the table names of the data tables to be determined and the field information of the data tables to be determined in the local cache.

[0088] In some specific implementations, the data parsing module 12 may specifically include:

[0089] A data acquisition unit, used to acquire data to be transmitted collected from data sources respectively corresponding to each of the data source connectors using a plurality of data source connectors in the Apache Flink;

[0090] A first data parsing unit, configured to parse the value of any field in the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and the data partition respectively corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined;

[0091] A second data parsing unit, configured to extract preset field information from the data to be transmitted using the data parsing configuration information, and parse the preset field information using a preset parsing rule, so as to determine the target data table name and data partition respectively corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined;

[0092] A third data parsing unit, configured to parse the preset regular expression corresponding to the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and the data partition respectively corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined;

[0093] The fourth data parsing unit is used to parse the code corresponding to the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained and the structural information of each data table to be determined.

[0094] In some specific implementations, the data transmission device based on Apache Flink may further include:

[0095] The information format adjustment unit is used to use preset format adjustment rules and adjust the information format of the data to be transmitted based on the field information corresponding to the table name of each target data table, so as to match the data to be transmitted after the information format adjustment with the field information corresponding to the table name of each target data table.

[0096] In some specific implementations, the data transmission device based on Apache Flink may further include:

[0097] An output state saving unit is used to save the output state corresponding to each of the target data tables to the state backend of the Apache Flink, so that when an abnormal situation occurs and restarts, the file system continues to output the data to be transmitted to the target data tables corresponding to the table names of the target data tables based on the current output state obtained from the state backend of the Apache Flink.

[0098] In some specific implementations, the data transmission device based on Apache Flink may further include:

[0099] A synchronization operation execution unit is used to use the data warehouse tool client to perform partition synchronization operations on the data corresponding to each target data table and the data corresponding to the file system, so as to keep the data corresponding to each target data table consistent with the data corresponding to the file system, and query the data in the file system based on the data query instructions issued by the data warehouse tool.

[0100] Furthermore, the present application also discloses an electronic device. Figure 5 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the data transmission method based on Apache Flink disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0101] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0102] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0103] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the data transmission method based on Apache Flink executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0104] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed data transmission method based on Apache Flink is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0105] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0106] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0107] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0108] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0109] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data transmission method based on Apache Flink, characterized in that: Applied to a target operator deployed in Apache Flink, the target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool, and the method includes: Establishing a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data table to be determined includes the table name of the data table to be determined and the field information of the data table to be determined; Acquire the data to be transmitted collected by the Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table names and data partitions respectively corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each of the data tables to be determined; A preset data writing component is used to write each of the data to be transmitted and the corresponding data partitions into a file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

2. The data transmission method based on Apache Flink according to claim 1, characterized in that: The establishing a connection with the data warehouse tool based on the tool address and the configuration file information to obtain structural information corresponding to all the to-be-determined data tables from the data warehouse tool includes: Determining the data warehouse tool based on the tool address, and establishing a communication connection line with the data warehouse tool using the configuration file information; The table names of the to-be-determined data tables and the field information of the to-be-determined data tables corresponding to all the to-be-determined data tables in the data warehouse tool are obtained from the data warehouse tool through the communication connection line, and the table names of the to-be-determined data tables and the field information of the to-be-determined data tables are stored in the local cache.

3. The data transmission method based on Apache Flink according to claim 1, characterized in that: The obtaining of the data to be transmitted collected by the Apache Flink, and parsing the data to be transmitted using the data parsing configuration information, so as to determine the target data table names and data partitions respectively corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each of the data tables to be determined, includes: Acquire data to be transmitted collected from data sources respectively corresponding to each of the data source connectors using several data source connectors in the Apache Flink; Parsing any field value in the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition respectively corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined; Or, using the data parsing configuration information to extract preset field information in the data to be transmitted, and using preset parsing rules to parse the preset field information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained by parsing and the structural information of each of the data tables to be determined; Or, using the data parsing configuration information to parse the preset regular expression corresponding to the data to be transmitted, so as to determine the target data table name and data partition respectively corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined; Or, the code corresponding to the data to be transmitted is parsed using the data parsing configuration information, so as to determine the target data table name and data partition respectively corresponding to the data to be transmitted based on the parsing result obtained by the parsing and the structural information of each data table to be determined.

4. The data transmission method based on Apache Flink according to claim 1, characterized in that: After the data to be transmitted is parsed by using the data parsing configuration information to determine the target data table names and data partitions respectively corresponding to the data to be transmitted based on the parsing results obtained by the parsing and the structural information of each of the data tables to be determined, the method further includes: The information format of the data to be transmitted is adjusted by using preset format adjustment rules and based on the field information corresponding to the table names of the target data tables, so as to match the data to be transmitted after the information format adjustment with the field information corresponding to the table names of the target data tables.

5. The data transmission method based on Apache Flink according to claim 1, characterized in that: After the preset data writing component is used to write each of the data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structure information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool, the method further includes: The output status corresponding to each of the target data tables is saved to the state backend of the Apache Flink, so that when an abnormal situation occurs and restart is performed, the file system continues to output the data to be transmitted to the target data tables corresponding to the table names of the target data tables based on the current output status obtained from the state backend of the Apache Flink.

6. The data transmission method based on Apache Flink according to claim 1, characterized in that: After the preset data writing component is used to write each of the data to be transmitted and the corresponding data partitions to the file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structure information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool, the method further includes: The data warehouse tool client is used to perform partition synchronization operations on the data corresponding to each target data table and the data corresponding to the file system to keep the data corresponding to each target data table consistent with the data corresponding to the file system, and the data in the file system is queried based on the data query instructions issued by the data warehouse tool.

7. The data transmission method based on Apache Flink according to any one of claims 1 to 6, characterized in that: The target operator is an operator that inherits the AbstractStreamOperator in the Apache Flink; the data warehouse tool is Hive; and the data to be transmitted is data in JSON format.

8. A data transmission device based on Apache Flink, characterized in that: Applied to a target operator deployed in Apache Flink, the target operator includes a tool address, configuration file information, and data parsing configuration information corresponding to a data warehouse tool, and the device includes: A connection establishment module, used to establish a connection with the data warehouse tool based on the tool address and the configuration file information, so as to obtain structural information corresponding to all the data tables to be determined from the data warehouse tool; the structural information of the data table to be determined includes the table name of the data table to be determined and the field information of the data table to be determined; A data parsing module, used to obtain the data to be transmitted collected by the Apache Flink, and parse the data to be transmitted using the data parsing configuration information, so as to determine the target data table name and data partition corresponding to the data to be transmitted respectively based on the parsing result obtained by the parsing and the structural information of each of the data tables to be determined; A data writing module is used to use a preset data writing component to write each of the data to be transmitted and the corresponding data partitions into a file system corresponding to the data warehouse tool, so that the file system stores each of the data to be transmitted in the corresponding data partition, and transmits the structural information of the target data table corresponding to each of the data to be transmitted to the data warehouse tool.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the data transmission method based on Apache Flink as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the steps of the data transmission method based on Apache Flink are implemented as described in any one of claims 1 to 7.

Images