Streaming data distribution method and system

Abstract

An embodiment of the present application provides a streaming data distribution method for use in a computer device at a data distribution layer, the method comprising: consuming a data stream in an upstream message subscription system and distributing the data in the data stream to corresponding downstream nodes; monitoring partition sensing messages associated with the data stream; and sending the partition sensing messages to the downstream nodes so that the downstream nodes perform corresponding data processing tasks. An embodiment of the present application is provided with a partition sensing mechanism to proactively provide partition sensing messages of the data stream in a timely manner. Compared with the traditional technology of detecting partition information through a timed script, the partition sensing mechanism can save computing performance.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of computer technology, and in particular, to a streaming data distribution method, apparatus, computer equipment, computer-readable storage medium, and streaming data distribution system. Background Art

[0002] The current streaming data distribution system is generally composed of a data transmission layer (such as a gateway), a data cache layer, a data distribution layer (controller) and a data storage terminal. When the data source has data to report, the data source will report the data through the data transmission layer, the data cache layer, the data distribution layer and finally flow into the data storage terminal.

[0003] However, the existing data distribution layer uses the Flume architecture, and different data streams are prone to mutual influence. In addition, the data flowing into the data storage terminal through the data distribution layer needs to be detected by a scheduled script, which consumes computing performance. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a streaming data distribution method, apparatus, computer equipment, computer-readable storage medium and streaming data distribution system, which can be used to solve the following problems: (1) the data streams affected by the Flume architecture; (2) the data flowing into the data storage terminal through the data distribution layer needs to detect the data partition information through a timing script, which consumes computing performance.

[0005] One aspect of an embodiment of the present application provides a streaming data distribution system, including one or more Flink clusters, each Flink cluster is used to: consume data streams in an upstream message subscription system and distribute the data in the data stream to corresponding downstream nodes; monitor partition sensing messages associated with the data stream; and send the partition sensing messages to the downstream nodes so that the downstream nodes perform corresponding data processing tasks.

[0006] Optionally, each Flink cluster includes a task executor and a task deployer; the task executor includes multiple data input modules, multiple data processing modules, multiple data output modules, and a partition message submission module; wherein the partition message submission module is used to: obtain the watermark of the data in each data output module and find the minimum watermark; generate the partition sensing message according to the minimum watermark; and send the partition sensing message to the downstream node so that the downstream node performs the corresponding data processing task.

[0007] Optionally, it is characterized in that the partition message submission module is also used to: send a cleanup instruction to the downstream node to notify the downstream node to clean up zombie files.

[0008] Optionally, each data input module is used to: consume a corresponding data stream; report the watermark of the corresponding data stream to the task deployer so that the task deployer generates a speed control instruction; receive the speed control instruction; and control the consumption speed of the corresponding data stream according to the speed control instruction to keep it consistent with the consumption speed of other data streams in other data input modules, so that the downstream node obtains all data within a time window each time.

[0009] Optionally, each data processing module includes a parsing module, an ETL module and an adaptation module, wherein: the parsing module is used to perform parsing operations on the corresponding data stream to obtain parsed data; the ETL module is used to perform corresponding ETL processing on the parsed data according to the stream identifier of the corresponding data stream to obtain ETL-processed data; the adaptation module is used to perform format conversion and / or type conversion on the ETL-processed data to obtain converted data.

[0010] Optionally, each data output module is used to dynamically allocate to obtain multiple partitions according to event time, metadata information of the data stream and metadata information of other data streams, so as to write corresponding data to corresponding directories in the downstream node.

[0011] Another aspect of an embodiment of the present application provides a streaming data distribution method for use in a computer device at a data distribution layer, the method comprising: consuming a data stream in an upstream message subscription system, and distributing the data in the data stream to corresponding downstream nodes; monitoring partition sensing messages associated with the data stream; and sending the partition sensing messages to the downstream nodes so that the downstream nodes perform corresponding data processing tasks.

[0012] Optionally, it also includes: generating a watermark for the data stream; reporting the watermark to a task deployer so that the task deployer generates a speed control instruction based on the watermark; receiving the speed control instruction; and controlling the consumption speed of the data stream based on the speed control instruction.

[0013] Optionally, it also includes: parsing the data stream to obtain parsed data; performing corresponding ETL processing on the parsed data according to the stream identifier of the data stream to obtain ETL-processed data; and performing format conversion and / or type conversion on the ETL-processed data to obtain converted data.

[0014] Optionally, the computer device is or is located in a Flink cluster; sending the partition sensing message to the downstream node so that the downstream node performs the corresponding data processing task, further comprising: obtaining all watermarks associated with the data stream in the Flink cluster and finding the minimum watermark; generating the partition sensing message according to the minimum watermark; and sending the partition sensing message to the downstream node so that the downstream node performs the corresponding data processing task.

[0015] Optionally, the method further includes: sending a cleanup instruction to the downstream node to notify the downstream node to clean up zombie files.

[0016] Optionally, it also includes: dynamically allocating to obtain multiple partitions based on event time, metadata information of the data stream and metadata information of other data streams, so as to write corresponding data to corresponding directories in the downstream node.

[0017] Another aspect of an embodiment of the present application provides a streaming data distribution device, which includes: a consumption module, which is used to consume the data stream in the upstream message subscription system and distribute the data in the data stream to the corresponding downstream node; a monitoring module, which is used to monitor the partition sensing message associated with the data stream; and a sending module, which is used to send the partition sensing message to the downstream node so that the downstream node performs the corresponding data processing task.

[0018] Another aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the streaming data distribution method as described in any one of the above items when executing the computer program.

[0019] Another aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it is used to implement the steps of the streaming data distribution method as described in any one of the above items.

[0020] The streaming data distribution method, apparatus, computer device, computer-readable storage medium, and streaming data distribution system provided by the embodiments of the present application have the following advantages: (1) Introducing the Flink architecture into the streaming data distribution system can avoid the mutual influence between different data streams. (2) A partition sensing mechanism is provided to proactively provide partition sensing messages of the data stream in a timely manner. Compared with the traditional technology of detecting partition information through a timed script, the partition sensing mechanism can save computing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A link diagram of a streaming data transmission link is schematically shown;

[0022] Figure 2 The following schematically shows an environment diagram of a streaming data distribution system according to Embodiment 1 of the present application;

[0023] Figure 3 A schematic diagram of partitioned sensing is shown schematically;

[0024] Figure 4 A schematic diagram of partitioning and slicing is schematically shown;

[0025] Figure 5 The workflow diagram of Transform is schematically shown;

[0026] Figure 6 A schematic diagram schematically shows the adaptation protocol and type;

[0027] Figure 7 A schematic diagram of dynamically allocating partitions is schematically shown;

[0028] Figure 8 The flowchart of the streaming data distribution method according to the second embodiment of the present application is schematically shown;

[0029] Fig. 9 The newly added flow chart of the streaming data distribution method according to the second embodiment of the present application is schematically shown;

[0030] Fig.10 The newly added flow chart of the streaming data distribution method according to the second embodiment of the present application is schematically shown;

[0031] Fig.11 for Figure 8 Sub-flow chart of step S804;

[0032] Fig.12 The newly added flow chart of the streaming data distribution method according to the second embodiment of the present application is schematically shown;

[0033] Fig.13 The newly added flow chart of the streaming data distribution method according to the second embodiment of the present application is schematically shown;

[0034] Fig.14 A block diagram schematically shows a streaming data distribution device according to Embodiment 3 of the present application; and

[0035] Fig.15 The hardware architecture diagram of a computer device suitable for implementing the streaming data distribution method according to the fourth embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0037] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0038] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0039] The following are some explanations of terms involved in this application:

[0040] Flink Cluster is a distributed system for stateful computation on unbounded and bounded data streams. Flink is designed to run in all common cluster environments, performing computations at memory speed and at any scale.

[0041] The task deployer (JobManager), as the Master node of the Flink cluster, is responsible for Flink cluster task scheduling and resource management.

[0042] Task executor (TaskManager), as the Worker node (slave node) of the Flink cluster. TaskManager receives the tasks to be deployed from JobManager and is responsible for the specific task execution and resource application and management of the corresponding tasks on each node.

[0043] The data input module (Source), as a data input interface, is used to consume one or more data streams from the corresponding topics (Topic) in the data cache layer 3.

[0044] The data processing module (Tranform) is used to process one or more data streams received by the Source.

[0045] The data output module (Sink), as a data output interface, is used to distribute the data obtained after the Tranform processing to the storage terminal of the data storage layer 5.

[0046] The partition message committer module (Commiter) is a partition committer module, which is used to generate partition sensing messages and send the partition sensing messages to the data storage layer 5.

[0047] Watermark is a mechanism proposed by Apache Flink for processing EventTime window calculations. It is essentially a timestamp.

[0048] The flow identifier (LogId) can be defined by three-segment semantics (such as department + project + business) so that the category to which the data belongs can be quickly locked. At the same time, the flow identifier can also be defined with other auxiliary information, such as creator information, etc. The data flow can be defined with a schema (the organization and structure of the database), such as fields, types, required or not, etc. The schema can be used for analysis and evaluation operations of the data flow. According to the defined schema, corresponding field values ​​can be written into the metadata information of the data flow, such as business scenarios, etc. Different business scenarios can be configured with different SLA (Service-Level Agreement) quality assurance. It should be noted that these field values ​​can be written and modified by users or managers.

[0049] Figure 1 The schematic diagram shows a streaming data transmission link according to an embodiment of the present application, and the streaming data transmission link is to provide streaming data transmission services, such as data collection and distribution for real-time streaming and offline streaming scenarios. In the real-time streaming scenario, corresponding to the data timeliness at the second level, it is mainly used to write data into databases such as kafka and hbase. In the offline streaming scenario, corresponding to the data timeliness at the hourly or daily level, it is mainly used to write data into databases such as HDFS and hive. The streaming data distribution system can be composed of the following parts: BFE layer 1, network routing layer 2, data buffer layer 3, data distribution layer 4, data storage layer 5, etc.

[0050] The BFE layer 1 may be implemented by one or more edge nodes, and is used to receive, process, and output reported data. The reported data may be data from different data sources, such as reported data from APP and Web.

[0051] The network routing layer 2 can be implemented by one or more gateway nodes, and is used to forward the data provided by the BFE layer 1 to the data buffer layer 3. Specifically, the network routing layer 2 is configured to be connected to the BFE layer 1, and can adapt to various business scenarios and data protocols, for example, it is configured to be compatible with APP and Web data that parses the HTTP (HyperText Transfer Protocol) protocol, and internal communication data of the GRPC protocol.

[0052] The data buffer layer 3 can be implemented through a message distribution subscription system or the above system cluster. In some embodiments, the data buffer layer 3 can be composed of multiple sets of Kafka clusters to play the role of data peak shaving and valley filling. Data of different importance, priority, and data throughput can be diverted to different Kafka clusters to ensure the value of different types of data and avoid system failures affecting the overall data.

[0053] The data distribution layer 4 can be implemented by a streaming data distribution system (composed of multiple traffic distribution node Collectors) for content conversion and distribution storage, that is, to ensure that data is obtained from the data buffer layer 3 and written to the corresponding storage terminal in the data storage layer 5. Specifically, the data distribution layer 4 is used for data distribution and landing, and the supported distribution scenarios include HDFS (Hadoop Distributed File System), Kafka, Hbase, ES (Elasticsearch), etc., and in the distribution process, due to the different requirements for the timeliness of data landing in different storage terminals, for example, the data writing of HDFS is the calculation and application of tasks on a daily basis, and the data writing of Kafka is generally the calculation and application of tasks on a second basis, which is usually used in scenarios such as real-time recommendation and real-time calculation. In response to the distribution requirements of different data scenarios, the data distribution layer 4 can manage services in groups according to the storage terminal. For example, the online will be divided into KafkaCollector group, HDFS Collector group, etc. Different Collector groups will obtain the data of the corresponding topic from the data buffer layer 3 and distribute it to the downstream.

[0054] The data storage layer 5 is used to store data and can be composed of databases in different forms. The database can be HDFS, ES, Hive, Kafka, Hbase, etc.

[0055] That is, the data flow of the streaming data transmission link is as follows: BFE layer 1 → network routing layer 2 → data buffer layer 3 → data distribution layer 4 → data storage layer 5. Through the streaming data transmission link, the data in the data source can be transmitted to the target terminal. Specifically, the data source can output a data stream with LogId as the stream identifier, report this data to the edge node through HTTP, RPC and other protocols, and pass through the gateway routing layer 2, data buffer layer 3, data distribution layer 4 in turn, and finally enter the storage terminal in the data storage layer 5.

[0056] Embodiment 1

[0057] This embodiment is intended to exemplarily describe a streaming data distribution system in the data distribution layer 4 .

[0058] The streaming data distribution system includes one or more Flink clusters, each of which is used to:

[0059] Consume the data stream in the upstream message subscription system and distribute the data in the data stream to the corresponding downstream nodes;

[0060] monitoring partition sensing messages associated with the data stream; and

[0061] The partition sensing message is sent to the downstream node so that the downstream node performs a corresponding data processing task.

[0062] The partition sensing information carries partition information, such as time partition information.

[0063] like Figure 2 As shown, the streaming data distribution system may exemplarily include three Flink clusters based on Yarn (Yet Another Resource Negotiator). The three Flink clusters may be responsible for one or more data streams under a Topic in the data cache layer 3, such as the data stream under the Kafka Topic, the data stream under the HDFS Topic, or the data stream under the Hbase Topic.

[0064] Continue reading Figure 2 , the process of pulling up a Flink cluster is as follows:

[0065] (1) When the user needs to configure a data stream A1, the MYSQL database can receive the following information provided by the user: system configuration information (System config), pipeline configuration information (Pipeline config), and metadata information (Metadata) of this data stream A1. (2) Lancer-control (control platform, which can perform control operations of the Flink cluster according to the configuration information of the data stream, such as scheduling control of the Flink cluster) will call the API (Application Programming Interface) of Lancer-service (service platform, which can write and read data stream configuration information in the database), obtain configuration information (such as System config) through Lance-service, and provide the configuration information to Saber (real-time computing platform, which can generate and submit scheduling instructions). (3) Saber submits the scheduling job to Yarn according to the configuration information. (4) After receiving the scheduling job, Yarn will pull up a Flink cluster according to the scheduling job, where the Flink cluster can get the data stream A1 from the data cache layer 3 and distribute the data stream A1.

[0066] Continue reading Figure 2 ,Each Flink cluster is equipped with a partition sensing mechanism, which actively sends partition sensing messages through the message service.

[0067] This embodiment has the following advantages: (1) Compared with the traditional Flume architecture, this embodiment introduces the Flink architecture into the streaming data distribution system, which can avoid the mutual influence between different data streams; (2) A partition sensing mechanism is provided to timely and proactively provide partition sensing messages of data streams. Compared with the traditional technology of detecting partition information through a timing script, the partition sensing mechanism can save computing performance. For the downstream node, after receiving the partition sensing message, it can determine the execution of the task based on the partition sensing message. Take the task executed on an hourly basis as an example: if the content of the partition sensing message includes 17:50 on August 7, 2020, it means that all the data before 17:00 on August 7, 2020 has been written to the downstream node. Therefore, the downstream node can perform corresponding tasks according to the data of the partition from 16:00 on August 7, 2020 to 17:00 on August 7, 2020, such as calculating the DAU (Daily Active User) in the time period from 16:00 on August 7, 2020 to 17:00 on August 7, 2020, or performing hourly incremental model training according to the data of the partition from 16:00 on August 7, 2020 to 17:00 on August 7, 2020.

[0068] like Figure 3 As shown, each Flink cluster includes two modules: JobManager and TaskManager. The TaskManager includes multiple Sources, multiple Transforms, multiple Sinks, and Committers.

[0069] The following introduces Committer, Source, Tranform, and Sink respectively.

[0070] (1) The Committer may provide partition sensing features.

[0071] The commiter is used to: obtain the watermark of the data in each sink and find the minimum watermark; generate the partition sensing message according to the minimum watermark; and send the partition sensing message to the downstream node so that the downstream node performs the corresponding data processing task. In this embodiment, each source generates a watermark, and the produced watermarks all flow to the commiter. It should be noted that the minimum watermark refers to the watermark with the earliest time among all watermarks. For example, when three watermarks are received and the time information of these three watermarks is 18:36 on August 7, 2020, 18:01 on August 7, 2020, and 18:45 on August 7, 2020, respectively), the commiter will write the watermark corresponding to "18:01 on August 7, 2020" into the partition sensing message. In this embodiment, a valid partition sensing message can be obtained through the commiter mechanism and watermark.

[0072] Continue reading Figure 3 The commiter is also used to send a cleanup instruction to the downstream node to notify the downstream node to clean up zombie files. Zombie files are generally files of a specific format generated at the time of device startup, etc., and the files of this format are invalid files. Figure 3 In the HDFS cluster, when the commiter detects that there are zombie files in a directory of the HDFS cluster, it issues a cleanup instruction. This embodiment can clean up invalid files of downstream nodes in a timely manner and increase available storage space.

[0073] The Committer is further used to append partition sensing messages to a metadata information storage node (eg, HIVEMetaStore).

[0074] (2) The Source may provide a partition and shard ordering feature.

[0075] In order to upgrade the tasks of the downstream nodes from the hourly level to the minute level, for example, the downstream node can perform the corresponding tasks according to the partition data from 19:21 to 19:25 on August 7, 2020 at 19:28 on August 7, 2020. This places extremely high demands on the synchronization of the data written to the downstream node, that is, before 19:28 on August 7, 2020, all data corresponding to 19:21 to 19:25 on August 7, 2020 must be written. In this embodiment, the above problem is solved by controlling the data consumption (inflow) speed of each Source. For example, each Source is used to: consume the corresponding data stream; report the watermark of the corresponding data stream to the JobManager so that the JobManager generates a speed control instruction; receive the speed control instruction; and control the consumption speed of the corresponding data stream according to the speed control instruction to keep consistent with the consumption speed of other data streams in other Sources, so that the downstream node obtains all data in a time window each time. like Figure 4 As shown, a Topic in the data cache layer 3 has three partitions. Generally speaking, one Partition corresponds to a Source in the Flink cluster. Some Sources consume quickly, for example, they have consumed until 19:46 on August 7, 2020; some Sources consume slowly, for example, they have consumed until 17:46 on August 7, 2020. This will lead to a problem that the data written to the downstream node has a large difference in time, and minute-level tasks cannot be executed in real time. In this embodiment, through the watermark reported by each Source, the JobManager can know the consumption progress of each Source, and thus issue corresponding speed control instructions to each Source respectively. After receiving the corresponding speed control instruction, each Source will limit the data consumption speed from the corresponding Partition, thereby ensuring that: the data streams in each Partition flow through the Flink cluster to the downstream node (such as the HDSF cluster) at approximately the same speed. As Figure 4 As shown in the figure, the data P1_T1, P2_T1, P3_T1, P4_T1, P5_T1, P6_T1, etc. corresponding to the T1 partition are stored in the HDFS cluster at approximately the same time. Since all the partition data of this time segment T1 are stored in the HDFS cluster almost at the same time, the corresponding data processing tasks can be performed for all the partition data of this time segment T1, and minute-level tasks can be performed.

[0076] (3) The Tranform, which can provide data processing and adaptation.

[0077] like Figure 5 As shown, each Tranform includes a parsing module (Parser), an ETL module (ETL) and an adapter module (Exporter). The parsing module is used to perform parsing operations on the corresponding data stream to obtain parsed data. The ETL module is used to perform corresponding ETL (Extract-Transform-Load) processing on the parsed data according to the stream identifier of the corresponding data stream to obtain ETL processed data. The adapter module is used to convert the data format of the ETL processed data to obtain a data stream after format conversion. It should be noted that each module can be a customizable / pluggable module. This embodiment includes the following technical advantages: (1) The corresponding data stream can be decompressed, deserialized, etc. through the parsing module. (2) Different data cleaning operations can be performed on different data streams through the ETL module. For example, the user can customize the data cleaning rules of the corresponding data stream, and send the metadata including the data cleaning rules to the ETL module through the Lance-Service via the broadcast module (ConfigBroadcast), so that the ETL module can perform the corresponding ETL operation on the corresponding data stream. (3) The adapter module can perform corresponding format conversion and category conversion on the data written to different downstream nodes. Figure 6 As shown, if the downstream node is an HDFS cluster, the adapter module needs to convert the data into Parquet format; if the downstream node is a Kafaka cluster, the adapter module needs to convert the data into Protubuf (Google Protocol Buffer, a sequence structured data format used by Google for data exchange) format; if the downstream node is an ES cluster, the adapter module needs to convert the data into Json (JavaScript Object Notation, JS object notation) format; if the downstream node is an Hbase cluster, the adapter module needs to convert the data into KeyValue (data stored in key values) format. It should be noted that the above examples are not intended to limit the scope of protection of this application.

[0078] (4) The Sink can provide a dynamic partitioning (Bucket) mechanism.

[0079] The Sink is used to dynamically allocate multiple partitions according to the event time (EventTime) and metadata information (Metadata) of multiple data streams to write the corresponding data to the corresponding directory in the downstream node. Figure 7As shown, in this embodiment, the partition assignment component (BucketAssigner) can send the corresponding LogId data received by the Sink to the corresponding directory of the downstream node.

[0080] Embodiment 2

[0081] Figure 8 The flowchart of the streaming data distribution method according to the second embodiment of the present application is schematically shown. The following is an exemplary description using a computer device at the data distribution layer as the execution subject. The computer device can be a server or a server cluster.

[0082] like Figure 8 As shown, the streaming data distribution method may include steps S800 to S804, wherein:

[0083] Step S800: Consume the data stream in the upstream message subscription system, and distribute the data in the data stream to the corresponding downstream nodes.

[0084] Step S802: monitoring the partition sensing message associated with the data flow.

[0085] Step S804: Send the partition sensing message to the downstream node so that the downstream node performs a corresponding data processing task.

[0086] like Fig. 9 As shown, the streaming data distribution method may also include steps S900 to S906, wherein: step S900, generating a watermark of the data stream; step S902, reporting the watermark to the task deployer so that the task deployer generates a speed control instruction according to the watermark; step S904, receiving the speed control instruction; step S906, controlling the consumption speed of the data stream according to the speed control instruction.

[0087] like Fig.10 As shown, the streaming data distribution method may also include steps S1000 to S1004, wherein: step S1000, performing a parsing operation on the data stream to obtain parsed data; step S1002, performing corresponding ETL processing on the parsed data according to the stream identifier of the data stream to obtain ETL processed data; step S1004, performing format conversion and / or type conversion on the ETL processed data to obtain converted data.

[0088] The computer device is a Flink cluster or is located in a Flink cluster.

[0089] like Fig.11As shown, step S804 may include steps S1100 to S1104, wherein: step S1100, obtaining all watermarks associated with the data stream in the Flink cluster and finding the minimum watermark; step S1102, generating the partition sensing message according to the minimum watermark; step S1104, sending the partition sensing message to the downstream node so that the downstream node performs the corresponding data processing task.

[0090] like Fig.12 As shown, the streaming data distribution method may further include step S1200, wherein step S1200 is: sending a cleanup instruction to the downstream node to notify the downstream node to clean up zombie files.

[0091] like Fig.13 As shown, the streaming data distribution method may also include step S1300, wherein step S1300 is: dynamically allocating to obtain multiple partitions according to event time, metadata information of the data stream and metadata information of other data streams, so as to write corresponding data into corresponding directories in the downstream nodes.

[0092] Embodiment 3

[0093] Fig.14 The block diagram of the streaming data distribution device according to the third embodiment of the present application is shown. The streaming data distribution device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.14 As shown, the streaming data distribution device 1400 may include the following components:

[0094] The consumption module 1410 consumes the data stream in the upstream message subscription system and distributes the data in the data stream to the corresponding downstream nodes.

[0095] The monitoring module 1420 monitors the partition sensing message associated with the data flow.

[0096] The sending module 1430 sends the partition sensing message to the downstream node so that the downstream node performs a corresponding data processing task.

[0097] Optionally, the streaming data distribution device 1400 also includes a control module, which is used to: generate a watermark for the data stream; report the watermark to the task deployer so that the task deployer generates a speed control instruction based on the watermark; receive the speed control instruction; and control the consumption speed of the data stream according to the speed control instruction.

[0098] Optionally, the streaming data distribution device 1400 further includes a data processing module. The data processing module is used to: perform a parsing operation on the data stream to obtain parsed data; perform corresponding ETL processing on the parsed data according to the stream identifier of the data stream to obtain ETL processed data; perform format conversion and / or type conversion on the ETL processed data to obtain converted data.

[0099] Optionally, the streaming data distribution device 1400 is or is located in a Flink cluster; the sending module is also used to: obtain all watermarks associated with the data stream in the Flink cluster and find the minimum watermark; generate the partition sensing message according to the minimum watermark; and send the partition sensing message to the downstream node so that the downstream node performs the corresponding data processing task.

[0100] Optionally, the sending module is further used to send a cleanup instruction to the downstream node to notify the downstream node to clean up zombie files.

[0101] Optionally, the streaming data distribution device 1400 further includes a dynamic allocation module. The dynamic allocation module is used to dynamically allocate to obtain multiple partitions according to event time, metadata information of the data stream and metadata information of other data streams, so as to write corresponding data into corresponding directories in the downstream nodes.

[0102] Embodiment 4

[0103] Fig.15 The schematic diagram of the hardware architecture of a computer device suitable for implementing the streaming data distribution method according to the fourth embodiment of the present application is shown. In this embodiment, the computer device. Moreover, the computer device 15 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a workstation, a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.15 As shown, the computer device 15 includes at least but is not limited to: a memory 1510, a processor 1520, and a network interface 1530 which can communicate with each other through a system bus. Among them:

[0104] The memory 1510 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 1510 can be an internal storage module of the computer device 15, such as a hard disk or memory of the computer device 15. In other embodiments, the memory 1510 can also be an external storage device of the computer device 15, such as a plug-in hard disk equipped on the computer device 15, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the memory 1510 can also include both the internal storage module of the computer device 15 and its external storage device. In this embodiment, the memory 1510 is generally used to store the operating system and various application software installed on the computer device 15, such as the program code of the streaming data distribution method, etc. In addition, the memory 1510 can also be used to temporarily store various types of data that have been output or are to be output.

[0105] In some embodiments, the processor 1520 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 1520 is generally used to control the overall operation of the computer device 15, such as performing control and processing related to data interaction or communication with the computer device 15. In this embodiment, the processor 1520 is used to run the program code stored in the memory 1510 or process data.

[0106] The network interface 1530 may include a wireless network interface or a wired network interface, and the network interface 1530 is generally used to establish a communication connection between the computer device 15 and other computer devices. For example, the network interface 1530 is used to connect the computer device 15 to an external terminal through a network, and to establish a data transmission channel and a communication connection between the computer device 15 and the external terminal. The network may be a wireless or wired network such as an Intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0107] It should be pointed out that Fig.15 Only a computer device having components 1510 - 1530 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0108] In this embodiment, the streaming data distribution method stored in the memory 1510 may also be divided into one or more program modules and executed by one or more processors (processor 1520 in this embodiment) to complete the present application.

[0109] Embodiment 5

[0110] This embodiment further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the streaming data distribution method in the embodiment are implemented.

[0111] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the streaming data distribution method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0112] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0113] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A streaming data distribution system, characterized in that: Includes one or more Flink clusters, each of which is used to: Consume the data stream in the upstream message subscription system and distribute the data in the data stream to the corresponding downstream nodes; monitoring partition sensing messages associated with the data stream; and Sending the partition sensing message to the downstream node so that the downstream node performs a corresponding data processing task; The monitoring of the partition sensing message associated with the data stream includes: generating a watermark of the data stream, obtaining the watermark of the data in each data output module, and finding the minimum watermark; and generating the partition sensing message according to the minimum watermark.

2. The streaming data distribution system according to claim 1, characterized in that: Each Flink cluster includes task executors and task deployers; The task executor includes a plurality of data input modules, a plurality of data processing modules, a plurality of data output modules, and a partition message submission module; Wherein, the partition message submission module is used to: Get the watermark of the data in each data output module and find the minimum watermark; Generating the partition sensing message according to the minimum watermark; and The partition sensing message is sent to the downstream node so that the downstream node performs a corresponding data processing task.

3. The streaming data distribution system according to claim 2, characterized in that: The partition message submission module is also used for: A cleanup instruction is issued to the downstream node to notify the downstream node to clean up zombie files.

4. The streaming data distribution system according to claim 2, characterized in that: Each data input module is used to: Consume the corresponding data stream; Reporting the watermark of the corresponding data stream to the task deployer so that the task deployer generates a speed control instruction; receiving the speed control instruction; and According to the speed control instruction, the consumption speed of the corresponding data stream is controlled to be consistent with the consumption speed of other data streams in other data input modules, so that the downstream node obtains all data within a time window each time.

5. The streaming data distribution system according to claim 2, characterized in that: Each data processing module includes a parsing module, an ETL module and an adaptation module, where: The parsing module is used to perform a parsing operation on the corresponding data stream to obtain parsed data; The ETL module is used to perform corresponding ETL processing on the parsed data according to the flow identifier of the corresponding data flow to obtain ETL-processed data; The adaptation module is used to perform format conversion and / or type conversion on the data processed by the ETL to obtain converted data.

6. The streaming data distribution system according to claim 2, characterized in that: Each data output module is used to: According to the event time, the metadata information of the data stream and the metadata information of other data streams, multiple partitions are dynamically allocated to write the corresponding data into the corresponding directories in the downstream nodes.

7. A streaming data distribution method, characterized in that: In a computer device used in a data distribution layer, the method comprises: Consume the data stream in the upstream message subscription system and distribute the data in the data stream to the corresponding downstream nodes; monitoring partition sensing messages associated with the data stream; and Sending the partition sensing message to the downstream node so that the downstream node performs a corresponding data processing task; The monitoring of the partition sensing message associated with the data stream includes: generating a watermark of the data stream, obtaining the watermark of the data in each data output module, and finding the minimum watermark; and generating the partition sensing message according to the minimum watermark.

8. The streaming data distribution method according to claim 7, characterized in that: Also includes: generating a watermark for the data stream; Reporting the watermark to a task deployer so that the task deployer generates a speed control instruction according to the watermark; receiving the speed control instruction; and According to the speed control instruction, the consumption speed of the data stream is controlled.

9. The streaming data distribution method according to claim 7, characterized in that: Also includes: Performing a parsing operation on the data stream to obtain parsed data; According to the flow identifier of the data flow, performing corresponding ETL processing on the parsed data to obtain ETL processed data; The data processed by the ETL is format converted and / or type converted to obtain converted data.

10. The streaming data distribution method according to claim 7, characterized in that: The computer device is or is located in a Flink cluster; Sending the partition sensing message to the downstream node so that the downstream node performs a corresponding data processing task, further comprising: Get all watermarks associated with the data stream in the Flink cluster and find the minimum watermark; Generating the partition sensing message according to the minimum watermark; and The partition sensing message is sent to the downstream node so that the downstream node performs a corresponding data processing task.

11. The streaming data distribution method according to claim 7, characterized in that: Also includes: A cleanup instruction is issued to the downstream node to notify the downstream node to clean up zombie files.

12. The streaming data distribution method according to claim 7, characterized in that: Also includes: According to the event time, the metadata information of the data stream and the metadata information of other data streams, multiple partitions are dynamically allocated to write the corresponding data into the corresponding directories in the downstream nodes.

13. A streaming data distribution device, characterized in that: The device comprises: A consumption module, used to consume the data stream in the upstream message subscription system and distribute the data in the data stream to the corresponding downstream nodes; A monitoring module, configured to monitor partition sensing messages associated with the data stream; and A sending module, used for sending the partition sensing message to the downstream node so that the downstream node performs a corresponding data processing task; The monitoring of the partition sensing message associated with the data stream includes: generating a watermark of the data stream, obtaining the watermark of the data in each data output module, and finding the minimum watermark; and generating the partition sensing message according to the minimum watermark.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it is used to implement the steps of the streaming data distribution method according to any one of claims 7 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it is used to implement the steps of the streaming data distribution method according to any one of claims 7 to 12.

Images