Flink-based data processing method, device and storage medium

Abstract

The application provides a Flink-based data processing method, device and storage medium. The method comprises the following steps: in response to a data calculation task, obtaining a target data stream, determining a plurality of to-be-processed data corresponding to the target data stream based on a preset mode; determining a time partition directory based on the to-be-processed data and an event time field value corresponding to the to-be-processed data, and determining a partition index corresponding to the to-be-processed data based on the time partition directory; writing the to-be-processed data into a corresponding time partition based on the partition index, and generating a result file corresponding to the to-be-processed data based on Flink. The method of the application achieves the technical effects of reducing the number of result files and improving the file reading and writing performance.

Description

Technical Field

[0001] This application relates to the Internet field, and in particular to a data processing method, device and storage medium based on Flink. Background Technology

[0002] With the development of internet technology, the amount of data in online information is constantly increasing, and the demand for data processing is constantly rising. Traditional data processing methods are batch processing, which has low applicability when dealing with data with high real-time requirements. Flink streaming data processing, which is designed for real-time data processing, has become a promising direction.

[0003] In existing technologies, Flink-based streaming data processing generates partition directories by partitioning the data and determines the corresponding partition directory based on the data to be processed. Result files are generated based on the data to be processed through parallel processing and written to the partition directory.

[0004] In existing technologies, the number of result files generated by streaming data processing is related to the parallelism of data processing, and the number of result files generated by streaming data processing is huge. A large number of indexes need to be generated in the file system, and the retrieval time when the file system performs read and write operations increases accordingly. Therefore, existing technologies have the technical problem of generating a large number of result files and low file read and write performance. Summary of the Invention

[0005] This application provides a data processing method, device, and storage medium based on Flink to solve the technical problems of large number of result files generated and low file read / write performance in the prior art.

[0006] Firstly, this application provides a data processing method based on Flink, including:

[0007] In response to a data computation task, the target data stream is acquired, and multiple data to be processed corresponding to the target data stream are determined based on a preset method.

[0008] Based on the data to be processed and the event time field value corresponding to the data to be processed, determine the time partition directory, and based on the time partition directory, determine the partition index corresponding to the data to be processed.

[0009] The data to be processed is written to the corresponding time partition based on the partition index, and the result file corresponding to the data to be processed is generated based on Flink.

[0010] Optionally, based on the data to be processed and the corresponding event time field value, a time partition directory is determined, and based on the time partition directory, the partition index corresponding to the data to be processed is determined, including:

[0011] At least one time partition directory is determined based on the amount of data to be processed; different time partition directories correspond to different partition indexes.

[0012] Based on the event time field value, determine the time partition directory corresponding to the data to be processed;

[0013] Based on the partition index corresponding to the time partition directory, determine the partition index corresponding to the data to be processed.

[0014] Optionally, based on the data to be processed and the corresponding event time field value, a time partition directory is determined, and based on the time partition directory, the partition index corresponding to the data to be processed is determined, including:

[0015] The first time partition directory and the second time partition directory are determined based on the amount of data to be processed; wherein, any partition index corresponding to the second time partition directory is greater than any partition index corresponding to the first time partition directory.

[0016] Determine if the event time field value is odd; if yes, determine if the time partition corresponding to the data to be processed belongs to the first time partition directory; otherwise, determine if the time partition corresponding to the data to be processed belongs to the second time partition directory.

[0017] Based on the partition index corresponding to the time partition directory, determine the partition index corresponding to the data to be processed.

[0018] Optionally, based on the partition index corresponding to the time partition directory, the partition index corresponding to the data to be processed is determined, including:

[0019] The partition index is determined based on the time-partition directory corresponding to the data to be processed, using a random pattern; or,

[0020] The partition index is determined based on the time partition directory corresponding to the data to be processed, using a polling model.

[0021] Optionally, the partition index is determined based on the time partition directory corresponding to the data to be processed, according to a random pattern, including:

[0022] Randomly select one partition index from the multiple partition indexes corresponding to the time partition directory as the partition index corresponding to the data to be processed.

[0023] Optionally, the partition index is determined based on the time partition directory corresponding to the data to be processed, using a polling pattern, including:

[0024] The multiple partition indexes corresponding to the time partition directory are sorted in ascending order, and each partition index is determined as the partition index corresponding to the data to be processed.

[0025] Optionally, the data to be processed is written to the corresponding time partition based on the partition index, and the result file corresponding to the data to be processed is generated based on Flink, including:

[0026] The time partition used for writing data to be processed is determined based on the partition index;

[0027] Execute the subtask corresponding to the time partition and write the data to be processed into the time partition;

[0028] Based on Flink, the result file is generated in the time partition directory corresponding to the time partition, and the data to be processed is written to the result file.

[0029] Secondly, this application provides a Flink-based data processing device, comprising:

[0030] The processing module is used to respond to data computing tasks, acquire the target data stream, and determine multiple data to be processed corresponding to the target data stream based on a preset method.

[0031] The determination module is used to determine the time partition directory based on the data to be processed and the event time field value corresponding to the data to be processed, and to determine the partition index corresponding to the data to be processed based on the time partition directory.

[0032] The generation module is used to write the data to be processed into the corresponding time partition based on the partition index, and generate the result file corresponding to the data to be processed based on Flink.

[0033] Optionally, the determining module is also used for:

[0034] At least one time partition directory is determined based on the amount of data to be processed; different time partition directories correspond to different partition indexes.

[0035] Based on the event time field value, determine the time partition directory corresponding to the data to be processed;

[0036] Based on the partition index corresponding to the time partition directory, determine the partition index corresponding to the data to be processed.

[0037] Optionally, the determining module is also used for:

[0038] The first time partition directory and the second time partition directory are determined based on the amount of data to be processed; wherein, any partition index corresponding to the second time partition directory is greater than any partition index corresponding to the first time partition directory.

[0039] Determine if the event time field value is odd; if yes, determine if the time partition corresponding to the data to be processed belongs to the first time partition directory; otherwise, determine if the time partition corresponding to the data to be processed belongs to the second time partition directory.

[0040] Based on the partition index corresponding to the time partition directory, determine the partition index corresponding to the data to be processed.

[0041] Optionally, the determining module is also used for:

[0042] The partition index is determined based on the time-partition directory corresponding to the data to be processed, using a random pattern; or,

[0043] The partition index is determined based on the time partition directory corresponding to the data to be processed, using a polling model.

[0044] Optionally, the determining module is also used for:

[0045] Randomly select one partition index from the multiple partition indexes corresponding to the time partition directory as the partition index corresponding to the data to be processed.

[0046] Optionally, the partition index is determined based on the time partition directory corresponding to the data to be processed, using a polling pattern, including:

[0047] The multiple partition indexes corresponding to the time partition directory are sorted in ascending order, and each partition index is determined as the partition index corresponding to the data to be processed.

[0048] Optionally, the generation module is also used for:

[0049] The time partition used for writing data to be processed is determined based on the partition index;

[0050] Execute the subtask corresponding to the time partition and write the data to be processed into the time partition;

[0051] Based on Flink, the result file is generated in the time partition directory corresponding to the time partition, and the data to be processed is written to the result file.

[0052] Thirdly, this application provides a Flink-based data processing device, including a processor and a memory;

[0053] The memory stores the instructions that the computer executes;

[0054] The processor executes computer execution instructions stored in memory, causing the electronic device to perform any of the Flink-based data processing methods in the first aspect.

[0055] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a Flink-based data processing method as described in any of the first aspects.

[0056] This application provides a data processing method, device, and storage medium based on Flink. The method includes: in response to a data computation task, acquiring a target data stream; determining multiple data items to be processed corresponding to the target data stream based on a preset method; determining a time partition directory based on the data items to be processed and the corresponding event time field values; determining a partition index corresponding to the data items to be processed based on the time partition directory; writing the data items to be processed into the corresponding time partitions based on the partition indexes; and generating result files corresponding to the data items to be processed based on Flink. This achieves the technical effect of reducing the number of result files generated while maintaining data processing parallelism and improving file read / write performance without increasing data latency. This is achieved by acquiring data items to be processed based on a target data stream, determining the corresponding event time field values ​​based on the data items to be processed, determining the corresponding time partition directory based on the event time field values; determining the partition index corresponding to the data items to be processed based on the time partition directory; determining the partition index of the data items to be processed based on the event time field values; and determining the partition index of the data items to be processed based on the event time field values. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0058] Figure 1 The data processing method flow based on Flink provided in the embodiments of this application Figure 1 ;

[0059] Figure 2 The flow of the Flink-based data processing method provided in the embodiments of this application Figure 2 ;

[0060] Figure 3 The flow of the Flink-based data processing method provided in the embodiments of this application Figure 3 ;

[0061] Figure 4 The flow of the Flink-based data processing method provided in the embodiments of this application Figure 4 ;

[0062] Figure 5 A schematic diagram of the structure of a Flink-based data processing device provided in this application embodiment;

[0063] Figure 6 This is a hardware structure diagram of a Flink-based data processing device provided in an embodiment of this application.

[0064] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0065] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0066] First, let me explain the terms used in this application:

[0067] Flink refers to a distributed, open-source platform that integrates stream and batch processing. At its core, Flink is a stream processing engine that provides data distribution, communication, and automatic fault tolerance. Flink builds batch processing on top of stream processing and natively supports iterative computation, memory management, and program optimization.

[0068] The Hadoop Distributed File System (HDFS) of the Hadoop framework is a distributed file storage system designed for accessing large files in a streaming manner.

[0069] Kafka refers to a distributed messaging system with multiple partitions and replicas. It is also a distributed stream processing platform, widely used for its high throughput, persistence, horizontal scalability, and support for streaming data processing.

[0070] Netcat, abbreviated as NC, is primarily used in debugging, data transmission, and even hacking. This tool allows data from one end of a network to be completely sent to another host for display or storage. Common applications include file transfer, instant messaging with friends, streaming media transmission, or as a standalone client for server verification.

[0071] Sink: Used to connect to external systems and submit data for writing; all output operations in Flink programs are performed using sinks.

[0072] Time partition granularity: refers to the minimum time interval for generating time partition directories.

[0073] Checkpoints are the core functionality of Flink's fault tolerance mechanism. They periodically generate snapshots based on the state of each Operator in the Stream according to the configuration, thus persistently storing this state data. When a Flink program crashes unexpectedly, it can selectively recover from these snapshots when rerunning the program, thereby correcting the interruption of program data state caused by the failure. Stream is the core data structure of Flink, and Operator is the operator.

[0074] In existing technologies, Flink-based streaming data processing generates partition directories by partitioning the data and determining the corresponding partition directory based on the data to be processed. Result files are then generated based on the data in parallel processing and written to the partition directories. However, existing technologies suffer from several drawbacks. Firstly, the number of result files generated is highly dependent on the parallelism of the data processing. Secondly, the sheer volume of result files requires the creation of numerous indexes in the file system, increasing retrieval time during read / write operations. Therefore, existing technologies suffer from the problem of generating a large number of result files while exhibiting low file read / write performance.

[0075] This application provides a data processing method, device, and storage medium based on Flink. The method includes: in response to a data computation task, acquiring a target data stream; determining multiple data points to be processed corresponding to the target data stream based on a preset method; determining a time partition directory based on the data to be processed and the corresponding event time field value; determining the partition index corresponding to the data to be processed based on the time partition directory; writing the data to be processed to the corresponding time partition based on the partition index; and generating result files corresponding to the data to be processed based on Flink. This achieves the technical effect of reducing the number of result files generated while maintaining data processing parallelism and improving file read / write performance without increasing data latency. This is achieved by acquiring data to be processed based on a target data stream, determining the corresponding event time field value based on the data to be processed, determining the corresponding time partition directory based on the event time field value; determining the partition index corresponding to the data to be processed based on the time partition directory; determining the partition index of the data to be processed based on the event time field value; and determining the partition index of the data to be processed based on the event time field value.

[0076] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0077] Figure 1 The data processing method flow based on Flink provided in the embodiments of this application Figure 1 .like Figure 1 As shown, the method includes:

[0078] S101. In response to the data computing task, acquire the target data stream and determine multiple data to be processed corresponding to the target data stream based on a preset method;

[0079] In this embodiment, the preset method includes: preset time interval and window; in the first example, the preset time interval is a checkpoint, and the value of the checkpoint is set to 30 minutes; that is, the data to be processed corresponding to the target data stream is determined based on the 30-minute time interval.

[0080] S102. Based on the data to be processed and the event time field value corresponding to the data to be processed, determine the time partition directory, and based on the time partition directory, determine the partition index corresponding to the data to be processed.

[0081] In this embodiment, in the second example, the time partition granularity corresponding to the time partition directory is one day, and the event time field value corresponding to the data to be processed is 2019-09-29 23:21:13. Therefore, the time partition corresponding to the data to be processed should be 2019 / 09 / 29.

[0082] S103. Write the data to be processed into the corresponding time partition based on the partition index, and generate the result file corresponding to the data to be processed based on Flink.

[0083] This application provides a data processing method based on Flink, which includes: in response to a data computation task, acquiring a target data stream, determining multiple data to be processed corresponding to the target data stream based on a preset method; determining a time partition directory based on the data to be processed and the event time field value corresponding to the data to be processed, determining the partition index corresponding to the data to be processed based on the time partition directory; writing the data to be processed into the corresponding time partition based on the partition index, and generating a result file corresponding to the data to be processed based on Flink. This method acquires the data to be processed based on the target data stream, determines the event time field value corresponding to the data to be processed, and determines the time partition directory corresponding to the data to be processed based on the event time field value. The partition index corresponding to the data to be processed can be determined based on the time partition directory, and the partition index of the data flow direction can be determined based on the event time field value of the data to be processed. The data to be processed is then written to the time partition corresponding to the partition index. By using the event time field value of the data to be processed to determine the partition index of the data flow direction, the number of result files generated for the data to be processed can be controlled. This reduces the number of files generated for data processing without increasing data latency, while maintaining data processing parallelism, thus solving the technical problems of large number of result files and low file read / write performance in existing technologies.

[0084] Figure 2 The flow of the Flink-based data processing method provided in the embodiments of this application Figure 2 .like Figure 2 As shown, the method includes:

[0085] S201. In response to the data computation task, acquire the target data stream and determine multiple data to be processed corresponding to the target data stream based on a preset method;

[0086] In this embodiment, the target data stream can be obtained by: reading data from a local file, obtaining data from HDFS, using Kafka as a data source, or using NC in Linux to generate data as a data source.

[0087] S202. Determine the first time partition directory and the second time partition directory based on the amount of data to be processed; wherein, any partition index corresponding to the second time partition directory is greater than any partition index corresponding to the first time partition directory.

[0088] In this embodiment, the total number of partition indexes corresponding to the first time partition directory and the second time partition directory corresponds to the number of subtasks in the data processing operator and is also the parallelism of the operator processing; in the third example, the parallelism of the operator is 200, so the partition index corresponding to the first time partition directory is in the range of 0 to 99, and the partition index corresponding to the second time partition directory is in the range of 100 to 199.

[0089] S203. Determine if the event time field value is odd; if yes, determine if the time partition corresponding to the data to be processed belongs to the first time partition directory; if no, determine if the time partition corresponding to the data to be processed belongs to the second time partition directory.

[0090] In this embodiment, in the fourth example, the event time field value corresponding to the data to be processed is 2019-12-1200:00:02, the time partition granularity of the time partition directory is 1 hour, and the time field value corresponding to the data to be processed is not odd, then it is determined that the time partition corresponding to the data to be processed belongs to the second time partition directory.

[0091] S204. Randomly select one partition index from the multiple partition indexes corresponding to the time partition directory as the partition index corresponding to the data to be processed.

[0092] In this embodiment, in the fifth example, there are 100 partition indices corresponding to the time partition directory, and the range of the partition indices is 0 to 99. Then, based on the random number generation function, a random number in the range of 0 to 99 is generated, and the generated random number is used as the value of the partition index corresponding to the data to be processed.

[0093] S205. Determine the time partition for writing the data to be processed based on the partition index; execute the subtask corresponding to the time partition to write the data to be processed to the time partition; generate the result file in the time partition directory corresponding to the time partition based on Flink, and write the data to be processed to the result file.

[0094] By executing S201 to S205, the partition index corresponding to the data to be processed can be determined based on the parity of the event time field value corresponding to the data to be processed, thereby achieving the technical effect of reducing the number of result files generated in the time partition directory.

[0095] Figure 3 The flow of the Flink-based data processing method provided in the embodiments of this application Figure 3 .like Figure 3 As shown, the method includes:

[0096] S301. In response to the data computing task, acquire the target data stream and determine multiple data to be processed corresponding to the target data stream based on a preset method;

[0097] S302. Determine the first time partition directory and the second time partition directory based on the amount of data to be processed; wherein, any partition index corresponding to the second time partition directory is greater than any partition index corresponding to the first time partition directory.

[0098] S303. Determine if the event time field value is odd; if yes, determine that the time partition corresponding to the data to be processed belongs to the first time partition directory; if no, determine that the time partition corresponding to the data to be processed belongs to the second time partition directory.

[0099] S304. Sort the multiple partition indexes corresponding to the time partition directory in ascending order, and determine the partition indexes as the partition indexes corresponding to the data to be processed one by one.

[0100] In this embodiment, in the sixth example, the range of the partition index is 0 to 199. The partition index and the data to be processed are matched one by one based on the order from smallest to largest. The event time field value corresponding to the first data to be processed is the first time value, and the partition index corresponding to the first data to be processed is 0. The event time field value corresponding to the second data to be processed is the second time value, which is after the first time value, and the partition index corresponding to the second data to be processed is 1.

[0101] In the seventh example, the total number of partition indexes is represented by numPartitions. The partition index range corresponding to the first time partition directory is 0 to numPartitions / 2-1, and the partition index range corresponding to the second time partition directory is numPartitions / 2 to numPartitions-1. The method for determining the partition index corresponding to the data to be processed can be as follows: use last_index to record the index value allocated to the data to be processed during the last allocation. When new data to be processed arrives, if the event time field value corresponding to the data to be processed is odd, then return (last_index+1)%(numPartitions / 2) as the partition index of the data to be processed; if the event time field value corresponding to the data to be processed is even, then return numPartitions-1–index as the partition index of the data to be processed. Thus, the partition index of the data to be processed with odd event time field values ​​is determined to be a partition index in the range of 0 to numPartitions / 2-1, and the partition index of the data to be processed with even event time field values ​​is determined to be a partition index in the range of numPartitions / 2 to numPartitions-1.

[0102] S305. Determine the time partition for writing the data to be processed based on the partition index; execute the subtask corresponding to the time partition to write the data to be processed to the time partition; generate the result file in the time partition directory corresponding to the time partition based on Flink, and write the data to be processed to the result file.

[0103] In this embodiment, the subtask corresponding to the time partition is executed, a partition file is generated, the data to be processed is written to the partition file, the time partition directory corresponding to the time partition is obtained, the time partition file containing the data to be processed is written to the time partition directory, and a result file is generated.

[0104] By executing S301 to S305, the parity of the event time field value corresponding to the data to be processed can be determined based on the data to be processed, and the partition index corresponding to the data to be processed can be determined by polling. This halves the number of result files generated in the time partition directory, thereby achieving the technical effect of reducing the number of result files generated.

[0105] Figure 4 The flow of the Flink-based data processing method provided in the embodiments of this application Figure 4 .like Figure 4 As shown, in this embodiment, the time partition granularity is 1 hour, meaning one time partition directory corresponds to each hour; the preset method is a time interval of 30 minutes, meaning the current data processing task is archived every 30 minutes; the parallelism of the operator performing the calculation is set to 500, meaning the total number of partition indices in the time partition directory corresponding to the data to be processed is 500, and the partition index corresponds to a time partition under the time partition directory, so the total number of time partitions under the time partition directory is 500; the specific implementation of this embodiment includes: acquiring the target data stream, determining multiple data to be processed corresponding to the target data stream based on the 30-minute time interval; acquiring the event time field value corresponding to the data to be processed based on the data to be processed, i.e., extracting the event time field value from the data to be processed; determining the time partition directory based on the event time field value, determining the first time partition directory and the second time partition directory based on the data to be processed, and determining the partition index; judging the event time field value; determining the time partition directory based on the event time field value; determining the first time partition directory and the second time partition directory based on the data to be processed, and determining the partition index; and judging the event time field value. If the time field value is odd, the data to be processed corresponds to the first time partition directory; otherwise, it corresponds to the second time partition directory. The partition index corresponding to the data to be processed is selected from multiple partition indexes corresponding to the time partition directories. The partition index is allocated by setting a random number or by round-robin. The result file of the data to be processed is generated based on the subtask corresponding to the partition index. In the specific implementation, the operator is represented by `sink`, the parallelism of the operator is represented by `sink.parallelism`, the total number of partition indexes is represented by `numPartitions`, the partition index range corresponding to the first time partition directory is 0 to `numPartitions / 2-1`, and the partition index range corresponding to the second time partition directory is `numPartitions / 2` to `numPartitions-1`.

[0106] This application provides a Flink-based data processing method, which includes: in response to a data computation task, acquiring a target data stream; determining multiple data to be processed corresponding to the target data stream based on a preset method; determining a time partition directory corresponding to the data to be processed, including a first time partition directory and a second time partition directory, based on the data to be processed and the event time field value corresponding to the data to be processed; wherein the partition index corresponding to the first time partition directory is always less than the partition index corresponding to the second time partition directory; determining the partition index corresponding to the data to be processed based on the time partition directory; determining whether the data to be processed corresponds to the first time partition directory or the second time partition directory by judging the parity of the event time field value, and determining the partition index corresponding to the data to be processed; writing the data to be processed into the corresponding time partition based on the partition index; and generating a result file corresponding to the data to be processed based on Flink. This method achieves the following: acquiring data to be processed based on the target data stream; determining the corresponding event time field value based on the data to be processed; determining the corresponding time partition directory based on the event time field value; and determining the parity of the event time field value. Data with odd-numbered event time field values ​​is assigned to the partition index in the first time partition directory, while data with even-numbered event time field values ​​is assigned to the partition index in the second time partition directory. The partition index is assigned by matching it with the data to be processed through random or round-robin methods. After the data to be processed obtains the corresponding partition index, it is processed through the subtask corresponding to the partition index, generating a result file in the time partition corresponding to the subtask, and writing the result file to the corresponding time partition directory. This reduces the number of result files generated in the time partition directory without changing the parallelism of data processing, solving the technical problems of large number of result files and low file read / write performance in existing technologies.

[0107] Figure 5 This is a schematic diagram of a Flink-based data processing device provided in an embodiment of this application. The device in this embodiment can be in the form of software and / or hardware. Figure 5 As shown in the figure, this application provides a Flink-based data processing device 500, which includes a processing module 501, a determination module 502, and a generation module 503.

[0108] Processing module 501 is used to respond to data computing tasks, acquire target data stream, and determine multiple data to be processed corresponding to target data stream based on a preset method;

[0109] The determination module 502 is used to determine the time partition directory based on the data to be processed and the event time field value corresponding to the data to be processed, and to determine the partition index corresponding to the data to be processed based on the time partition directory.

[0110] The generation module 503 is used to write the data to be processed into the corresponding time partition based on the partition index, and generate the result file corresponding to the data to be processed based on Flink.

[0111] In one possible implementation, the determining module 502 is further used for:

[0112] At least one time partition directory is determined based on the amount of data to be processed; different time partition directories correspond to different partition indexes.

[0113] Based on the event time field value, determine the time partition directory corresponding to the data to be processed;

[0114] Based on the partition index corresponding to the time partition directory, determine the partition index corresponding to the data to be processed.

[0115] In one possible implementation, the determining module 502 is further used for:

[0116] The first time partition directory and the second time partition directory are determined based on the amount of data to be processed; wherein, any partition index corresponding to the second time partition directory is greater than any partition index corresponding to the first time partition directory.

[0117] Determine if the event time field value is odd; if yes, determine if the time partition corresponding to the data to be processed belongs to the first time partition directory; otherwise, determine if the time partition corresponding to the data to be processed belongs to the second time partition directory.

[0118] Based on the partition index corresponding to the time partition directory, determine the partition index corresponding to the data to be processed.

[0119] In one possible implementation, the determining module 502 is further used for:

[0120] The partition index is determined based on the time-partition directory corresponding to the data to be processed, using a random pattern; or,

[0121] The partition index is determined based on the time partition directory corresponding to the data to be processed, using a polling model.

[0122] In one possible implementation, the determining module 502 is further used for:

[0123] Randomly select one partition index from the multiple partition indexes corresponding to the time partition directory as the partition index corresponding to the data to be processed.

[0124] One possible implementation involves determining the partition index based on the time-partitioned directory corresponding to the data to be processed, using a polling pattern.

[0125] The multiple partition indexes corresponding to the time partition directory are sorted in ascending order, and each partition index is determined as the partition index corresponding to the data to be processed.

[0126] In one possible implementation, the generation module 503 is further used for:

[0127] The time partition used for writing data to be processed is determined based on the partition index;

[0128] Execute the subtask corresponding to the time partition and write the data to be processed into the time partition;

[0129] Based on Flink, the result file is generated in the time partition directory corresponding to the time partition, and the data to be processed is written to the result file.

[0130] This application provides a Flink-based data processing device, comprising: a processing module, used to respond to a data computation task, acquire a target data stream, and determine multiple data to be processed corresponding to the target data stream based on a preset method; a determination module, used to determine the time partition directory corresponding to the data to be processed based on the data to be processed and the event time field value corresponding to the data to be processed, including a first time partition directory and a second time partition directory; wherein the partition index corresponding to the first time partition directory is always less than the partition index corresponding to the second time partition directory; determining the partition index corresponding to the data to be processed based on the time partition directory, and by judging the parity of the event time field value, determining whether the data to be processed corresponds to the first time partition directory or the second time partition directory, and determining the partition index corresponding to the data to be processed; and a generation module, used to write the data to be processed into the corresponding time partition based on the partition index, and generate the result file corresponding to the data to be processed based on Flink. This method achieves the following: acquiring data to be processed based on the target data stream; determining the corresponding event time field value based on the data to be processed; determining the corresponding time partition directory based on the event time field value; and determining the parity of the event time field value. Data with odd-numbered event time field values ​​is assigned to the partition index in the first time partition directory, while data with even-numbered event time field values ​​is assigned to the partition index in the second time partition directory. The partition index is assigned by matching it with the data to be processed through random or round-robin methods. After the data to be processed obtains the corresponding partition index, it is processed through the subtask corresponding to the partition index, generating a result file in the time partition corresponding to the subtask, and writing the result file to the corresponding time partition directory. This reduces the number of result files generated in the time partition directory without changing the parallelism of data processing, solving the technical problems of large number of result files and low file read / write performance in existing technologies.

[0131] Figure 6 This is a hardware structure diagram of a Flink-based data processing device provided in an embodiment of this application. Figure 6 As shown, the Flink-based data processing device 600 includes:

[0132] Processor 601 and memory 602;

[0133] The memory stores the instructions that the computer executes;

[0134] The processor executes computer execution instructions stored in memory 602, causing the Flink-based data processing device to perform the Flink-based data processing method described above.

[0135] It should be understood that the processor 601 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The memory 602 may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0136] This application also provides a computer-readable storage medium storing computer-executable instructions, which are used to implement a Flink-based data processing method when executed by a processor.

[0137] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0138] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0139] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0140] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0141] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0142] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0143] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0144] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0145] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method based on Flink, characterized in that, include: In response to a data computation task, a target data stream is acquired, and multiple data to be processed corresponding to the target data stream are determined based on a preset method. Based on the data to be processed and the event time field value corresponding to the data to be processed, a time partition directory is determined, and based on the time partition directory, a partition index corresponding to the data to be processed is determined. Based on the partition index, the data to be processed is written into the corresponding time partition, and the result file corresponding to the data to be processed is generated based on Flink. The step of determining a time partition directory based on the data to be processed and the event time field value corresponding to the data to be processed, and determining the partition index corresponding to the data to be processed based on the time partition directory, includes: A first time partition directory and a second time partition directory are determined based on the amount of data to be processed; wherein, any partition index corresponding to the second time partition directory is greater than any partition index corresponding to the first time partition directory. Determine whether the event time field value is odd; if yes, determine that the time partition corresponding to the data to be processed belongs to the first time partition directory; if no, determine that the time partition corresponding to the data to be processed belongs to the second time partition directory. Determining the partition index corresponding to the data to be processed based on the partition index corresponding to the time partition directory includes: determining the partition index based on the time partition directory corresponding to the data to be processed using a random mode; or, determining the partition index based on the time partition directory corresponding to the data to be processed using a polling mode.

2. The method according to claim 1, characterized in that, The step of determining a time partition directory based on the data to be processed and the event time field value corresponding to the data to be processed, and determining the partition index corresponding to the data to be processed based on the time partition directory, includes: At least one time partition directory is determined based on the amount of data to be processed; wherein different time partition directories correspond to different partition indexes; Based on the event time field value, determine the time partition directory corresponding to the data to be processed; Based on the partition index corresponding to the time partition directory, determine the partition index corresponding to the data to be processed.

3. The method according to claim 1, characterized in that, The step of determining the partition index based on the time partition directory corresponding to the data to be processed using a random pattern includes: One partition index is randomly selected from the multiple partition indexes corresponding to the time partition directory as the partition index corresponding to the data to be processed.

4. The method according to claim 1, characterized in that, The step of determining the partition index based on the polling mode according to the time partition directory corresponding to the data to be processed includes: The multiple partition indexes corresponding to the time partition directory are sorted in ascending order, and each partition index is determined as the partition index corresponding to the data to be processed.

5. The method according to claim 1, characterized in that, The process of writing the data to be processed into the corresponding time partition based on the partition index, and generating the result file corresponding to the data to be processed based on Flink, includes: The time partition used for writing the data to be processed is determined based on the partition index; Execute the subtask corresponding to the time partition and write the data to be processed into the time partition; Flink generates the result file in the time partition directory corresponding to the time partition and writes the data to be processed into the result file.

6. A data processing device based on Flink, characterized in that, include: The processing module is used to respond to a data computing task, acquire a target data stream, and determine multiple data to be processed corresponding to the target data stream based on a preset method. The determining module is configured to determine a time partition directory based on the data to be processed and the event time field value corresponding to the data to be processed, and to determine the partition index corresponding to the data to be processed based on the time partition directory, including: determining a first time partition directory and a second time partition directory based on the quantity of the data to be processed; wherein, any partition index corresponding to the second time partition directory is greater than any partition index corresponding to the first time partition directory; determining whether the event time field value is odd; if yes, determining that the time partition corresponding to the data to be processed belongs to the first time partition directory; if no, determining that the time partition corresponding to the data to be processed belongs to the second time partition directory; The generation module is used to write the data to be processed into the corresponding time partition based on the partition index, and generate the result file corresponding to the data to be processed based on Flink. The determining module is further configured to determine the partition index corresponding to the data to be processed based on the partition index corresponding to the time partition directory, including: determining the partition index based on the time partition directory corresponding to the data to be processed in a random mode; or, determining the partition index based on the time partition directory corresponding to the data to be processed in a polling mode.

7. A data processing device based on Flink, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the Flink-based data processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the Flink-based data processing method as described in any one of claims 1 to 5.

Images