Method, apparatus and non-transitory storage medium for multi-data stream parallel processing

Abstract

This application discloses a method, apparatus, and non-volatile storage medium for parallel processing of multiple data streams. The method includes: determining task configuration information for a target task stream, wherein the target task stream includes at least one data stream, and the task configuration information includes the association method between the data streams in the target task stream, data characteristic information of each data stream in the target task stream, and the data table type of the data streams; in the pre-write batching stage of the target task stream, performing a first type of operation on the data streams in the target task stream according to the task configuration information, wherein the first type of operation includes at least one of the following: data deduplication, data association; and in the data write merging stage of the target task stream, performing persistent disk write processing on the data streams in the target task stream according to the task configuration information. This application solves the technical problem in related technologies where real-time data stream association at the computation layer during data stream association results in extremely large cached data for computational tasks.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a method, apparatus, and non-volatile storage medium for parallel processing of multiple data streams. Background Technology

[0002] In related technologies, when associating data streams, real-time data streams are usually associated at the computing layer. This association requires the introduction of an external storage system, which leads to high storage and network transmission pressure, easily creating performance bottlenecks, and the amount of cached data for computing tasks is extremely large.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, and non-volatile storage medium for parallel processing of multiple data streams, which at least solves the technical problem in related technologies where real-time data stream association at the computing layer causes extremely large cached data for computing tasks when associating data streams.

[0005] According to one aspect of the embodiments of this application, a method for parallel processing of multiple data streams is provided, comprising: determining task configuration information of a target task stream, wherein the target task stream includes at least one data stream, and the task configuration information includes the association method between the data streams in the target task stream, the data characteristic information of each data stream in the target task stream, and the data table type of the data stream; in the pre-write batching stage of the target task stream, performing a first type of operation on the data streams in the target task stream according to the task configuration information, wherein the first type of operation includes at least one of the following: data deduplication, data association; and in the data write merging stage of the target task stream, performing persistent disk write processing on the data streams in the target task stream according to the task configuration information.

[0006] Optionally, the association method includes one of the following: normal write mode, wide table join mode, and data stream JOIN mode.

[0007] Optionally, when the association method is a wide table join mode, the steps of performing the first type of operation on the data stream in the target task flow based on the task configuration information include: determining whether there is target data with the same key value in the target task flow; if it is determined that there is target data with the same key value, determining whether the target data with the same key value belongs to the same data stream; if the target data with the same key value belongs to the same data stream, updating the data stream to which the target data belongs; if the target data with the same key value does not belong to the same data stream, performing data concatenation processing on the data stream to which the target data belongs.

[0008] Optionally, when the association method is data flow JOIN mode, the steps of performing the first type of operation on the data flow in the target task flow according to the task configuration information include: determining whether there is target data with the same key value in the target task flow; if it is determined that there is target data with the same key value, determining whether the target data with the same key value belongs to the same data table; if it is determined that the target data corresponds to the same data table, retaining the target data with a later data generation time; if it is determined that the data does not correspond to the same data table, performing data flow JOIN processing on the data flow corresponding to the data.

[0009] Optionally, the steps for performing data stream JOIN processing on the corresponding data stream include: determining the JOIN order between data streams and associating the data streams according to the JOIN order; when associating the data streams again according to the JOIN order, the association method is Inner.Join or Right.Join, and if there is a JOIN failure, returning and marking the target data as hidden data; when associating the data streams again according to the JOIN order, if the association method is Left.Join, returning the target data and adding a hidden mark to the data in the data stream that did not participate in the association.

[0010] Optionally, when the association method is data flow JOIN mode, the steps of persisting the data flow in the target task flow to disk based on the task configuration information include: determining the write merging logic corresponding to the table type information of the data flow, wherein the write merging logic includes the write merging method for the first type of data flow and the second type of data flow, the first type of data flow being the data flow with hidden markers in the corresponding data, and the second type of data flow being the data flow without hidden markers in the corresponding data; and persisting the data flow to disk based on the write merging logic corresponding to the data flow.

[0011] Optionally, when the association method is a wide table join mode, the steps for persisting the data stream in the target task flow to disk based on the task configuration information include: determining the target data with the same key value; determining the generation time of the target data and determining the target column where the latest generated target data is located; using the target column to overwrite the historical data, and persisting the data associated with the data stream to disk after overwriting.

[0012] Optionally, the task configuration information may also include parallelism; the steps for determining the configuration information of the target task flow include: obtaining a preset parallelism parameter; or, determining the number of data sources associated with the target task flow; and configuring the parallelism parameter of the target task flow according to the number of data sources.

[0013] Optionally, after the step of persisting the data stream in the target task flow to disk based on the task configuration information, the multi-data-stream parallel processing method further includes: determining the query type of the data query instruction and the table type of the data table corresponding to the data query instruction; and processing the data in the data table according to the query type and table type.

[0014] According to another aspect of the embodiments of this application, a multi-data-stream parallel processing apparatus is also provided, comprising: a first processing module, configured to determine task configuration information of a target task stream, wherein the target task stream includes at least one data stream, and the task configuration information includes the association method between the data streams in the target task stream, the data characteristic information of each data stream in the target task stream, and the data table type of the data stream; a second processing module, configured to perform a first type of operation on the data streams in the target task stream according to the task configuration information during the pre-write batching stage of the target task stream, wherein the first type of operation includes at least one of the following: data deduplication, data association; and a third processing module, configured to perform persistent disk write processing on the data streams in the target task stream according to the task configuration information during the data write merging stage of the target task stream.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, and the program controls the device where the non-volatile storage medium is located to execute a multi-data stream parallel processing method when it runs.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes a multi-data-stream parallel processing method during runtime.

[0017] In this embodiment, task configuration information for a target task flow is determined. The target task flow includes at least one data flow. The task configuration information includes the association method between the data flows in the target task flow, the data characteristic information of each data flow in the target task flow, and the data table type of the data flow. In the pre-write batching stage of the target task flow, a first type of operation is performed on the data flows in the target task flow according to the task configuration information. The first type of operation includes at least one of the following: data deduplication and data association. In the data write merging stage of the target task flow, the data flows in the target task flow are persistently written to disk according to the task configuration information. By associating the data flows in the pre-write batching stage, the association logic of the data flows is pushed down to the storage layer of the data lake. This achieves the technical effect of not having to execute all data association tasks in the computing layer, thereby solving the technical problem of extremely large data cache size caused by real-time data flow association in the computing layer when associating data flows in related technologies. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a schematic diagram of the structure of a computer device (mobile device) according to an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating a multi-data-stream parallel processing method provided according to an embodiment of this application;

[0021] Figure 3 This is a flowchart illustrating a multi-data stream processing procedure according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of a multi-data-stream parallel processing device provided according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0026] Hudi: In this application, it refers to a data lake.

[0027] Payload: Hudi Payload is a scalable data processing mechanism. Different payloads allow for customized data writing methods for complex scenarios, greatly increasing the flexibility of data processing. Payload is a utility class for performing operations such as deduplication, filtering, and merging of data when writing to and reading from Hudi tables.

[0028] State: A data structure stored in the state backend (state storage module, used to store the state of various objects) to meet the need for historical data during operator computation and to ensure fault tolerance using a checkpoint mechanism. State is used to store intermediate results or metadata of nodes in the computation process.

[0029] Compaction: Reconciles operations on differential data structures in Hudi, changing updates from row-based log files to column-based files.

[0030] Base file: In Hudi, the base file is stored as a columnar file.

[0031] Log file: In Hudi, incremental files such as log files are stored as line files.

[0032] Currently, thanks to the development of real-time computing frameworks such as Flink and Spark Streaming, and the rise of technologies like Kafka and MPP, the field of real-time computing has become increasingly sophisticated. With the promotion of technologies such as the Internet of Things and machine learning, real-time stream computing has been widely applied in areas such as intelligent recommendation, real-time fraud detection, complex event processing, and real-time machine learning. With the support of various application scenarios, traditional single data streams for statistical aggregation calculations can no longer meet the needs of complex business scenario indicator calculations; therefore, the problem of real-time data stream correlation and connectivity urgently needs to be solved.

[0033] In related technologies, common data stream JOIN connection schemes for associating data streams include: 1. Multi-task parallel processing: The data streams to be associated are written to an external storage system, and the critical data streams are queried from the external data storage points to complete the data association. 2. Using the Flink streaming computing framework, using Interval Join, Window Join, or caching the associated data using its own state. 3. Columnar storage: Each streaming task corresponds to a single real-time data source, and only certain columns related to that single data source are updated.

[0034] However, all of the above solutions have some drawbacks. For method 1, the problem is that it relies on external storage, increasing the operational burden; the large data volume puts pressure on storage and network I / O, easily creating a performance bottleneck. For method 2, Flink Interval Join or Window Join, data needs to be cached in memory, placing significant pressure on the task; and the large state size will lead to excessively long checkpoint times, affecting the timeliness of business data and the stability of Flink streaming tasks.

[0035] Moreover, the above association method has a write contention problem when multiple streams write to the table at the same time, requiring sequential acquisition of write locks, which can easily lead to deadlocks and performance loss.

[0036] Furthermore, in related technologies, when performing data association, the loading of both hot and cold data presents challenges. Because the hot data is stored in memory, it's impossible to set a reasonable TTL (time to live), and there's a possibility that cached data might not be updated in a timely manner, leading to data anomalies. Moreover, real-time data flow association at the computation layer in these technologies can result in an extremely large amount of cached data for computational tasks, causing task backpressure and other issues.

[0037] To address the aforementioned issues, this application provides relevant solutions, which are detailed below.

[0038] According to an embodiment of this application, a method embodiment of a multi-data-stream parallel processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a multi-data-stream parallel processing method is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0040] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the multi-data stream parallel processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned multi-data stream parallel processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0042] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0043] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0044] Under the above operating environment, embodiments of this application provide a method for parallel processing of multiple data streams, such as... Figure 2 As shown, the method includes the following steps:

[0045] Step S202: Determine the task configuration information of the target task flow, wherein the target task flow includes at least one data flow, and the task configuration information includes the association method between the data flows in the target task flow, the data characteristic information of each data flow in the target task flow, and the data table type of the data flow.

[0046] In the technical solution provided in step S202, the association method includes one of the following: normal write mode, wide table join mode, and data stream JOIN mode.

[0047] As an optional implementation, when configuring the target task flow, the configuration content may include:

[0048] The connection methods between data streams corresponding to the target task flow, such as: JOIN (data stream JOIN mode), common (normal write mode), wide-mode (wide table join mode), etc.

[0049] The schema (feature) of a single stream is used for data parsing and pattern matching, such as: t1.schema = {"type":"record","fields":[{"name":"field1","type":"string"},{"name":"field2","type":"string"}]}, where the schema includes the structure and type of each stream, such as field name, data type, length, constraints and other information.

[0050] For JOIN logic of data streams associated using the JOIN pattern, such as t1.left.join.t2, where t1 and t2 represent the two associated data streams.

[0051] In some embodiments of this application, the task configuration information further includes parallelism; the step of determining the configuration information of the target task flow includes: obtaining a preset parallelism parameter; or, determining the number of data sources associated with the target task flow; and configuring the parallelism parameter of the target task flow according to the number of data sources.

[0052] Specifically, the parallelism mentioned above refers to the number of data streams that can be processed simultaneously in the target task flow. When setting parallelism, users can configure global parallelism within the streaming computing framework or configure the parallelism of a single process task. During task execution, if the parallelism of a task is not individually configured, the global parallelism configuration is used as the default configuration for that task. If the user does not set global task parallelism or parallelism for a single task, the parallelism can be inferred based on the number of specific data sources. For example, for Kafka Source type data sources, the task parallelism configuration is based on the number of Topic partitions. For instance, the parallelism can be set to equal the number of data sources or the number of Topic partitions.

[0053] After configuration, all data streams to be associated can be connected. A streaming task can be configured to read the list of data source topics, and a partition discovery mechanism can be set up to dynamically obtain the number of partitions in the source topic. For example, when a partition is added, the added partition is registered for consumption and added to the data reading list, dynamically connecting to the data streams to be associated in real time.

[0054] After the task configuration settings for the target task flow are completed, the target task flow can load the configuration file and read the task configuration during the startup phase. Then, it can determine the connection method between data flows and load the Payload data processing class corresponding to different modes.

[0055] Step S204: In the pre-write batching stage of the target task flow, perform a first type of operation on the data flow in the target task flow according to the task configuration information. The first type of operation includes at least one of the following: data deduplication and data association.

[0056] During the pre-write batching stage, different data deduplication and association logics can be configured based on the connection method between data streams. For example, for a data stream in normal write mode, deduplication will only be performed on that data stream, and data will be updated, cleaned, and deleted according to the preset configuration scheme, but this data stream will not be connected or associated with other data streams.

[0057] In the technical solution provided in step S204, when the association method is a wide table join mode, the steps of performing the first type of operation on the data stream in the target task stream based on the task configuration information include: determining whether there is target data with the same key value in the target task stream; if it is determined that there is target data with the same key value, determining whether the target data with the same key value belongs to the same data stream; if the target data with the same key value belongs to the same data stream, updating the data stream to which the target data belongs; if the target data with the same key value does not belong to the same data stream, performing data concatenation processing on the data stream to which the target data belongs.

[0058] Specifically, because the data being processed at this point includes data written from different data streams, meaning each data entry may contain different columns, it's necessary to determine whether two records with the same recordKey (key value) come from the same data stream during data merging and deduplication. If they are confirmed to be from the same data stream, the data corresponding to that key value is updated, meaning the most recently generated data is retained. If not, data concatenation is performed. During data concatenation, if two data streams have overlapping columns, they are sorted according to the precombine.field field (pre-merge field), and the overlapping columns are applied as the latest data, meaning they are replaced with the latest data.

[0059] As an optional implementation, when the association method is data flow JOIN mode, the steps of performing the first type of operation on the data flow in the target task flow based on the task configuration information include: determining whether there is target data with the same key value in the target task flow; if it is determined that there is target data with the same key value, determining whether the target data with the same key value belongs to the same data table; if it is determined that the target data corresponds to the same data table, retaining the target data with a later data generation time; if it is determined that the data does not correspond to the same data table, performing data flow JOIN processing on the data flow corresponding to the data.

[0060] In some embodiments of this application, the step of performing data stream JOIN processing on the data stream corresponding to the data includes: determining the JOIN order between the data streams and associating the data streams according to the JOIN order; when associating the data streams again according to the JOIN order, the association method is Inner.Join or Right.Join, and if there is a JOIN failure, returning and marking the target data as hidden data; when associating the data streams again according to the JOIN order, if the association method is Left.Join, returning the target data and adding a hidden mark to the data in the data stream that did not participate in the association.

[0061] Specifically, when using the JOIN pattern to connect different data streams, the target task stream needs to process data that includes data written by different data streams. That is, the data may originate from different data tables. Therefore, at this stage, for two records with the same recordKey (key value), it is necessary to further determine whether their corresponding schemas are the same. If they are the same, it means the two data entries come from the same data table, and the latest data is used to overwrite the old data to complete the data update. If they are different, it means the two data entries come from different data tables. In this case, it is necessary to match the data stream schema with the configuration table schema to distinguish which data stream table each data entry comes from (the data stream table can be considered as a data stream in this application), and perform a data JOIN on the data stream tables according to the configured JOIN method, associating the streams in the JOIN order.

[0062] Subsequently, if the next sequential join of the related data flow tables is an Inner Join or Right Join, and there are cases where the Join did not result in a connection, the previously joined intermediate data (i.e., the data with the same key-value pair that was previously discovered) will be returned and marked as hidden data. If the next sequential join between the related data flow tables is a Left Join, the data can be returned without marking it as hidden. For other data that did not participate in the join, a hidden mark will be added before returning the data, and preparation will be made for the next stage of data writing.

[0063] It should be noted that returning data during the pre-write batching stage means that the data is allowed to undergo write merging processing.

[0064] Step S206: In the data writing and merging stage of the target task flow, the data flow in the target task flow is persisted to disk according to the task configuration information.

[0065] In the technical solution provided in step S206, during the data write-and-merge phase, the Hudi table can be used to determine the corresponding write data logic based on different data table types. For example, for Copy On Write table types, the base file can be copied again, and during the rewrite process, historical and incremental data can be updated to complete the persistent disk write. For Merge On Read table types, incremental data can be written to the log file first, and then during compaction, the data in the historical base file and the incremental log file can be merged and updated.

[0066] As an optional implementation, for data streams in normal write mode, the precombine.field field can be compared for two records with the same recordKey according to the configured payload, and the data that meets the payload retention rules can be persisted to disk.

[0067] In some embodiments of this application, when the association method is a wide table join mode, the step of persistently writing data streams in the target task flow to disk based on task configuration information includes: determining target data with the same key value; determining the generation time of the target data and determining the target column where the latest generated target data is located; using the target column to overwrite historical data, and then persistently writing data associated with the data stream to disk after overwriting.

[0068] Specifically, for data streams joined from wide tables, for two records with the same recordKey, the latest data's columns are used to overwrite the historical data, concatenating them to form the latest data before persistence to disk. For example, for a record in the base file with primary key 'key1', if the Spill Map finds data with the same key value in columns B, C, and D, but column A does not contain data with primary key 'key1', then the latest data will be used to update columns B, C, and D. After the update, the updated columns B, C, and D will be concatenated with column A again.

[0069] As an optional implementation, when the association method is data flow JOIN mode, the step of persisting the data flow in the target task flow to disk based on the task configuration information includes: determining the write merging logic corresponding to the table type information of the data flow, wherein the write merging logic includes the write merging method for the first type of data flow and the second type of data flow, the first type of data flow is the data flow with hidden markers in the corresponding data, and the second type of data flow is the data flow without hidden markers in the corresponding data; and persisting the data flow to disk based on the write merging logic corresponding to the data flow.

[0070] Specifically, for data streams connected using the JOIN mode, during persistent disk write processing, it's necessary to first determine the type of the data table to be processed before determining the corresponding write merging logic. For example, for Merge On Read table type data, during the compaction phase between the base file and the log file, for data with the same recordKey, if the incremental log file contains data without a hidden marker, and the corresponding existing data in the base file also does not contain data with the corresponding recordKey, then the aforementioned data with the same recordKey is determined as new associated data and written to the new base file corresponding to the instant time. If the base file contains existing data with the same recordKey, then the incremental data is used to overwrite and update the historical data, and written to the new base file corresponding to the instant time.

[0071] If the corresponding log file contains data with a hidden marker, and the base file contains data with the same recordKey, then the columns in the incremental data are updated in the historical data, concatenated into complete data, and persisted to disk. If the base file does not contain data with the aforementioned recordKey, then the hidden data is written to the log file corresponding to the new instant time, for use in data association of related data streams during the next compaction.

[0072] For Copy On Write table type data, the write process is equivalent to compaction, and only the base file is included. Therefore, the specific write logic is as follows: For data with the same recordKey, the old and new data are merged, and overlapping columns are updated using an overwrite method. For data without the same recordKey, it remains marked as hidden data and is written to the new base file corresponding to the instant time.

[0073] In some embodiments of this application, after the step of persistently writing the data stream in the target task stream to disk based on the task configuration information, the multi-data stream parallel processing method further includes: determining the query type of the data query instruction and the table type of the data table corresponding to the data query instruction; and processing the data in the data table according to the query type and the table type.

[0074] Specifically, during the data application phase, such as when performing data queries, different processing methods can be applied to the data in the tables depending on the type of table and the query command. For example, for a Copy on Write table, there is no need to merge incremental and historical data during the query; therefore, it is only necessary to filter out the data containing hidden tags, which yields the final real-time streaming wide table data after the association.

[0075] For Merge On Read tables, if the query type is SnapshotView, a merge operation needs to be performed between the historical base file and the incremental log file. The specific logic of the merge operation is the same as the compaction process in the write-merge phase. Then, data containing hidden markers is filtered out, resulting in the final real-time streaming wide table data after the association.

[0076] In summary, the complete process for processing multiple data streams in this application is as follows: Figure 3 As shown, it includes the following steps:

[0077] Step S302: Load the task configuration file in the streaming computing task and determine the association patterns between data streams and the data stream parsing characteristics;

[0078] Step S304: Load the payload for the corresponding mode according to the association mode configuration between data streams;

[0079] Step S306: Based on the association pattern of the data flow, perform data pre-deduplication and association operations during the data pre-writing and batching stage;

[0080] Step S308: Based on the association pattern of the data flow and the data table type, historical data and incremental data are merged during the data writing and disk disposal stage.

[0081] As can be seen, this application decentralizes the JOIN process of the data flow to the Hudi storage of the data lake and utilizes Hudi payload technology for wide table joins. This eliminates the need for data association between cached states within the computing engine and avoids the need for external storage for data caching. Therefore, no maintenance is required for the external storage system, and it does not place additional pressure on storage and network I / O, thus avoiding performance bottlenecks. Furthermore, this application allows for the writing of data to multiple related dimension tables for a single streaming computing task, avoiding the concurrent write lock issues that may arise from multiple tasks writing to a single table simultaneously. This improves task processing performance and avoids potential deadlock problems during concurrent writes.

[0082] Furthermore, by employing a method that determines the task configuration information of the target task flow, wherein the target task flow includes at least one data flow, and the task configuration information includes the association method between the data flows in the target task flow, the data characteristic information of each data flow in the target task flow, and the data table type of the data flow; in the pre-write batching stage of the target task flow, a first type of operation is performed on the data flows in the target task flow according to the task configuration information, wherein the first type of operation includes at least one of the following: data deduplication, data association; in the data write merging stage of the target task flow, the data flows in the target task flow are persistently written to disk according to the task configuration information, and by associating the data flows in the pre-write batching stage, the goal of sinking the association logic of the data flows down to the storage layer of the data lake is achieved, thereby realizing the technical effect that it is not necessary to execute all data association tasks in the computing layer, and thus solving the technical problem of the huge amount of data cached by the computing task caused by real-time data flow association in the computing layer when associating data flows in related technologies.

[0083] This application provides a multi-data-stream parallel processing device. Figure 4 This is a schematic diagram of the device. (For example...) Figure 4 As shown, the device includes: a first processing module 40, used to determine the task configuration information of the target task flow, wherein the target task flow includes at least one data flow, and the task configuration information includes the association method between the data flows in the target task flow, the data characteristic information of each data flow in the target task flow, and the data table type of the data flow; a second processing module 42, used to perform a first type of operation on the data flow in the target task flow according to the task configuration information during the pre-write batching stage of the target task flow, wherein the first type of operation includes at least one of the following: data deduplication, data association; and a third processing module 44, used to perform persistent disk write processing on the data flow in the target task flow according to the task configuration information during the data write merging stage of the target task flow.

[0084] In some embodiments of this application, the association method includes one of the following: normal write mode, wide table join mode, and data stream JOIN mode.

[0085] In some embodiments of this application, the task configuration information further includes parallelism; the step of the first processing module 40 in determining the configuration information of the target task flow includes: obtaining a preset parallelism parameter; or, determining the number of data sources associated with the target task flow; and configuring the parallelism parameter of the target task flow according to the number of data sources.

[0086] In some embodiments of this application, when the association method is a wide table join mode, the second processing module 42 performs the first type of operation on the data stream in the target task stream based on the task configuration information, including: determining whether there is target data with the same key value in the target task stream; if it is determined that there is target data with the same key value, determining whether the target data with the same key value belongs to the same data stream; if the target data with the same key value belongs to the same data stream, updating the data stream to which the target data belongs; and if the target data with the same key value does not belong to the same data stream, performing data splicing processing on the data stream to which the target data belongs.

[0087] In some embodiments of this application, when the association method is data flow JOIN mode, the second processing module 42 performs the first type of operation on the data flow in the target task flow according to the task configuration information, including: determining whether there is target data with the same key value in the target task flow; if it is determined that there is target data with the same key value, determining whether the target data with the same key value belongs to the same data table; if it is determined that the target data corresponds to the same data table, retaining the target data with a later data generation time; if it is determined that the data does not correspond to the same data table, performing data flow JOIN processing on the data flow corresponding to the data.

[0088] In some embodiments of this application, the second processing module 42 performs data stream JOIN processing on the data stream corresponding to the data stream, including: determining the JOIN order between data streams and associating the data streams according to the JOIN order; when associating the data streams again according to the JOIN order, the association method is Inner.Join or Right.Join, and if there is a JOIN association failure, returning and marking the target data as hidden data; when associating the data streams again according to the JOIN order, if the association method is Left.Join, returning the target data and adding a hidden mark to the data in the data stream that did not participate in the association.

[0089] In some embodiments of this application, when the association method is data flow JOIN mode, the third processing module 44 performs persistent disk processing on the data flow in the target task flow based on the task configuration information, including: determining the write merging logic corresponding to the table type information of the data flow, wherein the write merging logic includes the write merging method for the first type of data flow and the second type of data flow, the first type of data flow is the data flow with hidden markers in the corresponding data, and the second type of data flow is the data flow without hidden markers in the corresponding data; and performing persistent disk processing on the data flow according to the write merging logic corresponding to the data flow.

[0090] In some embodiments of this application, when the association method is a wide table join mode, the third processing module 44 performs persistent disk processing on the data stream in the target task stream based on the task configuration information, including: determining target data with the same key value; determining the generation time of the target data and determining the target column where the latest generated target data is located; using the target column to overwrite historical data, and performing persistent disk processing on the data associated with the data stream after overwriting.

[0091] In some embodiments of this application, after the step of persistently writing the data stream in the target task stream to disk based on the task configuration information, the multi-data stream parallel processing device is further configured to: determine the query type of the data query instruction and the table type of the data table corresponding to the data query instruction; and process the data in the data table according to the query type and the table type.

[0092] It should be noted that each module in the above-mentioned multi-data-stream parallel processing device can be a program module (for example, a set of program instructions that implements a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0093] According to an embodiment of this application, a non-volatile storage medium is provided. The non-volatile storage medium stores a program, wherein, during program execution, it controls the device where the non-volatile storage medium resides to perform the following multi-data-stream parallel processing method: determining task configuration information for a target task stream, wherein the target task stream includes at least one data stream, and the task configuration information includes the association method between the data streams in the target task stream, the data characteristic information of each data stream in the target task stream, and the data table type of the data stream; in the pre-write batching stage of the target task stream, performing a first type of operation on the data streams in the target task stream according to the task configuration information, wherein the first type of operation includes at least one of the following: data deduplication, data association; in the data write merging stage of the target task stream, performing persistent disk write processing on the data streams in the target task stream according to the task configuration information.

[0094] According to an embodiment of this application, an electronic device is provided, comprising a memory and a processor. The processor is used to run a program stored in the memory, wherein the program executes the following multi-data-stream parallel processing method: determining task configuration information of a target task stream, wherein the target task stream includes at least one data stream, and the task configuration information includes the association method between the data streams in the target task stream, the data characteristic information of each data stream in the target task stream, and the data table type of the data stream; in the pre-write batching stage of the target task stream, performing a first type of operation on the data streams in the target task stream according to the task configuration information, wherein the first type of operation includes at least one of the following: data deduplication, data association; in the data write merging stage of the target task stream, performing persistent disk processing on the data streams in the target task stream according to the task configuration information.

[0095] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0100] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for parallel processing of multiple data streams, characterized in that, include: Determine the task configuration information of the target task flow, wherein the target task flow includes at least one data flow, and the task configuration information includes the association method between the data flows in the target task flow, the data characteristic information of each data flow in the target task flow, and the data table type of the data flow; During the pre-write batching stage of the target task flow, a first type of operation is performed on the data flow in the target task flow according to the task configuration information, wherein the first type of operation includes at least one of the following: data deduplication, data association; During the data writing and merging phase of the target task flow, the data flow in the target task flow is persisted to disk according to the task configuration information. When the association method is the data flow JOIN mode, performing the first type of operation on the data flow in the target task flow according to the task configuration information includes: determining whether there is target data with the same key value in the target task flow; if the target data with the same key value is determined to exist, determining whether the target data with the same key value belongs to the same data table; if the target data corresponds to the same data table, retaining the target data with a later data generation time; if the data does not correspond to the same data table, performing data flow JOIN processing on the data flow corresponding to the data. The data stream JOIN process for the data stream corresponding to the data includes: Determine the JOIN order between the data streams and associate the data streams according to the JOIN order; if the data streams are associated again according to the JOIN order using the Inner.Join or Right.Join method, and a JOIN association fails, return and mark the target data as hidden data; if the data streams are associated again according to the JOIN order using the Left.Join method, return the target data and add a hidden mark to the data in the data streams that did not participate in the association. When the association method is the data flow JOIN mode, the persistence processing of the data flow in the target task flow according to the task configuration information includes: determining the write merging logic corresponding to the table type information of the data flow, wherein the write merging logic includes write merging methods for the first type of data flow and the second type of data flow, the first type of data flow being the data flow in which the hidden mark exists, and the second type of data flow being the data flow in which the hidden mark does not exist; and performing persistence processing of the data flow according to the write merging logic corresponding to the data flow.

2. The multi-data-stream parallel processing method according to claim 1, characterized in that, The association method includes one of the following: normal write mode, wide table join mode, and data stream JOIN mode.

3. The multi-data-stream parallel processing method according to claim 2, characterized in that, When the association method is the wide table join mode, the step of performing the first type of operation on the data stream in the target task flow based on the task configuration information includes: Determine whether there is target data with the same key value in the target task flow; If it is determined that target data with the same key value exists, determine whether the target data with the same key value belong to the same data stream; If the target data with the same key value belongs to the same data stream, the data stream to which the target data belongs is updated; If the target data with the same key value does not belong to the same data stream, perform data concatenation processing on the data stream to which the target data belongs.

4. The multi-data-stream parallel processing method according to claim 2, characterized in that, When the association method is the wide table join mode, the step of persisting the data stream in the target task flow to disk based on the task configuration information includes: Identify target data with the same key value; Determine the generation time of the target data, and determine the target column where the most recently generated target data is located; The target column is used to cover historical data, and after covering, the data associated with the data stream is persisted to disk.

5. The multi-data-stream parallel processing method according to claim 1, characterized in that, The task configuration information also includes parallelism; the step of determining the configuration information of the target task flow includes: Get the preset parallelism parameters; or, Determine the number of data sources associated with the target task flow; configure the parallelism parameter of the target task flow based on the number of data sources.

6. The multi-data-stream parallel processing method according to claim 1, characterized in that, After the step of persisting the data stream in the target task stream to disk based on the task configuration information, the multi-data-stream parallel processing method further includes: Determine the query type of the data query instruction and the table type of the data table corresponding to the data query instruction; The data in the data table is processed according to the query type and the table type.

7. A multi-data-stream parallel processing device, characterized in that, include: The first processing module is used to determine the task configuration information of the target task flow, wherein the target task flow includes at least one data flow, and the task configuration information includes the association method between the data flows in the target task flow, the data feature information of each data flow in the target task flow, and the data table type of the data flow; The second processing module is used to perform a first type of operation on the data stream in the target task flow according to the task configuration information during the pre-write batching stage of the target task flow. The first type of operation includes at least one of the following: data deduplication and data association. The third processing module is used to perform persistent disk write processing on the data stream in the target task stream according to the task configuration information during the data writing and merging stage of the target task stream. When the association method is the data flow JOIN mode, the second processing module is further configured to: determine whether there is target data with the same key value in the target task flow; if the target data with the same key value is determined to exist, determine whether the target data with the same key value belongs to the same data table; if the target data corresponds to the same data table, retain the target data with a later data generation time; if the data does not correspond to the same data table, perform data flow JOIN processing on the data flow corresponding to the data. The second processing module is further configured to: determine the JOIN order between the data streams, and associate the data streams according to the JOIN order; if the association method is Inner.Join or Right.Join when the data streams are associated again according to the JOIN order, and there is a JOIN association failure, return and mark the target data as hidden data; if the association method is Left.Join when the data streams are associated again according to the JOIN order, return the target data and add a hidden mark to the data in the data stream that did not participate in the association. When the association method is the data stream JOIN mode, the third processing module is further configured to: determine the write merging logic corresponding to the table type information of the data stream, wherein the write merging logic includes write merging methods for the first type of data stream and the second type of data stream, the first type of data stream being the data stream in which the hidden mark exists, and the second type of data stream being the data stream in which the hidden mark does not exist; and perform persistent disk write processing on the data stream according to the write merging logic corresponding to the data stream.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device where the non-volatile storage medium is located to execute the multi-data stream parallel processing method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the multi-data-stream parallel processing method according to any one of claims 1 to 6.

Images