High-concurrency data stream processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to a high-concurrency data stream processing method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring data streams to be processed from a plurality of data sources, storing the data streams to be processed to corresponding topics of a distributed message queue according to data source types, packaging job management nodes and task execution nodes into container instances, and deploying the container instances in a cluster to obtain a stream processing engine, generating a corresponding parallel computing task according to the number of partitions of each theme of the distributed message queue, reading a task data stream corresponding to the parallel computing task from the distributed message queue in parallel through a task execution node, standardizing the task data stream through a stream processing engine to obtain a standardized data stream, and sending the standardized data stream to the distributed message queue; and storing the standardized data stream to a distributed storage system. By adopting the method, the resource utilization rate can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a high-concurrency data stream processing method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] Data storage and processing architecture is a key infrastructure supporting modern digital business. In high-concurrency, massive data application scenarios, due to the high coupling between computing and storage resources, traditional distributed storage systems cannot achieve flexible dynamic scaling when facing high-concurrency data streams, resulting in low resource utilization efficiency and difficulty in fully adapting to dynamic changes in business load. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, computer device, and readable storage medium for high-concurrency data stream processing that can improve resource utilization in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a high-concurrency data stream processing method, including:

[0005] Data streams to be processed are obtained from multiple data sources and stored in the corresponding topics of the distributed message queue according to the data source type. Multiple partitions under each topic of the distributed message queue are used together to store the data streams to be processed.

[0006] The job management node and task execution node are encapsulated as container instances, and the container instances are deployed on the cluster to obtain the stream processing engine; the cluster is used to allocate fixed management resources to the job management node and computing resources to the task execution node;

[0007] Based on the number of partitions in each topic of the distributed message queue, corresponding parallel computing tasks are generated, and task execution nodes read the task data streams corresponding to the parallel computing tasks from the distributed message queue in parallel.

[0008] The task data stream is standardized by the stream processing engine to obtain a standardized data stream, which is then stored in a distributed storage system. The distributed storage system adjusts the allocation of cache resources based on the amount of data written and the frequency of access.

[0009] In one embodiment, the data stream to be processed is a real-time data stream or historical batch data; the step of standardizing the task data stream using a stream processing engine to obtain a standardized data stream includes:

[0010] The task data stream is standardized by executing standardized data transformation rules through the table application programming interface of the stream processing engine, resulting in a standardized data stream.

[0011] In one embodiment, the step of storing a standardized data stream to a distributed storage system includes:

[0012] Based on the user's historical access data, obtain low-frequency access data, medium-frequency access data, and high-frequency access data from the standardized data stream;

[0013] Low-frequency access data is stored in the object storage area of ​​the distributed storage system, medium-frequency access data is stored in the solid-state drive cache area of ​​the distributed storage system, and high-frequency access data is stored in the memory cache area of ​​the distributed storage system.

[0014] In one embodiment, the method further includes:

[0015] Get the workload and data stream processing rate of the stream processing engine;

[0016] Adjust the number of container instances corresponding to job management nodes and task execution nodes based on the workload.

[0017] Adjust the resources allocated to job management nodes and task execution nodes in the cluster based on the data stream processing rate.

[0018] In one embodiment, the method further includes:

[0019] During the standardization process of task data streams through the stream processing engine, a stream processing state snapshot is generated by the stream processing engine and stored in a distributed storage system.

[0020] In the event of a fault in the stream processing engine, a snapshot of the stream processing state is read from the distributed storage system, and the processing state of the stream processing engine is restored based on the snapshot. The process of standardizing the task data stream through the stream processing engine continues.

[0021] In one embodiment, the task data stream is standardized using a stream processing engine to obtain a standardized data stream, which includes:

[0022] Determine the time window based on business needs; the time window includes the window type and window size.

[0023] Determine the event time window corresponding to the standardized data stream based on the event timestamps carried by the standardized data stream;

[0024] Obtain the maximum event timestamp carried by the standardized data stream, and obtain the watermark timestamp based on the maximum event timestamp and the allowed delay threshold;

[0025] If the current watermark timestamp exceeds the end time of the current event time window, aggregate the standardized data stream corresponding to the current event time window.

[0026] In one embodiment, the method further includes:

[0027] For the standardized data stream corresponding to the same event time window, obtain the deduplication identifier of the standardized data stream and map the deduplication identifier to continuous index values;

[0028] Obtain the hash bucket number based on the deduplication identifier, and allocate a computing node for each hash bucket number using the stream processing engine;

[0029] When the event time window ends, the standardized data stream is deduplicated based on the continuous index value to obtain the local deduplication result. The local deduplication results corresponding to the deduplication identifier are then summarized by all computing nodes to obtain the global deduplication result.

[0030] Based on the global deduplication results, the standardized data stream is stored in a distributed storage system.

[0031] Secondly, this application also provides a high-concurrency data stream processing apparatus, comprising:

[0032] The data stream acquisition module is used to acquire data streams to be processed from multiple data sources and store the data streams to be processed in the corresponding topics of the distributed message queue according to the data source type; multiple partitions under each topic of the distributed message queue are used together to store the data streams to be processed.

[0033] The container encapsulation module is used to encapsulate job management nodes and task execution nodes into container instances, and deploy the container instances on the cluster to obtain the stream processing engine; the cluster is used to allocate fixed management resources to job management nodes and computing resources to task execution nodes;

[0034] The task generation module is used to generate corresponding parallel computing tasks based on the number of partitions in each topic of the distributed message queue, and to read the task data stream corresponding to the parallel computing tasks from the distributed message queue in parallel through the task execution node.

[0035] The data stream writing module is used to standardize the task data stream through the stream processing engine to obtain a standardized data stream, and then store the standardized data stream in the distributed storage system. The distributed storage system adjusts the allocation of cache resources according to the amount of data written and the access frequency.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of the first aspects.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method steps of any one of the first aspects.

[0038] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method steps of any one of the first aspects.

[0039] The aforementioned high-concurrency data stream processing method, apparatus, computer equipment, and readable storage medium, by acquiring data streams to be processed from multiple data sources, storing the data streams to be processed in the corresponding topics of a distributed message queue according to the data source type, encapsulating job management nodes and task execution nodes as container instances, and deploying the container instances in a cluster to obtain a stream processing engine, generating corresponding parallel computing tasks based on the number of partitions in each topic of the distributed message queue, reading the task data streams corresponding to the parallel computing tasks from the distributed message queue in parallel through the task execution nodes, standardizing the task data streams through the stream processing engine to obtain standardized data streams, and storing the standardized data streams in a distributed storage system, can improve resource utilization, reduce data processing latency, achieve real-time streaming ingestion, and thus ensure data reliability. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a diagram illustrating the application environment of a high-concurrency data stream processing method in one embodiment.

[0042] Figure 2 This is a flowchart illustrating a high-concurrency data stream processing method in one embodiment;

[0043] Figure 3 This is a flowchart illustrating a high-concurrency data stream processing method in another embodiment;

[0044] Figure 4 This is a structural block diagram of a high-concurrency data stream processing device in one embodiment;

[0045] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] The high-concurrency data stream processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 is used to obtain data streams to be processed from multiple data sources, store the data streams according to the data source type to the corresponding topics of a distributed message queue, encapsulate job management nodes and task execution nodes into container instances, deploy the container instances on a cluster to obtain a stream processing engine, generate corresponding parallel computing tasks based on the number of partitions in each topic of the distributed message queue, read the task data streams corresponding to the parallel computing tasks from the distributed message queue in parallel through the task execution nodes, standardize the task data streams through the stream processing engine to obtain standardized data streams, and store the standardized data streams in a distributed storage system. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0048] In one exemplary embodiment, such as Figure 2 As shown, a high-concurrency data stream processing method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 208. Wherein:

[0049] S202: Obtain the data stream to be processed from multiple data sources and store the data stream to be processed in the corresponding topic of the distributed message queue according to the data source type; multiple partitions under each topic of the distributed message queue are used together to store the data stream to be processed.

[0050] Optionally, multiple data sources refer to various sources that generate the data to be processed, including real-time data streams from sensors, batch data files from business systems, log data, etc., which are the original data sources for streaming data import. Distributed message queues (such as Kafka) are middleware that supports high-concurrency data buffering and distribution, used to temporarily store data flowing in from multiple data sources, avoiding overload caused by directly writing data to the processing engine. A topic refers to a data classification unit in a distributed message queue divided according to data source type, realizing data classification by source, which facilitates subsequent targeted processing. A partition is a parallel storage unit under a topic. By splitting data of the same topic into multiple sub-units, partitions support parallel read and write operations on multiple nodes, improving message queue throughput.

[0051] Optionally, for different types of data sources (such as real-time sensors and historical batch files), the data streams to be processed are collected through adaptive interfaces (such as APIs and file readers) to ensure compatible access to multi-source data. The collected data streams are mapped to the corresponding topics of the distributed message queue according to the data source type, ensuring that the data is isolated by source and avoiding processing chaos caused by mixed streams. Multiple partitions are configured under each topic, and the message queue uses a hash algorithm (such as hashing by data source ID) to evenly distribute the data streams to each partition, realizing parallel storage of multiple partitions on the same topic, supporting high-concurrency writes, and laying the foundation for subsequent parallel computing.

[0052] S204: Encapsulate the job management node and task execution node into container instances, and deploy the container instances on the cluster to obtain the stream processing engine; the cluster is used to allocate fixed management resources to the job management node and computing resources to the task execution node.

[0053] Optionally, the JobManager is a core component of the Flink stream processing engine, responsible for the lifecycle management of stream processing jobs (such as job submission, task scheduling, and fault recovery). The TaskManager is Flink's computation execution component, responsible for running specific parallel computing tasks (such as data reading, cleaning, and transformation), and serves as the engine's computing power carrier. Independent running units encapsulated using container technologies such as Docker package the JobManager / TaskManager and their dependencies (such as configuration files and runtime environments) to ensure "package once, run anywhere." The cluster refers to a Kubernetes (K8s) container cluster, responsible for the scheduling, resource allocation, and operation and maintenance management of container instances, and is the foundation for achieving elastic scaling. The distributed computing engine, composed of containerized JobManager nodes and multiple TaskManager nodes, features high throughput, low latency, and exact-once semantics.

[0054] Optionally, the job management node and task execution node are encapsulated as container instances. The job management node container contains only management components and is lightweight; the task execution node container integrates computational dependencies to ensure consistency of the computing environment. The container instances are deployed to the Kubernetes cluster using management tools, automatically creating Pods (the smallest scheduling unit in Kubernetes) for both the job management node and the task execution node. The cluster allocates fixed management resources to the job management node to ensure its stable operation; and allocates elastic computing resources to the task execution node to match its computationally intensive characteristics, providing computing power support for subsequent parallel task execution, while reserving space for automatic scaling.

[0055] S206: Generate corresponding parallel computing tasks based on the number of partitions in each topic of the distributed message queue, and read the task data stream corresponding to the parallel computing tasks from the distributed message queue in parallel through the task execution node.

[0056] Optionally, parallel computing tasks are data processing tasks generated based on the number of partitions in the message queue topic and executed in parallel. Each task corresponds to the reading and processing of data from one partition, which is the core of achieving Flink's high throughput. There is a one-to-one correspondence between the number of partitions and the degree of parallelism, with the degree of parallelism determining the computational parallelism of the task execution nodes. Each parallel computing task reads its own dedicated data stream from its corresponding partition, ensuring one task per partition and avoiding contention for data in the same partition among multiple tasks. The job management node reads the number of partitions for each topic in the distributed message queue and automatically generates the corresponding number of parallel computing tasks, maximizing the utilization of the computing power of the task execution nodes. The job management node distributes the parallel computing tasks to multiple task execution nodes, using Flink's task slot mechanism to achieve task isolation and avoid resource contention between tasks. Then, the task execution nodes use Flink's Source operator to read the task data stream from the corresponding partition in the message queue in parallel. Each task execution node only reads the data from its assigned partition, achieving multi-node, multi-task parallel reading and avoiding the throughput bottleneck of single-node data reading.

[0057] S208: The task data stream is standardized by the stream processing engine to obtain a standardized data stream, which is then stored in a distributed storage system. The distributed storage system adjusts the allocation of cache resources based on the amount of data written and the frequency of access.

[0058] Optionally, standardization processing refers to the cleaning, transformation, and aggregation operations performed on the task data stream by Flink to ensure the consistency of the output data. Data that conforms to a unified format after standardization processing can be directly written to the distributed storage system, avoiding storage failures or query anomalies caused by data format inconsistencies. The distributed storage system, integrating multi-level caching (memory, SSD), adaptive data sharding, and hot / cold data tiering mechanisms, is the core carrier for high-concurrency data persistence. It can dynamically adjust the cache level according to the data write volume and access frequency, optimizing storage performance and cost.

[0059] Optionally, the task execution nodes perform standardization processing on the task data stream to ensure the accuracy of out-of-order data processing and avoid standardization errors caused by out-of-order data. The standardized data stream is written to the distributed storage system through the Flink Sink operator. The write process adopts two-phase commit and multi-version concurrency control to ensure data consistency and avoid data conflicts during high-concurrency writes. The distributed storage system adjusts cache resources in real time according to data characteristics to achieve a balance between storage performance and cost.

[0060] In the aforementioned high-concurrency data stream processing method, data streams to be processed are obtained from multiple data sources, stored in corresponding topics of a distributed message queue according to data source type, job management nodes and task execution nodes are encapsulated as container instances, and the container instances are deployed in the cluster to obtain a stream processing engine. Based on the number of partitions in each topic of the distributed message queue, corresponding parallel computing tasks are generated. The task execution nodes read the task data streams corresponding to the parallel computing tasks from the distributed message queue in parallel. The stream processing engine performs standardization processing on the task data streams to obtain standardized data streams, which are then stored in a distributed storage system. This approach can improve resource utilization, reduce data processing latency, and achieve real-time streaming ingestion, thereby ensuring data reliability.

[0061] In an exemplary embodiment, the data stream to be processed is a real-time data stream or historical batch data; the step of standardizing the task data stream through the stream processing engine to obtain a standardized data stream includes: executing standardized data transformation rules through the table application programming interface of the stream processing engine to standardize the task data stream and obtain a standardized data stream.

[0062] Optionally, real-time data streams refer to continuous data generated in real time from data sources such as sensors and business systems, characterized by high concurrency and low latency access. Historical batch data refers to large-scale offline data that has been stored, characterized by large data volume and low access frequency. The table application programming interface refers to the Flink core data processing interface, which is a set of declarative APIs based on a relational table model. It supports defining data transformation rules through a unified syntax and can simultaneously adapt to the processing logic of real-time data streams and historical batch data, eliminating the need to develop separate code for the two types of data and achieving a unified stream and batch processing solution with a single interface.

[0063] Optionally, for real-time data streams, after accessing through the Flink Source operator, dynamic tables are defined based on the application programming interface (API). These dynamic tables are continuously updated as real-time data flows in. For historical batch data, after accessing through the Flink batch processing Source, it is mapped to a static table. Batch data is loaded all at once, and real-time and batch data need to be mapped to different processing models to achieve a unified table model adaptation for the two types of data. Then, standardized data transformation rules are executed to uniformly process real-time and historical batch data, including format unification, outlier filtering, and field alignment. The standardized data transformation rules are implemented using the same set of code, eliminating the need for separate development for real-time and historical batch data. After transformation, the output of the standardized data stream will have the dynamic table (real-time data) continuously generating streaming data, and the static table (historical batch data) generating batch processing result data. Both types of data are ultimately encapsulated into a standardized data stream with completely consistent field structure, data type, and format, which is then directly written to the distributed storage system through the Flink Sink operator.

[0064] In this embodiment, the standardized data transformation rules are executed through the table application programming interface of the stream processing engine to standardize the task data stream and obtain a standardized data stream. This enables unified stream and batch processing, reduces architectural complexity and development costs, improves the efficiency of standardized processing, and achieves high-concurrency streaming data import.

[0065] In an exemplary embodiment, the step of storing a standardized data stream to a distributed storage system includes: obtaining low-frequency access data, medium-frequency access data, and high-frequency access data from the standardized data stream based on the user's historical access data; storing the low-frequency access data in the object storage area of ​​the distributed storage system, storing the medium-frequency access data in the solid-state drive cache area of ​​the distributed storage system, and storing the high-frequency access data in the memory cache area of ​​the distributed storage system.

[0066] Optionally, before receiving standardized data streams, the distributed storage system first constructs an access frequency model based on users' historical access data. It then extracts key indicators from the historical access data to perform threshold determination. If the data access indicator reaches a high-frequency threshold (e.g., ≥10 accesses in the past hour), it is determined to be high-frequency access data; if it reaches a medium-frequency threshold (1-10 accesses in the past day), it is determined to be medium-frequency access data; and if it falls below a low-frequency threshold (<1 access in the past 7 days), it is determined to be low-frequency access data. Standardized data streams of different access frequencies are then directed and written to the corresponding storage levels of the distributed storage system.

[0067] Furthermore, high-frequency data is written to the memory cache, and a cache eviction policy is enabled. When memory is insufficient, the least accessed high-frequency data is automatically removed, ensuring that the memory always stores the most currently accessed data. Multi-version concurrency control is used during the write process to avoid high-concurrency write conflicts. Mid-frequency data is written to the SSD cache, and a read-ahead caching mechanism is enabled to preload potentially accessed related data, balancing access speed and storage cost. An adaptive compression algorithm is used during the write process to compress the data and reduce SSD storage usage. Low-frequency data is written to the object storage area, and high-compression encoding is performed before writing to further reduce storage costs. Cold / hot identifiers and access indexes are added to the data to facilitate quick location of data storage during subsequent accesses, avoiding latency caused by full scans.

[0068] Optionally, the tiered storage of a distributed storage system is not static and fixed. When user access behavior changes (such as low-frequency data suddenly being accessed at a high frequency due to business needs), a data migration mechanism is automatically triggered to migrate the data from the lower-level storage to the higher-level storage, and vice versa, to ensure that storage resources always match the actual access needs and avoid resource waste.

[0069] In this embodiment, by obtaining low-frequency, medium-frequency, and high-frequency access data from the standardized data stream based on the user's historical access data, storing the low-frequency access data in the object storage area of ​​the distributed storage system, storing the medium-frequency access data in the solid-state drive cache area of ​​the distributed storage system, and storing the high-frequency access data in the memory cache area of ​​the distributed storage system, storage costs can be reduced, resource utilization can be improved, and thus the high-concurrency real-time demand can be accurately matched.

[0070] In one exemplary embodiment, the method further includes: obtaining the workload and data stream processing rate of the stream processing engine; adjusting the number of container instances corresponding to the job management node and task execution node according to the workload; and adjusting the resources allocated by the cluster to the job management node and task execution node according to the data stream processing rate.

[0071] Optionally, the stream processing engine collects core load metrics of job management nodes and task execution nodes in real time, including job submission queue length and task scheduling latency for job management nodes, and CPU utilization, memory usage, and task backlog for task execution nodes. Additionally, the stream processing engine tracks the data stream processing rate of task execution nodes in real time, including events processed per second, data read rate, and data write rate, while also monitoring for backpressure to quantitatively reflect the matching degree between the engine's current data processing capacity and the inflow of data. Based on the collected business load data, the Kubernetes container orchestration platform dynamically scales up and down the number of container instances. When the business load is too high, existing container instances cannot handle the current tasks, requiring an increase in the number of instances; when the business load is too low, resources are idle, requiring a decrease in the number of instances.

[0072] Optionally, if the data stream processing rate is consistently lower than the data access rate and there is no trend of backpressure relief, it is determined that the current resources cannot support the processing demand, and the CPU / memory quota for the task execution node needs to be increased to improve the processing capacity of a single instance. If the data stream processing rate is significantly higher than the data access rate and this continues for more than 10 minutes, it is determined that the resources are over-allocated, and the resource quota for the task execution node needs to be reduced to release idle resources. The Kubernetes scheduler adjusts the resource configuration in real time according to resource demand. At the same time, the task execution node optimizes the task parallelism based on the resource adjustment results to ensure efficient matching of resources and tasks.

[0073] In this embodiment, by obtaining the business load and data stream processing rate of the stream processing engine, adjusting the number of container instances corresponding to the job management node and task execution node according to the business load, and adjusting the resources allocated by the cluster to the job management node and task execution node according to the data stream processing rate, resource utilization can be improved, processing stability under high concurrency scenarios can be ensured, thereby ensuring the reliability of high-concurrency data stream processing.

[0074] In an exemplary embodiment, the method further includes: during the standardization process of the task data stream by the stream processing engine, generating a stream processing state snapshot by the stream processing engine and storing the stream processing state snapshot in a distributed storage system; in the event of a failure of the stream processing engine, reading the stream processing state snapshot from the distributed storage system, restoring the processing state of the stream processing engine based on the stream processing state snapshot, and continuing the standardization process of the task data stream by the stream processing engine.

[0075] Optionally, when performing task data stream standardization processing, the stream processing engine automatically generates stream processing state snapshots through a distributed snapshot mechanism. These snapshots contain key state data of the stream processing engine during the standardization process, including business processing status and task metadata. The business processing status includes intermediate calculation results and window calculation status during the standardization transformation; the task metadata includes the current data stream partition and task execution node configuration information, ensuring accurate continuation of the processing flow after recovery and avoiding data duplication or omission. The generated stream processing state snapshots are stored in a distributed storage system. The distributed storage system assigns a unique version number to each snapshot and records the snapshot generation time and corresponding processing status version information, avoiding version conflicts when multiple snapshots are stored.

[0076] Optionally, when the stream processing engine fails, rapid recovery is achieved based on the stream processing state snapshot stored in the distributed storage system, continuing the standardized processing task data stream. Specifically, in the event of a failure, the latest valid snapshot stored in the distributed storage system is automatically located, the latest snapshot is selected based on the snapshot version number, and the snapshot integrity is verified to ensure the reliability of the data source for recovery. Kubernetes automatically restarts the container instances of the failed job management node and task execution node, restoring the engine's basic operating environment. The newly started engine instance reads the latest valid snapshot from the distributed storage system, loads the business processing state from the snapshot into local memory, and synchronizes the task metadata to the distributed message queue, ensuring that data is read from the point of termination before the failure after recovery, avoiding duplicate processing or data loss. After the state is loaded, the engine automatically restarts the standardized processing task, continuing to perform operations such as format unification and outlier filtering on the task data stream based on the recovered state.

[0077] In this embodiment, during the standardization process of the task data stream by the stream processing engine, a stream processing state snapshot is generated by the stream processing engine and stored in the distributed storage system. In the event of a fault in the stream processing engine, the stream processing state snapshot is read from the distributed storage system, and the processing state of the stream processing engine is restored based on the stream processing state snapshot. The process of continuing to standardize the task data stream by the stream processing engine can quickly restore the fault state, ensure business continuity, and ensure the security and reliability of state data.

[0078] In an exemplary embodiment, after standardizing the task data stream using a stream processing engine to obtain a standardized data stream, the process includes: determining a time window based on business requirements; the time window includes a window type and a window size; determining the event time window corresponding to the standardized data stream based on the event timestamps carried by the standardized data stream; obtaining the maximum event timestamp carried by the standardized data stream, and obtaining a watermark timestamp based on the maximum event timestamp and an allowed delay threshold; and aggregating the standardized data stream corresponding to the current event time window if the current watermark timestamp exceeds the end time of the current event time window.

[0079] Optionally, after receiving the standardized data stream, the stream processing engine defines the core parameters of the time window based on real-time analysis requirements, including window type and window size. Window types include scrolling windows and sliding windows. Scrolling windows have no overlap or gaps between windows, suitable for aggregation needs at fixed periods, avoiding redundant data calculations; sliding windows have overlapping windows, suitable for scenarios requiring high-frequency updates of aggregation results. The choice of window type directly relates to the granularity of business analysis, ensuring that the aggregation results align with the actual application scenario. The window size is configured based on the business's need to balance real-time performance with data integrity.

[0080] Furthermore, based on the event timestamps carried by the standardized data streams (i.e., the actual time when the data was generated, not the system time when the stream processing engine received the data), the data is allocated to the corresponding time windows. The standardized data streams carry unique event timestamps at the time of generation (e.g., when sensors collect data), and these timestamps directly determine the window to which the data belongs. By allocating event time windows, it is ensured that out-of-order data (e.g., data arriving at the engine with a 2-second delay due to network fluctuations, but whose event timestamp is still within the original window range) can be correctly allocated to its appropriate window, avoiding data omissions in statistics.

[0081] Optionally, the stream processing engine tracks the maximum event timestamp in the currently accessed standardized data stream in real time. This value reflects the current data stream's time progress. The threshold setting needs to balance data integrity and aggregation real-time performance, configured according to the business's tolerance for delayed data. A higher tolerance results in a larger threshold, ensuring more delayed data is included in the statistics; a lower tolerance results in a smaller threshold, ensuring rapid output of aggregation results. The watermark timestamp equals the maximum event timestamp plus the allowed delay threshold. The core function of the watermark is to mark the time boundary of the data stream, informing the engine that data with delays exceeding the threshold will no longer be included in the current window, providing a clear basis for aggregation triggering. Specifically, when the stream processing engine detects that the current watermark timestamp exceeds the end of the current event time window, it automatically triggers the aggregation of the standardized data stream in that window. If a very small number of delayed data exceeding the threshold still arrive after the watermark-triggered aggregation, the engine will route them to the side output stream for separate processing, ensuring the real-time performance of the main window's aggregation results while avoiding complete data loss.

[0082] In this embodiment, by determining the time window based on business needs, determining the event time window corresponding to the standardized data stream based on the event timestamps carried by the standardized data stream, obtaining the maximum event timestamp carried by the standardized data stream, and obtaining the watermark timestamp based on the maximum event timestamp and the allowable delay threshold, and aggregating the standardized data stream corresponding to the current event time window when the current watermark timestamp exceeds the end time of the current event time window, the accuracy of data aggregation can be improved, the business needs can be accurately adapted, and the architectural complexity of the stream processing engine can be reduced, thereby reducing development and maintenance costs.

[0083] In an exemplary embodiment, the method further includes: for a standardized data stream corresponding to the same event time window, obtaining a deduplication identifier for the standardized data stream, and mapping the deduplication identifier to a continuous index value; obtaining a hash bucket number based on the deduplication identifier, and allocating a computing node for each hash bucket number through the stream processing engine; when the event time window ends, performing deduplication processing on the standardized data stream based on the continuous index value to obtain a local deduplication result, and summarizing the local deduplication results corresponding to the deduplication identifier through all computing nodes to obtain a global deduplication result; and storing the standardized data stream to a distributed storage system based on the global deduplication result.

[0084] Optionally, for standardized data streams within the same event time window, deduplication identifier extraction and index mapping are performed. A unique business-specific deduplication identifier is extracted from the standardized data stream. This identifier is generated by combining business attributes and is used to distinguish duplicate data. Non-contiguous deduplication identifiers are converted into contiguous integer index values ​​through a dictionary mapping table. The bitmap marks the existence of data based on the contiguous integer index (a 1 in the corresponding bit indicates existence, and a 0 indicates non-existence). Directly using non-contiguous identifiers would lead to wasted bitmap space, while contiguous indexes can maximize the storage efficiency of the bitmap and reduce the amount of state storage.

[0085] Furthermore, hash bucketing is performed based on the deduplication identifier, and a compute node of the stream processing engine is assigned to each bucket. A hash operation is performed on the deduplication identifier, and a hash bucket number is obtained according to the preset number of hash buckets, ensuring that the same deduplication identifier always maps to the same hash bucket and avoiding duplicate data being scattered across different buckets. The stream processing engine binds each hash bucket number to its corresponding compute node, and after bucketing, each compute node only processes data from a portion of the hash buckets. When the event time window ends, two levels of deduplication are performed: local deduplication and global aggregation. Local deduplication refers to each compute node performing bitmap deduplication based on consecutive index values ​​on the standardized data stream within its assigned hash bucket. After the window ends, the index corresponding to "1" in the bitmap is the local deduplication result (only the first occurrence of data is retained). This process does not require storing complete data, only maintaining the bitmap, significantly reducing memory usage. The job management node of the stream processing engine collects the local deduplication results from all computing nodes and merges the bitmap bit by bit. After merging, the consecutive index values ​​corresponding to "1" in the global bitmap are mapped back to the original deduplication identifiers, thus obtaining the global deduplication result (identifiers of all non-repeating standardized data streams). This ensures that there is no duplicate data within the entire event time window and avoids incomplete deduplication caused by bucketing. Finally, the globally deduplicated standardized data stream is written to the distributed storage system.

[0086] In this embodiment, by obtaining the deduplication identifier of the standardized data stream corresponding to the same event time window, mapping the deduplication identifier to a continuous index value, obtaining the hash bucket number based on the deduplication identifier, and allocating a computing node for each hash bucket number through the stream processing engine, when the event time window ends, deduplication processing is performed on the standardized data stream according to the continuous index value to obtain the local deduplication result. The local deduplication results corresponding to the deduplication identifier are summarized by all computing nodes to obtain the global deduplication result. The standardized data stream is stored in the distributed storage system according to the global deduplication result, which can reduce the amount of state storage, improve resource utilization, and thus improve high-concurrency processing capabilities.

[0087] In one exemplary embodiment, such as Figure 3 As shown, a high-concurrency data stream processing method is provided, which includes the following steps:

[0088] (1) Data stream acquisition: Obtain the data stream to be processed from multiple data sources and store the data stream to be processed in the corresponding topic of the distributed message queue according to the data source type; multiple partitions under each topic of the distributed message queue are used to store the data stream to be processed. Among them, the data stream to be processed is real-time data stream or historical batch data.

[0089] (2) Parallel Business Processing: The job management node and task execution node are encapsulated as container instances, and the container instances are deployed in the cluster to obtain the stream processing engine. The cluster is used to allocate fixed management resources to the job management node and computing resources to the task execution node. Based on the number of partitions of each topic in the distributed message queue, corresponding parallel computing tasks are generated, and the task execution nodes read the task data streams corresponding to the parallel computing tasks from the distributed message queue in parallel. The standardized data transformation rules are executed through the table application programming interface of the stream processing engine to standardize the task data streams and obtain standardized data streams.

[0090] (3) Engine state recovery: During the process of standardizing the task data stream through the stream processing engine, the stream processing engine generates a stream processing state snapshot and stores the stream processing state snapshot in the distributed storage system; in the event of a fault in the stream processing engine, the stream processing state snapshot is read from the distributed storage system, and the processing state of the stream processing engine is restored according to the stream processing state snapshot, and the process of standardizing the task data stream through the stream processing engine continues.

[0091] (4) Time window aggregation: Determine the time window according to business needs; the time window includes window type and window size; determine the event time window corresponding to the standardized data stream according to the event timestamp carried by the standardized data stream; obtain the maximum event timestamp carried by the standardized data stream, and obtain the watermark timestamp according to the maximum event timestamp and the allowed delay threshold; when the current watermark timestamp exceeds the end time of the current event time window, aggregate the standardized data stream corresponding to the current event time window.

[0092] (5) Data stream deduplication: For the standardized data stream corresponding to the same event time window, obtain the deduplication identifier of the standardized data stream and map the deduplication identifier to a continuous index value; obtain the hash bucket number according to the deduplication identifier, and allocate a computing node for each hash bucket number through the stream processing engine; when the event time window ends, perform deduplication processing on the standardized data stream according to the continuous index value to obtain the local deduplication result, and summarize the local deduplication result corresponding to the deduplication identifier through all computing nodes to obtain the global deduplication result; store the standardized data stream to the distributed storage system according to the global deduplication result.

[0093] (6) Categorized storage of accessed data: Based on the user's historical access data, low-frequency access data, medium-frequency access data, and high-frequency access data are obtained from the standardized data stream; low-frequency access data is stored in the object storage area of ​​the distributed storage system, medium-frequency access data is stored in the solid-state drive cache area of ​​the distributed storage system, and high-frequency access data is stored in the memory cache area of ​​the distributed storage system. Among them, the distributed storage system adjusts the cache resource allocation according to the data write volume and access frequency.

[0094] (7) Dynamic adjustment of node resources: obtain the business load and data stream processing rate of the stream processing engine; adjust the number of container instances corresponding to the job management node and task execution node according to the business load; adjust the resources allocated by the cluster to the job management node and task execution node according to the data stream processing rate.

[0095] Optionally, for ease of understanding, the above method is illustrated using an architecture integrating a streaming engine and the distributed Hstore storage model. The streaming engine uses Flink as its core streaming computing engine. Flink boasts high throughput and low latency, supporting tens of millions of events per second. Simultaneously, Flink provides exactly-once semantics, effectively ensuring data consistency. In the architecture design, Flink is responsible for receiving and processing real-time data streams from various data sources. Utilizing its internal parallel processing mechanism and efficient event-driven model, it performs rapid data cleaning, transformation, and aggregation operations. The processed data is promptly written to the Hstore distributed storage system, ensuring the efficiency and reliability of the entire data processing chain. Unified streaming and batch processing logic is implemented through the Flink Table API and SQL. Since the batch processing layer and stream processing layer are independent, leading to higher development and maintenance costs, Flink's unified streaming and batch design allows developers to process both streaming and batch data simultaneously using a single codebase and configuration. For example, in the data cleaning phase, whether processing real-time incoming sensor data or processing historically accumulated batch data files, the same Flink Table API can be used to define data transformation rules. This not only reduces redundant code development and lowers system complexity, but also improves development efficiency, making the data processing workflow simpler and more efficient, and facilitating subsequent maintenance and expansion.

[0096] Furthermore, Flink's lightweight distributed checkpointing mechanism is integrated, combined with a multi-level cache in the hstore-opt storage mode, to achieve efficient persistence and fault recovery of state data. Flink's checkpointing mechanism periodically saves snapshots of the stream processing job's state. When a system failure occurs, it can quickly restore the state to its pre-failure state based on these snapshots and continue data processing. The hstore-opt multi-level cache provides high-performance storage support for state data. Frequently accessed state data can be read and written quickly in the memory cache, while less frequently accessed state data can be stored on SSDs or object storage. This optimizes storage costs and data access performance while ensuring the reliability of state data persistence.

[0097] Optionally, during elastic resource allocation, a Kubernetes-based Flink cloud-native deployment (such as ArceeOperator) can be used to achieve automatic scaling of JobManager and TaskManager, combined with Spot instances to reduce costs. Kubernetes, as a powerful container orchestration platform, provides a flexible deployment and management environment for Flink clusters. Fine-grained management of the Flink cluster through Arcee Operator allows for dynamic adjustment of the number of JobManager and TaskManager instances based on actual business load. When business traffic increases, the automatic scaling mechanism can quickly launch new TaskManager instances, allocating more computing resources to process data; conversely, when business traffic decreases, idle instances can be promptly reclaimed, releasing resources and reducing operating costs. Simultaneously, combined with the use of Spot instances, this further saves costs and improves resource utilization efficiency while ensuring system stability.

[0098] Furthermore, by leveraging Flink's backpressure mechanism in conjunction with the Kubernetes scheduler, the data stream processing rate and resource quotas are dynamically adjusted to prevent cluster overload. Flink's backpressure mechanism can monitor the pressure on the data stream at each processing stage in real time. When data backlog occurs at a stage, indicating impending overload, Flink can automatically detect and adjust the data stream processing rate. Simultaneously, it works in conjunction with the Kubernetes scheduler to dynamically allocate appropriate resource quotas to Flink jobs based on the current load. For example, it may increase CPU and memory allocation or prioritize jobs to ensure the entire cluster remains stable under high load, preventing data loss or system crashes due to insufficient resources.

[0099] Furthermore, Flink's EventTime windowing mechanism is employed to address out-of-order data issues, combined with watermarking technology for delayed data processing. In actual data acquisition and transmission, data may arrive out of order due to network latency, device performance differences, and other factors. Flink's EventTime windowing mechanism divides windows based on the timestamps carried by the data itself, rather than on the system time of data arrival at the processing node. This ensures that data is aggregated and calculated in the correct chronological order. Simultaneously, watermarking is used to mark delayed data in the data stream. When the watermark reaches a certain threshold, it triggers the calculation and output of the corresponding window, effectively processing data that arrives late but is still within an acceptable timeframe, ensuring the accuracy and integrity of data processing.

[0100] Furthermore, for deduplication scenarios (such as COUNT DISTINCT), Flink's BitMap reuse and bucketing aggregation techniques are introduced to reduce state storage and alleviate data skew. Deduplication is a common requirement in big data processing, but traditional deduplication algorithms can consume significant memory resources and easily lead to data skew. Flink's BitMap reuse technology leverages the efficiency and compressibility of bitmap data structures to provide a compact representation and storage of deduplicated data. Simultaneously, bucketing aggregation divides data into different buckets according to certain rules for aggregation calculations, avoiding a single node processing too much data, thus effectively mitigating data skew and improving processing performance and stability.

[0101] In this embodiment, by adopting an architecture that integrates a streaming engine and a distributed storage model, elastic resource scheduling for high-concurrency streaming data ingestion can be achieved in a cloud-native environment. This ensures the efficiency and consistency of data writing in the distributed storage system. At the same time, the storage model is optimized to reduce I / O overhead, improve data compression and partitioning efficiency, simplify the architectural complexity through integrated stream and batch design, enhance real-time data processing capabilities, and ensure compatibility with new distributed storage systems, enabling high-speed ingestion of massive amounts of data.

[0102] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0103] Based on the same inventive concept, this application also provides a high-concurrency data stream processing apparatus for implementing the high-concurrency data stream processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the high-concurrency data stream processing apparatus provided below can be found in the limitations of the high-concurrency data stream processing method described above, and will not be repeated here.

[0104] In one exemplary embodiment, such as Figure 4 As shown, a high-concurrency data stream processing device is provided, including: a data stream acquisition module 10, a container encapsulation module 20, a task generation module 30, and a data stream writing module 40, wherein:

[0105] The data stream acquisition module 10 is used to acquire data streams to be processed from multiple data sources and store the data streams to be processed in the corresponding topics of the distributed message queue according to the data source type; multiple partitions under each topic of the distributed message queue are used to store the data streams to be processed.

[0106] The container encapsulation module 20 is used to encapsulate the job management node and the task execution node into container instances, and deploy the container instances on the cluster to obtain the stream processing engine; the cluster is used to allocate fixed management resources to the job management node and computing resources to the task execution node.

[0107] The task generation module 30 is used to generate corresponding parallel computing tasks based on the number of partitions of each topic in the distributed message queue, and to read the task data stream corresponding to the parallel computing tasks from the distributed message queue in parallel through the task execution node.

[0108] The data stream writing module 40 is used to standardize the task data stream through the stream processing engine to obtain a standardized data stream, and then store the standardized data stream in the distributed storage system. The distributed storage system adjusts the allocation of cache resources according to the amount of data written and the access frequency.

[0109] In an exemplary embodiment, the data stream to be processed is a real-time data stream or historical batch data; the data stream writing module 40 is also used to execute standardized data transformation rules through the table application programming interface of the stream processing engine to standardize the task data stream and obtain a standardized data stream.

[0110] In an exemplary embodiment, the data stream writing module 40 is further configured to obtain low-frequency access data, medium-frequency access data and high-frequency access data in the standardized data stream based on the user's historical access data; store the low-frequency access data in the object storage area of ​​the distributed storage system, store the medium-frequency access data in the solid-state drive cache area of ​​the distributed storage system, and store the high-frequency access data in the memory cache area of ​​the distributed storage system.

[0111] In an exemplary embodiment, the data stream writing module 40 is further configured to obtain the business load and data stream processing rate of the stream processing engine; adjust the number of container instances corresponding to the job management node and task execution node according to the business load; and adjust the resources allocated by the cluster to the job management node and task execution node according to the data stream processing rate.

[0112] In an exemplary embodiment, the data stream writing module 40 is further configured to generate a stream processing state snapshot through the stream processing engine and store the stream processing state snapshot in a distributed storage system during the process of standardizing the task data stream through the stream processing engine; in the event of a failure of the stream processing engine, read the stream processing state snapshot from the distributed storage system and restore the processing state of the stream processing engine based on the stream processing state snapshot, and continue the process of standardizing the task data stream through the stream processing engine.

[0113] In an exemplary embodiment, the data stream writing module 40 is further configured to determine a time window based on business requirements; the time window includes a window type and a window size; determine the event time window corresponding to the standardized data stream based on the event timestamps carried by the standardized data stream; obtain the maximum event timestamp carried by the standardized data stream, and obtain the watermark timestamp based on the maximum event timestamp and the allowed delay threshold; and aggregate the standardized data stream corresponding to the current event time window if the current watermark timestamp exceeds the end time of the current event time window.

[0114] In an exemplary embodiment, the data stream writing module 40 is further configured to: obtain a deduplication identifier for the standardized data stream corresponding to the same event time window; map the deduplication identifier to a continuous index value; obtain a hash bucket number based on the deduplication identifier; allocate a computing node for each hash bucket number through the stream processing engine; when the event time window ends, perform deduplication processing on the standardized data stream based on the continuous index value to obtain a local deduplication result; and summarize the local deduplication results corresponding to the deduplication identifier through all computing nodes to obtain a global deduplication result; and store the standardized data stream to the distributed storage system based on the global deduplication result.

[0115] Each module in the aforementioned high-concurrency data stream processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0116] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a high-concurrency data stream processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0117] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: obtaining data streams to be processed from multiple data sources; storing the data streams to be processed into corresponding topics of a distributed message queue according to the data source type; multiple partitions under each topic of the distributed message queue are used to store the data streams to be processed; encapsulating job management nodes and task execution nodes into container instances, and deploying the container instances on a cluster to obtain a stream processing engine; the cluster is used to allocate fixed management resources to the job management nodes and computing resources to the task execution nodes; generating corresponding parallel computing tasks according to the number of partitions in each topic of the distributed message queue, and reading the task data streams corresponding to the parallel computing tasks from the distributed message queue in parallel through the task execution nodes; standardizing the task data streams through the stream processing engine to obtain standardized data streams, and storing the standardized data streams in a distributed storage system; the distributed storage system adjusts the allocation of cache resources according to the data write volume and access frequency.

[0119] In one embodiment, the data stream to be processed is a real-time data stream or historical batch data; the standardization of the task data stream by the stream processing engine to obtain a standardized data stream when the processor executes the computer program includes: executing standardized data transformation rules through the table application programming interface of the stream processing engine to standardize the task data stream and obtain a standardized data stream.

[0120] In one embodiment, storing a standardized data stream to a distributed storage system when the processor executes a computer program includes: acquiring low-frequency access data, medium-frequency access data, and high-frequency access data from the standardized data stream based on the user's historical access data; storing the low-frequency access data in the object storage area of ​​the distributed storage system, storing the medium-frequency access data in the solid-state drive cache area of ​​the distributed storage system, and storing the high-frequency access data in the memory cache area of ​​the distributed storage system.

[0121] In one embodiment, when the processor executes the computer program, it also performs the following steps: obtaining the workload and data stream processing rate of the stream processing engine; adjusting the number of container instances corresponding to the job management node and task execution node according to the workload; and adjusting the resources allocated by the cluster to the job management node and task execution node according to the data stream processing rate.

[0122] In one embodiment, when the processor executes the computer program, it further implements the following steps: during the process of standardizing the task data stream through the stream processing engine, a stream processing state snapshot is generated by the stream processing engine and stored in a distributed storage system; in the event of a failure of the stream processing engine, the stream processing state snapshot is read from the distributed storage system, and the processing state of the stream processing engine is restored based on the stream processing state snapshot, and the process of standardizing the task data stream through the stream processing engine continues.

[0123] In one embodiment, when a processor executes a computer program, it involves standardizing a task data stream using a stream processing engine to obtain a standardized data stream. This process includes: determining a time window based on business requirements; the time window includes a window type and a window size; determining the event time window corresponding to the standardized data stream based on the event timestamps carried by the standardized data stream; obtaining the maximum event timestamp carried by the standardized data stream; and obtaining a watermark timestamp based on the maximum event timestamp and an allowed delay threshold; and aggregating the standardized data stream corresponding to the current event time window if the current watermark timestamp exceeds the end time of the current event time window.

[0124] In one embodiment, when the processor executes the computer program, it further implements the following steps: for the standardized data stream corresponding to the same event time window, obtain the deduplication identifier of the standardized data stream, and map the deduplication identifier to a continuous index value; obtain the hash bucket number according to the deduplication identifier, and allocate a computing node for each hash bucket number through the stream processing engine; when the event time window ends, perform deduplication processing on the standardized data stream according to the continuous index value to obtain a local deduplication result, and summarize the local deduplication results corresponding to the deduplication identifier through all computing nodes to obtain a global deduplication result; and store the standardized data stream to a distributed storage system according to the global deduplication result.

[0125] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: obtaining data streams to be processed from multiple data sources, storing the data streams to be processed in corresponding topics of a distributed message queue according to the data source type; multiple partitions under each topic of the distributed message queue are used together to store the data streams to be processed; encapsulating job management nodes and task execution nodes into container instances, and deploying the container instances on a cluster to obtain a stream processing engine; the cluster is used to allocate fixed management resources to job management nodes and computing resources to task execution nodes; generating corresponding parallel computing tasks according to the number of partitions in each topic of the distributed message queue, and reading the task data streams corresponding to the parallel computing tasks from the distributed message queue in parallel through the task execution nodes; standardizing the task data streams through the stream processing engine to obtain standardized data streams, and storing the standardized data streams in a distributed storage system; the distributed storage system adjusts the allocation of cache resources according to the amount of data written and the access frequency.

[0126] In one embodiment, the data stream to be processed is a real-time data stream or historical batch data; when the computer program is executed by the processor, the task data stream is standardized by the stream processing engine to obtain a standardized data stream, including: executing standardized data transformation rules through the table application programming interface of the stream processing engine to standardize the task data stream and obtain a standardized data stream.

[0127] In one embodiment, the storage of a standardized data stream to a distributed storage system when a computer program is executed by a processor includes: acquiring low-frequency access data, medium-frequency access data, and high-frequency access data from the standardized data stream based on the user's historical access data; storing the low-frequency access data in the object storage area of ​​the distributed storage system, storing the medium-frequency access data in the solid-state drive cache area of ​​the distributed storage system, and storing the high-frequency access data in the memory cache area of ​​the distributed storage system.

[0128] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the workload and data stream processing rate of the stream processing engine; adjusting the number of container instances corresponding to the job management node and task execution node according to the workload; and adjusting the resources allocated by the cluster to the job management node and task execution node according to the data stream processing rate.

[0129] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: during the standardization process of the task data stream by the stream processing engine, a stream processing state snapshot is generated by the stream processing engine and stored in a distributed storage system; in the event of a failure of the stream processing engine, the stream processing state snapshot is read from the distributed storage system, and the processing state of the stream processing engine is restored based on the stream processing state snapshot, and the process of standardizing the task data stream by the stream processing engine continues.

[0130] In one embodiment, when a computer program is executed by a processor, it involves standardizing a task data stream through a stream processing engine to obtain a standardized data stream. This process includes: determining a time window based on business requirements; the time window includes a window type and a window size; determining the event time window corresponding to the standardized data stream based on the event timestamps carried by the standardized data stream; obtaining the maximum event timestamp carried by the standardized data stream; and obtaining a watermark timestamp based on the maximum event timestamp and an allowed delay threshold; and aggregating the standardized data stream corresponding to the current event time window if the current watermark timestamp exceeds the end time of the current event time window.

[0131] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for the standardized data stream corresponding to the same event time window, obtain the deduplication identifier of the standardized data stream, and map the deduplication identifier to a continuous index value; obtain the hash bucket number according to the deduplication identifier, and allocate a computing node for each hash bucket number through the stream processing engine; when the event time window ends, perform deduplication processing on the standardized data stream according to the continuous index value to obtain a local deduplication result, and summarize the local deduplication results corresponding to the deduplication identifier through all computing nodes to obtain a global deduplication result; and store the standardized data stream to a distributed storage system according to the global deduplication result.

[0132] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: acquiring data streams to be processed from multiple data sources, storing the data streams to be processed in corresponding topics of a distributed message queue according to the data source type; multiple partitions under each topic of the distributed message queue are used together to store the data streams to be processed; encapsulating job management nodes and task execution nodes into container instances, and deploying the container instances on a cluster to obtain a stream processing engine; the cluster is used to allocate fixed management resources to job management nodes and computing resources to task execution nodes; generating corresponding parallel computing tasks according to the number of partitions in each topic of the distributed message queue, and reading the task data streams corresponding to the parallel computing tasks from the distributed message queue in parallel through the task execution nodes; standardizing the task data streams through the stream processing engine to obtain standardized data streams, and storing the standardized data streams in a distributed storage system; the distributed storage system adjusts the allocation of cache resources according to the data write volume and access frequency.

[0133] In one embodiment, the data stream to be processed is a real-time data stream or historical batch data; when the computer program is executed by the processor, the task data stream is standardized by the stream processing engine to obtain a standardized data stream, including: executing standardized data transformation rules through the table application programming interface of the stream processing engine to standardize the task data stream and obtain a standardized data stream.

[0134] In one embodiment, the storage of a standardized data stream to a distributed storage system when a computer program is executed by a processor includes: acquiring low-frequency access data, medium-frequency access data, and high-frequency access data from the standardized data stream based on the user's historical access data; storing the low-frequency access data in the object storage area of ​​the distributed storage system, storing the medium-frequency access data in the solid-state drive cache area of ​​the distributed storage system, and storing the high-frequency access data in the memory cache area of ​​the distributed storage system.

[0135] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the workload and data stream processing rate of the stream processing engine; adjusting the number of container instances corresponding to the job management node and task execution node according to the workload; and adjusting the resources allocated by the cluster to the job management node and task execution node according to the data stream processing rate.

[0136] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: during the standardization process of the task data stream by the stream processing engine, a stream processing state snapshot is generated by the stream processing engine and stored in a distributed storage system; in the event of a failure of the stream processing engine, the stream processing state snapshot is read from the distributed storage system, and the processing state of the stream processing engine is restored based on the stream processing state snapshot, and the process of standardizing the task data stream by the stream processing engine continues.

[0137] In one embodiment, when a computer program is executed by a processor, it involves standardizing a task data stream through a stream processing engine to obtain a standardized data stream. This process includes: determining a time window based on business requirements; the time window includes a window type and a window size; determining the event time window corresponding to the standardized data stream based on the event timestamps carried by the standardized data stream; obtaining the maximum event timestamp carried by the standardized data stream; and obtaining a watermark timestamp based on the maximum event timestamp and an allowed delay threshold; and aggregating the standardized data stream corresponding to the current event time window if the current watermark timestamp exceeds the end time of the current event time window.

[0138] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for the standardized data stream corresponding to the same event time window, obtain the deduplication identifier of the standardized data stream, and map the deduplication identifier to a continuous index value; obtain the hash bucket number according to the deduplication identifier, and allocate a computing node for each hash bucket number through the stream processing engine; when the event time window ends, perform deduplication processing on the standardized data stream according to the continuous index value to obtain a local deduplication result, and summarize the local deduplication results corresponding to the deduplication identifier through all computing nodes to obtain a global deduplication result; and store the standardized data stream to a distributed storage system according to the global deduplication result.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A high-concurrency data stream processing method, characterized in that, The method includes: Data streams to be processed are obtained from multiple data sources and stored in corresponding topics of a distributed message queue according to the data source type; multiple partitions under each topic of the distributed message queue are used together to store the data streams to be processed. The job management node and task execution node are encapsulated as container instances, and the container instances are deployed in a cluster to obtain a stream processing engine; the cluster is used to allocate fixed management resources to the job management node and computing resources to the task execution node; Based on the number of partitions in each topic of the distributed message queue, corresponding parallel computing tasks are generated, and the task execution nodes read the task data stream corresponding to the parallel computing tasks from the distributed message queue in parallel. The task data stream is standardized by the stream processing engine to obtain a standardized data stream, which is then stored in a distributed storage system. The distributed storage system adjusts the allocation of cache resources based on the amount of data written and the access frequency.

2. The method according to claim 1, characterized in that, The data stream to be processed is a real-time data stream or historical batch data; the standardization process of the task data stream by the stream processing engine to obtain a standardized data stream includes: The task data stream is standardized by executing standardized data transformation rules through the table application programming interface of the stream processing engine to obtain a standardized data stream.

3. The method according to claim 1, characterized in that, The step of storing the standardized data stream to a distributed storage system includes: Based on the user's historical access data, obtain low-frequency access data, medium-frequency access data, and high-frequency access data from the standardized data stream; The low-frequency access data is stored in the object storage area of ​​the distributed storage system, the medium-frequency access data is stored in the solid-state drive cache area of ​​the distributed storage system, and the high-frequency access data is stored in the memory cache area of ​​the distributed storage system.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the business load and data stream processing rate of the stream processing engine; Adjust the number of container instances corresponding to the job management node and the task execution node according to the business load; Based on the data stream processing rate, the resources allocated by the cluster to the job management node and the task execution node are adjusted.

5. The method according to claim 1, characterized in that, The method further includes: During the standardization process of the task data stream by the stream processing engine, a stream processing state snapshot is generated by the stream processing engine and stored in the distributed storage system. In the event of a fault in the stream processing engine, the process involves reading the stream processing state snapshot from the distributed storage system, restoring the processing state of the stream processing engine based on the stream processing state snapshot, and continuing to standardize the task data stream through the stream processing engine.

6. The method according to claim 1, characterized in that, The step of standardizing the task data stream using the stream processing engine to obtain a standardized data stream includes: The time window is determined based on business requirements; the time window includes window type and window size. Based on the event timestamps carried by the standardized data stream, determine the event time window corresponding to the standardized data stream; Obtain the maximum event timestamp carried by the standardized data stream, and obtain the watermark timestamp based on the maximum event timestamp and the allowed delay threshold; If the current watermark timestamp exceeds the end time of the current event time window, the standardized data stream corresponding to the current event time window is aggregated.

7. The method according to claim 6, characterized in that, The method further includes: For the standardized data stream corresponding to the same event time window, obtain the deduplication identifier of the standardized data stream and map the deduplication identifier to a continuous index value; The hash bucket number is obtained based on the deduplication identifier, and a computing node is allocated to each hash bucket number by the stream processing engine. When the event time window ends, the standardized data stream is deduplicated according to the continuous index value to obtain a local deduplication result. The local deduplication results corresponding to the deduplication identifiers are then summarized by all computing nodes to obtain a global deduplication result. The standardized data stream is stored in a distributed storage system based on the global deduplication result.

8. A high-concurrency data stream processing device, characterized in that, The device includes: The data stream acquisition module is used to acquire data streams to be processed from multiple data sources and store the data streams to be processed in the corresponding topics of the distributed message queue according to the data source type; multiple partitions under each topic of the distributed message queue are used to store the data streams to be processed. The container encapsulation module is used to encapsulate job management nodes and task execution nodes into container instances, and deploy the container instances on the cluster to obtain a stream processing engine; the cluster is used to allocate fixed management resources to the job management nodes and computing resources to the task execution nodes. The task generation module is used to generate corresponding parallel computing tasks based on the number of partitions of each topic in the distributed message queue, and to read the task data stream corresponding to the parallel computing task from the distributed message queue in parallel through the task execution node; The data stream writing module is used to standardize the task data stream through the stream processing engine to obtain a standardized data stream, and to store the standardized data stream in a distributed storage system; the distributed storage system adjusts the allocation of cache resources according to the amount of data written and the access frequency.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images