Data processing method and device, electronic equipment and nonvolatile storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and nonvolatile storage medium. The method comprises the following steps: obtaining a to-be-processed data stream from a data source, and determining a data processing task corresponding to the to-be-processed data stream; in the case that a processing delay required by the data processing task is lower than a preset delay threshold, distributing the to-be-processed data stream to a first data processing layer; performing target sampling processing on the to-be-processed data stream by using an operator of the first data processing layer to obtain a sampled data stream; and performing a processing operation corresponding to the data processing task on the sampled data stream to obtain a data processing result. The application solves the technical problems of high processing delay and poor expansibility of real-time data stream caused by the deficiencies of full-amount processing or traditional sampling methods in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data analysis, in particular to a data processing method and device, electronic equipment and nonvolatile storage medium. BACKGROUND

[0002] With the development of the Internet of Things and 5G (5th Generation Mobile Communication Technology), the speed of data generation is rapidly increasing, and the traditional batch processing data analysis method is limited because it cannot meet the real-time demand. Although the stream computing framework (such as Apache Flink, Apache Storm, etc.) in the related art can support real-time data stream processing to some extent, there are still deficiencies in processing delay, scalability, resource utilization efficiency and algorithm support.

[0003] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0004] The embodiments of the present application provide a data processing method and device, electronic equipment and nonvolatile storage medium, to at least solve the technical problems of high processing delay and poor scalability of real-time data stream caused by the deficiencies of full processing or traditional sampling method in the related art.

[0005] According to an aspect of the embodiments of the present application, a data processing method is provided, comprising: obtaining a to-be-processed data stream from a data source, and determining a data processing task corresponding to the to-be-processed data stream; in a case where a processing delay required by the data processing task is lower than a preset delay threshold, distributing the to-be-processed data stream to a first data processing layer; using an operator of the first data processing layer to perform target sampling processing on the to-be-processed data stream, to obtain a sampled data stream, wherein the data amount of the sampled data stream is less than the data amount of the to-be-processed data stream, and the relative order of data samples in the sampled data stream is consistent with the relative order of data samples in the to-be-processed data stream; and performing a processing operation corresponding to the data processing task on the sampled data stream, to obtain a data processing result.

[0006] Optionally, the target sampling processing on the to-be-processed data stream comprises: initializing a data storage pool, wherein the data storage pool is used to store selected data samples, each data sample in the to-be-processed data stream comprises: a data item, a first timestamp corresponding to the data item, and a serial number; sequentially obtaining data samples from the to-be-processed data stream, and at each time of obtaining a data sample, clearing data samples in the data storage pool whose difference between a second timestamp and the first timestamp exceeds a preset time window length, wherein the second timestamp is a timestamp corresponding to the time of obtaining the data sample from the to-be-processed data stream; determining a storage strategy according to the storage state of the data storage pool, and storing the newly obtained data sample from the to-be-processed data stream into the data storage pool according to the storage strategy, to finally obtain a sampled data stream.

[0007] Optionally, storing the newly obtained data sample from the to-be-processed data stream into the data storage pool according to the storage strategy comprises: in a case where the amount of data samples already stored in the data storage pool is less than the data sample capacity of the data storage pool, inserting the newly obtained data sample from the to-be-processed data stream into the data storage pool in the order of the size of the serial number; and in a case where the amount of data samples already stored in the data storage pool is not less than the data sample capacity, performing a replacement operation of the data sample according to a target replacement rule, wherein the replacement operation is used to select a data sample in the data storage pool and determine whether to replace the selected data sample in the data storage pool with the newly obtained data sample from the to-be-processed data stream.

[0008] Optionally, performing the replacement operation of the data sample according to the target replacement rule comprises: determining a basic replacement probability according to the data sample capacity and the total number of data samples already obtained from the to-be-processed data stream; determining an aging parameter corresponding to each data sample in the data storage pool according to the first timestamp of each data sample, wherein the aging parameter is used to represent the size of the difference between the current timestamp and the first timestamp of the data sample, the larger the difference, the larger the aging parameter, and the worse the aging of the data sample is represented; determining a weighted probability distribution of the data samples in the data storage pool according to the aging parameter, wherein the weighted probability distribution is used to represent the probability of each data sample in the data storage pool being selected, the larger the aging parameter, the larger the probability of the data sample being selected; randomly selecting a data sample in the data storage pool as a replacement target according to the weighted probability distribution, and randomly replacing the selected replacement target with the newly obtained data sample from the to-be-processed data stream according to the basic replacement probability.

[0009] Optionally, the operator is a basic processing unit for data processing in the stream processing system; the method further includes: monitoring running indexes corresponding to the plurality of operators in the system, wherein the running indexes are used to represent load states of the operators; predicting system load trend data corresponding to a second time period according to the running indexes in a first time period, wherein the second time period is a time period immediately after the first time period; and adjusting parallelism degrees of the plurality of operators according to the predicted system load trend data, so as to optimize processing capacity and resource utilization of the system, wherein the parallelism degree is used to represent a number of instances of the operator running simultaneously in the system.

[0010] Optionally, the running indexes include distribution data of keys of a key business, wherein the key is used to uniquely identify data or group data; the method further includes: determining whether there is a data skew problem in the system by analyzing the distribution data of the keys, wherein the data skew problem includes at least one of the following: a data amount and / or processing time corresponding to a key exceeds an average value by more than a preset range; and in a case where it is determined that there is a data skew problem in the system, processing a data processing task corresponding to a key causing the data skew problem by using a skew mitigation strategy, wherein the skew mitigation strategy is used to optimize load balancing of the data processing task to alleviate the data skew problem; and the skew mitigation strategy includes at least one of the following: a two-stage aggregation strategy, a broadcast small table strategy, and a directional scheduling strategy, the two-stage aggregation strategy is used to first add a random salt value to the key causing the data skew problem to scatter to a plurality of instances, and then combine results after local aggregation, the broadcast small table strategy is used to broadcast a small table to reduce redistribution pressure of a large table in a case where the data skew problem is caused by a key of a small table hotspot in a join operation, and the directional scheduling strategy is used to schedule the data processing task corresponding to the key causing the data skew problem to an operator with the lowest load.

[0011] Optionally, the method further includes: determining a hardware acceleration requirement of a data processing task corresponding to the to-be-processed data stream, wherein the hardware acceleration requirement is used to represent a dependency requirement of the data processing task on a specific type of hardware resource; obtaining a load state of a hardware resource corresponding to the hardware acceleration requirement, and determining a scheduling score of the hardware resource in a heterogeneous hardware resource pool according to a task feature of the data processing task and the load state, wherein the hardware resource includes at least one of the following: a graphics processing unit and a field programmable gate array, and the scheduling score is used to represent availability and execution efficiency of the hardware resource for the data processing task under the load state; determining a hardware resource with the highest scheduling score in the heterogeneous hardware resource pool as a target resource, and scheduling the data processing task to the target resource for execution.

[0012] Optionally, the method further comprises: in a case that a processing delay required by the data processing task is higher than a preset delay threshold, or a complexity of the data processing task is higher than a preset complexity threshold, distributing the to-be-processed data stream to a second data processing layer; and performing, by an operator of the second data processing layer, a processing operation corresponding to the data processing task on the to-be-processed data stream to obtain the data processing result.

[0013] According to another aspect of the embodiments of the present application, a data processing apparatus is further provided, comprising: a data stream acquisition module, configured to acquire a to-be-processed data stream from a data source and determine a data processing task corresponding to the to-be-processed data stream; a data stream distribution module, configured to distribute the to-be-processed data stream to a first data processing layer in a case that a processing delay required by the data processing task is lower than a preset delay threshold; a data stream sampling module, configured to perform target sampling processing on the to-be-processed data stream by an operator of the first data processing layer to obtain a sampled data stream, wherein a data amount of the sampled data stream is smaller than a data amount of the to-be-processed data stream, and a relative order of data samples in the sampled data stream is consistent with a relative order of data samples in the to-be-processed data stream; and a data task processing module, configured to perform a processing operation corresponding to the data processing task on the sampled data stream to obtain a data processing result.

[0014] According to still another aspect of the embodiments of the present application, an electronic device is further provided, comprising: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program performs a data processing method when running.

[0015] According to yet another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, comprising a stored computer program, wherein a device in which the non-volatile storage medium is located performs a data processing method by running the computer program.

[0016] According to still another aspect of the embodiments of the present application, a computer program product is further provided, comprising a computer program, the computer program being configured to implement steps of a data processing method when executed by a processor.

[0017] In the embodiment of the present application, the data stream to be processed from the data source is obtained, and the data processing task corresponding to the data stream to be processed is determined; in the case that the processing delay required by the data processing task is lower than the preset delay threshold, the data stream to be processed is distributed to the first data processing layer; the target sampling processing of the data stream to be processed is performed by using the operator of the first data processing layer, and the sampled data stream is obtained, wherein the data amount of the sampled data stream is less than the data amount of the data stream to be processed, and the relative order of the data samples in the sampled data stream is consistent with the relative order of the data samples in the data stream to be processed; the processing operation corresponding to the data processing task is performed on the sampled data stream to obtain the data processing result. Through the layered stream computing architecture and the improved stream data sampling algorithm considering timeliness and order preservation, the high-speed incoming data is efficiently and representatively processed, the purpose of reducing the delay of stream data processing and balancing the low-delay response and the deep complex analysis requirement is achieved, and the technical problems of high processing delay and poor scalability of real-time data stream caused by the deficiency of full processing or traditional sampling method in the related art are solved. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and its description, which serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0019] Figure 1 is a hardware structure block diagram of a computer terminal (or electronic equipment) for implementing a data processing method according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a data processing method flow according to an embodiment of the present application;

[0021] Figure 3 is a structural schematic diagram of a real-time big data analysis processing system based on a layered stream computing framework according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a layered stream computing architecture according to an embodiment of the present application;

[0023] Figure 5 is a flow schematic diagram of a TOPS sampling algorithm according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a dynamic load balancing mechanism according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of a distributed processing combined with edge computing according to an embodiment of the present application;

[0026] Figure 8 is a structural schematic diagram of a data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0029] In order to facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns related to the embodiments of the present application are explained as follows:

[0030] Stream Computing: a computing method that processes data in units of data streams. Data is continuously generated and flows, and a computing engine continuously processes it.

[0031] Apache Flink: a distributed computing framework for stream processing and batch processing.

[0032] Reservoir Sampling: a probability sampling algorithm that ensures that each element has the same probability of being retained in the case of continuous growth of data stream.

[0033] Flink SQL: an SQL query interface in the Flink framework, supporting query and analysis of stream data and static data.

[0034] Heterogeneous Computing: refers to the integration of different types of computing resources, such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and FPGA (Field Programmable Gate Array), etc., for optimizing task processing.

[0035] In related technologies, the stream processing system may have the following problems when performing large-scale real-time data stream processing:

[0036] 1) Due to the lack of full processing or traditional sampling method, the processing delay is too high, and the key timing information and recent data features are lost;

[0037] 2) In related technologies, when the stream processing system faces complex conditions such as dramatic fluctuations in data traffic and skewed data distribution, its load balancing mechanism responds slowly and the adjustment strategy is rough, which cannot quickly respond to sudden data peaks, and the expansion requires several minutes, affecting real-time performance, and causing poor system stability and low expansion efficiency;

[0038] 3) In scenarios requiring millisecond-level response, existing frameworks often have to choose between low latency and complex analysis (especially machine learning model inference) requirements, and cannot meet both requirements;

[0039] 4) In related technologies, the homogeneous computing resource allocation mode or the simple heterogeneous resource manual configuration method cannot match the needs of different types of tasks, resulting in resource waste and low utilization, especially when processing compute-intensive or specific accelerator optimization tasks, the performance bottleneck is obvious.

[0040] To solve the above problems, the related solutions are provided in the embodiments of the present application, which are described in detail below.

[0041] According to the embodiments of the present application, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from here.

[0042] The method embodiment provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or electronic equipment) for implementing the data processing method is shown. As Figure 1As shown, the computer terminal 10 (or electronic device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0043] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or electronic device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0045] The transmission device 106 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module that is configured to communicate with the Internet wirelessly.

[0046] The display can be a liquid crystal display (LCD) that is touch screen type, for example, which can enable a user to interact with a user interface of the computer terminal 10 (or the electronic device).

[0047] In the above operating environment, the embodiments of the present application provide a data processing method, Figure 2 is a schematic diagram of a method flow of a data processing method according to the embodiments of the present application, as shown in the figure, the method includes the following steps: Figure 2

[0048] In step S202, a to-be-processed data stream from a data source is obtained, and a data processing task corresponding to the to-be-processed data stream is determined.

[0049] In step S204, the to-be-processed data stream is distributed to the first data processing layer when a processing delay required by the data processing task is lower than a preset delay threshold.

[0050] In step S206, an operator of the first data processing layer is used to perform target sampling processing on the to-be-processed data stream to obtain a sampled data stream, wherein a data amount of the sampled data stream is smaller than a data amount of the to-be-processed data stream, and a relative order of data samples in the sampled data stream is consistent with a relative order of data samples in the to-be-processed data stream.

[0051] In step S208, a processing operation corresponding to the data processing task is performed on the sampled data stream to obtain a data processing result.

[0052] Through the above steps, through the layered stream computing architecture and using an improved stream data sampling algorithm that takes into account timeliness and order preservation, high-speed incoming data is efficiently and representatively processed, the purpose of reducing the delay of stream data processing and balancing low-delay response and deep complex analysis requirements is achieved, and thus the technical problems of high processing delay of real-time data stream and poor scalability caused by the deficiencies of full-volume processing or traditional sampling methods in related technologies are solved.

[0053] ​The real-time big data analysis processing method based on stream computing provided in the embodiments of the present application can be applied to various scenarios that need to process large-scale real-time data streams, such as financial transactions, network security, intelligent manufacturing, and smart cities; the data processing method in steps S202 to S208 of the embodiments of the present application is further introduced below.

[0054] Figure 3 is a structural schematic diagram of a real-time big data analysis processing system based on a hierarchical stream computing framework provided according to the embodiments of the present application, as Figure 3 indicated, the system architecture mainly includes:

[0055] 1) data source: such as Internet of Things devices, sensor networks, transaction systems, user behavior logs, network traffic probes, etc., continuously generating heterogeneous real-time data streams;

[0056] 2) data access layer: a high-throughput, low-latency message queue system (such as Apache Kafka) can be used, responsible for reliable reception, buffering, and preliminary distribution of data.

[0057] 3) real-time processing layer (first data processing layer): based on stream processing engines such as Apache Flink, the core runs the target sampling processing TOPS (Time-aware Order-Preserving Reservoir Sampling) algorithm proposed in the embodiments of the present application, and performs computing tasks that require extremely low latency, such as real-time filtering, simple aggregation, and basic pattern detection;

[0058] 4) near-real-time processing layer (second data processing layer): also based on Apache Flink or combined with other distributed computing frameworks (such as Spark Streaming), processes more complete data streams or data processed by specific strategies from the data access layer, and performs complex correlation analysis, machine learning model inference, deep learning training, etc.

[0059] 5) resource manager: built-in predictive skew-aware heterogeneous load balancer PSHLB (Predictive Skew-aware Heterogeneous Load Balancer), responsible for global monitoring, intelligent decision-making, dynamic resource allocation, task scheduling (including cross-layer, cross-heterogeneous hardware), and load balancing;

[0060] 6) heterogeneous computing resource pool: including CPU clusters, GPU server clusters (for accelerating parallel computing and AI tasks), and possibly FPGA cards (for accelerating specific algorithms);

[0061] 7) Results storage and display system: used to persistently store analysis results (such as time series databases and NoSQL databases) and provide real-time dashboards, reports and other visual interfaces.

[0062] In this embodiment, the aforementioned components are interconnected via a high-speed network and, under the unified coordination of PSHLB, constitute a highly efficient, intelligent, and flexible distributed real-time big data analysis and processing system. The processing flow of this system is described below.

[0063] First, a data access layer is used to continuously receive real-time data streams (data streams to be processed) from various data sources. For example, Kafka Connectors or custom producers can be used to access data from different sources. Then, the data streams to be processed can be preprocessed, including but not limited to: performing deserialization, basic verification, and timestamp injection / calibration.

[0064] In this embodiment, a layered streaming computing architecture can be adopted, based on the data processing tasks corresponding to the data stream to be processed, such as... Figure 4 As shown, the data processing task is clearly divided into a real-time processing layer (first data processing layer) and a near-real-time processing layer (second data processing layer), aiming to effectively balance the two different requirements of extremely low latency response and deep, complex analysis. For example, according to a preset strategy (such as Topic routing), the raw or lightly processed data stream can be distributed to the Kafka Topic corresponding to the real-time processing layer and / or the near-real-time processing layer. In this embodiment, to support order preservation, a monotonically increasing sequence number can be added to the sample data in the data stream.

[0065] Specifically, when the processing latency required by the data processing task is lower than a preset latency threshold (such as performing computational tasks requiring extremely low latency, such as real-time filtering, simple aggregation, basic pattern detection, etc.), the data stream to be processed is distributed to the first data processing layer. In the first data processing layer, target sampling processing TOPS is used to efficiently process the rapidly flowing data. However, when the processing latency required by the data processing task is higher than a preset latency threshold, or when the complexity of the data processing task is higher than a preset complexity threshold (such as performing complex correlation analysis, machine learning model inference, deep learning training, etc.), the data stream to be processed is distributed to the second data processing layer. Complex analytical logic, including data mining and machine learning algorithms, is integrated using more comprehensive data stream data (such as unsampled or sampled using different strategies) to obtain deeper business insights.

[0066] The TOPS target sampling algorithm used in the real-time processing layer (first data processing layer) will be described in detail below.

[0067] In the real-time processing layer (first data processing layer), after the operator receives the data stream to be processed from the corresponding Topic in the data access layer, it applies the Time-Aware Order-Preserving Reservoir Sampling (TOPS) algorithm (i.e., target sampling processing algorithm) to sample the input data stream. This algorithm aims to efficiently extract representative samples from large-scale data streams while preserving the temporal information and recent features of the samples, so as to overcome the shortcomings of traditional sampling methods in related technologies in preserving recent data features and maintaining event order. The specific algorithm steps are as follows.

[0068] In some embodiments of this application, target sampling processing of the data stream to be processed includes: initializing a data storage pool, wherein the data storage pool is used to store selected data samples, and each data sample in the data stream to be processed includes: a data item, a first timestamp corresponding to the data item, and a sequence number; sequentially acquiring data samples from the data stream to be processed, and clearing data samples in the data storage pool whose difference between the second timestamp and the first timestamp exceeds a preset time window length each time a data sample is acquired, wherein the second timestamp is the timestamp corresponding to the moment when the data sample is acquired from the data stream to be processed; determining a storage strategy based on the storage state of the data storage pool, and storing newly acquired data samples from the data stream to be processed into the data storage pool according to the storage strategy, ultimately obtaining a sampled data stream.

[0069] Specifically, such as Figure 5 As shown, initialization is performed first: a reservoir (i.e., data storage pool) R of fixed size (i.e., data sample capacity) k is maintained to store the selected data samples. In this embodiment, each data sample is stored in the form of a tuple (including: data item, first timestamp, and sequence number); at the same time, a counter (current index) is maintained to record the total number of data samples processed; and a time window W is set to limit the time range of data processing in stream processing.

[0070] When performing target sampling on the data stream, data samples are sequentially obtained from the data stream to be processed, and their first timestamp and sequence number are obtained. Each time a data sample is obtained, the counter (current index) is incremented by 1. At the same time, expired sample data in the reservoir (i.e., data storage pool) is cleaned up, and samples in the reservoir (i.e., data storage pool) whose first timestamp is earlier than (current time - W) are removed to ensure that the samples are within the time window, that is, except for data samples in the data storage pool whose difference between the second timestamp and the first timestamp exceeds the preset time window length.

[0071] Then, based on whether the storage pool (i.e., the data storage pool) is full, the newly acquired data samples from the data stream to be processed can be stored in the data storage pool according to different storage strategies. The specific steps are as follows.

[0072] In some embodiments of the present application, storing the newly acquired data sample from the to-be-processed data stream into the data storage pool according to the storage strategy comprises: in a case where the amount of the data samples already stored in the data storage pool is less than the data sample capacity of the data storage pool, inserting the newly acquired data sample from the to-be-processed data stream into the data storage pool according to the size order of the sequence numbers; and in a case where the amount of the data samples already stored in the data storage pool is not less than the data sample capacity, performing a replacement operation of the data samples according to the target replacement rule, wherein the replacement operation is used to select one data sample in the data storage pool and determine whether to replace the selected data sample in the data storage pool with the newly acquired data sample from the to-be-processed data stream.

[0073] Specifically, if the amount R of the data samples already stored in the data storage pool is less than the data sample capacity K of the data storage pool, the new data sample can be inserted into the data storage pool in the order of the sequence numbers, so as to maintain the relative order of the samples in the data storage pool; if the amount R of the data samples is not less than the data sample capacity K, i.e., in a case where the data storage pool is full, the replacement of the data samples is needed, and the specific steps are as follows.

[0074] In some embodiments of the present application, performing the replacement operation of the data samples according to the target replacement rule comprises the following steps: determining a basic replacement probability according to the data sample capacity and the total number of the data samples already acquired from the to-be-processed data stream; determining an aging parameter corresponding to each data sample in the data storage pool according to the first time stamp of each data sample, wherein the aging parameter is used to represent the difference between the current time stamp and the first time stamp of the data sample, and the larger the difference is, the larger the aging parameter is, and the worse the aging of the data sample is represented; determining a weighted probability distribution corresponding to the data samples in the data storage pool according to the aging parameter, wherein the weighted probability distribution is used to represent the probability of each data sample in the data storage pool being selected, and the larger the aging parameter is, the larger the probability of the data sample being selected is; selecting a data sample in the data storage pool as a replacement target according to the weighted probability distribution, and randomly replacing the selected replacement target with the newly acquired data sample from the to-be-processed data stream according to the basic replacement probability.

[0075] Specifically, first, a base replacement probability p_base = data sample capacity k / current index (i.e. the total number of data samples that have been obtained from the data stream to be processed) is calculated; then, according to the first timestamp of each data sample in the data storage pool, the timeliness parameter (similar to "age", which can be expressed as the current timestamp - the first timestamp of the sample) corresponding to each data sample is determined; then, the weighted probability distribution of the data samples in the data storage pool can be determined according to the timeliness parameter. In this embodiment, a weighting function (for example, the greater the timeliness parameter, the higher the weight) can be designed to adjust the probability of each sample data being selected as a replacement candidate according to its timeliness parameter, with the goal of making sample data with a larger timeliness parameter (i.e. worse timeliness) have a slightly higher probability than p_base of being selected, while sample data with a smaller timeliness parameter have a slightly lower probability than p_base of being selected, but the overall expected replacement probability still tends to p_base.

[0076] After the weighted probability distribution is determined, a data sample with index j in the data storage pool can be randomly selected as a potential replacement target according to the weighted probability distribution; then, it is determined whether to perform replacement with a probability of p_base (or according to the weighted comprehensive probability). If it is decided to replace, the sample with index j in R is replaced with a new data sample, and it is ensured that the relative order by sequence number is maintained in R after replacement (the position may need to be adjusted). After processing by the TOPS algorithm, a representative sample stream (i.e. a sampled data stream) with a size not exceeding k, containing recent (within the time window W), and maintaining the original relative order can be finally output.

[0077] The TOPS algorithm described above ensures that the final sample set strictly maintains the relative time order of samples in the original data stream by introducing and comparing the sequence identifiers of data items during reservoir sampling, solving the problem of complete loss of order information in traditional reservoir sampling. In combination with the timeliness weighted replacement mechanism, when the reservoir is full and samples need to be replaced, it is not a random replacement with equal probability, but a dynamic weighting mechanism based on the "age" of the sample (the length of time from the current time) is introduced, so that recently arrived data items have a higher probability of entering or remaining in the sample pool, thereby enabling the sample set to more sensitively reflect the latest state and short-term trend of the data stream. At the same time, in cooperation with the time window, the sample is finally constrained by the sliding time window, ensuring that the sample set always represents the effective data within the specified time range, and the weighted replacement mechanism optimizes the representativeness of the samples within the window. Through the TOPS algorithm, the recent dynamics and event order information, which are crucial for real-time analysis, can be more effectively preserved while significantly reducing the amount of data.

[0078] The embodiments of the present application significantly reduce the amount of data that needs to be processed in real time through the TOPS sampling algorithm, thereby greatly reducing the processing delay, while preserving the time sequence correlation information and recent data characteristics that are essential for real-time analysis (especially complex event processing (CEP) that relies on order and anomaly detection that requires rapid response to recent changes) to the maximum extent through order preservation and timeliness weighting; enhances dynamic adaptability, can quickly respond to changes in data flow rate and content mutations, and the sample set can be quickly updated to reflect the latest data distribution and pattern; has computational efficiency, keeps the core operation of the algorithm at a low time complexity (approximately O(1) or O(log k) depending on the order preservation implementation), and is fully applicable to high-speed data stream processing scenarios. Compared with related technologies, which often have to compromise between low latency (by sacrificing information, such as losing order or oversampling) and high information fidelity (by processing full-amount data at the expense of latency). TOPS achieves a better balance between the two through its unique mechanism, especially suitable for real-time big data analysis applications that have strict requirements for both timeliness and timeliness.

[0079] After obtaining the sampled data stream by performing target sampling processing on the to-be-processed data stream, the sampled data stream can be further processed to obtain the final data processing result.

[0080] In the present embodiment, after receiving the sampled data stream output from the TOPS sampling operator, the subsequent operators (such as window operators) in the first data processing layer can be used for real-time data aggregation. Based on the ordered sample stream with higher information density after TOPS sampling, sliding time window or rolling time window technology is used for aggregation calculation (such as counting, summation, average value), or hash index technology can also be applied to speed up window lookup, further reducing the delay of aggregation calculation.

[0081] Further, complex event processing (CEP) corresponding to the data processing task can be performed on the aggregated result stream or the TOPS-sampled sample stream directly received, for example, Flink CEP library or custom ProcessFunction can be used to define and detect complex event patterns (such as specific sequence of transaction behavior, continuous occurrence of device abnormal state, etc.) on the ordered and timeliness strong sample stream. Once a matching pattern is detected, an alarm is generated or a downstream action is triggered.

[0082] The embodiments of the present application use the TOPS sampling algorithm to compress the end-to-end delay to milliseconds while preserving the recent dynamic characteristics of the data stream and the relative order of event occurrence, so that the accuracy and depth of real-time analysis are significantly improved, which is much better than the traditional sampling or full-amount processing scheme in related technologies.

[0083] In addition, a dynamic load balancing mechanism based on multi-dimensional index prediction and resource perception (predictive skew perception heterogeneous load balancer PSHLB) is introduced in the embodiments of the present application. The dynamic load balancing mechanism can adjust the computing resource allocation, task scheduling and parallel strategy actively and intelligently according to multi-dimensional refined indexes such as real-time characteristics (rate, distribution) of data flow, operator processing state (delay, queue), system resource (CPU / GPU / FPGA load) and in combination with short-term load prediction, so as to solve key problems such as response lag, poor skew processing and extensive utilization of heterogeneous resources in the existing stream processing system. The following will be specifically introduced.

[0084] In some embodiments of the present application, the operator is a basic processing unit for data processing in the stream processing system; the method further comprises the following steps: monitoring running indexes corresponding to a plurality of operators in the system, wherein the running indexes are used to represent the load state of the operators; predicting system load trend data corresponding to a second time period according to the running indexes in a first time period, wherein the second time period is a time period immediately after the first time period; and adjusting the parallelism of the plurality of operators according to the predicted system load trend data, so as to optimize the processing capacity and resource utilization of the system, wherein the parallelism is used to represent the number of instances of the operator running simultaneously in the system.

[0085] Specifically, the resource manager (built-in predictive skew perception heterogeneous load balancer PSHLB) can periodically (for example, every 5 seconds) or event-driven based on key indexes (such as operator delay exceeding threshold, queue length surging) to monitor and predict the state of the system in multiple dimensions, such as Figure 6 As shown in FIG. 2, real-time running indexes of the entire Flink cluster and each operator can be continuously collected, including but not limited to: input / output rate, end-to-end delay, CPU / memory / network usage, TaskManager internal queue length and waiting time, Checkpoint state and time consumption, distribution of key business keys (through sampling or approximate calculation), and real-time load and availability of GPU / FPGA and other heterogeneous resources.

[0086] Then, a light time series model (such as exponential smoothing, ARIMA) can be used to predict the system load trend in a short time window (second time period, such as the next 1-5 minutes) in the future based on historical input rate, periodic pattern (day / week pattern) and other data (i.e. running indexes in the first time period); and the adjustment (increase or decrease) of the operator parallelism is triggered in advance according to the predicted load trend rather than only the current load. For example, if a traffic peak is predicted, the parallelism of the related operator is increased in advance to avoid delay soaring due to insufficient resources when the peak arrives. In the embodiments, the parallelism of the job operator can be dynamically adjusted through the Rescaling API of Flink.

[0087] By making the decision of scaling the operator parallelism in advance based on the predicted load trend, rather than waiting for the load to actually change and then responding passively, the delay spikes and the hysteresis and oscillation of resource allocation are effectively avoided.

[0088] Meanwhile, the KEY distribution data of the key business can be monitored to automatically perform the tilt processing, and the tilt processing specifically includes the following.

[0089] In some embodiments of the present application, the running index includes: distribution data of a key of a key business, wherein the key is used to uniquely identify data or group data; the method further includes: analyzing the distribution data of the key to determine whether there is a data tilt problem in the system, wherein the data tilt problem includes at least one of the following: the data volume corresponding to the key and / or the processing time exceeds the average value by more than a preset range; in the case where it is determined that there is a data tilt problem in the system, a tilt relief strategy is used to process the data processing task corresponding to the key that causes the data tilt problem, wherein the tilt relief strategy is used to optimize the load balancing of the data processing task to alleviate the data tilt problem; wherein the tilt relief strategy includes at least one of the following: a two-stage aggregation strategy, a broadcast small table strategy, and a directional scheduling strategy, the two-stage aggregation strategy is used to first add a random salt value to the key that causes the data tilt problem to scatter to multiple instances, and then combine the results after local aggregation, the broadcast small table strategy is used to reduce the redistribution pressure of the large table by broadcasting the small table in the case where the data tilt problem is caused by the key of the small table hotspot in the join operation, and the directional scheduling strategy is used to schedule the data processing task corresponding to the key that causes the data tilt problem to the operator with the lowest load.

[0090] Specifically, the Key distribution index can be analyzed in real time, and once the data tilt (for example, the data volume / processing time of a certain Key is much higher than the average value) is detected, the tilt relief strategy is automatically triggered. In this embodiment, the optional tilt relief strategy includes but is not limited to: 1) two-stage aggregation: first, add a random salt (Salt) to the tilt Key to scatter to multiple instances, and then combine the results after local aggregation; 2) broadcast small table: if the tilt is caused by the small table hotspot Key in the join operation, try to broadcast the small table; 3) directional scheduling: preferentially schedule the subtask processing the tilt Key to the TaskManager with lower current load or stronger computing power.

[0091] By building real-time detection logic for data tilt and automatically triggering the corresponding relief strategy, such as adding salt to the hotspot Key for re-partitioning, dynamically adjusting task allocation to disperse pressure, or directing the tilt processing task to a specific node, the stable operation ability and overall throughput of the system in the face of severe load fluctuations and uneven data distribution are improved.

[0092] In addition, the embodiments of the present application can also integrate CPU, GPU, FPGA and other heterogeneous computing resources, and through the intelligent scheduling capability built in the PSHLB dynamic load balancing mechanism, the specific type (such as rule matching, model inference, complex calculation) and real-time state of the analysis task are analyzed, and the task or subtask is automatically scheduled to the most suitable hardware accelerator for execution, thereby significantly improving the processing performance of the specific task and the overall system efficiency. The specific steps are as follows.

[0093] In some embodiments of the present application, the method further comprises the steps of: determining the hardware acceleration requirement of the data processing task corresponding to the to-be-processed data stream, wherein the hardware acceleration requirement is used to represent the dependency requirement of the data processing task on a specific type of hardware resource; obtaining the load state of the hardware resource corresponding to the hardware acceleration requirement, and determining the scheduling score of the hardware resource in the heterogeneous hardware resource pool according to the task characteristics of the data processing task and the load state, wherein the hardware resource comprises at least one of the following: a graphics processing unit, a field programmable gate array, and the scheduling score is used to represent the availability and execution efficiency of the hardware resource for the data processing task under the load state; determining the hardware resource with the highest scheduling score in the heterogeneous hardware resource pool as the target resource, and scheduling the data processing task to the target resource for execution.

[0094] Specifically, first, the hardware acceleration requirement of the data processing task corresponding to the to-be-processed data stream is determined. For tasks marked as requiring specific hardware acceleration (for example, a certain operator is specified to be suitable for GPU acceleration through code annotation or configuration), the PSHLB queries the state of the heterogeneous resource pool. If the target resource is available and the load is reasonable, the operator or its subtask is scheduled to the GPU / FPGA for execution through a specific task scheduling interface (such as an extended Flink scheduler or an external collaboration framework). If the resource is not available or is overloaded, it is returned to the CPU for execution or waits. In this embodiment, the scheduling and resource allocation of heterogeneous tasks can be performed through the interface interaction with the heterogeneous computing resource management system (such as the device plugin of Kubernetes, the GPU scheduling support of YARN) or the specific framework (such as TensorFlow on Flink).

[0095] Through the task-aware heterogeneous resource intelligent scheduling, the computing characteristics of the task (such as being marked as "AI inference", "parallel computing intensive") can be identified, and the real-time availability and load of the heterogeneous hardware (GPU / FPGA) are combined to intelligently and dynamically schedule the appropriate task or subtask to the optimal hardware accelerator for execution. Thus, the resource utilization efficiency and cost-effectiveness are maximized, and various computing resources such as CPU, GPU, FPGA are used on demand and efficiently, avoiding the bottleneck or idling of a single type of resource, and significantly improving the hardware return on investment.

[0096] The PSHLB mechanism with predictive, tilt-awareness and supporting heterogeneous resources in the embodiments of the present application enables the system to actively and intelligently and finely cope with load fluctuation, data tilt and diversified computing requirements, and guarantees the stability, high throughput and resource utilization of the system under high pressure, and the comprehensive effect exceeds the resource management capability of the stream processing framework itself or simple combination in the related art.

[0097] In addition, in the embodiments of the present application, distributed intelligent processing combined with edge computing can also be implemented, as shown in Figure 7 The lightweight sampling idea of TOPS is applied to the edge node for preprocessing, and the monitoring and decision range of PSHLB is extended to edge-cloud resources to realize global collaborative scheduling; the introduction of edge computing significantly reduces the amount of data transmitted to the cloud through local preprocessing (TOPS idea is applied) and intelligent filtering, reduces network delay and bandwidth cost, thereby further reducing end-to-end delay and saving core network bandwidth; at the same time, edge-cloud collaborative computing, combined with the extended edge-aware PSHLB intelligent scheduling, can optimize task layout according to the global state, improve the response speed to local events, and provide certain local processing capability when the network is unstable; and partial data processing at the edge can reduce the need to transmit sensitive raw data to the cloud, protect data privacy, and help meet the compliance requirements of specific industries.

[0098] The TOPS algorithm is applied to the real-time processing layer to realize efficient and low-delay processing, and the innovative PSHLB mechanism is used as the central coordinator of the entire layered system to uniformly manage the load balancing, task scheduling and resource allocation of the real-time layer, near-real-time layer and underlying heterogeneous resources; the real-time layer focuses on quickly extracting key timing information with the help of TOPS, and the near-real-time layer can use stronger resources (including heterogeneous hardware) for in-depth analysis under the intelligent scheduling of PSHLB. PSHLB can perform differentiated and priority-aware resource allocation according to the different SLA requirements of the two layers and the global resource status, and achieve global optimization; the entire layered system can simultaneously achieve extremely low processing delay (real-time layer) and powerful complex analysis capability (near-real-time layer) which is difficult to achieve by traditional single architecture or simple layering, effectively solving the limitations of related art in balancing real-time, analysis depth and resource efficiency. These technical effects enable the present scheme to provide efficient and reliable solutions far exceeding related art in application scenarios such as financial real-time risk control, network security threat detection, intelligent manufacturing process monitoring, smart city traffic management, which have extremely high requirements for data processing real-time, accuracy and complexity.

[0099] According to the embodiments of the present application, an embodiment of a data processing device is also provided. Figure 8 is a structural schematic diagram of a data processing device according to the embodiments of the present application. As shown inFigure 8 As shown, the apparatus comprises:

[0100] a data stream acquisition module 80, configured to acquire a to-be-processed data stream from a data source, and determine a data processing task corresponding to the to-be-processed data stream;

[0101] a data stream distribution module 82, configured to distribute the to-be-processed data stream to a first data processing layer in a case where a processing delay required by the data processing task is lower than a preset delay threshold;

[0102] a data stream sampling module 84, configured to perform target sampling processing on the to-be-processed data stream by using an operator of the first data processing layer, to obtain a sampled data stream, wherein a data amount of the sampled data stream is smaller than a data amount of the to-be-processed data stream, and a relative order of data samples in the sampled data stream is consistent with a relative order of data samples in the to-be-processed data stream;

[0103] a data task processing module 86, configured to perform a processing operation corresponding to the data processing task on the sampled data stream, to obtain a data processing result.

[0104] Optionally, the target sampling processing on the to-be-processed data stream comprises: initializing a data storage pool, wherein the data storage pool is used to store selected data samples, and each data sample in the to-be-processed data stream comprises: a data item, a first timestamp corresponding to the data item, and a serial number; sequentially acquiring data samples from the to-be-processed data stream, and clearing, at each time of acquiring a data sample, data samples in the data storage pool whose second timestamp and the first timestamp have a difference exceeding a preset time window length, wherein the second timestamp is a timestamp corresponding to a time of acquiring the data sample from the to-be-processed data stream; determining a storage strategy according to a storage state of the data storage pool, and storing a newly acquired data sample from the to-be-processed data stream into the data storage pool according to the storage strategy, to finally obtain the sampled data stream.

[0105] Optionally, the storing the newly acquired data sample from the to-be-processed data stream into the data storage pool according to the storage strategy comprises: in a case where an amount of data samples already stored in the data storage pool is smaller than a data sample capacity of the data storage pool, inserting the newly acquired data sample from the to-be-processed data stream into the data storage pool according to a size order of the serial numbers; and in a case where the amount of data samples already stored in the data storage pool is not smaller than the data sample capacity, performing a replacement operation of data samples according to a target replacement rule, wherein the replacement operation is used to select a data sample in the data storage pool, and determine whether to replace the selected data sample in the data storage pool with the newly acquired data sample from the to-be-processed data stream.

[0106] Optionally, according to the target replacement rule, the replacement operation of the data sample comprises: determining a basic replacement probability according to a data sample capacity and a total number of data samples that have been obtained from the to-be-processed data stream; determining an aging parameter corresponding to each data sample in the data storage pool according to a first timestamp of each data sample, wherein the aging parameter is used to represent a difference between a current timestamp and the first timestamp of the data sample, and the larger the difference is, the larger the aging parameter is, and the worse the aging of the data sample is represented; determining a weighted probability distribution corresponding to the data sample in the data storage pool according to the aging parameter, wherein the weighted probability distribution is used to represent a probability of each data sample in the data storage pool being selected, and the larger the aging parameter is, the larger the probability of the data sample being selected is; and randomly selecting a data sample in the data storage pool as a replacement target according to the weighted probability distribution, and randomly replacing the selected replacement target with a newly obtained data sample from the to-be-processed data stream according to the basic replacement probability.

[0107] Optionally, the operator is a basic processing unit for data processing in the stream processing system; and the data processing apparatus is further configured to: monitor running indexes corresponding to a plurality of operators in the system, wherein the running indexes are used to represent load states of the operators; predict system load trend data corresponding to a second time period according to the running indexes in a first time period, wherein the second time period is a time period immediately after the first time period; and adjust parallelism degrees of the plurality of operators according to the predicted system load trend data, so as to optimize processing capability and resource utilization of the system, wherein the parallelism degree is used to represent a number of instances of the operator running in the system at the same time.

[0108] Optionally, the running indexes comprise distribution data of keys of a key business, wherein the key is used to uniquely identify data or group data; and the data processing apparatus is further configured to: determine whether there is a data skew problem in the system by analyzing the distribution data of the keys, wherein the data skew problem comprises at least one of the following: a data amount and / or processing time corresponding to a key exceeds an average value by more than a preset range; and in a case where it is determined that there is a data skew problem in the system, process a data processing task corresponding to a key causing the data skew problem by using a skew mitigation strategy, wherein the skew mitigation strategy is used to optimize load balancing of the data processing task, so as to mitigate the data skew problem; and the skew mitigation strategy comprises at least one of the following: a two-stage aggregation strategy, a broadcast small table strategy, and a directional scheduling strategy, wherein the two-stage aggregation strategy is used to first add a random salt value to the key causing the data skew problem to scatter to a plurality of instances, perform local aggregation, and then combine results, the broadcast small table strategy is used to broadcast a small table to reduce redistribution pressure of a large table in a case where the data skew problem is caused by a key of a small table hotspot in a join operation, and the directional scheduling strategy is used to schedule the data processing task corresponding to the key causing the data skew problem to an operator with the lowest load.

[0109] Optionally, the data processing apparatus is further configured to: determine a hardware acceleration requirement of the data processing task corresponding to the data stream to be processed, wherein the hardware acceleration requirement is used to represent a dependency requirement of the data processing task on a specific type of hardware resource; acquire a load state of the hardware resource corresponding to the hardware acceleration requirement, and determine a scheduling score of the hardware resource in the pool of heterogeneous hardware resources according to a task feature of the data processing task and the load state, wherein the hardware resource comprises at least one of a graphics processing unit and a field programmable gate array, and the scheduling score is used to represent availability and execution efficiency of the hardware resource for the data processing task under the load state; determine a hardware resource with the highest scheduling score in the pool of heterogeneous hardware resources as a target resource, and schedule the data processing task to the target resource for execution.

[0110] Optionally, the data stream distribution module 82 is further configured to: in a case where a processing delay required by the data processing task is higher than a preset delay threshold or a complexity of the data processing task is higher than a preset complexity threshold, distribute the data stream to be processed to the second data processing layer; and perform a processing operation corresponding to the data processing task on the data stream to be processed by using an operator of the second data processing layer to obtain a data processing result.

[0111] It should be noted that each module in the data processing apparatus described above can be a program module (for example, a program instruction set for implementing a certain specific function) or a hardware module. For the latter, it can be in the form of, but not limited to, a processor, or the functions of the above modules are implemented by a processor.

[0112] It should be noted that the data processing apparatus provided in the embodiment can be used to execute the data processing method shown in Figure 2 The related explanations and descriptions of the data processing method described above are also applicable to the embodiments of the present application, and will not be repeated here.

[0113] The embodiments of the present application also provide a non-volatile storage medium, which comprises a stored computer program, wherein a device in which the non-volatile storage medium is located executes the following data processing method by running the computer program: acquiring a data stream to be processed from a data source and determining a data processing task corresponding to the data stream to be processed; in a case where a processing delay required by the data processing task is lower than a preset delay threshold, distributing the data stream to be processed to a first data processing layer; performing target sampling processing on the data stream to be processed by using an operator of the first data processing layer to obtain a sampled data stream, wherein a data amount of the sampled data stream is smaller than a data amount of the data stream to be processed, and a relative order of data samples in the sampled data stream is consistent with a relative order of data samples in the data stream to be processed; and performing a processing operation corresponding to the data processing task on the sampled data stream to obtain a data processing result.

[0114] The embodiment of the present application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the data processing method described in various embodiments of the present application: obtaining a to-be-processed data stream from a data source, and determining a data processing task corresponding to the to-be-processed data stream; in the case where a processing delay required by the data processing task is lower than a preset delay threshold, distributing the to-be-processed data stream to a first data processing layer; performing target sampling processing on the to-be-processed data stream by using an operator of the first data processing layer, to obtain a sampled data stream, wherein a data amount of the sampled data stream is less than a data amount of the to-be-processed data stream, and a relative order of data samples in the sampled data stream is consistent with a relative order of data samples in the to-be-processed data stream; and performing a processing operation corresponding to the data processing task on the sampled data stream, to obtain a data processing result.

[0115] The above embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0116] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0117] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only illustrative, and for example, the division of units can be a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0118] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0119] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0120] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk and various program code storage media.

[0121] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises: acquiring a to-be-processed data stream from a data source, and determining a data processing task corresponding to the to-be-processed data stream; in a case where a processing delay required by the data processing task is lower than a preset delay threshold, distributing the to-be-processed data stream to a first data processing layer; performing target sampling processing on the to-be-processed data stream by using an operator of the first data processing layer to obtain a sampled data stream, wherein a data amount of the sampled data stream is smaller than a data amount of the to-be-processed data stream, and a relative order of data samples in the sampled data stream is consistent with a relative order of the data samples in the to-be-processed data stream; wherein performing target sampling processing on the to-be-processed data stream comprises: initializing a data storage pool, wherein the data storage pool is used to store selected data samples, each of the data samples in the to-be-processed data stream comprises: a data item, a first timestamp corresponding to the data item, and a serial number; sequentially acquiring the data samples from the to-be-processed data stream, and clearing, each time the data samples are acquired, the data samples in the data storage pool whose difference between a second timestamp and the first timestamp exceeds a preset time window length, wherein the second timestamp is a timestamp corresponding to a time when the data samples are acquired from the to-be-processed data stream; determining a storage strategy according to a storage state of the data storage pool, and storing the data samples newly acquired from the to-be-processed data stream into the data storage pool according to the storage strategy, to finally obtain the sampled data stream, comprising: in a case where an amount of data samples already stored in the data storage pool is smaller than a data sample capacity of the data storage pool, inserting the data samples newly acquired from the to-be-processed data stream into the data storage pool according to a size order of the serial numbers; in a case where the amount of data samples already stored in the data storage pool is not smaller than the data sample capacity, performing a replacement operation of data samples according to a target replacement rule, wherein the replacement operation is used to select one data sample in the data storage pool, and to decide whether to replace the selected data sample in the data storage pool with the data sample newly acquired from the to-be-processed data stream; performing a processing operation corresponding to the data processing task on the sampled data stream to obtain a data processing result.

2. The data processing method according to claim 1, characterized in that, Performing a replacement operation of data samples according to a target replacement rule comprises: determining a basic replacement probability according to the data sample capacity and a total number of the data samples already acquired from the to-be-processed data stream; determining an aging parameter corresponding to each data sample in the data storage pool according to the first timestamp of each data sample, wherein the aging parameter is used to represent a difference between a current timestamp and the first timestamp of the data sample, the larger the difference is, the larger the aging parameter is, and the worse the aging of the data sample is represented. According to the timeliness parameter, a weighted probability distribution corresponding to a data sample in the data storage pool is determined, wherein the weighted probability distribution is used to represent a probability of each data sample in the data storage pool being selected, and the greater the timeliness parameter, the greater the probability of the data sample being selected; According to the weighted probability distribution, a data sample in the data storage pool is randomly selected as a replacement target, and the selected replacement target is randomly replaced by a new data sample obtained from the to-be-processed data stream according to the basic replacement probability.

3. The data processing method of claim 1, wherein, The operator is a basic processing unit for data processing in a stream processing system; the method further comprises: Monitoring running indexes corresponding to a plurality of operators in the system, wherein the running indexes are used to represent load states of the operators; According to the running indexes in a first time period, system load trend data corresponding to a second time period is predicted, wherein the second time period is a time period immediately after the first time period; According to the predicted system load trend data, parallelism of the plurality of operators is adjusted to optimize processing capacity and resource utilization of the system, wherein the parallelism is used to represent a number of instances of an operator running simultaneously in the system.

4. The data processing method according to claim 3, characterized in that, The running indexes include distribution data of keys of a key business, wherein a key is used to uniquely identify data or group data; the method further comprises: By analyzing the distribution data of the keys, it is determined whether there is a data skew problem in the system, wherein the data skew problem includes at least one of the following: a data amount and / or processing time corresponding to a key exceeds an average value by more than a preset range; In a case where it is determined that there is the data skew problem in the system, a skew mitigation strategy is used to process a data processing task corresponding to a key causing the data skew problem, wherein the skew mitigation strategy is used to optimize load balancing of the data processing task to alleviate the data skew problem; The skew mitigation strategy includes at least one of the following: a two-stage aggregation strategy, a broadcast small table strategy, and a directional scheduling strategy, the two-stage aggregation strategy is used to first add a random salt value to a key causing the data skew problem to be scattered to a plurality of instances, perform local aggregation, and then combine results, the broadcast small table strategy is used to broadcast a small table to reduce redistribution pressure of a large table in a case where the data skew problem is caused by a key of a small table hotspot in a join operation, and the directional scheduling strategy is used to schedule a data processing task corresponding to a key causing the data skew problem to an operator with the lowest load.

5. The data processing method according to claim 3, characterized in that, The method further comprises: Determining a hardware acceleration requirement of a data processing task corresponding to the to-be-processed data stream, wherein the hardware acceleration requirement is used to represent a dependency requirement of the data processing task on a specific type of hardware resource; obtaining a load state of a hardware resource corresponding to the hardware acceleration requirement, and determining a scheduling score of the hardware resource in the heterogeneous hardware resource pool according to a task feature of the data processing task and the load state, wherein the hardware resource comprises at least one of a graphics processing unit and a field programmable gate array, and the scheduling score is used to represent availability and execution efficiency of the hardware resource for the data processing task under the load state; determining the hardware resource with the highest scheduling score in the heterogeneous hardware resource pool as a target resource, and scheduling the data processing task to the target resource for execution.

6. The data processing method of claim 1, wherein, The method further comprises: in a case where processing delay required by the data processing task is higher than a preset delay threshold or complexity of the data processing task is higher than a preset complexity threshold, distributing the to-be-processed data stream to a second data processing layer, wherein processing efficiency of the second data processing layer is lower than that of the first data processing layer, and logical complexity of data analysis of the second data processing layer is higher than that of the first data processing layer; performing, by using an operator of the second data processing layer, a processing operation corresponding to the data processing task on the to-be-processed data stream to obtain a data processing result.

7. A data processing apparatus, characterized by, comprise: a data stream acquisition module configured to acquire a to-be-processed data stream from a data source and determine a data processing task corresponding to the to-be-processed data stream; a data stream distribution module configured to, in a case where processing delay required by the data processing task is lower than a preset delay threshold, distribute the to-be-processed data stream to a first data processing layer; a data stream sampling module configured to perform target sampling processing on the to-be-processed data stream by using an operator of the first data processing layer to obtain a sampled data stream, wherein a data amount of the sampled data stream is smaller than that of the to-be-processed data stream, and a relative order of data samples in the sampled data stream is consistent with that of the to-be-processed data stream; The target sampling processing on the to-be-processed data stream comprises: initializing a data storage pool, wherein the data storage pool is used to store selected data samples, each of the data samples in the to-be-processed data stream comprises: a data item, a first timestamp corresponding to the data item, and a sequence number; sequentially obtaining the data samples from the to-be-processed data stream, and clearing the data samples in the data storage pool whose difference between a second timestamp and the first timestamp exceeds a preset time window length each time the data samples are obtained, wherein the second timestamp is a timestamp corresponding to a time when the data samples are obtained from the to-be-processed data stream; determining a storage strategy according to a storage state of the data storage pool, and storing the newly obtained data samples from the to-be-processed data stream into the data storage pool according to the storage strategy, to finally obtain the sampled data stream, which comprises: in a case where an amount of the data samples already stored in the data storage pool is less than a data sample capacity of the data storage pool, inserting the newly obtained data samples from the to-be-processed data stream into the data storage pool according to a size order of the sequence numbers; and in a case where the amount of the data samples already stored in the data storage pool is not less than the data sample capacity, performing a replacement operation of data samples according to a target replacement rule, wherein the replacement operation is used to select one data sample in the data storage pool and determine whether to replace the selected data sample in the data storage pool with the newly obtained data sample from the to-be-processed data stream. The data task processing module is configured to perform a processing operation corresponding to the data processing task on the sampled data stream to obtain a data processing result.

8. An electronic device, comprising: The data processing method comprises: The memory and the processor are configured to run a program stored in the memory, wherein the program performs the data processing method in any one of claims 1 to 6 when running.

9. A non-volatile storage medium, comprising: The non-volatile storage medium comprises a stored computer program, wherein a device where the non-volatile storage medium is located performs the data processing method in any one of claims 1 to 6 by running the computer program.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the data processing method in any one of claims 1 to 6.

Images