Stream data processing method

Abstract

This application provides a streaming data processing method applied to a data processor. The method includes: acquiring streaming data from a distributed streaming processing cluster; the streaming data includes at least one data fragment; sending the streaming data to a first storage unit of a graphics processor via remote direct memory access; sending a first prompt message to a second storage unit of the graphics processor based on a preset address mapping relationship; the first prompt message indicating that at least one first data fragment has been sent; acquiring a second prompt message from the graphics processor; the second prompt message indicating that at least one second data fragment in the first data fragment has been verified; determining the occupancy rate of the first storage unit based on the number of the first and second data fragments respectively; and adjusting the streaming data acquisition rate based on the occupancy rate.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of information technology, and in particular to a method for processing streaming data. Background Technology

[0002] With the explosive growth of the Internet of Things, social media, and Internet (Web) services, the demands on throughput and latency for real-time streaming data processing are becoming increasingly stringent, and distributed message queue systems are gradually becoming the standard entry point for large-scale data stream pipelines. In scenarios where high-performance graphics processing units (GPUs) are used to accelerate data analysis, the data stream typically needs to be pulled, parsed, and copied to the GPU memory by the central processing unit (CPU) before entering the GPU memory, thus limiting end-to-end throughput and latency lower bounds. Summary of the Invention

[0003] In view of this, embodiments of this application provide at least one streaming data processing method.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a streaming data processing method applied to a data processor, including: Acquire streaming data from a distributed stream processing cluster; the streaming data includes at least one data shard. Send streaming data to the first memory unit of the graphics processor via remote direct memory access; Based on a preset address mapping relationship, a first prompt message is sent to the second storage unit of the graphics processor; the first prompt message indicates that at least one first data segment has been sent. Obtain a second prompt message from the graphics processor; the second prompt message indicates that at least one of the second data segments in the first data segment has been verified. The utilization rate of the first storage unit is determined based on the number of the first data fragment and the second data fragment, respectively. Adjust the acquisition rate of streaming data based on occupancy rate.

[0005] This application provides a streaming data processing method applied to a graphics processor, the graphics processor including a first storage unit and a second storage unit; the method includes: Acquire streaming data sent by the data processor to the first storage unit; the streaming data includes at least one data fragment; The data processor sends a first notification message to the second storage unit; the first notification message indicates that at least one first data fragment has been sent. In response to receiving the first prompt information, at least one first data fragment is verified; Based on the preset address mapping relationship, a second prompt message is sent to the data processor; the second prompt message indicates that at least one of the second data fragments in the first data fragment has been verified.

[0006] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the implementation flow of a streaming data processing method provided in an embodiment of this application. Figure 1 ; Figure 2 This is a schematic diagram of the implementation flow of a streaming data processing method provided in an embodiment of this application. Figure 2 ; Figure 3 This is a schematic diagram of the composition architecture of a streaming data processing system provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the implementation process of a GPU-accelerated processing method for Kafka streaming data provided in an embodiment of this application.

[0008] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] In the following description, the terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first / second / third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application. It should also be noted that, for ease of description, only the parts relevant to the application are shown in the accompanying drawings.

[0011] This application provides a streaming data processing method applied to a data processing unit (DPU). In implementation, the data processor runs in an electronic device. Exemplarily, the electronic device can refer to a server, laptop, tablet, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or any other device with data processing capabilities.

[0012] like Figure 1 As shown, the method includes the following steps S101 to S106: Step S101: Obtain streaming data from the distributed stream processing cluster; the streaming data includes at least one data shard.

[0013] Here, the distributed stream processing cluster can correspond to the distributed message queue system described above. For example, the distributed stream processing cluster can include a Kafka cluster corresponding to the distributed messaging system Apache Kafka.

[0014] Taking a Kafka cluster as an example, a Kafka cluster receives and maintains messages (records) published by producers, and consumers pull records from the Broker nodes in the cluster in the form of data streams for consumption.

[0015] Each data shard corresponds to a set of messages contained in a data retrieval round. Taking a Kafka cluster as an example, each data shard corresponds to a basic unit of message storage and transmission (RecordBatch), and each RecordBatch contains at least one record. By processing messages in batches, high throughput for streaming data processing can be achieved.

[0016] In implementation, an electronic device with a data processor can serve as a node in a distributed stream processing cluster, and its corresponding service process can run on the CPU of the electronic device; or, the nodes of the distributed stream processing cluster may be different from the electronic device with a data processor, and this application does not limit this.

[0017] Step S102: Send streaming data to the first storage unit of the graphics processor via remote direct memory access.

[0018] Here, a graphics processor is a processing unit used to process streaming data to provide corresponding services. In implementation, the graphics processor may run in the same or different electronic devices as the data processor, and this application embodiment does not limit this.

[0019] Remote Direct Memory Access (RDMA) is a direct memory access technology that enables zero-copy data transfer and low latency by supporting zero-copy transfer and kernel bypass.

[0020] The first storage unit corresponds to the graphics processor's memory space, such as video memory. In implementation, the DPU can pull streaming data from the distributed stream processing cluster and write the streaming data to a preset area in the GPU's video memory via the RDMA engine; alternatively, it can write it to any area in the GPU's memory space.

[0021] Understandably, using RDMA to directly transmit streaming data end-to-end (DPU to CPU) can effectively reduce the transmission latency caused by the CPU pulling, parsing, and then copying the data.

[0022] Step S103: Based on the preset address mapping relationship, send a first prompt message to the second storage unit of the graphics processor; the first prompt message indicates that at least one first data segment has been sent.

[0023] Here, the preset address mapping relationship is the mapping relationship between the GPU's second storage unit and the address space accessible by the DPU. The second storage unit is the hardware storage structure inside the GPU, and the preset address mapping relationship can map this hardware storage structure to the DPU's memory address space.

[0024] For example, the second storage unit can be a GPU register. The DPU can write the first prompt message to a GPU register, such as a doorbell register, through a memory-mapped I / O (MMIO) mechanism that supports the high-speed serial computer extended bus standard (PCIe). Furthermore, the GPU polls the doorbell register until it finds an updated first prompt message.

[0025] The first data fragment is the first data fragment that the DPU has written to the GPU's first storage unit. It can be understood that the DPU can pull multiple data fragments from the distributed stream processing cluster multiple times (pulling at least one data fragment each time), and write these multiple data fragments to the first storage unit in batches according to the batches pulled. Therefore, after completing the writing of each first data fragment, the DPU can update the first notification information sent to the second storage unit, allowing the GPU to obtain the latest data writing progress.

[0026] In implementation, the first prompt information can be the data identifier of the first data segment, or it can be an indication (such as a pointer) pointing to the address of the first data segment in the first storage unit.

[0027] Step S104: Obtain a second prompt message from the graphics processor; the second prompt message indicates that at least one second data segment in the first data segment has been verified.

[0028] Here, the second cue message can be sent from the GPU to the DPU based on a preset address mapping relationship. For example, the GPU can also send the second cue message to the DPU's reverse Doorbell register via the PCIe MMIO mechanism.

[0029] In implementation, after receiving the first prompt information, the GPU can read and verify the data fragments in the first storage unit until the first data fragment has been read completely and / or an update to the first prompt information (indicating an update to the data fragment to be read) is detected. Taking each data fragment corresponding to one RecordBatch as an example, the GPU only sends the second prompt information indicating the completion of the RecordBatch verification to the DPU when all records in the RecordBatch have been verified.

[0030] It's understandable that the DPU's data writing and the GPU's data verification are performed separately. Therefore, it's possible that after the DPU writes multiple data fragments, the GPU has only completed the verification of a portion of those fragments; that is, the verification progress of the data fragments is not earlier than the writing progress. In this situation, there's a possibility that the unread data in the first storage unit may gradually increase until the storage unit occupancy rate becomes too high, leading to reduced GPU processing performance and data loss.

[0031] In implementation, the second prompt information can be the data identifier of the second data segment, or it can be an indication (such as a pointer) pointing to the address of the second data segment in the first storage unit.

[0032] Step S105: Determine the occupancy rate of the first storage unit based on the number of the first data fragment and the second data fragment respectively.

[0033] Here, since the second data fragment has been verified, the storage space in the first storage unit used to store the second data fragment can be considered as reusable (overwriteable) unoccupied space. The number of first data fragments reflects the progress of the DPU sending (writing) data fragments, and the number of second data fragments reflects the progress of the GPU verifying data fragments. Therefore, the occupancy rate of the first storage unit can be determined based on the respective numbers of the first and second data fragments; that is, the proportion of storage space in the first storage unit corresponding to data fragments that have been written but not yet verified.

[0034] Step S106: Adjust the acquisition rate of streaming data based on the occupancy rate.

[0035] Here, after the DPU obtains the current occupancy rate of the first storage unit, it can reduce the rate of acquiring streaming data when the occupancy rate is high, thereby reducing the rate at which the DPU sends data fragments to the first storage unit, and thus reducing the possibility of data loss due to insufficient available storage space for unwritten data; conversely, it can increase the rate of acquiring streaming data when the occupancy rate is low, which indicates that the GPU is reading and verifying data fragments more efficiently. Therefore, it can increase the rate at which the DPU sends data fragments to the first storage unit, thereby improving the utilization rate and processing efficiency of GPU storage resources.

[0036] In implementation, when the difference between the current occupancy rate of the first storage unit and the average occupancy rate over the target duration exceeds a preset threshold (such as 5%, 10%), the rate of acquiring streaming data can be adjusted. This improves the stability of data processing by the GPU based on the first storage unit.

[0037] In implementation, the rate at which streaming data is acquired can be adjusted if the difference between the current occupancy rate of the first storage unit and the preset target occupancy rate (determined according to requirements) exceeds a preset threshold. This improves the adaptability of GPU processing performance to the needs of the current scenario.

[0038] In this embodiment, on the one hand, the DPU receives streaming data from the distributed stream processing cluster and sends it directly to the GPU via RDMA. Furthermore, after completing data transmission, the DPU updates the data transmission progress to the GPU's storage space in real time through address mapping. This effectively reduces end-to-end transmission latency and throughput limitations caused by CPU performance bottlenecks in streaming data processing scenarios where CPU intervention is not required. On the other hand, the DPU can adjust the streaming data acquisition rate in real time based on the transmission and verification progress of data fragments in the first storage unit. This improves data transmission stability while further enhancing the GPU's storage resource utilization and processing efficiency.

[0039] In some embodiments, the first storage unit includes a plurality of sub-storage units, and step S102 above may include the following step S111: Step S111: Through remote direct memory access, each data fragment is sent to a sub-storage unit whose status identifier represents the data fragment that can be stored, according to the original order of each data fragment in the streaming data.

[0040] Here, each sub-storage unit in the first storage unit can be used to store a data fragment; furthermore, according to the original order of each data fragment in the streaming data, storing each data fragment into sequentially arranged sub-storage units can enable the GPU to read and verify data according to the original timing of the streaming data, thereby improving the timeliness and traceability of performing computing tasks based on streaming data.

[0041] The status identifiers of each sub-storage unit are used to characterize the current application status of each sub-storage unit. It is understandable that by obtaining the status identifiers of each sub-storage unit, the writing and verification progress of data fragments in the first storage unit can be more accurately grasped.

[0042] For example, the status flag may include, but is not limited to, at least one of the following: a FREE flag indicating that a storable data fragment is available; a WRITING flag indicating that the DPU is writing a data fragment; a READY flag indicating that the DPU has completed writing the data fragment and is awaiting GPU verification; a PROCESSING flag indicating that the data fragment is being read or verified by the GPU; and a DONE flag indicating that the GPU verification is complete.

[0043] In implementation, each sub-storage unit includes at least a first storage area and a second storage area; wherein, the first storage area is used to store the metadata of each data fragment, and the second storage area is used to store the data content of each data fragment. The first storage area may include the status identifier of the corresponding sub-storage unit.

[0044] Step S103 above may include the following step S112: Step S112: Based on the preset address mapping relationship, the first prompt information pointing to the first sub-storage unit is sent to the second storage unit; the first sub-storage unit is a sub-storage unit that stores the first data fragment, and the status identifier of the first sub-storage unit indicates that the data fragment stored therein is pending verification.

[0045] Here, the first prompt information is used to indicate the location of the first sub-storage unit within the first storage unit. Before the writing of the corresponding first data fragment is completed in the first sub-storage unit, its status flag may include a FREE flag indicating that the data fragment can be stored, or a WRITING flag indicating that the DPU is writing the data fragment. Until the writing of the first data fragment is completed, the status flag of the first sub-storage unit can be switched to a READY flag indicating that the DPU has completed the data fragment writing and is awaiting GPU verification.

[0046] In implementation, the DPU can switch the status flag of the first sub-memory unit through atomic operation instructions. For example, after completing the writing of the first data fragment, the DPU can switch the status flag of the first sub-memory unit through a Compare And Swap (CAS) operation.

[0047] In this embodiment, the DPU completes the process from sending the first data fragment (start writing) to sending a prompt message to the GPU (write complete) based on the original order of each data fragment in the streaming data and the status identifier of each sub-storage unit in the first storage unit. This improves the real-time performance and accuracy of forward communication in bidirectional communication between the DPU and GPU by modifying the status identifiers representing the application status of each sub-storage unit when batch processing streaming data using a first storage unit containing multiple sub-storage units.

[0048] In some embodiments, the plurality of sub-storage cells in the first storage cell are arranged in a circular arrangement, and step S105 above may include the following steps S121 to S124: Step S121: Based on the first position of the first sub-storage unit in the first storage unit, determine the first number corresponding to the first data fragment.

[0049] For example, the first storage unit can be a ring buffer of the GPU, which includes multiple slots (slots are sub-storage units), each slot being used to store a data fragment.

[0050] The first prompt information is used to indicate the first position of the first sub-storage unit in the first storage unit. Since the multiple sub-storage units in the first storage unit are arranged in a ring, the first number of the first data fragments that have been written can be obtained based on the starting position of the data fragments written in the first storage unit and the first position of the first sub-storage unit in the first storage unit.

[0051] During implementation, while the DPU sends the first prompt message to the GPU, it can also maintain the first prompt message locally to obtain the real-time progress of the data that has been written.

[0052] Step S122: Obtain the second sub-storage unit of the second data segment in the second storage unit at the second location.

[0053] Here, the second prompt information is used to indicate the second sub-storage unit's second location within the first storage unit.

[0054] Step S123: Based on the second position, determine the second quantity corresponding to the second data fragment.

[0055] Since the multiple sub-storage units in the first storage unit are arranged in a ring, the second number of the verified second data fragments can be obtained based on the starting position of the verified data fragments in the first storage unit and the second position of the second sub-storage unit in the first storage unit.

[0056] Step S124: Determine the occupancy rate based on the difference between the first quantity and the second quantity, and the total number of sub-storage units in the first storage unit.

[0057] In implementation, the difference between the first quantity and the second quantity can be obtained by subtracting the second quantity from the first quantity; alternatively, the difference between the first quantity and the second quantity can be obtained by directly counting the number of sub-memory units between the first and second locations.

[0058] Furthermore, the ratio between the difference between the first quantity and the second quantity and the total number of sub-storage units in the first storage unit is the occupancy rate of the first storage unit.

[0059] In this embodiment, the first storage unit of the ring storage structure is used to store data fragments. Based on the latest position of the written data (first position) and the latest position of the verified data (second position), the occupancy rate of the first storage unit can be obtained quickly and accurately, thereby improving the efficiency and accuracy of adjusting the data acquisition rate based on the occupancy rate.

[0060] In some embodiments, step S106 may include steps S131 to S132: Step S131: In response to the occupancy rate not being less than the first occupancy rate threshold, shrink the receiving window for receiving streaming data from the distributed streaming processing cluster and / or reduce the frequency of sending data acquisition requests to the distributed streaming processing cluster to reduce the acquisition frequency of streaming data.

[0061] Here, the first occupancy threshold is a preset maximum threshold. If the current occupancy rate of the first storage unit is not less than the first occupancy threshold, it indicates that the current occupancy rate of the first storage unit is high, and there is a possibility of data loss or reduced processing performance.

[0062] Furthermore, by shrinking the receiving window for receiving streaming data from the distributed stream processing cluster, the size of the streaming data received by the DPU can be reduced, thereby reducing the frequency of streaming data acquisition (overall dimension). And / or, by reducing the frequency of sending data retrieval requests to the distributed stream processing cluster, the frequency of the distributed stream processing cluster responding to requests and feeding back stream data to the DPU can be reduced, thereby reducing the frequency of stream data retrieval.

[0063] In implementation, the DPU can also reduce the overall streaming data acquisition frequency by pausing at least one round of data shard acquisition to extend the time interval between at least two data acquisition requests. This reduces the frequency of sending data acquisition requests to the distributed stream processing cluster. For example, the number of data acquisition requests sent per second can be reduced from 10 to 5, thus halving the streaming data acquisition frequency.

[0064] In implementation, the first occupancy rate threshold can be represented as HWM. For example, the first occupancy rate threshold can take values ​​such as 70% or 80%, and this application embodiment does not limit this.

[0065] Step S132: In response to the occupancy rate being less than the second occupancy rate threshold, increase the data size of each stream data acquisition from the distributed stream processing cluster to improve the stream data acquisition rate; the first occupancy rate threshold is greater than the second occupancy rate threshold.

[0066] Here, the second occupancy threshold is a preset minimum threshold. If the current occupancy rate of the first storage unit is less than the second occupancy threshold, it indicates that the current occupancy rate of the first storage unit is low. This can be addressed by increasing the data size of the streaming data retrieved from the distributed stream processing cluster each time (which can be represented as max). bytes This increases the overall acquisition frequency of streaming data and further accelerates the writing of data fragments to the first storage unit, thereby improving the GPU's processing efficiency and accelerating data consumption.

[0067] In implementation, the second occupancy rate threshold can be represented as LWM. For example, the second occupancy rate threshold can be 25%, 30%, etc., and this application embodiment does not limit it in this way.

[0068] In some implementations, a third occupancy threshold, denoted as MWM, may be set to maintain the current streaming data acquisition frequency. This third occupancy threshold is lower than the first occupancy threshold and higher than the second occupancy threshold. For example, MWM can take values ​​such as 50% or 60%, but this application does not limit the specific values.

[0069] Furthermore, the DPU can maintain the current frequency of sending data retrieval requests to the distributed stream processing cluster and the size of the stream data retrieved from the distributed stream processing cluster each time, in response to a occupancy rate that is less than a first occupancy rate threshold but not less than a third occupancy rate threshold.

[0070] In this embodiment, different methods are employed to increase or decrease the acquisition frequency of streaming data based on the relationship between the occupancy rate of the first storage unit and the first and second occupancy rate thresholds. Thus, when the occupancy rate of the first storage unit is in different ranges, different adjustment methods can be used to increase or decrease the data flow rate from the DPU side or the cluster side. This further enhances the flexibility of flow rate adjustment and achieves more precise backpressure control, building upon the backpressure control implemented using the DPU.

[0071] In some embodiments, each data slice includes at least one sub-data piece, and step S101 above may include steps S141 to S142: Step S141: Send a first data acquisition request carrying a first data identifier to the distributed stream processing cluster; the first data identifier is the data identifier of the third data fragment in the stream data; the first data fragment is at least one data fragment in the stream data that is located after the third data fragment and adjacent to the third data fragment.

[0072] Here, each piece of data in each data segment can correspond to a message (i.e., a record) in the above embodiment.

[0073] The first data fragment and the third data fragment are arranged sequentially in the streaming data. The third data fragment corresponds to the data fragment that has passed the verification before the first data fragment in the streaming data. That is, the first data identifier corresponds to the largest identifier of the data fragment that has passed the verification before the first data fragment is obtained (the identifier in the streaming data is different from the identifier dimension corresponding to the position of the sub-storage unit in the first storage unit, but they correspond one-to-one).

[0074] In implementation, the first data identifier can correspond to the offset in the Kafka message stream, that is, the position number of the data shard in the message queue. In the Kafka architecture, messages within each partition are assigned consecutive offsets sequentially, and consumers can track consumption progress through the offset. For example, in a distributed stream processing cluster, if the current consumption has reached the third data shard with an offset of 1000, then the first data identifier is 1000, and the DPU can request at least one subsequent adjacent data shard based on this first data identifier.

[0075] It is understandable that by introducing a first data identifier, the precise location of the data fragment corresponding to the data acquisition request can be achieved, which can reduce the data redundancy or omission problem caused by fuzzy requests in related solutions.

[0076] Step S142: Receive the first data fragment returned by the distributed stream processing cluster.

[0077] In implementation, the distributed stream processing cluster responds to the first data acquisition request and, based on the first data identifier, provides at least one sub-data located after the third data shard in the stream data to the DPU in the form of the first data shard (such as RecordBatch).

[0078] The streaming data processing method may further include the following steps S143 to S145: Step S143: In response to obtaining the second prompt information, obtain the target percentage of the first sub-data that has passed the verification in each second data segment.

[0079] The second prompt message indicates that the second data shard has been verified. However, when the proportion of the first sub-data that passed the verification in the second data shard is in different ranges, the DPU can further determine the subsequent data request strategy.

[0080] The target percentage is a crucial criterion for judging data quality and determining whether data needs to be reacquired. The first sub-data is the valid data that passes verification in the second data shard, and the validity of this first sub-data is determined by GPU verification. For example, if the second data shard contains 100 sub-data entries, and 98 of them pass GPU verification, then the target percentage is 98%.

[0081] In some implementations, after the GPU completes the verification of the second data fragment, it can directly write the verification result (i.e., whether each sub-data piece passes the verification) into the DPU's memory via direct peer-to-peer (Direct P2P) transmission and send a second prompt message to the DPU. In response to receiving the second prompt message, the DPU retrieves the verification result of the second data fragment from memory and calculates the target percentage of the first sub-data piece.

[0082] Step S144: Determine the second data identifier based on the target percentage.

[0083] The second data identifier is used to indicate the data offset corresponding to the next round of data retrieval request.

[0084] During implementation, the second data identifier corresponding to the target percentage range can be determined based on the range in which the target percentage is located.

[0085] During implementation, it can also be determined whether it is the same as the first data identifier based on the relationship between the target percentage and the preset ratio.

[0086] Step S145: Send a second data acquisition request carrying a second data identifier to the distributed stream processing cluster; the second data identifier may be the same as or different from the first data identifier.

[0087] It is understandable that the second data identifier can be continuous with the first data identifier, indicating that subsequent data is retrieved from the current data acquisition position; when the second data identifier is reset to the first data identifier, it indicates that the current data is retrieved again.

[0088] By dynamically adjusting the second data identifier, an adaptive data fetching strategy can be implemented, enhancing the fault tolerance and stability of the DPU and GPU when consuming data. In the face of network jitter or data corruption, the DPU can quickly locate and recover erroneous data (by repeatedly fetching), thereby ensuring the high availability of streaming data processing functions.

[0089] The structure of the second data retrieval request is the same as that of the first data retrieval request, but the starting position of the data fragment to be retrieved is determined by the second data identifier. The second data retrieval request is placed in the next round after the first data retrieval request to ensure that consumption starts from the correct position.

[0090] By utilizing a dynamic control mechanism that determines the second data identifier based on proportion, the DPU can reuse the data identifiers corresponding to historical requests and re-fetch previously unprocessed data, achieving precise data retry and recovery, and improving the integrity and consistency of data processing. Furthermore, by enabling the DPU to independently complete request sending and error recovery, the entire streaming data processing process is decoupled from the CPU, achieving end-to-end CPU-free operation.

[0091] In this embodiment, by introducing a first data identifier and a second data identifier, combined with the collaborative working mechanism of the DPU and GPU, precise positioning, efficient transmission, and error recovery of streaming data are achieved. This further improves the real-time performance and reliability of data processing while reducing performance bottlenecks and latency issues caused by CPU involvement. It is understandable that for data processing scenarios with high requirements for timeliness and stability, such as the Internet of Things, financial transactions, and online video, the DPU's error recovery mechanism demonstrates significant advantages.

[0092] In some embodiments, step S144 may include step S151: Step S151: In response to the existence of a fourth data segment in at least one second data segment with a target proportion not less than the minimum preset ratio but less than the maximum preset ratio, the data identifier of the fifth data segment located before the first fourth data segment in the streaming data is determined as the second data identifier.

[0093] Here, the minimum preset ratio and the maximum preset ratio are used to define the effective range of the target percentage. The minimum preset ratio can be considered as the lowest pass rate without error alarms. When the target percentage is less than the minimum preset ratio, it indicates that there may be a fault in the current transmission link, and an alarm needs to be issued. The maximum preset ratio can be considered as the lowest pass rate for the data fragment to pass the verification.

[0094] For example, the minimum preset ratio can be set to 90%, and the maximum preset ratio can be set to 99%. Only when the target percentage is within the range of less than 90% but not less than 99% will the data segment be determined as the fourth data segment. In implementation, the minimum and maximum preset ratios can be flexibly adjusted according to the actual application scenario to adapt to different data characteristics and business needs.

[0095] The fourth data shard represents a data shard in the second data shard that has intermediate characteristics. It neither has the conditions to trigger a fault alarm nor can it be considered to have passed the verification.

[0096] The fifth data shard is a data shard in the streaming data that is located before the first fourth data shard and immediately adjacent to the first fourth data shard.

[0097] By using the data identifier corresponding to the fifth data shard as the second data identifier, the fourth data shard can be used as the backtracking starting point for the failed batch, that is, the recovery benchmark when there is a data error.

[0098] It is understood that, in this embodiment, the judgment is based on the target proportion of verified sub-data in the data shard. In actual implementation, the judgment can also be based on the target proportion of unverified sub-data in the data shard. This embodiment does not limit this. Correspondingly, the judgment condition (preset ratio) is the failure rate.

[0099] In this embodiment, in response to the existence of the fourth data fragment, the data identifier of the fifth data fragment preceding the fourth data fragment is selected as the second data identifier. This allows for the re-acquisition of data fragments at the point where the first sub-data has not reached the upper bound indicating that the data fragment has passed verification, but where there are no abnormalities in the data link, thus achieving accurate recovery of erroneous data.

[0100] In some embodiments, step S144 may include steps S161 to S162: Step S161: In response to the target proportion of each second data segment not being less than the maximum preset ratio, the data identifier of the sixth data segment in the streaming data is determined as the second data identifier; the sixth data segment is the data segment located after the second data segment and adjacent to the second data segment.

[0101] Here, the sixth data fragment is the data fragment in the streaming data that follows and is directly adjacent to the second data fragment. It serves as the next data unit to be processed, thereby improving the sequentiality and logical consistency of data processing and reducing the possibility of data loss or corruption due to skipping intermediate data fragments. If the target proportion of the second data fragment is not less than the maximum preset ratio, it indicates that the second data fragment has met the verification conditions, and further, subsequent data fragments can be acquired.

[0102] Understandably, by using the above method, the data identifier in the data acquisition request can be switched only when the data quality meets the expected standard, thereby improving the reliability and stability of data processing and effectively supporting the application's demand for high-quality data.

[0103] Step S162: Send the second sub-data that failed the verification in each second data segment to the target processor; run the application that needs to use the second data segment on the target processor.

[0104] Here, for the second sub-data that fails the verification, it is separated from the current processing link and sent to the designated target processor (such as CPU or GPU) so that the target processor can receive and process these abnormal or low-quality data for further analysis, repair or archiving.

[0105] The second sub-data refers to the data units identified as failing verification within the second data segment. These second sub-data units may fail verification due to network transmission errors, hardware malfunctions, or protocol parsing anomalies. Even if these second sub-data units cannot participate in the real-time processing of the main data stream, they may still have potential analytical value; therefore, they can be collected for subsequent optimization processing.

[0106] In some implementations, the target processor can be a standalone server, virtual machine, or container instance, running customized applications such as data repair, log analysis, or alarm systems, isolated from the main data processing path, thereby reducing the possibility of it affecting normal business processes.

[0107] In this embodiment, by sending the second sub-data that failed verification in each second data segment to the target processor, timely isolation and targeted processing of abnormal data can be achieved. Thus, when a high percentage of verified data passes, sending the data that failed verification to the application layer for processing allows the application layer to compensate for a small amount of data that failed verification and continue acquiring the next batch of data. This further balances application performance and data processing efficiency while preventing the second sub-data that failed verification from polluting the main data stream and ensuring consumption stability.

[0108] In some embodiments, the streaming data processing method may further include the following step S171: Step S171: In response to the existence of a seventh data fragment in at least one second data fragment with a target proportion less than the minimum preset ratio, stop sending data acquisition requests to the distributed stream processing cluster and send an alarm message to the central processing unit.

[0109] Here, when the target proportion of the seventh data fragment is less than the minimum preset ratio, it indicates that the data fragment may have quality problems or transmission link abnormalities, which is a key element in triggering subsequent alarm and recovery processes.

[0110] Considering that data anomalies may be caused by transmission link failures, stopping the sending of data retrieval requests to the distributed stream processing cluster can reduce the possibility of data fragments with a target proportion lower than the minimum preset ratio continuing to be written, thereby reducing resource waste and the probability of error propagation.

[0111] The central processing unit (CPU) receiving alarm information can run on the same device as the data processor or on a different device. During implementation, alarm information can include, but is not limited to, key parameters such as the type of data anomaly, the time of occurrence, the affected partition, and the difference between the target percentage and the minimum preset ratio. This facilitates operations and maintenance personnel in quickly locating the root cause of the problem and performing fault recovery. Furthermore, after receiving an alarm, the operations and maintenance team can promptly repair the fault and reduce the impact of data loss or service interruption by checking the running status of nodes in the distributed stream processing cluster, network connection quality, or the producer-side data generation logic.

[0112] In this embodiment, the DPU side performs real-time identification and alarms for the seventh data fragment with an excessively low target proportion based on a minimum preset ratio. This effectively reduces the impact of the seventh data fragment on the overall data processing performance and prevents cascading failures caused by error accumulation, further improving the reliability and fault tolerance of the entire distributed stream processing system.

[0113] In some embodiments, the streaming data processing method may further include one of the following steps S181 to S182: Step S181: In response to a failure of the graphics processor, a third data acquisition request carrying a third data identifier is sent to the distributed stream processing cluster; the third data identifier is the next data identifier of the largest data identifier corresponding to the data fragment that the graphics processor has verified.

[0114] Here, the graphics processing unit (GPU) is responsible for performing data verification, protocol parsing, and parallel computing tasks, and is the component that bears the main computational load in the complete data link. When the GPU fails, such as a Compute Unified Device Architecture (CUDA) context exception, a memory access error, or a kernel execution interruption, a corresponding recovery mechanism must be activated to ensure that data is not lost.

[0115] The third data identifier is the data identifier corresponding to the data fragment used to indicate the starting point of data recovery, so that the recovery process starts after the last successfully processed data fragment, avoiding duplicate processing or missing data.

[0116] In implementation, the largest data identifier corresponding to the data fragment that the GPU has successfully verified can be represented as `committed_offset`, and correspondingly, the third data identifier can be represented as `committed_offset+1`. For example, if the data identifier corresponding to the last successfully verified data fragment in the streaming data is 1000, then the third data identifier is 1001. The DPU will then retrieve the data from this position to achieve accurate recovery. This process does not require CPU intervention and can be directly coordinated by the DPU, further decoupling the data link from the CPU.

[0117] Understandably, the data processing progress before the failure can be accurately located based on the third data identifier. Using the subsequent data fragments corresponding to this progress as the starting point for recovery can avoid data reprocessing or loss, thereby ensuring the integrity and consistency of data processing and achieving end-to-end at least once semantics.

[0118] Step S182: In response to a data processor failure, a fourth data acquisition request carrying a fourth data identifier is sent to the distributed stream processing cluster; the fourth data identifier is the data identifier carried in the last data acquisition request sent to the distributed stream processing cluster before the data processor failure.

[0119] Here, the data processor, as a key component responsible for coordination and control in the entire data link, needs to activate the corresponding recovery mechanism to ensure the consistency of the consumer group's state (i.e., the consistency of DPU data writing and GPU data verification progress) when it fails, such as when the processor core crashes, the network engine fails, or the RDMA channel is interrupted.

[0120] The fourth data identifier reflects the location of the last successfully retrieved data fragment before the failure occurred. For example, if the data identifier carried in the last data retrieval request sent by the DPU before the failure was 2000, then the fourth data identifier is 2000, and the DPU will retrieve data again from data identifier 2000 (inclusive).

[0121] Understandably, after a DPU failure, a fourth data identifier is generated based on the data identifier carried in the last data retrieval request sent before the data processor failure, and a fourth data retrieval request carrying the fourth data identifier is sent to the distributed stream processing cluster. This allows for accurate recovery of the consumption progress on the DPU side, preventing data duplication or omission, thereby maintaining the stability and consistency of the consumer group (DPU and GPU), and ultimately achieving a complete consumer group protocol closed loop and a data link with zero CPU involvement.

[0122] Because of the data interaction and state synchronization relationship between the distributed stream processing cluster and the DPU, the DPU, acting as a consumer agent, can periodically send data retrieval requests to the distributed stream processing cluster to pull data. After receiving the second hint information from the GPU, it submits the verified data offset to the cluster. Therefore, in the event of a failure, the DPU can indicate the recovery starting point to the distributed stream processing cluster by sending a request carrying a fourth data identifier. The distributed stream processing cluster then returns the corresponding data fragments, ensuring the continuity of consumption progress.

[0123] In this embodiment, corresponding data recovery mechanisms are determined based on third and fourth data identifiers for fault scenarios of both GPU and DPU. This enables precise responses to faults at different levels, ensuring the robustness and availability of the entire data processing chain and constructing a truly CPU-independent, closed-loop, end-to-end GPU streaming data processing architecture.

[0124] In some embodiments, step S101 may include step S191: Step S191: Based on the partition information, pull streaming data from the first target partition of the distributed stream processing cluster.

[0125] Here, partition information is key metadata used to describe how streaming data is distributed within a distributed stream processing cluster. For example, partition information may include, but is not limited to, partition identifier (ID), associated topic, current offset, and replica location, used to guide data consumers to pull data from specific partitions.

[0126] The first target partition is the initial data partition assigned to the consumer based on the subscription strategy and load balancing algorithm. By accurately locating the first target partition based on the partition information, the data retrieval process can be ensured to be stable and efficient, thereby avoiding the waste of resources caused by blindly scanning all partitions, further improving data acquisition efficiency, and reducing latency.

[0127] The streaming data processing method may further include the following steps S192 to S194: Step S192: In response to the update of partition information, stop pulling streaming data from the distributed stream processing cluster.

[0128] Here, partitions are typically allocated after a consumer joins a consumer group. Partition information updates may occur during consumer group rebalancing or when cluster nodes dynamically adjust partition allocations, such as when a new consumer joins, an existing consumer leaves, or a node migrates a partition due to a failure. In these cases, the original partition allocation scheme becomes invalid and needs to be renegotiated.

[0129] By responding to partition information updates and stopping data retrieval from the original partition, the possibility of reading invalid or duplicate data can be reduced, ensuring semantic consistency of data consumption, reducing data loss or duplicate processing caused by partition information updates, thereby maintaining the reliability of data processing and supporting continuous service in high availability scenarios.

[0130] Step S193: Obtain the third prompt information from the graphics processor; the third prompt information indicates that the data fragments obtained before the partition information was updated have been verified.

[0131] In practice, the third prompt message can be delivered through the aforementioned reverse Doorbell mechanism or by writing to a dedicated memory area.

[0132] Understandably, the third prompt message enables all data shards fetched before the partition information is updated to be processed completely, reducing data loss or incomplete verification caused by premature switching, achieving CPU-free collaboration between devices, thereby ensuring the integrity of data processing and further improving the fault tolerance and stability of the data transmission link.

[0133] Step S194: Based on the updated partition information, continue to pull streaming data from the second target partition of the distributed stream processing cluster.

[0134] Here, the second target partition is the new partition that the consumer should consume based on the new allocation result after the partition information is updated. The second target partition may be the same as the first target partition (e.g., only the offset is adjusted) or it may be different (e.g., it is allocated to a new partition after rebalancing).

[0135] In this embodiment, in response to the update of partition information, the DPU, after ensuring that data fragment verification is completed before the partition information is updated, continues to pull streaming data from the corresponding target partition based on the updated partition information. This enables the DPU to subsequently pull streaming data from the second target partition based on the updated partition information, thereby achieving seamless data stream source switching. This ensures that consumers can continuously receive data in a dynamic environment, achieving complete offloading of the consumer group protocol, thus decoupling the CPU from the Rebalance process, and ultimately realizing an end-to-end CPU-free data path.

[0136] In conjunction with the above-described streaming data processing method applied to a data processor, this application provides a streaming data processing method applied to a graphics processor, the graphics processor including a first storage unit and a second storage unit; like Figure 2 As shown, the streaming data processing method includes the following steps S201 to S204: Step S201: Obtain the streaming data sent by the data processor to the first storage unit; the streaming data includes at least one data fragment.

[0137] Step S202: Obtain the first prompt information sent by the data processor to the second storage unit; the first prompt information indicates that at least one first data fragment has been sent.

[0138] Step S203: In response to receiving the first prompt information, verify at least one first data fragment.

[0139] Here, the GPU can respond to the first prompt message and initiate the verification process for the first data slice.

[0140] In implementation, the GPU can use multiple kernels to perform data segmentation, data decoding, data verification, and data recording of the first data slice in series.

[0141] In the data decoding step, the GPU can use multi-threading to process multiple sub-data in a data slice in parallel, thereby improving data processing efficiency; there is a one-to-one correspondence between threads and sub-data.

[0142] Step S204: Based on the preset address mapping relationship, send a second prompt message to the data processor; the second prompt message indicates that at least one of the second data fragments in the first data fragment has been verified.

[0143] As a mainstream distributed message queue system, Apache Kafka faces the following bottlenecks in GPU-accelerated analytics scenarios, such as real-time data analysis, as the Kafka data stream enters the GPU memory: (1) High overhead: The overhead of CPU context switching and system memory copying during real-time streaming data processing; (2) PCIe bandwidth is occupied bidirectionally: data first enters the host memory and then the GPU memory, making one round trip on PCIe; (3) CPU performance is limited: The CPU core is occupied by input-output intensive tasks such as Kafka protocol parsing, cyclic redundancy check (CRC32 check), consumer offset management, and consumer group protocols (such as JoinGroup, Heartbeat, Rebalance, etc.), making it difficult to use for business computing; (4) Incomplete CPU-free: In related technologies, most solutions that use GPU to accelerate the processing of Kafka streaming data only cover the data interaction (Fetch) path, which is used to accelerate data retrieval, parsing, verification, and listening to the consumption data of Kafka Topic. However, consumer group management protocols, such as Rebalance, run on the CPU, constituting an incomplete "semi-CPU-free" solution.

[0144] To address the above bottlenecks, the following solutions are proposed in related technologies: (1) Traditional CPU path: After the CPU completes all Kafka protocol parsing, cyclic redundancy check and offset management, the data is copied to the GPU via Host to Device (H2D) method. Its end-to-end latency is usually more than 100 microseconds. Although placing some protocols on the CPU can achieve local acceleration, the CPU is still the absolute bottleneck; (2) Using RDMA technology: It only provides zero-copy primitives for the "transport layer" to bypass the host memory transfer, but it is not aware of any application layer protocols such as Kafka. It must rely on the upper layer, use GPUDirect RDMA applications, and call the specific interface of GPUDirect RDMA to provide complete semantic control flow. (3) DPU (transport layer, which is different from GPU computing layer) performs network protocol offloading: mainly for general network tasks that the system can automatically close the loop and do not require manual intervention. Since it does not involve semantic-level offloading of application layer protocols such as Kafka, the processed data still needs to be returned to the CPU, that is, the overall process still requires the participation of the CPU. (4) Parallel CRC32 and columnar conversion on GPU: It is assumed that the input data already exists in the GPU memory, without considering cross-device coordination issues such as synchronization between DPU writing and GPU consumption, back pressure of the ring buffer, and error recovery.

[0145] In summary, none of the above four solutions can achieve a complete closed-loop operation independent of the CPU.

[0146] To address the aforementioned cross-device collaboration issues, consumer semantic consistency issues, and complete protocol coverage issues, this application provides a GPU-accelerated processing method for Kafka streaming data, which is applied to a streaming data processing system. This method provides a collaboration protocol, consistency guarantee mechanism, and complete Kafka protocol coverage solution that can operate independently in a closed loop without CPU.

[0147] like Figure 3 As shown, the streaming data processing system 300 consists of a DPU 310, a GPU 320, a Kafka Broker cluster 330, and logical paths L1, L2, L3, L4, and L5.

[0148] During implementation, the DPU310 incorporates a network engine 311 (such as ConnectX), a multi-core processor 312 (such as Arm), an RDMA engine 313, and a two-stage offset state machine 314, which serves as the "network proxy + coordination hub" for the entire data link in this solution. The GPU320 includes a CUDA core 321, video memory 322, and registers 323, and has GPU Direct P2P capability. In this solution, it serves as the main component for "protocol parsing + computation". The Kafka Broker cluster 330 can run on the CPU. The CPU only participates in system initialization, memory allocation, kernel loading, and anomaly monitoring. It is not in the data transmission path during the overall system operation.

[0149] Correspondingly, logical path L1 is used for data transmission between the Kafka Broker (which can run on the CPU) and the DPU, carrying data fetch requests, responses, and consumer group protocols via Transmission Control Protocol / Internet Protocol (TCP / IP). Logical path L2 is used for DPU to write data to the GPU's Ring Buffer; Logical path L3 is used for forward doorbell notifications between the DPU and GPU; during implementation, notifications can be advanced via corresponding write pointers. Logical path L4 is used for reverse Doorbell notification between the DPU and the GPU; in implementation, notification can be advanced through the corresponding read pointer. The logical path L5 is used to carry consumer group protocols between the DPU and the Kafka Broker, such as JoinGroup, SyncGroup, Heartbeat, and OffsetCommit.

[0150] Understandably, none of the five logical paths pass through the CPU. The CPU is only used to allocate a RingBuffer and register it as an RDMA region during initialization, and it is removed from the data transmission path during runtime.

[0151] Combined with the aforementioned streaming data processing system 300, such as Figure 4 As shown, the GPU-accelerated processing method for Kafka streaming data includes the following steps S401 to S408; wherein steps S402 to S407 can be periodically looped during the steady-state operation of the streaming data processing system 300, and steps S401 and S408 can be executed only when the corresponding event is triggered: Step S401: System initialization.

[0152] In implementation, the CPU can call functions that dynamically allocate memory on the GPU memory (such as cudaMalloc) to pre-allocate a ring buffer in the GPU memory. Through the kernel components of GPUDirect RDMA (such as nvidia-peermem), network cards based on communication protocols such as InfiniBand and RDMA over Converged Ethernet (RoCE) can bypass the CPU and directly perform RDMA read and write operations on the GPU memory, registering it as an RDMA-accessible area. The DPU loads the Kafka consumer group configuration and initiates a JoinGroup; furthermore, it establishes a bidirectional Doorbell channel between the DPU and the GPU through PCIe MMIO mapping.

[0153] Step S402: DPU-side Fetch and Preprocessing.

[0154] During implementation, a lightweight Kafka client module can run on the DPU, consisting of the following sub-modules: (1) Fetch request generator: Based on the currently subscribed partition list (corresponding to the partition information in the above embodiment) and the "confirmed offset" (corresponding to the first data identifier in the above embodiment), construct a KafkaFetchRequest binary message (corresponding to the data acquisition request in the above embodiment) and run it on the DPU Arm core.

[0155] (2) TCP / IP Hardware Offload Engine: The built-in network engine ConnectX completes TCP handshake, flow control and packet reassembly. After the Fetch response arrives, the network engine automatically strips the header identifier bits from L2 to L4 and writes the Kafka application layer payload, i.e. the byte array in the value field of the message, into the DPU's internal staging buffer.

[0156] (3) RecordBatch boundary identifier: used to scan the fixed offset position of the payload (such as the batch_length field of the RecordBatch Header in the Kafka protocol, which is located at the offset of bytes 8 to 11, a total of four bytes), thereby cutting the continuous stream data into independent batches (each batch can correspond to a data fragment in the above embodiment). This step corresponds to O(1) complexity.

[0157] (4) Back pressure controller: Reads the read pointer of the GPU-side Ring Buffer (maintained locally in the DPU via a reverse Doorbell, corresponding to the second prompt information in the above embodiment), calculates the current Ring buffer occupancy rate, and dynamically adjusts the Fetch rhythm accordingly. The back pressure control process based on occupancy rate can be found in steps S121 to S124 and steps S131 to S132 of the above streaming data processing method, and will not be repeated here.

[0158] During implementation, the entire backpressure decision can be made independently by the DPU Arm core, without the CPU being aware of it.

[0159] Step S403: GPUDirect RDMA directly transfers data to the Ring Buffer.

[0160] During implementation, the DPU's built-in RDMA engine is used to write the identified RecordBatch directly to the GPU's pre-allocated RingBuffer via GPUDirect RDMA using Direct Memory Access (DMA).

[0161] The overall design of the Ring Buffer is as follows: It has a multi-slot structure, with each slot containing a fixed-length metadata header and a variable-length data area (corresponding to the first storage area and the second storage area in the above embodiments, respectively). For example, the 32-byte metadata header may include: producer batch processing limit batch_size, consumer processing progress tentative_offset, partition number partition_id, write timestamp, and status word state (corresponding to the status identifier in the above embodiments).

[0162] The state word corresponds to a five-state model, which can be represented as follows: from FREE state (00) to WRITING state (01), from WRITING state (01) to READY state (02), from READY state (02) to PROCESSING state (03), from PROCESSING state (03) to DONE state (04), and from DONE state (04) to FREE state (00). In implementation, the state transition of each slot can be completed through atomic CAS operations of the GPU and DPU respectively, avoiding data races.

[0163] Data processing progress is maintained using a three-pointer model. The write pointer (write_ptr) is exclusively maintained by the DPU, the read pointer (read_ptr) is written to the DPU register by the GPU and maintained by the DPU, and the ack pointer (ack_ptr) is maintained by the DPU for offset commit. The three pointers wrap around each other in a circular fashion, with the corresponding timing relationship: ack_ptr later than read_ptr, and read_ptr later than write_ptr, thus ensuring at-least-once processing.

[0164] Step S404: Notify the GPU via Doorbell.

[0165] Here, after the DPU completes writing a batch of data and sets the corresponding slot status word to READY, it updates write_ptr and writes a new value to the GPU's Doorbell register. On the GPU side, a continuously running CUDA Kernel (such as the Persistent Kernel) can poll the Doorbell register. Upon detecting that write_ptr has been pushed forward, it uses the parsing Kernel (which can be started only once initially and called whenever needed for data processing) to process the new batch.

[0166] In implementation, the Doorbell register can be mapped to a specific address in the GPU's video memory. The DPU's RDMA write operations can directly address this address, thus reducing the single Doorbell latency to the hundreds of nanoseconds. For example, address mapping can be achieved through CUDA driver interfaces (such as cuMemHostRegister combined with cudaHostGetDevicePointer). Compared to the traditional Doorbell mechanism, which points from the CPU to the peripheral, this solution implements a direct Doorbell from peripheral to peripheral (i.e., from the DPU to the GPU).

[0167] Step S405: Parallel parsing of CUDA Kernel on the GPU side.

[0168] Here, the GPU's parsing process can be executed in series by four CUDA Kernels.

[0169] For example, Kernel-1 is first used for prefix sum address calculation: based on the number of messages in RecordBatchHeader (Record Count) and the variable length (varint) of each Record, the starting offset of each Record is calculated in O(logn) time using a parallel Prefix-Sum algorithm; Then, Kernel-2 is used for multi-threaded independent decoding: each CUDA thread is responsible for one record, extracting the key, value, timestamp, and header fields according to the Kafka protocol specification. There are no dependencies between threads, and it can be fully parallelized on thousands of cores; Furthermore, Kernel-3 is used to perform Warp-level parallel CRC32: a Warp consisting of 32 threads collaboratively processes the CRC32 of a message and outputs an error bitmap, where each bit in the bitmap represents the verification result of a message; Finally, Kernel-4 writes the validated Scatter fields to the column vector: filtering out failed validation messages based on the Bitmap, and writing the validated Scatter fields to the pre-allocated columnar buffer (Key column, Value column, Timestamp column). After completion, the columnar buffer can be mounted as tabular data (such as cuDF DataFrame) with zero copy using protocols for different GPU array libraries (such as the cuda_array_interface protocol).

[0170] Step S406: The GPU generates a CRC bitmap and sends a reverse Doorbell notification to the DPU.

[0171] Here, the CRC bitmap is the Error Bitmap generated by Kernel-3 mentioned above, which is written by the GPU to the pre-registered RDMA memory inside the DPU via GPUDirect P2P. At the same time, the GPU writes a signal of "Message verification completed in batch N" (corresponding to the second prompt message mentioned above) to the DPU's reverse Doorbell register. Once the verification of a batch of messages (i.e., a data fragment) is completed, a corresponding completion signal can be sent.

[0172] During implementation, the error recovery decision state machine can be maintained on the DPU side: If the failure rate (i.e., the proportion of records that fail to pass in batch N to the total number of records in this batch, similar to the target proportion of records that pass the verification in the above embodiment) is less than 1%, then the offset corresponding to the message is marked as committable, and the messages (records) that fail the verification are added to the skip list (skipped for submission in the next round of Fetch, and compensated by the application layer). If the failure rate is greater than or equal to 1% but less than 10%, the batch can be re-fetched. The DPU can automatically resend the FetchRequest and reset the tentative_offset corresponding to the batch.

[0173] When the failure rate is greater than or equal to 10%, the link can be considered damaged. The DPU will then alert the CPU and suspend consumption, awaiting administrator intervention.

[0174] During implementation, the entire state machine can run entirely on the DPU Arm core, achieving zero CPU involvement, and only waking up the CPU to intervene in the most severe "link damage" scenario.

[0175] Step S407: DPU two-stage offset submission.

[0176] During implementation, the DPU can be used to maintain the following three offset levels: Last_fetched_offset: The offset used in the most recent FetchRequest.

[0177] tentative_offset: The maximum offset that has been written to the Ring Buffer but has not yet been verified by the GPU (corresponding to stage one).

[0178] committed_offset: The maximum offset that the GPU has confirmed has passed the verification and can be committed to the Kafka Broker (corresponding to phase two).

[0179] The timing sequence for each water level in the entire data link is as follows: t1: After the DPU writes a batch of data to the Ring Buffer, it advances the tentative_offset; t2: After the GPU verification passes, the reverse Doorbell is triggered, advancing committed_offset; t3: When the DPU timer (e.g., every 100 milliseconds) is triggered, committed_offset is submitted to the Broker node that maintains the consumer offsets topic.

[0180] In the event of a failure, fault recovery is guaranteed based on the offset water level maintained above: In the event of a GPU crash recovery, the CPU detects a CUDA anomaly and restarts the GPU context. The DPU rolls back the committed_offset and re-fetches from committed_offset+1, thus ensuring at-least-once semantics. In the event of a DPU crash recovery, the CPU restarts the DPU unloading program, and the DPU pulls its own last committed_offset (corresponding to Last_fetched_offset) from the Kafka Broker as the initial position, automatically resuming consumption.

[0181] Building upon this, we can further enhance the semantics of exactly-Once by maintaining a deduplication bitmap in the columnar buffer generated by Kernel-4. For example, it can be indexed by producer ID and sequence number. The GPU queries the bitmap before writing to the Scatter, skipping already processed messages. This, combined with the two-stage offset, enables end-to-end exactly-Once.

[0182] Step S408, Exception Handling and Rebalance Response Here, exception handling and rebalance responses are handled by the DPU, which differs from GPU-accelerated Kafka solutions in related technologies. Instead, it is a consumer group protocol implementation on the DPU Arm core.

[0183] Regarding JoinGroup and SyncGroup protocols: When the DPU starts, it can participate in partition allocation by sending a JoinGroupRequest to the GroupCoordinator and save the allocation results (assigned partitions) in the DPU state machine. Furthermore, the DPU notifies the CPU of the list of partitions that this node should consume through the PCIe doorbell (this notification can be done only once during initialization). Based on this, the CPU allocates the corresponding Ring Buffer area in the GPU memory for these partitions. After that, the CPU no longer participates in any consumer group logic.

[0184] For the Heartbeat protocol: Heartbeat sending can be triggered by a hardware timer on the DPU side (e.g., every 3 seconds), and completed by the Arm core in interrupt context. In implementation, this timer can be independent of CPU scheduling, so that even if the CPU is blocked due to busy operations, it will not be kicked out of the consumer group due to Heartbeat timeout.

[0185] For the rebalance coordination process: When the DPU receives an error code (such as REBALANCE_IN_PROGRESS), the error recovery state machine described above can perform the following operations: Pause Fetch, i.e., stop sending new FetchRequests; send a "flush" doorbell to the GPU, waiting for all batches being processed in the Ring Buffer to complete verification, i.e., read_ptr catches up with write_ptr; commit committed_offset to the Broker; rejoin the JoinGroup to accept new partition allocations; if the number of partitions changes, notify the CPU to adjust the Ring Buffer partition mapping, otherwise skip this step; resume Fetch. Throughout the entire rebalance process, the CPU only participates in memory mapping adjustment once when the number of partitions changes, without the intervention of runtime data paths.

[0186] In this embodiment, compared with the traditional CPU-involved path scheme, this scheme brings significant improvements in multiple dimensions, which can greatly reduce end-to-end latency and CPU utilization. More importantly, there are two system-level differences: First, the schemes in related technologies still require CPU participation in consumer group protocol processing during Rebalance, while this scheme completely sinks the protocol to the DPU, achieving complete CPU-free operation; Second, the Kafka consumer consistency semantics in related technologies depend on manual maintenance at the application layer, while this invention has built-in guarantees at the system layer (through the DPU two-stage state machine + GPU verification), which can significantly simplify the development of upper-layer applications.

[0187] This application also proposes a computer program including computer-readable code, which, when run in an electronic device, enables a processor in the electronic device to execute methods provided in this application.

[0188] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the method provided in this application.

[0189] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the method provided in this application when executed by a processor.

[0190] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and devices according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0193] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

[0194] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0195] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0196] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0197] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0198] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0199] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0200] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0201] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks. The above descriptions are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A streaming data processing method, applied to a data processor, the method comprising: Obtain streaming data from a distributed stream processing cluster; The streaming data includes at least one data fragment; The streaming data is sent to the first storage unit of the graphics processor via remote direct memory access; Based on a preset address mapping relationship, a first prompt message is sent to the second storage unit of the graphics processor; The first notification message indicates that at least one first data fragment has been successfully sent; Obtain the second prompt information from the graphics processor; The second prompt message indicates that at least one of the second data fragments in the first data fragment has been verified. The utilization rate of the first storage unit is determined based on the number of the first data fragment and the second data fragment, respectively. Based on the occupancy rate, the acquisition rate of the streaming data is adjusted.

2. The method according to claim 1, wherein the first storage unit comprises a plurality of sub-storage units; The step of sending the streaming data to the first storage unit of the graphics processor via remote direct memory access includes: By means of remote direct memory access, each data fragment is sent to a sub-storage unit whose status identifier represents a data fragment that can be stored, according to the original order of each data fragment in the streaming data. The step of sending a first prompt message to the second storage unit of the graphics processor based on a preset address mapping relationship includes: Based on the preset address mapping relationship, the first prompt information pointing to the first sub-storage unit is sent to the second storage unit; the first sub-storage unit is a sub-storage unit that stores the first data fragment, and the status identifier of the first sub-storage unit indicates that the data fragment stored therein is pending verification.

3. The method according to claim 2, wherein the plurality of sub-storage units in the first storage unit are arranged in a ring; the step of determining the occupancy rate of the first storage unit based on the respective quantities of the first data fragment and the second data fragment includes: Based on the first position of the first sub-storage unit in the first storage unit, determine the first number of the first data fragments; Obtain the second location of the second sub-storage unit storing the second data fragment within the second storage unit; Based on the second position, determine the second quantity corresponding to the second data fragment; The occupancy rate is determined based on the difference between the first quantity and the second quantity, and the total number of sub-storage units in the first storage unit.

4. The method according to any one of claims 1 to 3, wherein adjusting the acquisition rate of the streaming data based on the occupancy rate comprises: In response to the occupancy rate being not less than a first occupancy rate threshold, the receiving window for receiving the streaming data from the distributed streaming processing cluster is reduced, and / or the frequency of sending data acquisition requests to the distributed streaming processing cluster is reduced, so as to reduce the acquisition frequency of the streaming data. In response to the occupancy rate being less than a second occupancy rate threshold, the data size of the streaming data obtained from the distributed streaming processing cluster each time is increased to improve the acquisition rate of the streaming data; the first occupancy rate threshold is greater than the second occupancy rate threshold.

5. The method according to any one of claims 1 to 3, wherein each data segment includes at least one sub-data item; The acquisition of streaming data from the distributed stream processing cluster includes: Send a first data acquisition request carrying a first data identifier to the distributed stream processing cluster; the first data identifier is the data identifier of the third data fragment in the stream data; the first data fragment is at least one data fragment in the stream data that is located after and adjacent to the third data fragment; Receive the first data fragment returned by the distributed stream processing cluster; The method further includes: In response to receiving the second prompt information, the target percentage of the first sub-data that has passed the verification in each second data segment is obtained; Based on the target proportion, a second data identifier is determined; Send a second data acquisition request carrying the second data identifier to the distributed stream processing cluster; the second data identifier may be the same as or different from the first data identifier.

6. The method according to claim 5, wherein determining the second data identifier based on the target proportion includes: In response to the existence of a fourth data segment in the at least one second data segment whose target proportion is not less than the minimum preset ratio but less than the maximum preset ratio, the data identifier of the fifth data segment located before the first fourth data segment in the streaming data is determined as the second data identifier.

7. The method according to claim 5, wherein determining the second data identifier based on the target proportion includes: In response to the target proportion of each second data segment being not less than the maximum preset ratio, the data identifier of the sixth data segment in the streaming data is determined as the second data identifier; the sixth data segment is a data segment located after the second data segment and adjacent to the second data segment; The second sub-data that fails verification in each second data segment is sent to the target processor; the target processor runs an application that needs to use the second data segment.

8. The method according to claim 5, further comprising: In response to the existence of a seventh data fragment in at least one second data fragment where the target proportion is less than a minimum preset ratio, the sending of data acquisition requests to the distributed stream processing cluster is stopped, and an alarm message is sent to the central processing unit.

9. The method of claim 5, further comprising one of the following: In response to a failure of the graphics processor, a third data acquisition request carrying a third data identifier is sent to the distributed stream processing cluster; the third data identifier is the next data identifier of the largest data identifier corresponding to the data fragment that the graphics processor has verified. In response to a failure of the data processor, a fourth data acquisition request carrying a fourth data identifier is sent to the distributed stream processing cluster; the fourth data identifier is the data identifier carried in the last data acquisition request sent to the distributed stream processing cluster before the data processor failed.

10. The method according to any one of claims 1 to 3, wherein acquiring streaming data from the distributed stream processing cluster comprises: Based on partition information, the streaming data is pulled from the first target partition of the distributed stream processing cluster; The method further includes: In response to an update to the partition information, the retrieval of streaming data from the distributed stream processing cluster is stopped. Obtain a third prompt message from the graphics processor; the third prompt message indicates that the data fragments acquired before the partition information was updated have been verified. Based on the updated partition information, the streaming data continues to be pulled from the second target partition of the distributed stream processing cluster.

11. A streaming data processing method applied to a graphics processor, the graphics processor comprising a first storage unit and a second storage unit; the method comprising: Acquire the streaming data sent by the data processor to the first storage unit; The streaming data includes at least one data fragment; Obtain the first prompt information sent by the data processor to the second storage unit; The first notification message indicates that at least one first data fragment has been successfully sent; In response to receiving the first prompt information, the at least one first data fragment is verified; Based on the preset address mapping relationship, a second prompt message is sent to the data processor; The second prompt message indicates that at least one of the second data fragments in the first data fragment has been verified.

Images