Method, device and equipment for processing stream computing data and computer storage medium

Abstract

The embodiment of the application provides a kind of stream computing data processing method, device, equipment and computer storage medium, wherein the method comprises: calculating the time range of target event occurrence and window water line according to event data in window, and processing stream computing data based on the time range of target event occurrence and window water line calculated.The stream computing data processing method according to the embodiment of the application can avoid window to calculate in advance, and guarantee the integrity of window data.

Description

Technical Field

[0001] This application belongs to the field of Internet technology, and in particular relates to a method, apparatus, device and computer storage medium for processing streaming computing data. Background Technology

[0002] Currently, in real-time streaming computing for big data, data processing is performed sequentially. For out-of-order issues, mainstream streaming computing frameworks use a combination of watermarks and windows. However, during processing, the maximum window latency is manually set by developers. Improper settings can affect the integrity of the computational data, and the existing window triggering conditions are easily affected by dirty data, also leading to incomplete computational data and impacting the final calculation results. Summary of the Invention

[0003] This application provides a method, apparatus, device, and computer storage medium for processing streaming computing data. It can calculate the time range of events and the window watermark based on the event data of the window, thereby deleting problematic data and avoiding premature calculation of the window, thus ensuring the integrity of the data.

[0004] In a first aspect, embodiments of this application provide a method for processing streaming computing data, the method comprising: Retrieve event data within the window. The event data within the window includes the occurrence time of multiple events, the delay of multiple events, the target event data, and the maximum delay. The target event data includes the occurrence time of the target event and the delay of the target event. Determine the time range of events within the window based on the time of event occurrence; Delete the target event data when the time of the target event is outside the time range of the event occurrence; Window latency is calculated based on the latency of multiple events; The target window water level is determined based on window latency, target event latency, maximum event time and maximum latency, where the maximum event time is the maximum time in the event data within the window. Streaming data is processed based on the target window water level and the time range of events occurring within the window.

[0005] Secondly, embodiments of this application provide a streaming computing data processing apparatus, the apparatus comprising: The acquisition module is used to acquire event data within the window. The event data within the window includes the occurrence time of multiple events, the delay of multiple events, the target event data, and the maximum delay. The target event data includes the occurrence time of the target event and the delay of the target event. The first determination module is used to determine the time range of events occurring within the window based on the time of the event occurrence. The deletion module is used to delete the target event data when the time of the target event is outside the time range of the event occurrence; The first calculation module is used to calculate window latency based on the latency of multiple events; The second determining module is used to determine the target window water level based on the window delay, the target event delay, the maximum event time and the maximum delay, where the maximum event time is the maximum time in the event data within the window. The processing module is used to process streaming computing data based on the target window water level and the time range of events occurring within the window.

[0006] Thirdly, embodiments of this application provide a streaming computing data processing device, the device comprising: Processor, and memory storing computer program instructions; The processor reads and executes computer program instructions to implement the streaming computing data processing method of the first aspect.

[0007] Fourthly, embodiments of this application provide a computer storage medium. The computer storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, they implement the first aspect of the method for processing streaming computing data.

[0008] The streaming computing data processing method, apparatus, device, and computer storage medium of this application embodiment can determine the time range of an event based on event data within a window, delete problematic data, and simultaneously calculate the window's water level data and process the streaming computing data according to the window's water level, thus ensuring the integrity of the data calculated in the window. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating a method for processing streaming computing data provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a streaming computing data processing device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a streaming computing data processing device provided in an embodiment of this application. Detailed Implementation

[0011] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0012] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0013] Currently, in real-time streaming computing for big data, mainstream streaming computing frameworks use a combination of watermarks and windows to handle out-of-order issues. However, in these current methods, the presence of outlier data can cause windows to be calculated prematurely, leading to incomplete data within those windows.

[0014] To address the problems of the prior art, embodiments of this application provide a method, apparatus, device, and computer storage medium for processing streaming computing data. The method for processing streaming computing data provided in this application embodiment will be described first below.

[0015] Figure 1 A flowchart illustrating a method for processing streaming computing data according to an embodiment of this application is shown. Figure 1 As shown, the method may include the following steps: S110. Obtain event data within the window. The event data within the window includes the occurrence time of multiple events, the delay of multiple events, the target event data, and the maximum delay. The target event data includes the occurrence time of the target event and the delay of the target event.

[0016] In some embodiments, the event data acquired within the window is data sorted in ascending order according to the events in which they occurred, as shown in Table 1 for example: Table 1 Event Data within the Window The data is arranged in ascending order: the first column is the event sequence number; the second column is the event occurrence time; the third column is the actual event processing time; the fourth column is the event latency; the fifth column is the event window latency; and the sixth column is the event content. The window latency data is calculated based on the event latency data.

[0017] In some embodiments, the window start time is `window_start_time`, the window end time is `window_end_time`, the window time interval is `interval = window_end_time - window_start_time`, and the window is defined as (`window_start_time`, `window_end_time`). The event occurrence time is `event_time`, the event processing time is `event_process_time`, and the latency is... .

[0018] S120. Determine the time range of events within the window based on the time of event occurrence.

[0019] In some embodiments, a time range is determined based on the time when events occur within the window, and this time range includes the time when events occur for non-abnormal data.

[0020] S130. When the time of the target event is outside the time range of the event, delete the target event data.

[0021] In some embodiments, when the occurrence time of a target event is outside the time range of events occurring within the window, it is determined to be abnormal data and the target event data is deleted.

[0022] S140. Calculate window delay based on the delay of multiple events.

[0023] In some embodiments, the window latency corresponding to the target event is calculated based on the latency of multiple events within the window.

[0024] S150. Determine the target window water level based on window delay, target event delay, maximum event time and maximum delay. The maximum event time is the maximum time in the event data within the window.

[0025] In some embodiments, the target window watermark is watermark_time, the maximum event time is currentMaxTimestamp, and the maximum latency is [missing information]. The formula for calculating the window water level is as follows: S160. Process streaming data based on the target window water level and the time range of events occurring within the window.

[0026] In some embodiments, the determination is made based on the target window watermark and the time range of events occurring within the window. When watermark_time ≥ window_end_time, window calculation is triggered.

[0027] The streaming computing data processing method provided in this application can calculate the time range of the target event based on the event data in the window, delete abnormal target event data, and determine the window water level based on the time data in the window, thereby avoiding premature calculation of the window caused by abnormal data and ensuring the integrity of the window data.

[0028] In some embodiments, determining the time range of events occurring within a window based on the time of event occurrence includes: calculating the upper and lower quantiles based on the time of event occurrence; and determining the time range of events occurring within the window based on the upper and lower quantiles. When the window contains n data entries, the event occurrence time T is denoted as... Based on the data in Table 1, n=5, lower quantile The calculation formula is as follows: Substitute the data from Table 1, The calculation yields: The first and second terms represent the times when the events of the first and second data points in the n data points occurred. For ease of calculation, 10:00:00 on January 2, 2020 is taken as time 0, 10:00:01 on January 2, 2020 is taken as time 1, and so on.

[0029] Upper Hand The calculation formula is as follows: Substitute the data from Table 1, The calculation yields: Calculate the quantile distance (IQR) based on the upper and lower quantiles: IQR = - =4.5 - 0.5 = 4, The formula for calculating the upper limit (max) of the time range is as follows: The formula for calculating the lower limit (min) of the time range is as follows: Ultimately, the calculated time range for events occurring within the window is (-5.5, 10.5).

[0030] In some embodiments, calculating the window delay based on multiple event delays includes: calculating the average of the multiple event delays as the window delay. Based on the data in Table 1, Expected value of latency data This is the window delay. The calculation formula is as follows: Substitution Calculated .

[0031] In some embodiments, determining the target window level based on window delay, target event delay, maximum event time, and maximum delay includes: determining a maximum delay adjustment step size based on window delay, target event delay, and maximum delay; determining the target maximum delay based on the maximum adjustment step size and maximum delay; and determining the target window level based on the target maximum delay and maximum event time. The maximum delay adjustment step size is... The calculation formula is as follows: in, For the delay of the target event, The calculation formula is: Its initial value is maxOutOfOrderness. This represents the maximum delay corresponding to the previous event.

[0032] The streaming computing data processing method provided in this application can calculate the time range of the target event based on the event data in the window, delete abnormal target event data, calculate the window delay based on the event data in the window, and further calculate the maximum adjustment step size. The window water level is determined based on the maximum adjustment step size. Streaming computing based on the calculated event occurrence time range and window water level can avoid the existence of abnormal data and enable the window to be calculated in advance, thus ensuring the integrity of the window data.

[0033] Based on the above-described method for processing streaming computing data, this application also proposes a device for processing streaming computing data, the details of which are as follows: Figure 2 This is a schematic diagram of a streaming computing data processing device provided in an embodiment of this application. Figure 2 As shown, the device includes an acquisition module 210, a first determination module 220, a deletion module 230, a first calculation module 240, a second determination module 250, and a processing module 260.

[0034] The acquisition module 210 is used to acquire event data within the window. The event data within the window includes the occurrence time of multiple events, the delay of multiple events, the target event data, and the maximum delay. The target event data includes the occurrence time of the target event and the delay of the target event.

[0035] The first determining module 220 is used to determine the time range of events occurring within the window based on the time of the event.

[0036] The deletion module 230 is used to delete target event data when the time of the target event is outside the time range of the event occurrence.

[0037] The first calculation module 240 is used to calculate window delay based on the delay of multiple events.

[0038] The second determining module 250 is used to determine the target window water level based on the window delay, the target event delay, the maximum event time and the maximum delay, where the maximum event time is the maximum time in the event data within the window.

[0039] Processing module 260 is used to process streaming computing data based on the target window water level and the time range of events occurring within the window.

[0040] In some embodiments, the first determining module 220 is used to determine the time range of the event within the window based on the time of the event, including: calculating the upper quantile and the lower quantile based on the time of the event; and determining the time range of the event within the window based on the upper quantile and the lower quantile.

[0041] In some embodiments, the first calculation module 240 is used to calculate the window delay based on the delay of multiple events, including: calculating the average of the delays of multiple events as the window delay.

[0042] In some embodiments, the second determining module 250 is used to determine the target window level based on the window delay, the target event delay, the maximum event time, and the maximum delay, including: adjusting the maximum delay step size based on the window delay, the target event delay, and the maximum delay; determining the target maximum delay based on the maximum adjustment step size and the maximum delay; and determining the target window level based on the target maximum delay and the maximum event time.

[0043] The streaming computing data processing apparatus provided in this application embodiment can determine the time range of event occurrence based on event data within a window, delete abnormal data, calculate window latency, and further calculate the water level data of the window. Streaming computing based on the calculated time range of event occurrence and the water level of the window can ensure data integrity.

[0044] Figure 3A schematic diagram of the hardware structure for processing streaming computing data provided in an embodiment of this application is shown.

[0045] The device for processing streaming computing data may include a processor 301 and a memory 302 storing computer program instructions.

[0046] Specifically, the processor 301 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0047] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 302 may include removable or non-removable (or fixed) media, or memory 302 may be non-volatile solid-state memory. Memory 302 may be internal or external to the integrated gateway disaster recovery device.

[0048] In one example, memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 302 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0049] The processor 301 reads and executes computer program instructions stored in the memory 302 to achieve... Figure 1 The method / steps S110 to S160 in the illustrated embodiment are completed, and the desired outcome is achieved. Figure 1 The technical effects achieved by executing the methods / steps shown in the examples are not elaborated here for the sake of brevity.

[0050] In one example, the device for processing streaming computing data may further include a communication interface 303 and a bus 310. Wherein, as Figure 3 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 310 and complete communication with each other.

[0051] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0052] Bus 310 includes hardware, software, or both, that couples components of a device for processing streaming computing data together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 310 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0053] The streaming computing data processing device can calculate the time range of the target event and the window water level based on the event data within the window, and execute the streaming computing data processing method in this embodiment of the application, thereby achieving a combination of... Figure 1 The method described is for processing streaming computing data.

[0054] Furthermore, in conjunction with the streaming data processing methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the streaming data processing methods in the above embodiments.

[0055] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0056] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0057] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0058] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0059] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for processing streaming computing data, characterized in that, include: Acquire event data within a window, the event data within the window including the occurrence time of multiple events, the delay of multiple events, target event data, and maximum delay, the target event data including the occurrence time of the target event and the delay of the target event; The maximum delay is the maximum window delay; The time range of events occurring within the window is determined based on the time of the event's occurrence. If the time of the target event is outside the time range of the event, delete the target event data; Calculate window latency based on the latency of the aforementioned multiple events; The target window water level is determined based on the window delay, the target event delay, the maximum event time and the maximum delay, wherein the maximum event time is the maximum time in the event data within the window. The streaming data is processed based on the target window water level and the time range of events occurring within the window.

2. The method according to claim 1, characterized in that, Determining the time range of events occurring within the window based on the time of the event includes: Calculate the upper and lower quantiles based on the time of the event. The time range of events occurring within the window is determined based on the upper quantile and the lower quantile.

3. The method according to claim 1, characterized in that, The calculation of window latency based on the latency of the multiple events includes: The average value of the delays of the multiple events is calculated as the window delay.

4. The method according to claim 1, characterized in that, The step of determining the target window water level based on the window delay, the target event delay, the maximum event time, and the maximum delay includes: The maximum delay adjustment step size is determined based on the window delay, the target event delay, and the maximum delay; the maximum delay is the maximum window delay. The target maximum delay is determined based on the maximum delay adjustment step size and the maximum delay. The target window water level is determined based on the maximum delay of the target and the maximum time of the event.

5. A processing device for streaming computing data, characterized in that, The device includes: The acquisition module is used to acquire event data within a window. The event data within the window includes the occurrence time of multiple events, the delay of multiple events, target event data, and the maximum delay. The target event data includes the occurrence time of the target event and the delay of the target event. The first determining module is used to determine the time range of the event occurring within the window based on the time of the event's occurrence. The deletion module is used to delete the target event data when the time of the target event is outside the time range of the event occurrence; The first calculation module is used to calculate the window delay based on the delay of the multiple events; The second determining module is used to determine the target window water level based on the window delay, the target event delay, the maximum event time and the maximum delay, wherein the maximum event time is the maximum time in the event data within the window; The processing module is used to process streaming computing data based on the target window water level and the time range of events occurring within the window.

6. The apparatus according to claim 5, characterized in that, The first determining module is used to determine the time range of events occurring within the window based on the time of the event, including: Calculate the upper and lower quantiles based on the time of the event. The time range of events occurring within the window is determined based on the upper quantile and the lower quantile.

7. The apparatus according to claim 5, characterized in that, The first calculation module is used to calculate window latency based on the latency of the plurality of events, including: The average value of the delays of the multiple events is calculated as the window delay.

8. The apparatus according to claim 5, characterized in that, The second determining module is used to determine the target window water level based on the window delay, the target event delay, the maximum event time, and the maximum delay, including: The maximum delay adjustment step size is determined based on the window delay, the target event delay, and the maximum delay. The target maximum delay is determined based on the maximum delay adjustment step size and the maximum delay. The target window water level is determined based on the maximum delay of the target and the maximum time of the event.

9. A device for processing streaming computing data, characterized in that, The device for processing streaming computing data includes: a processor and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the streaming computing data processing method as described in any one of claims 1-4.

10. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the streaming computing data processing method as described in any one of claims 1-4.

Images