Coded stream processing

Abstract

Coded stream processing is provided. A system and method for encoding streaming data to facilitate recovery from faulty or slow processor (s) is described herein. A batch of processing stream data may be partitioned into a plurality of data blocks. The parity block (s) are used for a plurality of data blocks. The plurality of data blocks and parity block (s) may be provided to a processor for processing. Processed data of at least some (e.g., one or more) data blocks of the plurality of data blocks and processed data of parity block (s) are received. When it is determined that processed data for a predefined number of data blocks has not been received within a predefined time period, the parity block (s) is received based at least in part on the received processed parity block (s) and the received processed data block (s). The processed data is determined for the particular data block (s) of the particular processor (s) from which the processed data is not received.

Description

This application is a divisional application of the invention patent application entitled "Encoded Stream Processing", filed on August 18, 2020, with application number 202080069815.9. Background Technology

[0001] Stream processing allows for the real-time processing of multiple data streams from multiple sources. A “stream” or stream processing refers to receiving a stream of data, processing the data by multiple processors, and then returning the processed data as a single stream. Summary of the Invention

[0002] This document describes a system for encoding streaming data, comprising: a processing system including a processor and a memory storing computer-executable instructions thereon, which, when executed by the processor, cause the processing system to perform the following operations: receiving processed data of at least some of a plurality of data blocks; receiving processed data of at least one parity block; determining, at least in part, processed data of one or more specific data blocks from which processed data has not been received by one or more specific processors when it is determined that processed data for a predefined number of data blocks has not been received within a predefined time period; and providing the received plurality of processed data blocks and the determined processed data.

[0003] This summary is provided to introduce, in a simplified form, some concepts that will be further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0004] Figure 1 This is a functional block diagram illustrating a system for encoding streaming data.

[0005] Figure 2 This is a functional block diagram showing the streaming data encoding component.

[0006] Figure 3 This is a functional block diagram showing the streaming data destination component.

[0007] Figure 4 This is a flowchart illustrating a method for encoding streaming data.

[0008] Figure 5 This is a flowchart illustrating a method for processing encoded stream data.

[0009] Figure 6 This is a flowchart illustrating a method for encoding streaming data.

[0010] Figure 7 This is a flowchart illustrating a method for processing encoded stream data.

[0011] Figure 8 This is a functional block diagram illustrating an exemplary computing system. Detailed Implementation

[0012] Various techniques relating to encoded stream processing will now be described with reference to the accompanying drawings, wherein the same reference numerals are used throughout to refer to the same elements. In the following description, numerous specific details are set forth for illustrative purposes in order to provide a thorough understanding of one or more aspects. However, it will be apparent, however, that these aspects can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form for ease of description of one or more aspects. Furthermore, it should be understood that functions described as being performed by certain system components can be performed by multiple components. Similarly, for example, components can be configured to perform functions described as being performed by multiple components.

[0013] This topic discloses various products and processes that support performing or being configured to perform various actions related to encoded stream processing. Below are one or more exemplary systems and methods.

[0014] The aspects disclosed in this subject matter relate to technical problems concerning error correction for stream processing. Technical features associated with solving this problem include: dividing a batch of processing stream data into multiple data blocks; generating at least one parity block for the multiple data blocks; providing the multiple data blocks and one or more parity blocks to a processor for processing; receiving processed data from at least some of the multiple data blocks; receiving processed data from at least one parity block; when it is determined that processed data for a predefined number of data blocks has not been received within a predefined time period, determining, at least in part, processed data for one or more specific data blocks from which processed data has not been received, for one or more specific processors; and providing the received processed data blocks and the determined processed data. Accordingly, these technical features demonstrate the technical effects of more efficiently and effectively recovering from multiple processor (e.g., compute node) failures or slow processing in a stream processing system, for example, reducing network bandwidth consumption and / or the consumption of compute node processing resources.

[0015] Furthermore, the term "or" is intended to mean inclusive "or," not exclusive "or." That is, unless otherwise stated or clear from the context, the phrase "X adopts A or B" is intended to mean any natural inclusive permutation. Specifically, the phrase "X adopts A or B" is satisfied by any of the following instances: X adopts A; X adopts B; or X adopts both A and B. Additionally, the articles "a" and "an" used in this application and the appended claims should generally be interpreted as meaning "one or more," unless otherwise stated or clear from the context for the singular form.

[0016] As used herein, the terms “component” and “system” and their various forms (e.g., component, system, subsystem, etc.) are intended to refer to computer-related entities that can be hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an instance, an executable program, an execution thread, a program, and / or a computer. For instance, an application running on a computer and the computer itself can both be components. One or more components may reside in an execution process and / or execution thread, and components may be located on one computer and / or distributed across two or more computers. Furthermore, as used herein, the term “exemplary” is intended to indicate illustration or example of something and is not intended to indicate preference.

[0017] In some embodiments, the techniques disclosed herein are used in conjunction with pull-based stream processing employing anchor-based techniques, which facilitate one-and-only-once processing by the output processor(s). An "anchor" is generated by the input processor (e.g., along with the data in the output stream) and describes a point in the output stream of the input processor such that other data units in the output stream(s) precede or follow any given anchor. For the purposes of discussion, data "flows" downstream from the input processor through any intermediate compute processor(s)(s) to the output processor. Requests for data are sent upstream from the output processor through any intermediate compute processor(s)(s) to the input processor.

[0018] To reduce latency, streaming data can be divided into blocks, with each block processed in parallel by a processor (e.g., an intermediate compute processor). When performing parallel processing of streaming data, some processors used for (multiple) partitions may fail for a period of time and / or some processors may be slower due to hardware and / or software issues, noisy neighbors, etc.

[0019] This document describes a system and method for encoding streaming data to include one or more parity blocks. In the event that processed streaming data is not received in a timely manner from one or more processors, the encoding destination can utilize the parity block(s) to deduce the processed streaming data associated with the one or more processors from which the processed streaming data was not received. The encoding destination can then provide the deduced processed streaming data along with the received processed streaming data. By encoding the streaming data, the system can accommodate a predefined number of problematic processor(s) and / or communication paths(s), thereby allowing the processing of streaming data without a significant increase in latency and / or interruption.

[0020] refer to Figure 1 The diagram illustrates a system 100 for encoding streaming data. System 100 includes a streaming data encoding component 110, multiple processors 120, and a streaming data destination component 130. The streaming data encoding component 110 and the streaming data destination component 130 utilize an encoding scheme to successfully recover from failures and / or delayed arrival of processed data from one or more processors 120.

[0021] In some embodiments, the streaming data encoding component 110 is a component of an input node (not shown) of a streaming data processing system. For example, when sending data to multiple processors 120, a client may utilize an encoding client library. In some embodiments, the streaming data encoding component 110 is a component of a compute node / intermediate node (not shown) of a streaming data processing system.

[0022] In the event that processed streaming data is not received in a timely manner from one or more processors 120, the streaming data sink component 130 can use encoded information (e.g., parity blocks) to deduce the streaming data associated with the one or more processors 120 that did not receive processed streaming data from it in a timely manner. The streaming data sink component 130 can then provide the deduced streaming data (e.g., in place of the one or more processors 120 that did not receive processed streaming data from it in a timely manner) together with processed streaming data from other processors(multiple)120, for example, to a data storage device 140. In this way, the system 100 can accommodate a predefined number of problematic processors(multiple)120, thereby allowing streaming data to be processed without significant interruption.

[0023] Turning Figure 2 The diagram illustrates a streaming data encoding component 110. The received streaming data is segmented (e.g., divided) into multiple blocks (M data blocks) by a streaming data segmentation component 210. In some embodiments, the number of blocks (M) is predefined, for example, at least in part based on the number of processors 120.

[0024] Next, the parity block generation component 220 uses a parity algorithm to generate one or more parity blocks (P parity blocks) associated with multiple blocks. In some embodiments, the Reed-Solomon parity algorithm is employed. The Reed-Solomon algorithm derives redundant data from the coefficients of a polynomial (e.g., of degree M) generated by multiplying the data by a generator polynomial. In some embodiments, the Reed-Solomon algorithm represents the maximum efficiency scenario of M data units with P redundant data units (e.g., the total number of units processed K = M + P). Specifically, given the number of data block units M and the fault tolerance P, the Reed-Solomon algorithm generates P redundant data units. The Reed-Solomon algorithm provides a minimum amount of redundant data (e.g., K = M + P) for data recovery in the event of P failures (e.g., the loss of P data units and / or (multiple) processor 120 failures). When data recovery is required, the still-usable values ​​can be interpolated back into the polynomial expression / algorithm to solve for the lost values ​​and reconstruct the lost data accordingly.

[0025] For illustrative purposes, and not for limitation, consider a scenario using six data units (M=6). In this scenario, a representative Reed-Solomon example is given for six data units (D0, D1, D2, D3, D4, D5) and a fault tolerance of 2 (P1, P2), where the exemplary polynomials for parity blocks P1 and P2 are: Equation (1) is P0 = D0 + D1 + D2 + D3 + D4 + D5 P1=g 0 D0+ g 1 D1+ g 2 D2+ g 3 D3+ g 4 D4+ g 5 Equation D5 (2) Where g x The values ​​are configurable coefficients of the polynomial. In this scenario, a tolerance of 2 is recoverable. For example, if all six data units are received and two parity units fail, no parity unit is needed; however, the parity unit can be regenerated based on equations (1) and (2). If a single data unit, such as D1, is not received, then given P0 and P1, one of the two linear equations (1) and (2) can be solved for D1. If two data units, such as D1 and D2, are not received, then given P0 and P1, two linear equations (1) and (2) can be solved for D1 and D2.

[0026] In some embodiments, the number of parity blocks generated is proportional to the amount of unreceived processed data (e.g., faulty and / or slow processors 120) from which the streaming data sink component 130 can recover. That is, a single parity block can be used to recover from a single faulty processor 120, two parity blocks can be used to recover from up to two faulty processors 120, and so on. In some embodiments, increasing the number of parity blocks increases transmission and / or processing costs.

[0027] In some embodiments, one or more parity blocks are generated across all multiple data blocks (M blocks). For example, streaming data can be divided into one hundred data blocks and two parity blocks are generated across these one hundred data blocks. System 100 can accommodate up to two faulty (or slow) processors 120 out of a maximum of 102 processors 120 (when parity blocks are processed separately from data blocks) or 100 processors 120 (when parity blocks are processed together with data blocks). In the event that processed data is not received in a timely manner, processing by the streaming data sink component 130 may involve processed data received from other processors 120, including the received processed parity blocks. The latency associated with data recovery performed by the streaming data sink component 130 increases proportionally to the number of data blocks associated with the parity blocks.

[0028] In some embodiments, multiple data blocks are grouped into parity groups, wherein one or more parity blocks are generated for data blocks belonging to a specific parity group. By generating parity blocks for groups, the processing of the streaming data sink component 130 can be limited to the data of a specific group and the use of the parity blocks(s) associated with that specific group if processed data is not received in a timely manner.

[0029] Once the streaming data has been segmented into data blocks and one or more parity blocks have been generated, the output component 230 can provide the data blocks and / or (multiple) parity blocks to the processor 120. In some embodiments, when provided to the processor 120, the parity blocks are included together with the data blocks (e.g., the parity blocks are appended to the data blocks). In some embodiments, the processor 120 processes the parity blocks as well as the data blocks. In some embodiments, the (multiple) parity blocks are provided separately to the processor(s) 120.

[0030] Next, refer to Figure 3 The diagram shows the streaming data receiver component 130. The input component 310 receives processed data from the processor 120 and stores the data in the receiving data storage device 320.

[0031] Given the expected data (e.g., M processors 120), the off-time data determination component 330 monitors the reception of processed data (e.g., processed data received from x processors 120). The off-time data determination component 330 can determine that no processed data was received from a predefined number of processors 120 within a predefined time period. For example, no fewer than all but more than a predefined number of processed data blocks (or a specific group of processed data blocks) were received from one or more specific processors 120 within the predefined time period.

[0032] In some embodiments, the predefined number of processed data blocks is the expected number of data blocks (M) minus the number of received processed parity blocks, thereby enabling the non-timely data determination component 330 to recover / process data blocks associated with the specific processor 120 from which it has not received processed data blocks.

[0033] In some embodiments, the predefined number of processed data blocks is the expected number of data blocks for a set minus the number of parity blocks received for that set, thereby enabling the non-timely data determination component 330 to recover / process data blocks associated with a specific processor 120 from which it has not received processed data blocks.

[0034] In some embodiments, given the latency and / or performance requirements of the Service Level Agreement (SLA) for streaming data processing, a predefined time period is based at least in part on the expected amount of time (e.g., based on historical data associated with the performance of the streaming data sink component 130) for recovering / processing multiple data blocks for non-timely (e.g., faulty / slow) processing of multiple processors 120. In some embodiments, the predefined time period is based at least in part on the number of data blocks in the group and / or the number of parity blocks for recovering / processing multiple data blocks for non-timely (e.g., faulty / slow) processing of multiple processors 120.

[0035] In some embodiments, the predefined time period is static. In some embodiments, the predefined time period is dynamically adjusted based on the processor(s) current processing conditions(s) of the processor(s). In some embodiments, the predefined time period is at least partially based on the user(s) user-configurable settings(s). For example, a user can request the use of additional(or fewer) parity(s) blocks(s), which may result in increased(or decreased) processing and / or transmission costs.

[0036] When the non-timely data determination component 330 determines that no processed data has been received from a predefined number of processors 120 within a predefined time period, a data recovery process can begin. As described above, the data determination component 330 can use one or more received processed parity blocks and the received processed data to determine the processed data for (multiple) data blocks of (multiple) specific processors (e.g., (multiple) faulty or slow processors 120) from which processed data has not been received. For example, when no processed data has been received for one data block, one parity block can be used. When no processed data has been received for two data blocks, two parity blocks can be used, and so on.

[0037] Once it has been determined that no processed data blocks for the processed data have been received from the processor(s) (e.g., the fault processor(s) 120(s)) for the processed data, the output component 350 can combine the processed data derived by the data determination component 330 with the received processed data (e.g., stored in the receiving data storage device 320) and provide the combined processed data to, for example, the data storage device 140.

[0038] By efficiently and effectively selecting a certain number of data blocks and an associated number of parity blocks, system 100 can balance cost and / or latency requirements. In some embodiments, by selecting and / or dynamically adjusting the grouping strategy (e.g., the number of data blocks according to the number of parity blocks) and the time period for starting recovery, system 100 can effectively respond to unexpected parallel processing problems (e.g., a faulty processor 120, a slow processor 120, and / or a faulty / noisy communication path).

[0039] Figure 4-7 Exemplary methods related to encoded stream processing are shown. While these methods are shown and described as a series of actions performed sequentially, it should be understood and appreciated that these methods are not limited by the order of the sequence. For example, some actions may occur in a different order than those described herein. Furthermore, one action may occur concurrently with another. Moreover, in some cases, not all actions are required to implement the methods described herein.

[0040] Furthermore, the actions described herein can be computer-executable instructions, which can be implemented by one or more processors and / or stored on a computer-readable medium. Computer-executable instructions can include routines, subroutines, programs, threads of execution, etc. Moreover, the results of these actions can be stored on a computer-readable medium, displayed on a display device, etc.

[0041] refer to Figure 4 The diagram illustrates a method 400 for encoding streaming data. In some embodiments, method 400 is performed by streaming data encoding component 110.

[0042] At 410, a batch of processing stream data is divided into multiple data blocks. At 420, one or more parity blocks are generated for the multiple data blocks. The multiple data blocks and one or more parity blocks are then provided to the processor for processing.

[0043] Turning Figure 5 This illustrates a method 500 for encoding streaming data processing. In some embodiments, method 500 is performed by streaming data receiver component 130.

[0044] At 510, processed data of at least some of the data blocks (e.g., one or more data blocks) from a plurality of data blocks is received. At 520, processed data of at least one parity block from one or more parity blocks is received.

[0045] At 530, it is determined that processed data for a predefined number of data blocks has not been received within a predefined time period. At 540, when it is determined that processed data for a predefined number of data blocks has not been received within a predefined time period, processed data for one or more specific data blocks from which processed data has not been received is determined, at least in part, based on processed data from at least one or more parity blocks received and processed data from at least some of the received data blocks. At 540, the received processed data blocks and the determined processed data are provided (e.g., stored).

[0046] Next, refer to Figure 6 The diagram illustrates a method 600 for encoding streaming data. In some embodiments, method 600 is performed by streaming data encoding component 110.

[0047] At 610, a batch of processing stream data is divided into multiple data blocks. At 620, the multiple data blocks are grouped into multiple groups. At 630, for each group, one or more parity blocks are generated for the multiple data blocks of that specific group. At 640, the multiple data blocks are provided to the processor for processing. At 650, one or more parity blocks are provided to the processor(s) ...

[0048] Turning Figure 7 This illustrates a method 700 for encoding streaming data processing. In some embodiments, method 700 is performed by streaming data receiver component 130.

[0049] At 710, processed data of at least one data block from a plurality of data blocks is received. At 720, processed data of at least one parity block from one or more parity blocks is received.

[0050] At 730, it is determined that no processed data for a predefined number of data blocks for a specific group has been received within a predefined time period. At 740, when it is determined that processed data for a predefined number of data blocks for a specific group has not been received within a predefined time period, processed data for one or more specific data blocks of the specific group from which processed data has not been received is determined, at least in part, based on processed data from one or more parity blocks of the specific group and processed data from at least one data block of the specific group. At 740, the received processed data blocks and the determined processed data are provided (e.g., stored).

[0051] This document describes a system for encoding streaming data, comprising: a processing system including a processor and a memory storing computer-executable instructions thereon, which, when executed by the processor, cause the processing system to: receive processed data of at least some of a plurality of data blocks; receive processed data of at least one parity block; when it is determined that processed data for a predefined number of data blocks has not been received within a predefined time period, determine, at least in part, processed data for one or more specific data blocks from which processed data has not been received, for one or more specific processors; and provide the received plurality of processed data blocks and the determined processed data.

[0052] The system may include a memory that also stores computer-executable instructions that, when executed by a processor, cause the processing system to: divide a batch of processing stream data into multiple data blocks; generate at least one parity block for the multiple data blocks; and provide the multiple data blocks and one or more parity blocks to the processor for processing.

[0053] The system may further include a predefined number of data blocks that is at least partially based on the number of data blocks of processed data that have been received. The system may also include a predefined number of data blocks that is at least partially based on the number of parity data blocks of processed data that have been received. The system may further include at least one parity block generated based on the Reed-Solomon parity algorithm.

[0054] The system may also include a subset of data blocks that are processed substantially in parallel. The system may also include a predefined time period that is at least partially based on the number of data blocks into which the streaming data is divided and the number of parity blocks generated. The system may also include a predefined time period that is dynamically adjusted at least partially based on at least one of the following: determining the expected amount of time, latency requirements, or performance requirements for processing data for one or more specific data blocks from which no processed data has been received.

[0055] This document describes a method for coded streaming data processing, comprising: receiving processed data of at least some of a plurality of data blocks; receiving processed data of at least one parity block; when it is determined that processed data for a predefined number of data blocks has not been received within a predefined time period, determining, at least in part, processed data for one or more specific data blocks from which processed data has not been received, for one or more specific processors; and providing the received plurality of processed data blocks and the determined processed data.

[0056] The method may further include: dividing a batch of processing stream data into multiple data blocks; generating at least one parity block for the multiple data blocks; and providing the multiple data blocks and one or more parity blocks to a processor for processing. The method may further include a predefined number of data blocks that is at least partially based on the number of data blocks of processed data that have been received. The method may further include a predefined number of data blocks that is at least partially based on the number of parity data blocks of processed data that have been received.

[0057] The method may further include generating at least one parity block based on the Reed-Solomon parity algorithm. The method may further include defining a predefined time period based at least in part on the number of data blocks into which the streaming data is divided and the number of parity blocks generated. The method may further include dynamically adjusting the predefined time period based at least in part on the expected amount of time for determining processed data for one or more specific data blocks from which no processed data has been received.

[0058] This document describes a computer storage medium storing computer-readable instructions that, when executed, cause a computing device to: receive processed data of at least some of a plurality of data blocks; receive processed data of at least one parity block; when it is determined that processed data for a predefined number of data blocks has not been received within a predefined time period, determine, at least in part, processed data for one or more specific data blocks from which processed data has not been received, for one or more specific processors; and provide the received plurality of processed data blocks and the determined processed data.

[0059] The computer storage medium can also store computer-readable instructions that, when executed, cause the computing device to: divide a batch of processing stream data into multiple data blocks; generate at least one parity block for the multiple data blocks; and provide the multiple data blocks and one or more parity blocks to the processor for processing.

[0060] The computer storage medium may further include a predefined number of data blocks that is at least partially based on the number of data blocks of processed data that have been received. The computer storage medium may also include a predefined number of data blocks that is at least partially based on the number of parity data blocks of processed data that have been received. The computer storage medium may further include at least one parity block generated based on the Reed-Solomon parity algorithm.

[0061] refer to Figure 8 Example general-purpose computer, processing system, or computing device 802 is shown (e.g., mobile phone, desktop computer, laptop computer, tablet computer, watch, server, handheld device, programmable consumer or industrial electronics, set-top box, gaming system, computing node, etc.). For example, computing device 802 can be used in system 100 for encoding streaming data.

[0062] Computer 802 includes one or more processors 820, memory 830, system bus 840, mass storage device(s) 850, and one or more interface components 870. System bus 840 is at least communicatively coupled to the aforementioned system components. However, it should be understood that in its simplest form, computer 802 may include one or more processors 820 coupled to memory 830, which execute various computer-executable actions, instructions, and / or components stored in memory 830. These instructions may be, for example, instructions for implementing functions described as being performed by the aforementioned one or more components, or instructions for implementing the aforementioned one or more methods.

[0063] The processors 820 may be implemented using general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, it may be any processor, controller, microcontroller, or state machine. The processors 820 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a multi-core processor, one or more microprocessors combined with a DSP core, or any other such configuration. In one embodiment, the processors 820 may be a graphics processor.

[0064] Computer 802 may include or otherwise interact with various computer-readable media to facilitate control of computer 802 in implementing one or more aspects of the claimed subject matter. Computer-readable media may be any available medium accessible by computer 802, and includes volatile and non-volatile media, as well as removable and non-removable media. Computer-readable media may include two distinct and mutually exclusive types: computer storage media and communication media.

[0065] Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes storage devices such as memory devices (e.g., random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), etc.), magnetic storage devices (e.g., hard disks, floppy disks, cassette tapes, magnetic tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), and solid-state devices (e.g., solid-state drives (SSDs), flash memory drives (e.g., cards, sticks, key drives), etc.), or any other similar media that, in contrast to transmission and delivery, store desired information accessible by a computer 802. Accordingly, computer storage media does not include modulated data signals and signals described with respect to communication media.

[0066] Communication media contain computer-readable instructions, data structures, program modules, or other data in modulated data signals (such as carrier waves) or other transmission mechanisms, and include any information transmission medium. The term "modulated data signal" refers to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media).

[0067] Memory 830 and mass storage device(s) 850 are examples of computer-readable storage media. Depending on the exact configuration and type of the computing device, memory 830 may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or some combination of both. For example, the basic input / output system (BIOS) (including basic routines such as transferring information between components within computer 802 during startup) may be stored in non-volatile memory, while volatile memory may act as external cache memory for processing by processor(s) 820, and so on.

[0068] Multiple mass storage devices 850 include removable / non-removable, volatile / non-volatile computer storage media relative to memory 830 for storing large amounts of data. For example, multiple mass storage devices 850 include, but are not limited to, one or more devices such as disk or optical disk drives, floppy disk drives, flash memory, solid-state drives, or memory sticks.

[0069] The memory 830 and the mass storage device(s) 850 may include or store an operating system 860, one or more applications 862, one or more program modules 864, and data 866 therein. The operating system 860 is used to control and allocate the resources of the computer 802. The applications 862 include one or both system and application software, and can utilize the resource management of the operating system 860 by means of the program modules 864 and data 866 stored in the memory 830 and / or the mass storage device(s) 850 to perform one or more actions. Accordingly, the applications 862 can transform the general-purpose computer 802 into a special-purpose machine according to the logic provided therein.

[0070] All or part of the claimed subject matter may be implemented using standard programming and / or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to perform the disclosed functions. By way of example and not limitation, system 100 or a portion thereof may be or form part of application 862 and includes one or more modules 864 and data 866 stored in memory and / or (multiple) mass storage devices 850, which may perform their functions when executed by one or more processors 820.

[0071] In some embodiments, the processor(s) 820 may correspond to a system-on-a-chip (SOC) or similar architecture, including, or in other words, integrating both hardware and software on a single integrated circuit substrate. Here, the processor(s) 820 may include one or more processors, at least similar to the processor(s) 820 and memory 830, as well as memory, etc. Conventional processors include a minimal amount of hardware and software and rely heavily on external hardware and software. In contrast, a processor's SOC implementation is more powerful because it embeds hardware and software that implements specific functions with minimal or no dependence on external hardware and software. For example, system 100 and / or associated functions can be embedded within the hardware of an SOC architecture.

[0072] Computer 802 also includes one or more interface components 870, which are communicatively coupled to system bus 840 and facilitate interaction with computer 802. For example, interface component 870 may be a port (e.g., serial, parallel, PCMCIA, USB, FireWire, etc.) or an interface card (e.g., audio, video, etc.). In one example implementation, interface component 870 may be implemented as a user input / output interface, enabling a user to input commands and information into computer 802 via one or more input devices (e.g., pointing devices such as a mouse, trackball, pen, touchpad, keyboard, microphone, joystick, game pad, satellite dish, scanner, camera, other computer, etc.) (e.g., by one or more gestures or voice input). In another example implementation, interface component 870 may be implemented as an output peripheral interface supplying output to a display (e.g., LCD, LED, plasma, etc.), speakers, printer, and / or other computer, etc. Furthermore, interface component 870 may be implemented as a network interface to enable communication with other computing devices (not shown), such as via wired or wireless communication links.

[0073] The above description includes examples of aspects of the claimed subject matter. It is certainly impossible to describe every conceivable combination of components or methods in order to describe the claimed subject matter, but those skilled in the art will recognize that many further combinations and permutations of the disclosed subject matter are possible. Accordingly, the disclosed subject matter is intended to encompass all such variations, modifications, and changes that fall within the spirit and scope of the appended claims. Furthermore, with regard to the term "comprising" as used in the detailed description or claims, such a term is intended to be interpreted as an inclusive term in a manner similar to how the term "comprising" is interpreted when used as a transitional word in a claim.

Claims

1. A system comprising: At least one processor; as well as A memory having stored computer-executable instructions that, when executed by the at least one processor, cause the system to: Divide a batch of streaming data into multiple data blocks; Generate at least one parity check block for the plurality of data blocks; as well as The plurality of data blocks and the at least one parity block are provided to a plurality of streaming data processors, which are configured to perform parallel processing on the streaming data to generate processed data, wherein the at least one parity block provides a basis for the recovery of at least some data in the processed data. The at least one processor and the memory are implemented on the input node of a streaming data processing system including the streaming data processor.

2. The system of claim 1, wherein the computer-executable instructions, when executed by the at least one processor, cause the system to: The number of data blocks is defined based at least on the number of the plurality of streaming data processors.

3. The system of claim 2, wherein the computer-executable instructions, when executed by the at least one processor, cause the system to: The coefficients of a polynomial are used to derive the at least one parity block, the polynomial having a degree corresponding to the number of the plurality of data blocks.

4. The system of claim 3, wherein the computer-executable instructions, when executed by the at least one processor, cause the system to: The Reed-Solomon parity algorithm is used to determine the at least one parity block.

5. The system of claim 1, wherein the computer-executable instructions, when executed by the at least one processor, cause the system to: The at least one parity block is generated by applying the parity check to all of the plurality of data blocks.

6. The system of claim 1, wherein the computer-executable instructions, when executed by the at least one processor, cause the system to: Divide the corresponding data blocks into corresponding parity check groups; and For each parity check group, a corresponding parity check block is generated, and each parity check block provides the basis for the recovery of processed data for the data blocks of the corresponding parity check group.

7. The system of claim 1, further comprising the plurality of stream data processors configured to use the at least one parity block to recover at least some of the processed data.

8. One or more computer storage media storing computer-readable instructions, which, when executed, cause a computing device to perform operations including: Divide a batch of streaming data into multiple data blocks; Generate at least one parity check block for the plurality of data blocks; as well as The plurality of data blocks and the at least one parity block are provided to a plurality of streaming data processors configured to perform parallel processing on the streaming data to generate processed data, wherein the at least one parity block provides a basis for the recovery of at least some data in the processed data, and the number of the plurality of data blocks is defined at least based on the number of the plurality of streaming data processors.

9. The operation further comprises: The at least one parity block is derived from the plurality of data blocks using a polynomial having a degree corresponding to the number of the plurality of data blocks.

10. The operation further comprises: The Reed-Solomon parity algorithm is used to determine the at least one parity block.

11. The operation further comprises: The at least one parity block is generated by applying the parity check to all of the plurality of data blocks.

12. The computer storage medium according to claim 8, wherein the operation further comprises: The corresponding data blocks are divided into corresponding parity check groups; as well as For each parity check group, a corresponding parity check block is generated, and each parity check block provides the basis for the recovery of processed data for the data blocks of the corresponding parity check group.

13. One or more computer storage media according to claim 8, wherein the at least one parity block comprises a plurality of parity blocks, the plurality of parity blocks providing a basis for recovery from failure of at least two of the plurality of streaming data processors.

14. One or more computer storage media according to claim 8, wherein the generation comprises: Determine the value of the at least one parity block, the value of which allows at least some of the data in the processed data to be recovered by solving a linear equation.

15. A method comprising: Divide a batch of streaming data into multiple data blocks; The corresponding data blocks are divided into corresponding parity check groups; For each parity check group in the parity check group, generate a corresponding parity check block; as well as The plurality of data blocks and the corresponding parity blocks are provided to a plurality of stream data processors configured to perform parallel processing on the stream data to generate processed data, wherein each parity block provides a basis for the recovery of processed data for a corresponding data block of a corresponding parity group.

16. The method of claim 15, further comprising: The number of data blocks is defined based at least on the number of the plurality of streaming data processors.

17. The method of claim 16, further comprising: The corresponding parity check block is derived using polynomials.

18. The method of claim 15, further comprising: The Reed-Solomon parity algorithm is used to determine the corresponding parity block.

19. The method of claim 15, further comprising: The corresponding parity check block is generated so that the processed data can be recovered by solving the equation.

20. The method of claim 15, performed by an input node of a streaming data processing system including the streaming data processor.

Images