Data processing method and device for stream computing system

Abstract

The embodiments of the present disclosure disclose a data processing method and device for a streaming computing system. A specific implementation of the method includes: obtaining a confirmation message returned by a downstream processing system of the streaming computing system, wherein the confirmation message includes an identifier for indicating that the downstream processing system has completed processing of the data, locating a data structure corresponding to the identifier of the downstream data shard where the processed data is located in the streaming computing system, and finding the corresponding upstream data shard in the data structure based on the identifier of the upstream data shard where the processed data is located; traversing and deleting data in the upstream data shard found whose position identifier is not greater than the position identifier of the processed data in the upstream data shard. This implementation implements batch cleaning of processed data in the streaming computing system, and improves the data cleaning efficiency of the streaming computing system.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, specifically to the field of data processing, and more particularly to a data processing method and apparatus for a streaming computing system. Background Art

[0002] Stream computing is a data processing model that, in contrast to batch computing, processes data immediately upon the occurrence of an event, rather than caching the data for batch processing.

[0003] During the computing process, the streaming computing system continuously generates data for real-time processing, and it is necessary to process the processed data in a timely manner to avoid data accumulation and loss. Summary of the Invention

[0004] The embodiments of the present disclosure provide a data processing method and apparatus for a stream computing system.

[0005] In a first aspect, an embodiment of the present disclosure provides a data processing method for a streaming computing system, the method comprising: obtaining a confirmation message returned by a downstream processing system of the streaming computing system, the confirmation message comprising an identifier for indicating that the downstream processing system has completed processing of the data, the identifier of the processed data comprising an identifier of the downstream data shard where the processed data is located, an identifier of the upstream data shard where the processed data is located, and an identifier of the position of the processed data in the upstream data shard; locating a data structure corresponding to the identifier of the downstream data shard where the processed data is located in the streaming computing system, and finding the corresponding upstream data shard in the data structure according to the identifier of the upstream data shard where the processed data is located; traversing and deleting data in the found upstream data shard whose position identifier is not greater than the position identifier of the processed data in the upstream data shard.

[0006] In some embodiments, the confirmation message is generated in the following manner: in response to receiving a data cleansing request generated by a downstream processing system after completing data processing, the confirmation message is generated based on an identification of the data that has completed processing.

[0007] In some embodiments, the confirmation message is generated as follows: in response to detecting an abnormal restart of the downstream processing system, an identification of the data that has been processed is determined based on the data processing progress recorded in the downstream processing system, and a confirmation message is generated based on the identification of the data that has been processed.

[0008] In some embodiments, it also includes: in response to detecting an abnormal restart of the streaming computing system, reading a pre-stored system state and determining from the context information of the system state the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred, and restoring the data structure identified by the identifier of the downstream data shard bound to the preset processing unit, wherein the system state records the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred.

[0009] In some embodiments, it also includes: in response to detecting that an abnormal restart of the streaming computing system occurs, comparing the location identifier carried by the data to be transmitted into the streaming computing system with the location identifier of the data already transmitted into the streaming computing system, and transmitting the data to be transmitted into the streaming computing system whose location identifier carried by the data to be transmitted into the streaming computing system is greater than the location identifier of the data already transmitted into the streaming computing system, wherein the location identifier carried by the data to be transmitted into the streaming computing system is used to indicate the location of the data to be transmitted into the streaming computing system in the data shard to which it belongs.

[0010] In a second aspect, an embodiment of the present disclosure provides a data processing device for a streaming computing system, the device comprising: an acquisition unit, configured to acquire a confirmation message returned by a downstream processing system of the streaming computing system, the confirmation message comprising an identifier for indicating that the downstream processing system has completed processing of the data, the identifier of the processed data comprising an identifier of the downstream data shard where the processed data is located, an identifier of the upstream data shard where the processed data is located, and an identifier of the position of the processed data in the upstream data shard; a search unit, configured to locate a data structure corresponding to the identifier of the downstream data shard where the processed data is located in the streaming computing system, and to find the corresponding upstream data shard in the data structure according to the identifier of the upstream data shard where the processed data is located; a deletion unit, configured to traverse and delete data in the found upstream data shard whose position identifier is not greater than the position identifier of the processed data in the upstream data shard.

[0011] In some embodiments, the confirmation message is generated in the following manner: in response to receiving a data cleansing request generated by a downstream processing system after completing data processing, the confirmation message is generated based on an identification of the data that has completed processing.

[0012] In some embodiments, the confirmation message is generated as follows: in response to detecting an abnormal restart of the downstream processing system, an identification of the data that has been processed is determined based on the data processing progress recorded in the downstream processing system, and a confirmation message is generated based on the identification of the data that has been processed.

[0013] In some embodiments, it also includes: a recovery unit, which is configured to read a pre-stored system state and determine from the context information of the system state the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred in response to detecting an abnormal restart of the streaming computing system, and restore the data structure identified by the identifier of the downstream data shard bound to the preset processing unit, wherein the system state records the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred.

[0014] In some embodiments, it also includes: a rewriting unit, which is configured to compare the location identifier carried by the data to be transmitted into the streaming computing system with the location identifier of the data already transmitted into the streaming computing system in response to detecting an abnormal restart of the streaming computing system, and transmit the data to be transmitted into the streaming computing system whose location identifier carried by the data to be transmitted into the streaming computing system is greater than the location identifier of the data already transmitted into the streaming computing system, wherein the location identifier carried by the data to be transmitted into the streaming computing system is used to indicate the location of the data to be transmitted into the streaming computing system in the data shard to which it belongs.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation method in the first aspect.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0017] The data processing method and device of the streaming computing system provided by the embodiments of the present disclosure obtain a confirmation message returned by the downstream processing system of the streaming computing system, wherein the confirmation message includes an identifier for indicating that the downstream processing system has completed processing of the data, and the identifier of the processed data includes an identifier of the downstream data shard where the processed data is located, an identifier of the upstream data shard where the processed data is located, and an identifier of the position of the processed data in the upstream data shard. Then, the data structure corresponding to the identifier of the downstream data shard where the processed data is located is located in the streaming computing system, and the corresponding upstream data shard is found in the data structure according to the identifier of the upstream data shard where the processed data is located. Finally, the data whose position identifier in the found upstream data shard is not greater than the position identifier of the processed data in the upstream data shard is traversed and deleted, thereby realizing batch cleaning of the processed data in the streaming computing system and improving the data cleaning efficiency of the streaming computing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings:

[0019] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0020] Figure 2 is a flow chart of an embodiment of a data processing method of a stream computing system according to the present disclosure;

[0021] Figure 3 is a schematic diagram of a data organization method of a data processing method of a stream computing system according to an embodiment of the present disclosure;

[0022] Figure 4 is a flowchart of another embodiment of a data processing method of a stream computing system according to the present disclosure;

[0023] Figure 5 is a flowchart of another embodiment of a data processing method of a stream computing system according to the present disclosure;

[0024] Figure 6 is a structural diagram of an embodiment of a data processing device of a stream computing system according to the present disclosure;

[0025] Figure 7 It is a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] The present disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0027] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] Figure 1 An exemplary architecture 100 is shown to which the data processing method of a stream computing system or a data processing device of a stream computing system of the present disclosure can be applied.

[0029] like Figure 1As shown, the system architecture 100 may include a stream computing system 101, a network 102, and a downstream processing system 103. The network 102 is used as a medium for providing a communication link between the stream computing system 101 and the downstream processing system 103.

[0030] The stream computing system 101 can receive external task data. The stream computing system 101 can also perform calculations on the task data and send the processed data results to the downstream processing system 103 in real time. The stream computing system 101 can also clean up the data that has been processed by the downstream processing system.

[0031] The downstream processing system 103 can use the results processed by the stream computing system 101 to perform analysis and other processing to generate the calculation results of the downstream processing system 103. The downstream processing system 103 can generate a data cleaning request and send it to the stream computing system 101.

[0032] It should be noted that the above-mentioned streaming computing system 101 and downstream processing system 103 can be hardware or software systems that implement the corresponding functions. When the streaming computing system 101 and downstream processing system 103 are hardware systems, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the streaming computing system 101 and downstream processing system 103 are hardware systems, they can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or as a single software or software module. No specific limitations are given here.

[0033] It should be understood that Figure 1 The number of stream computing systems 101, networks 102, and downstream processing systems 103 in FIG. 1 is merely illustrative. Any number of stream computing systems 101, networks 102, and downstream processing systems 103 may be provided as needed.

[0034] Continue to refer Figure 2 , shows a process 200 of an embodiment of a data processing method of a stream computing system according to the present disclosure. The data processing method of the stream computing system includes the following steps:

[0035] Step 201: Obtain a confirmation message returned by a downstream processing system of the stream computing system.

[0036] The confirmation message includes an identifier for indicating that the downstream processing system has completed processing of the data. The identifier of the data that has completed processing includes an identifier of the downstream data shard where the data that has completed processing is located, an identifier of the upstream data shard where the data that has completed processing is located, and an identifier of the position of the data that has completed processing in the upstream data shard.

[0037] In this embodiment, the execution subject of the data processing method of the stream computing system (such as Figure 1 The server 105 shown can obtain an identifier indicating that the downstream processing system has completed processing data from a confirmation message returned by the downstream processing system of the stream computing system. The confirmation message may include an identifier indicating that the downstream processing system has completed processing data. As an example, the confirmation message may be implemented using an acknowledgment character (ACK). The downstream processing system of the stream computing system may be a system that receives data or calculation results from the stream computing system and performs further processing or output. Data shards may be mutually independent data subsets divided according to certain rules. The identifier of an upstream data shard can be used to identify independent data shards in the stream computing system. As an example, this can be achieved by numbering the data shard sets in the stream computing system. The identifier of a downstream data shard can be used to identify data shards in the downstream processing system. As an example, this can be achieved by numbering the data shards in the downstream processing system, where the data structure of the downstream data system may be organized in the form of multiple data shards. The location identifier of the data in the upstream data shard can be used to indicate the logical address of the data in the data shards of the stream computing system. As an example, this can be achieved by numbering the data in the data shards of the stream computing system.

[0038] In some optional implementations of this embodiment, the confirmation message may be generated as follows: in response to receiving a data cleanup request generated after the downstream processing system completes data processing, a confirmation message is generated based on an identification of the data that has been processed.

[0039] In this optional approach, a data purge request can be used to indicate the purge of processed data. In practice, when a preset system purge time is reached, the execution entity can generate a confirmation message including the identifier of the currently processed data based on the identifier of the currently processed data. Each time the preset system purge time is reached, the downstream processing system generates a data purge request, and the execution entity updates the identifier of the currently processed data included in the confirmation message based on the identifier of the currently processed data.

[0040] This implementation can timely update the identifier of the processed data included in the confirmation message, avoiding repeated cleaning of the processed data.

[0041] In some optional implementations of this embodiment, the confirmation message can be generated in the following manner: in response to detecting an abnormal restart of the downstream processing system, determining the identification of the data that has been processed based on the data processing progress recorded in the downstream processing system, and generating a confirmation message based on the identification of the data that has been processed.

[0042] In this optional approach, the data processing progress can be used to record the identification of completed data. In practice, when a network failure or power outage occurs and the downstream processing system restarts, the execution entity can determine the completion status of data processing based on the pre-stored data processing progress and generate a confirmation message based on the identification of the latest completed data recorded in the data processing progress.

[0043] This implementation can promptly restore data when a failure occurs and update the identifier of the processed data included in the confirmation message, thereby avoiding the loss of records of processed data and the inability to generate accurate confirmation messages when a system failure occurs.

[0044] Step 202: locate the data structure corresponding to the identifier of the downstream data shard where the processed data is located in the stream computing system, and find the corresponding upstream data shard in the data structure according to the identifier of the upstream data shard where the processed data is located.

[0045] In this embodiment, the data structure may be a collection of data elements with structural characteristics. The execution subject may represent the data structure diagram corresponding to the identifier of the downstream data shard by using the index relationship structure diagram between the identifier of the upstream data shard where the data is located and the identifier of the downstream data shard where the data is located, for example: Figure 3 shown.

[0046] Figure 3 This is a schematic diagram of the organization of a data structure of a data processing method of a stream computing system according to an embodiment of the present disclosure. Figure 3 In the schematic diagram, the execution entity locates data structure A corresponding to the downstream data shard identified as 1. Data structure B is the specific structure of a downstream data shard identified as 1 in data structure A. The data in data structure A can be organized in the form of multiple data shards, each of which can be identified as a downstream data shard. Each data shard includes multiple processed data that carry data identifiers.

[0047] The execution subject then finds the corresponding upstream data shard based on the identifier of the upstream data shard where the processed data is located. For example, the identifier of the data is (5, a, 3), where "a" is the identifier of the upstream data shard where the processed data is located. The execution subject can find the upstream data shard with the identifier a of the corresponding upstream data shard in the data structure B. The identifier of the data also includes the identifier of the processing unit that processes the processed data and the position identifier of the processed data in the upstream data shard. The processing unit can be a unit that processes and analyzes the data, and the execution subject processes the data with the processing unit as a unit. For example, the identifier of the data is (5, a, 3), where "5" is the identifier of the processing unit used to process the processed data and "3" is the position identifier of the processed data in the upstream data shard.

[0048] Step 203: traverse and delete the data in the upstream data slice whose position identifier is not greater than the position identifier of the processed data in the upstream data slice.

[0049] In this embodiment, the position identifier in the upstream data slice can be used to indicate the logical address of the data in the upstream data slice. As an example, the data in the upstream data slice can be numbered sequentially according to the time sequence of the data arriving in the upstream data slice. The execution entity searches the data structure in step 202 for the upstream data slice where the data is located, locates the position of the processed data in the upstream data slice based on the position identifier of the processed data in the upstream data slice, and accesses and deletes data in the upstream data slice whose position identifier is not greater than the position identifier of the processed data from the front to the back.

[0050] The data storage method provided by the above-mentioned embodiment of the present disclosure obtains a confirmation message returned by a downstream processing system of a streaming computing system, and then locates a data structure corresponding to an identifier of a downstream data shard where the processed data is located in the streaming computing system according to an identifier included in the confirmation message for indicating that the downstream processing system has completed processing the data, and finds the corresponding upstream data shard in the data structure according to the identifier of the upstream data shard where the processed data is located, and finally traverses and deletes the data in the upstream data shard found whose position identifier is not greater than the position identifier of the processed data in the upstream data shard, thereby realizing batch cleaning of processed data in the streaming computing system and improving data cleaning efficiency.

[0051] Further references Figure 4 , which shows a process 400 of another embodiment of a data processing method of a stream computing system. The process 400 of the data processing method of the stream computing system includes the following steps:

[0052] Step 401: Obtain a confirmation message returned by a downstream processing system of the stream computing system.

[0053] Step 402: locate the data structure corresponding to the identifier of the downstream data shard where the processed data is located in the streaming computing system, and find the corresponding upstream data shard in the data structure according to the identifier of the upstream data shard where the processed data is located.

[0054] Step 403: traverse and delete the data in the upstream data slice whose position identifier is not greater than the position identifier of the processed data in the upstream data slice.

[0055] The above steps 401, 402 and 403 are respectively consistent with steps 201, 202 and 203 in the aforementioned embodiment. The above descriptions of steps 201, 202 and 203 also apply to steps 401, 402 and 403, and are not repeated here.

[0056] Step 404, in response to detecting an abnormal restart of the streaming computing system, read the pre-stored system state and determine from the context information of the system state the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred, and restore the data structure identified by the downstream data shard identifier bound to the preset processing unit.

[0057] In this embodiment, the above-mentioned execution subject can pre-bind the identifier of the downstream data shard with the corresponding preset processing unit. The preset processing unit here can be the processing unit that was processing data when the failure occurred. When the streaming computing system restarts abnormally, the above-mentioned execution subject can find the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred based on the context information of the pre-stored system state. The preset processing unit will restore the data in the data structure corresponding to the identifier of the downstream data shard and the corresponding state of the data based on the data in the pre-stored data structure and the corresponding state of the data.

[0058] from Figure 4 As can be seen, process 400 of the data processing method for a streaming computing system in this embodiment enables data recovery via a preset processing unit bound to the identifier of a downstream data shard when the streaming computing system fails and restarts. Thus, the solution described in this embodiment enables data recovery at the preset processing unit level when a streaming computing system fails, eliminating the need to recover all data and enabling the data of the streaming computing system to be quickly and accurately restored to its pre-failure state.

[0059] Further references Figure 5, which shows a process 500 of another embodiment of a data processing method of a stream computing system. The process 500 of the data processing method of the stream computing system includes the following steps:

[0060] Step 501: Obtain a confirmation message returned by a downstream processing system of the stream computing system.

[0061] Step 502: locate the data structure corresponding to the identifier of the downstream data shard where the processed data is located in the streaming computing system, and find the corresponding upstream data shard in the data structure according to the identifier of the upstream data shard where the processed data is located.

[0062] Step 503: traverse and delete the data in the upstream data slice whose position identifier is not greater than the position identifier of the processed data in the upstream data slice.

[0063] The above steps 501, 502 and 503 are respectively consistent with steps 201, 202 and 203 in the aforementioned embodiment. The above descriptions of steps 201, 202 and 203 also apply to steps 501, 502 and 503, and are not repeated here.

[0064] Step 504: In response to detecting that an abnormal restart of the streaming computing system has occurred, the location identifier carried by the data to be transmitted to the streaming computing system is compared with the location identifier of the data already transmitted to the streaming computing system, and the data to be transmitted to the streaming computing system whose location identifier carried by the data to be transmitted to the streaming computing system is greater than the location identifier of the data already transmitted to the streaming computing system is transmitted to the streaming computing system.

[0065] In this embodiment, the location identifier carried by the data to be transmitted to the streaming computing system indicates the location of the data to be transmitted to the streaming computing system within the data shard to which it belongs. When the streaming computing system restarts abnormally, to ensure data processing accuracy, the execution entity can restore the data in the processing unit that was processing data at the time of the failure. The data in the processing unit that was processing data at the time of the failure may include a portion of processed data and a portion of unprocessed data. The execution entity uses the restored data as the data to be transmitted to the streaming computing system, compares the location identifiers carried by the data to be transmitted to the streaming computing system with the location identifiers carried by the data already transmitted to the streaming computing system, and then filters out the data to be transmitted to the streaming computing system that carries a smaller location identifier than the data already transmitted to the streaming computing system. Data to be transmitted to the streaming computing system that carries a larger location identifier than the data already transmitted to the streaming computing system is then passed to the streaming computing system. This embodiment prevents the streaming computing system from processing data repeatedly, implements the non-duplication feature of the streaming computing system, and ensures the accuracy of data processing by the streaming computing system.

[0066] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data processing device for a stream computing system. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0067] like Figure 6 As shown, the data processing device 600 of the stream computing system provided in this embodiment includes an acquisition unit 601, a search unit 602, and a deletion unit 603. The acquisition unit 601 is configured to acquire a confirmation message returned by the downstream processing system of the stream computing system, the confirmation message including an identifier for indicating that the downstream processing system has completed processing of the data, the identifier of the processed data including an identifier of the downstream data shard where the processed data is located, an identifier of the upstream data shard where the processed data is located, and an identifier of the position of the processed data in the upstream data shard; the search unit 602 is configured to locate the data structure corresponding to the identifier of the downstream data shard where the processed data is located in the stream computing system, and find the corresponding upstream data shard in the data structure according to the identifier of the upstream data shard where the processed data is located; and the deletion unit 603 is configured to traverse and delete the data in the upstream data shard found whose position identifier is not greater than the position identifier of the processed data in the upstream data shard.

[0068] In this embodiment, in the data processing device 600 of the stream computing system, the specific processing of the acquisition unit 601, the search unit 602 and the deletion unit 603 and the technical effects thereof can be referred to in the respective Figure 2 The relevant descriptions of step 201, step 202 and step 203 in the corresponding embodiment are not repeated here.

[0069] In some optional implementations of this embodiment, the confirmation message may be generated as follows: in response to receiving a data cleanup request generated after the downstream processing system completes data processing, a confirmation message is generated based on an identification of the data that has been processed.

[0070] In some optional implementations of this embodiment, the confirmation message can be generated in the following manner: in response to detecting an abnormal restart of the downstream processing system, determining the identification of the data that has been processed based on the data processing progress recorded in the downstream processing system, and generating a confirmation message based on the identification of the data that has been processed.

[0071] In some embodiments, the above-mentioned device 600 may also include: a recovery unit (not shown in the figure), which is configured to read a pre-stored system state and determine from the context information of the system state the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred in response to detecting an abnormal restart of the streaming computing system, and restore the data structure identified by the identifier of the downstream data shard bound to the preset processing unit, wherein the system state records the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred.

[0072] In some embodiments, the above-mentioned device 600 may also include: a rewriting unit (not shown in the figure), which is configured to compare the location identifier carried by the data to be transmitted into the streaming computing system with the location identifier of the data already transmitted into the streaming computing system in response to detecting an abnormal restart of the streaming computing system, and transmit the data to be transmitted into the streaming computing system whose location identifier carried by the data to be transmitted into the streaming computing system is greater than the location identifier of the data already transmitted into the streaming computing system, wherein the location identifier carried by the data to be transmitted into the streaming computing system is used to indicate the location of the data to be transmitted into the streaming computing system in the data shard to which it belongs.

[0073] The device provided by the above-mentioned embodiment of the present disclosure obtains the confirmation message returned by the downstream processing system of the streaming computing system through the acquisition unit 601, the search unit 602 locates the data structure corresponding to the identifier of the downstream data shard where the data that has been processed is located in the streaming computing system, and finds the corresponding upstream data shard in the data structure according to the identifier of the upstream data shard where the data that has been processed is located, and the deletion unit 603 traverses and deletes the data in the upstream data shard found whose position identifier is not greater than the position identifier of the data that has been processed in the upstream data shard, thereby realizing the system status of batch cleaning of the data that has been processed and improving the efficiency of cleaning data.

[0074] Reference below Figure 7 , refer to the following Figure 7 , which shows an electronic device (eg, Figure 1 Schematic diagram of the structure of the server of the streaming computing system 700. Figure 7 The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0075] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0076] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0077] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0078] It should be noted that the computer-readable medium of the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer-readable medium may be conveyed using any suitable medium, including but not limited to wires, optical cables, RF (Radio Frequency), or any suitable combination thereof.

[0079] The computer-readable medium may be included in the electronic device, or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: obtains a confirmation message returned by a downstream processing system of the streaming computing system, the confirmation message including an identifier for indicating that the downstream processing system has completed processing of the data, the identifier of the processed data including an identifier of the downstream data shard where the processed data is located, an identifier of the upstream data shard where the processed data is located, and an identifier of the position of the processed data in the upstream data shard; locates a data structure corresponding to the identifier of the downstream data shard where the processed data is located in the streaming computing system, and finds the corresponding upstream data shard in the data structure according to the identifier of the upstream data shard where the processed data is located; and traverses and deletes data in the upstream data shard found whose position identifier is not greater than the position identifier of the processed data in the upstream data shard.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to the various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0081] The units involved in the embodiments described in the present disclosure may be implemented in software or in hardware. The units described may also be provided in a processor, for example, may be described as: a processor comprising an acquisition unit, a search unit, and a deletion unit. The names of these units do not, in some cases, constitute a limitation on the units themselves. For example, the acquisition unit may also be described as a "unit for obtaining a confirmation message returned by a downstream processing system of a streaming computing system."

[0082] The above description is merely a preferred embodiment of the present disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A data processing method for a stream computing system, comprising: Obtaining a confirmation message returned by a downstream processing system of the stream computing system, the confirmation message including an identifier for indicating that the downstream processing system has completed processing of the data, the identifier of the processed data including an identifier of the downstream data shard where the processed data is located, an identifier of the upstream data shard where the processed data is located, and a position identifier of the processed data in the upstream data shard, the identifier of the upstream data shard being used to identify an independent data shard in the stream computing system, and the position identifier being used to indicate a logical address of the processed data in the independent data shard; Locating, in the stream computing system, a data structure corresponding to an identifier of a downstream data shard where the processed data is located, and searching, in the data structure, for a corresponding upstream data shard based on the identifier of the upstream data shard where the processed data is located; The data whose position identifier in the upstream data shard is found and traversed and deleted is not greater than the position identifier of the processed data in the upstream data shard.

2. The method according to claim 1, wherein The confirmation message is generated as follows: In response to receiving a data cleanup request generated by the downstream processing system after completing data processing, the confirmation message is generated based on an identification of the data that has completed processing.

3. The method according to claim 1, wherein The confirmation message is generated as follows: In response to detecting that the downstream processing system is abnormally restarted, an identifier of the processed data is determined based on the data processing progress recorded in the downstream processing system, and the confirmation message is generated based on the identifier of the processed data.

4. The method according to claim 1, wherein The method further comprises: In response to detecting that an abnormal restart of the streaming computing system occurs, a pre-stored system state is read and the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred is determined from the context information of the system state, and the data structure identified by the identifier of the downstream data shard bound to the preset processing unit is restored, wherein the system state records the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred.

5. The method according to claim 1, wherein The method further comprises: In response to detecting an abnormal restart of the streaming computing system, the location identifier carried by the data to be transmitted into the streaming computing system is compared with the location identifier of the data already transmitted into the streaming computing system, and the data to be transmitted into the streaming computing system whose location identifier carried by the data to be transmitted into the streaming computing system is greater than the location identifier of the data already transmitted into the streaming computing system is transmitted into the streaming computing system, wherein the location identifier carried by the data to be transmitted into the streaming computing system is used to indicate the location of the data to be transmitted into the streaming computing system in the data shard to which it belongs.

6. A data processing device for a stream computing system, comprising: an acquisition unit configured to acquire a confirmation message returned by a downstream processing system of the stream computing system, the confirmation message including an identifier for indicating that the downstream processing system has completed processing of the data, the identifier of the processed data including an identifier of the downstream data shard where the processed data is located, an identifier of the upstream data shard where the processed data is located, and a position identifier of the processed data in the upstream data shard, the identifier of the upstream data shard being used to identify an independent data shard in the stream computing system, and the position identifier being used to indicate a logical address of the processed data in the independent data shard; a search unit configured to locate, in the stream computing system, a data structure corresponding to an identifier of a downstream data shard where the processed data is located, and to search, in the data structure, for a corresponding upstream data shard based on the identifier of the upstream data shard where the processed data is located; The deleting unit is configured to traverse and delete the data whose position identifier in the upstream data shard is not greater than the position identifier of the processed data in the upstream data shard.

7. The device according to claim 6, wherein The confirmation message is generated as follows: In response to receiving a data cleanup request generated by the downstream processing system after completing data processing, the confirmation message is generated based on an identification of the data that has completed processing.

8. The device according to claim 6, wherein The confirmation message is generated as follows: In response to detecting that the downstream processing system is abnormally restarted, an identifier of the processed data is determined based on the data processing progress recorded in the downstream processing system, and the confirmation message is generated based on the identifier of the processed data.

9. The device according to claim 6, wherein The device further comprises: A recovery unit is configured to, in response to detecting an abnormal restart of the streaming computing system, read a pre-stored system state and determine from the context information of the system state the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred, and restore the data structure identified by the identifier of the downstream data shard bound to the preset processing unit, wherein the system state records the identifier of the downstream data shard bound to the preset processing unit that was processing data when the failure occurred.

10. The device according to claim 6, wherein The device further comprises: The rewriting unit is configured to compare the location identifier carried by the data to be transmitted into the streaming computing system with the location identifier of the data already transmitted into the streaming computing system in response to detecting an abnormal restart of the streaming computing system, and transmit the data to be transmitted into the streaming computing system whose location identifier carried by the data to be transmitted into the streaming computing system is greater than the location identifier of the data already transmitted into the streaming computing system, wherein the location identifier carried by the data to be transmitted into the streaming computing system is used to indicate the location of the data to be transmitted into the streaming computing system in the data shard to which it belongs.

11. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

12. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images