Streaming data processing method, streaming processing system, electronic device and computer storage medium

Abstract

The embodiments of the present application provide a method for processing streaming data, a streaming processing system, an electronic device, and a computer storage medium. The method for processing streaming data applied to the first data source node of the first streaming data includes: receiving the first streaming data; pulling the second streaming data according to a preset time period; comparing the currently pulled second streaming data with the historically pulled second streaming data to determine an incremental data identifier corresponding to the incremental data of the currently pulled second streaming data relative to the historically pulled second streaming data; filtering multiple pieces of data in the received first streaming data according to the incremental data identifier to determine a partial first streaming data corresponding to the incremental data identifier, and performing a deserialization operation on the partial first streaming data; and transmitting the partial first streaming data on which the deserialization operation is performed to a data connection node through a network, so that the data connection node connects the second streaming data and the partial first streaming data based on the data identifier.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technologies, and in particular, to a method for processing streaming data, a streaming processing system, an electronic device, and a computer storage medium. Background Art

[0002] A data warehouse can directly obtain various streaming data involved during operation in real time from a data source, and merge the obtained streaming data based on a streaming computing task to obtain multi-dimensional data corresponding to the operation process. Further, the data required by a user can be obtained by analyzing, summarizing, or tracing back the multi-dimensional data.

[0003] Currently, a two-stream connection (also known as two-stream join) method is generally used for merging streaming data. However, the two-stream connection consumes a large amount of processing resources, resulting in a high cost for the two-stream connection and a long time consumption.

[0004] Based on this, the technical problem that needs to be solved urgently in the prior art is to provide a streaming data processing solution with a lower cost. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide a streaming data processing solution to at least partially solve the above problems.

[0006] According to a first aspect of the embodiments of the present application, a method for processing streaming data is provided, which is applied to a first data source node of first streaming data, and includes: receiving the first streaming data; pulling second streaming data according to a preset time period, where the first streaming data and the second streaming data include multiple pieces of data, and each piece of data corresponds to a data identifier; comparing the currently pulled second streaming data with the historically pulled second streaming data to determine an incremental data identifier corresponding to the incremental data of the currently pulled second streaming data relative to the historically pulled second streaming data; filtering multiple pieces of data in the received first streaming data according to the incremental data identifier to determine a partial first streaming data corresponding to the incremental data identifier, and performing a deserialization operation on the partial first streaming data; and transmitting the partial first streaming data on which the deserialization operation is performed to a data connection node through a network, so that the data connection node connects the second streaming data and the partial first streaming data based on the data identifier.

[0007] According to a second aspect of the embodiments of the present application, there is provided a stream processing system, including a first data source node corresponding to a first stream of data, a second data source node corresponding to a second stream of data, and a data connection node; the first data source node is configured to transmit, through the network, a part of the first stream of data that has performed deserialization operations to the data connection node by using the method described above; the second data source node is configured to perform the deserialization operation on the second stream of data and transmit the second stream of data that has performed the deserialization operation to the data connection node through the network; the data connection node is configured to connect the second stream of data and the part of the first stream of data based on a data identifier.

[0008] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including: a processor, a memory, a communication interface, and a communication bus, where the processor, the memory, and the communication interface complete communication with each other through the communication bus; the memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method described in the first aspect.

[0009] According to a fourth aspect of the embodiments of the present application, there is provided a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method described in the first aspect.

[0010] According to a fifth aspect of the embodiments of the present application, there is provided a computer program product, including computer instructions, and the computer instructions instruct a computing device to perform operations corresponding to the method described in the first aspect.

[0011] According to the streaming data processing solution provided by the embodiments of the present application, through the first data source node, while receiving the first streaming data, the second streaming data is also pulled at a preset time period. Both the first streaming data and the second streaming data include multiple pieces of data, and each piece of data corresponds to a data identifier. Moreover, the currently pulled second streaming data can be compared with the historically pulled second streaming data to determine the incremental data identifier corresponding to the incremental data of the currently pulled second streaming data relative to the historically pulled second streaming data. Thus, multiple pieces of data in the received first streaming data can be filtered according to the incremental data identifier to determine a part of the first streaming data corresponding to the incremental data identifier, and a deserialization operation is performed on the part of the first streaming data, greatly reducing the number of data of the first streaming data that needs to perform the deserialization operation, thereby greatly reducing the resources consumed by performing the deserialization operation on the first streaming data. Subsequently, the part of the first streaming data to which the deserialization operation will be performed is transmitted to the data connection node through the network, so that the data connection node connects the second streaming data and the part of the first streaming data based on the data identifier, and can also reduce the network transmission resources consumed by transmitting the first streaming data, as well as reduce the processing resources consumed by the data connection node for processing the first streaming data, greatly reducing the resources consumed in the streaming data processing process and shortening the processing time. Brief Description of the Drawings

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the embodiments of the present application, and those of ordinary skill in the art can also obtain other drawings based on these drawings.

[0013] Figure 1A 、 Figure 1B shows a schematic diagram of the principle of dual-stream connection;

[0014] Figure 2 is a schematic diagram of an exemplary system for the streaming data processing method applicable to the embodiments of the present application;

[0015] Figure 3 is a schematic diagram of the architecture of a streaming processing system according to the embodiments of the present application;

[0016] Figure 4A is a flowchart of the steps of a streaming data processing method according to the embodiments of the present application;

[0017] Figure 4B For Figure 4A a schematic diagram of the usage scenario of the shown embodiment;

[0018] Figure 5Ais a flowchart of another method for processing stream data according to an embodiment of the present application;

[0019] Figure 5B This is an architectural diagram of a first data source node according to an embodiment of the present application;

[0020] Figure 6 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0022] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0023] In order to explain this solution more clearly, a commonly used dual-stream connection (dual-stream join) is used for explanation.

[0024] The commonly used dual-stream join (dual-stream join) can be divided into two types: hash join and broadcast join.

[0025] See also Figure 1A , which shows a schematic diagram of the principle of hash join. As shown in the figure, the stream processing system may include two data source nodes corresponding to stream data A and stream data B respectively, and multiple data connection nodes join.

[0026] The data source node streamA corresponding to data source A and the data source node streamB corresponding to data source B can process the data identification keys in stream data A and stream data B according to the same hash rule during the Shuffle process, and ensure that the data corresponding to the same data identification key (i.e. key1, key2, key3 in the figure) is sent to the same data connection node (join node). The join node can store the various data of streamA and streamB (corresponding to stataA and stataB in the figure), and continuously perform data reverse query on stream data A and stream data B based on the data identification key. Finally, the result is output if the connection condition is met. The connection condition can be that the data identification key in stream data A and stream data B is the same.

[0027] See alsoFigure 1B , showing a schematic diagram of the principle of a broadcast connection. As shown in the figure, the stream processing system may include two data source nodes corresponding to stream data A and stream data B respectively, and multiple data connection nodes join.

[0028] The data source node of stream data B can broadcast multiple pieces of data (i.e., key1, key2, key3 in the figure) included in data stream B during the Shuffle process, so that multiple data connection nodes join cache all the data corresponding to data stream B, namely satatB. The data source node corresponding to data stream A can evenly and randomly batch data stream A to multiple data connection nodes join in a polling manner during the Shuffle process. The join node can store stataA and continuously query the partially issued data stream A according to the data identifier key of data stream B, and finally output the result when the connection condition is met. The connection condition can be that the data identifiers key in data stream A and data stream B are the same.

[0029] When performing a dual-stream connection through the above two methods, both data stream A and data stream B need to be issued to the JOIN node. However, before being issued to the join node, operations such as reading, serializing, deserializing, and network transmission need to be performed on the data stream, so as to convert the received stream data into a format that the data connection node can process. However, there is generally a distinction between large and small streams when performing a dual-stream connection. The data volume of the large stream may be hundreds or thousands of times that of the small stream, that is, only one percent or one-thousandth of the data in the large stream is used for connection with the small stream and output, resulting in only one percent or one-thousandth of the operations such as reading, serializing, deserializing, and network transmission on the large stream data being effective, which causes a huge waste of resources.

[0030] Therefore, an embodiment of the present application provides a method for processing stream data to minimize resource waste.

[0031] Figure 2 showing an exemplary system applicable to the method for processing stream data according to an embodiment of the present application. As Figure 2 shown, the system 200 may include a cloud server 202, a communication network 204, and / or one or more user devices 206, Figure 2 exemplified as multiple user devices in the figure.

[0032] The cloud server 202 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 202 can perform any suitable function. For example, in some embodiments, the cloud server 202 can be used for dual-stream connection. As an alternative example, in some embodiments, the cloud server 202 can be used to receive and process the first stream data and the second stream data. As another example, in some embodiments, the cloud server 202 can be used to perform a dual-stream connection on the first stream data and the second stream data, perform data analysis based on the dual-stream connection result, and send the analysis result to the user device.

[0033] In some embodiments, the communication network 204 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 204 can include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. The user device 206 can be connected to the communication network 204 through one or more communication links (e.g., communication link 212), and the communication network 204 can be linked to the cloud server 202 via one or more communication links (e.g., communication link 214). The communication link can be any communication link suitable for transmitting data between the user device 206 and the cloud server 202, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0034] The user device 206 can include any one or more user devices for generating part of the first stream data or for generating part of the second stream data or for presenting the data analysis result corresponding to the connection result of the first stream data and the second stream data. In some embodiments, the user device 206 can include any suitable type of device. For example, in some embodiments, the user device 206 can include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a game console, a media player, a vehicle entertainment system, and / or any other suitable type of user device.

[0035] Based on the above system, embodiments of the present application provide a method for processing stream data, which will be described below through multiple embodiments.

[0036] See Figure 3, which shows a stream processing system, including a first data source node 301 corresponding to the first stream data, a second data source node 302 corresponding to the second stream data, and a data connection node 303.

[0037] The first data source node 301 is configured to receive the first stream data and the second stream data, perform deserialization operations on part of the first stream data according to the second stream data, and transmit the part of the first stream data on which the deserialization operations are performed to the data connection node 303 through a network.

[0038] The second data source node 302 is configured to perform the deserialization operations on the second stream data, and transmit the deserialized second stream data to the data connection node 303 through a network.

[0039] The data connection node 303 is configured to connect the second stream data and the part of the first stream data based on a data identifier.

[0040] In this embodiment, stream processing is an important big data processing means, and its main feature is that the data it processes is continuous and arrives in real time, that is, stream data. The stream processing system will include multiple types of nodes, such as the data source nodes, data connection nodes, etc. mentioned above, and may also include but are not limited to UNION, CONNECT, CoGroup, etc. This embodiment does not limit this.

[0041] In this embodiment, the first data source node or the second data source node can be used to read the corresponding first stream data or second stream data, perform serialization processing and storage on the read stream data, perform deserialization processing and field injection on the stored stream data, etc., obtain the stream data format that the data connection node can process, and transmit the stream data to the data connection node through a network.

[0042] In this embodiment, the data connection node can connect and output the data with the same data identifier in the first stream data and the second stream data.

[0043] Specifically, refer to Figure 4A , which shows a step flowchart of the first data source node 301 processing the first stream data. As shown in the figure, it includes:

[0044] S401. Receive the first stream data.

[0045] S402. Pull the second stream data according to a preset time period. The first stream data and the second stream data include multiple pieces of data, and each piece of data corresponds to a data identifier.

[0046] Exemplarily, refer to Figure 4B, the "large flow" in the figure corresponds to the first data source node 301, the "small flow" in the figure corresponds to the second data source node 302, join corresponds to the data connection node 303, the data identifiers in the first flow data can be k1-k9, and r1-r9 are the first data contents corresponding to the data identifiers respectively. The data identifiers in the second flow data can be k1-k3, and l1-l3 are the second data contents corresponding to the data identifiers respectively.

[0047] In this embodiment, the preset time period can be set by those skilled in the art according to the real-time processing requirements of the streaming data. If the real-time processing requirements for the streaming data are high, the time period can be set shorter; otherwise, the time period can be set longer. This embodiment does not limit this.

[0048] S403. Compare the currently pulled second flow data with the historically pulled second flow data to determine the incremental data identifiers corresponding to the incremental data of the currently pulled second flow data relative to the historically pulled second flow data.

[0049] Figure 4B The figure shown can be a schematic diagram of the first pull of the second flow data according to the preset time period. Then the corresponding historically pulled second flow data is zero, and the determined incremental data identifiers are k1, k2, and k3, which are stored in the cache cache.

[0050] If at Figure 4B On the basis of the second flow data shown, if the second flow data pulled again includes (k3, l3), (k4, l4), then the determined incremental data identifier is k4.

[0051] S404. Filter multiple pieces of data in the received first flow data according to the incremental data identifiers, determine the partial first flow data corresponding to the incremental data identifiers, and perform a deserialization operation on the partial first flow data.

[0052] See Figure 4B , the first flow data can be filtered according to the determined incremental data identifiers k1, k2, and k3 to obtain the partial first flow data corresponding to k1, k2, and k3, and a deserialization operation can be performed on this partial first flow data.

[0053] Serialization: It refers to converting an object into binary content, which is essentially a byte[]. Because after serialization, the byte[] can be saved to a file, or the byte[] can be transmitted over the network to a remote location (IO). In this way, it is equivalent to converting the object into binary data that can be stored in a file or transmitted over the network.

[0054] Deserialization: Opposite to serialization, deserialization refers to converting a binary data (i.e., a byte[] array) back into an object. Through deserialization, the binary data saved to a file or received over a network can be converted back into an object. In this embodiment, the object converted after performing the deserialization operation can be recognized and processed by the data connection node.

[0055] S405. Transmit a part of the first stream data on which the deserialization operation is performed to the data connection node over a network, so that the data connection node connects the second stream data and the part of the first stream data based on the data identifier.

[0056] The first data source node, the second data source node, and the data connection node can be communicatively connected over a network. Then, the first data source node can transmit a part of the first stream data on which the deserialization operation is performed to the data connection node over the network. Similarly, the second data source node can also transmit the second stream data on which the deserialization operation is performed to the data connection node over the network. For the specific network transmission method, reference can be made to the related art and will not be elaborated here.

[0057] See Figure 4B , the data connection node can store the received part of the first stream data and store the second stream data, and connect the second stream data and the part of the first stream data according to the data identifiers k1, k2, k3.

[0058] The solution provided in this embodiment, through the first data source node, while receiving the first stream data, also pulls the second stream data at a preset time period. The first stream data and the second stream data include multiple pieces of data, and each piece of data corresponds to a data identifier; and it can compare the currently pulled second stream data with the historically pulled second stream data to determine the incremental data identifier corresponding to the incremental data of the currently pulled second stream data relative to the historically pulled second stream data; thus, it can filter multiple pieces of data in the received first stream data according to the incremental data identifier, determine the part of the first stream data corresponding to the incremental data identifier, and perform a deserialization operation on the part of the first stream data, greatly reducing the amount of data of the first stream data that needs to perform the deserialization operation, thereby greatly reducing the resources consumed for performing the deserialization operation on the first stream data. Subsequently, the part of the first stream data on which the deserialization operation will be performed is transmitted to the data connection node over a network, so that the data connection node connects the second stream data and the part of the first stream data based on the data identifier, and can also reduce the network transmission resources consumed for transmitting the first stream data, and reduce the amount of data stored by the data connection node for the first stream data (such as Figure 4BThe full amount of data of the small stream state shown, and the large stream state only stores partial first stream data corresponding to the filtered k1, k2, and k3), and the processing resources consumed for processing the first stream data can greatly reduce the resources consumed in the process of processing stream data and shorten the processing time.

[0059] See Figure 5A , which shows a flowchart of steps for another first data source node to process the first stream data. Figure 5B Shows a schematic diagram of the architecture of a first data source node, as Figure 5A shown, the first data source node may include a parallel main thread, a second asynchronous thread, and a first asynchronous thread. The method includes:

[0060] S501. Receive the first stream data;

[0061] Specifically, see Figure 5B , the first data source node may include a source and a parser. The source is used to receive the first stream data, and the subsequent steps are executed by the parser. The three parallel processes are also deployed in the parser. The architecture of the parser can be divided into three layers, namely the data layer, the filtering layer, and the backtracking layer. The main thread can receive the first stream data through the large stream reciver interface of the data layer. Figure 5B The dotted part in [] does not belong to the first data source node. Next can be a data connection node or other nodes located between the first data source node and the data connection node. This embodiment does not limit this.

[0062] In addition, it should be noted that step S501 is not executed by the parser, so it is not shown in the figure; the main thread and the first asynchronous thread in the figure can be started during the initialization process, and the second asynchronous thread can be started by the trigger of the incremental data identifier.

[0063] S502. Through the first asynchronous thread, pull the second stream data according to a preset time period, and store the data identifiers corresponding to multiple pieces of data of the pulled second stream data in a compressed cache manner.

[0064] Specifically, after the first asynchronous thread pulls multiple pieces of data of the second stream data, it can merge and deduplicate according to the data identifiers of the multiple pieces of data, and compress and store the merged and deduplicated data identifiers.

[0065] Similarly, see Figure 5B , the first asynchronous thread of the first data source node can receive the first stream data through the small stream pull interface of the data layer.

[0066] The first asynchronous thread pulls the second stream of data according to a preset time period and can be stored in the storage unit Compress cache of the filtering layer in a compressed cache manner to reduce the consumed storage resources. In addition, in order to save storage resources, only the data identifiers included in the second stream of data are pulled and stored.

[0067] Furthermore, since the data identifiers of the second stream of data generally show an increasing trend, the currently pulled second stream of data and the previously pulled second stream of data (i.e., historical second stream of data) can be stored in a compressed cache manner to further reduce the consumed storage resources.

[0068] S503. Through the first asynchronous thread, compare the currently pulled second stream of data with the historical second stream of data that has been pulled to determine the incremental data identifier corresponding to the incremental data of the currently pulled second stream of data relative to the historical second stream of data.

[0069] Optionally, in this embodiment, the number of the first asynchronous threads includes multiple. The multiple first asynchronous threads pull part of the second stream of data simultaneously according to a preset time period. Step 503 may include: through the multiple first asynchronous threads, use the pull time point when starting to pull the second stream of data in the current period as a time demarcation point, compare the respective parts of the currently pulled second stream of data with the historical second stream of data pulled before the time demarcation point to determine the incremental data identifiers corresponding to the multiple first asynchronous threads respectively; through the multiple first asynchronous threads, write the respective corresponding incremental data identifiers into the same incremental storage unit.

[0070] In this embodiment, through multiple first asynchronous threads, the second stream of data can be pulled using a distributed scheme, improving the processing efficiency. By controlling the multiple first asynchronous threads to pull the second stream of data simultaneously and determining the incremental data identifier according to the time demarcation point, the accuracy of the incremental data identifiers respectively determined by the subsequent multiple first asynchronous threads is ensured.

[0071] S504. If there is the incremental data identifier, trigger the second asynchronous thread to perform a backtracking operation on multiple pieces of data in the received first stream of data according to the incremental data identifier to determine the partial first stream of data corresponding to the incremental data identifier, and store each piece of data corresponding to the partial first stream of data into the cache queue.

[0072] Specifically, referring to Figure 5B , the second asynchronous thread can perform a backtracking operation on the first stream of data through the lookup interface and filter the first stream of data according to the incremental data identifier through the filter filter to determine the partial first stream of data corresponding to the incremental data identifier.

[0073] After storing all the data corresponding to the partial first-stream data corresponding to the incremental data identifier in the cache queue, the execution of the second asynchronous thread can be stopped until a new incremental data identifier appears and then starts again.

[0074] The method for determining the incremental data identifier will be described below:

[0075] Solution 1

[0076] After receiving the first-stream data, remotely store the first-stream data; when determining the partial first-stream data, based on the remote call protocol, query the remotely stored first-stream data according to the incremental data identifier, obtain and parse the partial first-stream data corresponding to the incremental data identifier, and perform a deserialization operation on the partial first-stream data.

[0077] Specifically, through the second asynchronous thread, the first-stream data can be stored in a node dedicated to storing data (such as Figure 5B source in), and query through the call of the lookup interface to obtain and parse the partial first-stream data corresponding to the incremental data identifier; alternatively, the first-stream data can be stored in a third-party database, and query from the third-party database according to the incremental data identifier key, store each piece of data corresponding to the partial first-stream data corresponding to the incremental data identifier in the cache list, and parse and perform a deserialization operation on the partial first-stream data through the main thread.

[0078] Solution 2

[0079] Fully parse multiple pieces of data in the received first-stream data; during the process of performing the deserialization operation, according to the incremental data identifier, perform the deserialization operation on the partial data corresponding to the incremental data identifier among the parsed multiple pieces of data.

[0080] When adopting Solution 2, the second asynchronous thread is no longer executed, and step S501 can be executed by the main thread, and step 504 operation can be omitted. Specifically, the main thread can receive and parse the first-stream data and execute step S505. During the process of performing the deserialization operation, the main thread performs the deserialization operation on the partial data corresponding to the incremental data identifier among the parsed multiple pieces of data according to the incremental data identifier.

[0081] Solution 2 is mainly applicable to situations where query cannot be performed or query according to the data identifier is not possible. This solution requires full parsing of the first-stream data and consumes more resources compared to other solutions.

[0082] Solution 3

[0083] After receiving the first stream of data, store the first stream of data locally at the first data source node in the order of reception time; according to the reception time of the second stream of data corresponding to the incremental data identifier, query the first stream of data stored locally at the first data source node, obtain and parse the partial first stream of data corresponding to the incremental data identifier, and perform a deserialization operation on the partial first stream of data.

[0084] In Solution 3, the second asynchronous thread can store the first stream of data locally at the first data source node in the order of reception time, and according to the reception time of the second stream of data corresponding to the incremental data identifier, query the first stream of data stored locally at the first data source node, obtain and store each piece of data corresponding to the partial first stream of data corresponding to the incremental data identifier in the cache list, and the main thread parses and performs a deserialization operation on the partial first stream of data.

[0085] Specifically, the first stream of data within a preset life cycle can be stored locally at the first data source node in the order of reception time. The specific time of the life cycle can be set by those skilled in the art. The life cycle can specifically be the time length value from the reception of each piece of the first stream of data to the current time.

[0086] In addition, the first stream of data can be hierarchically stored in the way of time - data identifier in the order of reception time. Subsequently, the first stream of data can be directly cleared according to time.

[0087] S505. Through the main thread, read the data included in the cache queue, and perform the deserialization operation on each piece of the read - out data.

[0088] The method for specifically performing the deserialization operation can refer to the related technology and will not be elaborated here.

[0089] S506. Through the main thread, delete the data identifier corresponding to each piece of data that has undergone the deserialization operation from the incremental data identifier.

[0090] S507. Transmit the partial first stream of data that has undergone the deserialization operation to the data connection node through the network, so that the data connection node connects the second stream of data and the partial first stream of data based on the data identifier.

[0091] When using the solution provided in this embodiment for dual - stream connection, the overall resources consumed decrease from 2000 cu to 250 cu, a decrease of 87% (one cu is equal to a logical core of a computer). Specifically:

[0092] 1. The deserialization of large - stream data and the data transmission over the network decrease by 250 times.

[0093] 2. The size of a single checkpoint is reduced from 2.2T to 2.5G, a reduction of 900 times, which can greatly reduce the consumption of HDFS. This part cannot be reflected in resource statistics.

[0094] Checkpoint: Generally in the field of big data distributed computing, it is used to check the current consumption progress of consumers and is the main auxiliary means to achieve data consistency and integrity. In the field of stream computing, it refers to each computing node checking its current consumption point and saving it to the local disk or performing remote persistence.

[0095] Refer to Figure 6 , which shows a schematic structural diagram of an electronic device according to an embodiment of the present application. The specific implementation of the electronic device is not limited in the specific embodiments of the present application.

[0096] As Figure 6 shown, the electronic device may include: a processor 602, a communication interface 604, a memory 606, and a communication bus 608.

[0097] Among them:

[0098] The processor 602, the communication interface 604, and the memory 606 communicate with each other through the communication bus 608.

[0099] The communication interface 604 is used to communicate with other electronic devices or servers.

[0100] The processor 602 is used to execute the program 610, and specifically can execute the relevant steps in the above-mentioned embodiments of the stream data processing method.

[0101] Specifically, the program 610 may include program code, and the program code includes computer operation instructions.

[0102] The processor 602 may be a CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the intelligent device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0103] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0104] The program 610 can specifically be used to cause the processor 602 to perform operations corresponding to the flow data processing method described in any one of the foregoing method embodiments.

[0105] For the specific implementation of each step in the program 610, reference may be made to the corresponding descriptions in the corresponding steps and units in the foregoing method embodiments, and they have corresponding beneficial effects, which will not be elaborated here. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the devices and modules described above can refer to the corresponding process descriptions in the foregoing method embodiments, which will not be repeated here.

[0106] The embodiment of the present application also provides a computer program product, including computer instructions, where the computer instructions direct a computing device to perform operations corresponding to any one of the foregoing flow data processing methods in multiple method embodiments.

[0107] It should be noted that according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0108] The method according to the embodiments of the present application can be implemented in hardware, firmware, or be implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or be implemented as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as a RAM, a ROM, a flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for implementing the method shown herein.

[0109] Those of ordinary skill in the art will appreciate that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the embodiments of this application.

[0110] The above embodiments are only used to illustrate the embodiments of this application, rather than to limit the embodiments of this application. Those of ordinary skill in the relevant technical field can also make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of this application. The patent protection scope of the embodiments of this application shall be defined by the claims.

Claims

1. A method for processing streaming data, applied to a first data source node of first streaming data, comprising: Receiving the first streaming data; Pulling second streaming data according to a preset time period, wherein the first streaming data and the second streaming data include multiple pieces of data, and each piece of data corresponds to a data identifier; Comparing the currently pulled second streaming data with the historically pulled second streaming data to determine an incremental data identifier corresponding to the incremental data of the currently pulled second streaming data relative to the historically pulled second streaming data; Filtering multiple pieces of data in the received first streaming data according to the incremental data identifier to determine partial first streaming data corresponding to the incremental data identifier, and performing a deserialization operation on the partial first streaming data; Transmitting the partial first streaming data on which the deserialization operation has been performed to a data connection node via a network, so that the data connection node connects the second streaming data and the partial first streaming data based on the data identifier.

2. The method according to claim 1, wherein The first data source node includes a main thread and a second asynchronous thread in parallel. The main thread is used to perform the deserialization operation. Then, filtering multiple pieces of data in the received first streaming data according to the incremental data identifier to determine partial first streaming data corresponding to the incremental data identifier includes: If the incremental data identifier exists, triggering the second asynchronous thread to perform a backtracking operation on multiple pieces of data in the received first streaming data according to the incremental data identifier to determine partial first streaming data corresponding to the incremental data identifier, and storing each piece of data corresponding to the partial first streaming data in a cache queue.

3. The method according to claim 2, wherein, Performing the deserialization operation on the partial first streaming data includes: Reading the data included in the cache queue through the main thread, and performing the deserialization operation on each piece of data read out; Deleting the data identifier corresponding to each piece of data on which the deserialization operation has been performed from the incremental data identifier through the main thread.

4. The method according to claim 2, wherein, The first data source node further includes a first asynchronous thread in parallel with the main thread and the second asynchronous thread. Pulling the second streaming data according to a preset time period includes: Pulling the second streaming data according to a preset time period through the first asynchronous thread, and storing the data identifiers corresponding to multiple pieces of data of the pulled second streaming data in a compressed cache manner.

5. The method according to claim 4, wherein The number of the first asynchronous threads includes multiple. The multiple first asynchronous threads pull partial second streaming data simultaneously according to a preset time period. Comparing the currently pulled second streaming data with the historically pulled second streaming data to determine an incremental data identifier corresponding to the incremental data of the currently pulled second streaming data relative to the historically pulled second streaming data includes: Using, by the multiple first asynchronous threads, the pulling time point when pulling the second streaming data at the start of the current period as a time demarcation point, and comparing the partial currently pulled second streaming data pulled by each of them with the historically pulled second streaming data before the time demarcation point to determine the incremental data identifier corresponding to each of the multiple first asynchronous threads; Write the respective corresponding incremental data identifiers into the same incremental storage unit through multiple said first asynchronous threads.

6. The method according to claim 1, wherein After receiving the first stream of data, the method further includes: Remotely store the first stream of data; Filter multiple pieces of data in the received first stream of data according to the incremental data identifier, determine a part of the first stream of data corresponding to the incremental data identifier, and perform a deserialization operation on the part of the first stream of data, including: Based on a remote call protocol, query the first stream of data remotely stored according to the incremental data identifier, obtain and parse a part of the first stream of data corresponding to the incremental data identifier, and perform a deserialization operation on the part of the first stream of data.

7. The method according to claim 1, wherein After receiving the first stream of data, the method further includes: Store the first stream of data locally at the first data source node in the order of reception time; Filter multiple pieces of data in the received first stream of data according to the incremental data identifier, determine a part of the first stream of data corresponding to the incremental data identifier, and perform a deserialization operation on the part of the first stream of data, including: Query the first stream of data stored locally at the first data source node according to the reception time of the second stream of data corresponding to the incremental data identifier, obtain and parse a part of the first stream of data corresponding to the incremental data identifier, and perform a deserialization operation on the part of the first stream of data.

8. The method according to claim 1, wherein Filter multiple pieces of data in the received first stream of data according to the incremental data identifier, determine a part of the first stream of data corresponding to the incremental data identifier, and perform a deserialization operation on the part of the first stream of data, including: Parse multiple pieces of data in the received first stream of data; According to the incremental data identifier, perform the deserialization operation on the part of the data corresponding to the incremental data identifier among the parsed multiple pieces of data.

9. A stream processing system, including a first data source node corresponding to a first stream of data, a second data source node corresponding to a second stream of data, and a data connection node; The first data source node is configured to transmit, through the network, a part of the first stream of data on which the deserialization operation is performed to the data connection node by using the method according to any one of claims 1-8; The second data source node is configured to perform the deserialization operation on the second stream of data and transmit the second stream of data on which the deserialization operation is performed to the data connection node through the network; The data connection node is configured to connect the second stream of data and the part of the first stream of data based on a data identifier.

10. An electronic device, comprising: A processor, a memory, a communication interface, and a communication bus, through which the processor, the memory, and the communication interface complete communication with each other; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the method according to any one of claims 1-8.

11. A computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method according to any one of claims 1-8.

12. A computer program product comprising computer instructions that direct a computing device to perform operations corresponding to the method according to any one of claims 1-8.

Images