A streaming data processing method, device, equipment and medium based on data lake

Abstract

The present application discloses a streaming data processing method, apparatus, device and medium based on a data lake, the method comprising: a stream processing system can obtain a first data set to be processed from a data source. The obtained first data set is then processed in parallel to obtain multiple second data sets. The multiple second data sets are sent to a cache service system so that the cache service system stores the multiple second data sets. After obtaining the data, the stream processing system can send it to the cache service system without waiting for a fixed time, and the data written to the cache service system is immediately visible, thereby reducing the delay in waiting for data to be visible and also reducing the delay in querying data based on the cache service system.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and specifically to a streaming data processing method, apparatus, device, and medium based on a data lake. Background Art

[0002] A data lake is an evolving, scalable data processing architecture for storing, processing, and analyzing big data. It supports storage of various data structures, such as unstructured and multi-structured data. It can quickly acquire raw data from various data sources and dynamically prepare data when users access it. By integrating with various external heterogeneous data sources, it supports a wide range of enterprise applications.

[0003] In Online Analytical Processing (OLAP), jobs are typically categorized into streaming and batch jobs, specifically stream data processing and batch data processing. Currently, streaming data processing based on data lakes only achieves minute-level data visibility, failing to meet the demands of data queries requiring high real-time performance. When processing streaming data, streaming data processing engines typically use micro-batches to process small amounts of data and then write the processed data to a storage system. The time consumed by these micro-batches is typically a fixed interval of one minute or more, also known as the checkpoint interval. Therefore, when receiving data, the streaming data processing engine must wait for the checkpoint interval to expire, regardless of the volume. That is, even if the data received by the streaming data processing engine is the last data in a batch and consists of only one data item, it still needs to wait for the checkpoint interval to expire before writing the data to the storage system. Visibility is limited to data written to the storage system, resulting in latency issues for data queries based on the storage system. This limits data visibility to minutes, failing to meet the demands of data queries requiring high real-time performance. Summary of the Invention

[0004] In view of this, the present application provides a streaming data processing method, apparatus, device and medium based on a data lake to reduce the latency of subsequent data queries.

[0005] To achieve the above objectives, the technical solutions provided by this application are as follows:

[0006] In a first aspect of the present application, a method for stream data processing based on a data lake is provided. The method is applied to a stream processing system and includes:

[0007] Obtaining a first data set from a data source;

[0008] performing parallel processing on the first data set to obtain multiple second data sets;

[0009] The multiple second data sets are sent to a cache service system so that the cache service system stores the multiple second data sets.

[0010] In a second aspect of the present application, a streaming data processing method based on a data lake is provided. The method is applied to a cache service system, wherein the cache service system includes a master service node and multiple slave service nodes. The method includes:

[0011] The primary service node receives a write data request sent by a stream processing system, where the stream processing system includes a plurality of task managers;

[0012] The master service node sends service node information to the stream processing system, where the service node information includes multiple slave service nodes corresponding to the multiple task managers;

[0013] A plurality of slave service nodes receive a plurality of second data sets sent by the stream processing system and merge and store the plurality of second data sets, and the plurality of slave service nodes correspond one to one to the plurality of second data sets.

[0014] In a third aspect of the present application, a streaming data processing device based on a data lake is provided, the device being applied to a stream processing system, the device comprising:

[0015] an acquiring unit, configured to acquire a first data set from a data source;

[0016] a processing unit, configured to perform parallel processing on the first data set to obtain a plurality of second data sets;

[0017] The first sending unit is configured to send the plurality of second data sets to a cache service system so that the cache service system stores the plurality of second data sets.

[0018] In a fourth aspect of the present application, a streaming data processing device based on a data lake is provided, the device being applied to a cache service system, the device comprising:

[0019] A first receiving unit, configured to receive a write data request sent by a stream processing system, wherein the stream processing system includes a plurality of task managers;

[0020] A second sending unit, configured to send service node information to the stream processing system, wherein the service node information includes a plurality of slave service nodes corresponding to the plurality of task managers;

[0021] The second receiving unit is configured to receive and merge multiple second data sets sent by the stream processing system for storage, wherein the multiple slave service nodes correspond to the multiple second data sets in a one-to-one manner.

[0022] In a fifth aspect of the present application, an electronic device is provided, the device comprising: a processor and a memory;

[0023] The memory is used to store instructions or computer programs;

[0024] The processor is configured to execute the instructions or computer program in the memory so that the electronic device executes the method described in the first or second aspect above.

[0025] In a sixth aspect of the present application, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a device, the device executes the method described in the first or second aspect above.

[0026] In a seventh aspect of the present application, a computer program product is provided, wherein the computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0027] It can be seen that this application has the following beneficial effects:

[0028] In the above-mentioned implementation of the present application, in order to implement streaming data processing, the stream processing system can obtain a first data set to be processed from a data source. The obtained first data set is then processed in parallel to obtain multiple second data sets. The multiple second data sets are sent to the cache service system so that the cache service system can store the multiple second data sets. After obtaining the data, the stream processing system can send it to the cache service system without waiting for a fixed period of time. The data written to the cache service system is immediately visible, thereby reducing the delay in waiting for data to be visible and also reducing the delay in querying data based on the cache service system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 A flowchart of a data lake-based streaming data processing method provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the structure of a stream processing method provided in an embodiment of the present application;

[0032] Figure 3A flowchart of another data lake-based streaming data processing method provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a data lake-based streaming data processing device provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of another data lake-based streaming data processing device provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0037] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, the technical background involved in the embodiments of the present application will be introduced below.

[0038] A data lake is an evolving, scalable data processing architecture for storing, processing, and analyzing big data. It supports storage of various data structures, such as unstructured and multi-structured data. It can quickly acquire raw data from various data sources and dynamically prepare data when users access it. By integrating with various external heterogeneous data sources, it supports a wide range of enterprise applications.

[0039] Currently, streaming data processing based on data lakes can only achieve minute-level data visibility and cannot meet the needs of data queries with high real-time requirements. The main reason is that when the streaming data processing engine performs streaming data processing, it must wait for a fixed time greater than or equal to one minute, and obtain small-scale data for processing during the waiting time. Even if the last batch of data received by the streaming data processing engine is only one piece of data, it is necessary to wait for a fixed time before the received data can be written to the storage system, resulting in a delay problem for subsequent data queries based on the storage system. Typically, the storage system can be a distributed file system (Hadoop Distributed File System, HDFS), which includes multiple files that can be used to store data processed by the streaming processing engine. The data stored in HDFS is visible.

[0040] Based on this, an embodiment of the present application provides a streaming processing method based on a data lake to reduce the delay of subsequent data queries. In specific implementation, the stream processing system can obtain a first data set to be processed from a data source. The obtained first data set is then processed in parallel to obtain multiple second data sets. The multiple second data sets are sent to the cache service system so that the cache service system stores the multiple second data sets. After obtaining the data, the stream processing system can send it to the cache service system without waiting for a fixed time, and the data written to the cache service system is immediately visible, thereby reducing the delay in waiting for data to be visible, and also reducing the delay in querying data based on the cache service system.

[0041] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, a detailed introduction will be given below with reference to the accompanying drawings.

[0042] See also Figure 1 , Figure 1 A flowchart of a data lake-based streaming data processing method provided in an embodiment of the present application.

[0043] The method may be performed by a stream processing system, wherein the stream processing system may be a stream processing engine. The method may include the following steps:

[0044] S101: Acquire a first data set from a data source.

[0045] The stream processing system can obtain data of different structure types from various data sources for processing. The data sources can be data files, databases, message queues, etc., each corresponding to data of different structures. In other words, the first data set can include data of various types to be processed.

[0046] S102: Parallel processing is performed on the first data set to obtain multiple second data sets.

[0047] When a stream processing system acquires raw data, i.e., a first data set, it can perform parallel processing on the data in the first data set to obtain multiple processed second data sets. In one possible implementation, the stream processing system can include multiple task managers. After the stream processing system acquires the first data set, it can divide the first data set into multiple data groups based on data processing rules, with the data in each data group corresponding to the same data processing rule. For example, the data processing rule can include key value information, data structure type, data source, etc., that is, the data in the first data set can be divided based on the key value, structure type, data source to which the data belongs, etc., to obtain multiple data groups. The stream processing system can then use multiple task managers to process the multiple data groups to obtain multiple second data sets. The multiple task managers, multiple data groups, and multiple second data sets are in a one-to-one correspondence. Optionally, the job manager of the stream processing system can divide the first data set based on the data processing rules.

[0048] In one possible implementation, after the stream processing system groups the first data set to obtain multiple data groups, it can distribute each data group to a corresponding task manager. The multiple task managers then process the data in the multiple data groups in parallel, thereby obtaining multiple second data sets. When the task managers process the data in the data groups, they can associate the data, filter out invalid data, and so on. For example, when obtaining sales information for an e-commerce platform, it is necessary to associate the users who purchased products on the e-commerce platform with the products purchased by the users, and filter out data on products that users purchased but then received a refund.

[0049] S103: Send the multiple second data sets to the cache service system, so that the cache service system stores the multiple second data sets.

[0050] After processing the first data set, multiple second data sets are obtained. The stream processing system can then send the multiple second data sets to the cache service system so that the cache service system can store the multiple second data sets. In one possible implementation, the cache service system may include a master service node and multiple slave service nodes, wherein the master service node and the slave service nodes may be servers having a cache for storing data. In a specific implementation, after the stream processing system processes the data in the data group and obtains the second data set, when it wants to send the second data set to the cache service system, it can first send a data write request to the master service node, that is, send a request to the cache service system for the second data set. After the master service node responds to the request, it can send service node information to the stream processing system, wherein the service node information includes multiple slave service nodes corresponding to the multiple task managers of the stream processing system. That is, each task manager that processes the second data set corresponds to a slave service node. The stream processing system can use the task manager to send the second data set to the slave service node corresponding to the task manager, where the second data set is processed by the task manager.

[0051] Optionally, after processing the data in the data group to obtain the second data set, the task manager can send a request to write data to the main service node. After receiving the request sent by the task manager, the main service node can send service node information to the task manager, that is, send the slave service node information corresponding to the task manager, so that the task manager can send the second data set to the corresponding slave service node.

[0052] It should be noted that the size of the data sent by the task manager to the slave service node can be preconfigured by the task manager. For example, the task manager can be configured to send data to the slave service node upon receiving a piece of data, or it can be configured to send data to the slave service node upon receiving a preset piece of data. In other words, the size of the second data set is not limited in this embodiment, and can include one or multiple pieces of data.

[0053] After the second data set is received from the service node and stored, related information may also be stored, including the number of data items in the stored second data set, the data size, and the corresponding task manager identifier.

[0054] In one possible implementation, the stream processing system can also obtain the status of sending the second data set, that is, whether the second data set is sent successfully. When the status is failure to send, a request to write data can be sent to the main service node again, and the main service node can send new service node information to the stream processing system, wherein the new service node information includes a new slave service node corresponding to the task manager that failed to send the second data set. The new slave service node is a different service node from the original slave service node corresponding to the task manager. The task manager is used to send the second data set to the new slave service node. That is, when the task manager fails to send the second data set to the slave service node, the task manager can resend a request to write data to the main service node so that the main service node assigns a new slave service node to the task manager, and the task manager sends the second data set to the new slave service node again.

[0055] Optionally, the stream processing system can also obtain the status of the second data set being stored by the slave service node. If the slave service node fails to store the second data set, it can send a storage failure message to the stream processing system, thereby triggering the stream processing system to resend the second data set. Specifically, the stream processing system can resend a data write request to the master service node, and the master service node can send new service node information to the stream processing system, instructing it to send the second data set to the new slave service node for storage. Optionally, obtaining the status of sending the second data set or receiving the request to store the second data set from the slave service node can be performed by the task manager of the stream processing system.

[0056] It should be noted that when the task manager sends the second data set to the slave service node, each sending task can correspond to a unique task identifier, so as to distinguish each sending task, prevent the successfully sent task from being re-executed, or the failed sent task from being missed and re-executed, and facilitate recording the execution status of each sending task. After the sending tasks of multiple second data sets are completed, the stream processing system can obtain the status set of sending multiple second data sets and submit the status set to the cache service system. For example, the stream processing system can send a request for submission status to the main service node of the cache service system, and after receiving the instruction sent by the main service node to allow submission status, it will send the status set of sending multiple second data sets to the main service node.

[0057] In one possible implementation, when the slave service node stores data in the second data set, it can be determined whether the size of the data stored in the slave service node is greater than a preset threshold, that is, exceeds the size of data allowed to be stored. At this time, the slave service node can send the stored data to the storage system for storage, and the slave service node can clear the stored data so that new data can be stored again later. The data stored in the slave service node can include part or all of the data in the second data set stored this time, and can also include data that has been stored previously. Optionally, the storage system can be a distributed file system, and the distributed file system can include multiple files for storing data sent by the cache service system.

[0058] Optionally, when storing data, the cache service system can be divided into an alterable cache and a read-only cache. In addition to basic data read and write capabilities, the data stored in the alterable cache can also be modified. The alterable cache can store data in a row-based format, while the read-only cache can store data in a column-based format. For example, row-based storage can store different attribute information corresponding to the same user, including age, gender, education level, and job. Column-based storage can store data corresponding to the same attribute, such as different job types under the job attribute. Preferably, the cache service system can first store data in the alterable cache. When the storage space in the alterable cache exceeds a threshold, the data can be sent to the read-only cache for storage and the data in the alterable cache can be cleared. Storing data in a cached format can improve user data query efficiency and enhance data diversity. For example, users can perform point-to-point data searches or narrow range searches based on attribute information. When the data stored in the read-only cache exceeds a threshold, the data in the read-only cache can be asynchronously written to a distributed file system for storage and the data in the read-only cache can be cleared.

[0059] Based on the above method embodiment, the present application embodiment provides a structural diagram of a streaming data processing method. Figure 2 , Figure 2 A structural diagram of a streaming processing method provided in an embodiment of the present application.

[0060] The structure includes a data source 201, a stream processing system 202, a cache service system 203 and a distributed file system 204; wherein, the stream processing system 202 includes task manager 1, task manager 2, ..., task manager m, the cache service system 203 includes a master service node, slave service node 1, slave service node 2, ..., slave service node n, and the distributed file system 204 includes file 1, file 2, ..., file w, wherein m, n, and w are all positive integers.

[0061] The stream processing system can obtain a first data set from a data source, and then divide the first data set into multiple data groups and distribute them to corresponding multiple task managers. For any task manager i, after processing the data in the data group to obtain the second data set, it can send a request to write data to the master service node. After receiving the request sent by task manager i, the master service node can send service node information to task manager i, that is, send the slave service node j corresponding to the task manager, so that task manager i can send the second data set to the corresponding slave service node j. When the size of the data stored in the slave service node j exceeds a preset threshold, the stored data can be sent to file h in the distributed file system. File h stores the data sent by the slave service node, so that the slave service node can clear its memory to receive and store new data.

[0062] The data loss processing method provided in the embodiments of this application allows the stream processing system to send data to the cache service system immediately after receiving and processing it, without waiting for a fixed interval. This ensures data visibility and reduces latency. Furthermore, when data is stored in memory, the efficiency of data reading, writing, and querying can be improved, and multiple data query methods can be implemented to meet the needs of various scenarios.

[0063] Based on the above method embodiment, the present application embodiment also provides a streaming data processing method based on a data lake. Figure 3 , Figure 3 A flowchart of another data lake-based streaming data processing method provided in an embodiment of the present application.

[0064] The method may be performed by a cache service system, wherein the cache service system includes a master service node and multiple slave service nodes. The method may include the following steps:

[0065] S301: The primary service node receives a write data request sent by the stream processing system.

[0066] According to the above embodiment, after the stream processing system obtains the first data set, it can process the first data set in parallel to obtain multiple second data sets, and then send a request to write data to the primary service node.

[0067] S302: The primary service node sends service node information to the stream processing system.

[0068] The stream processing system includes multiple task managers, which can process the data groups obtained by grouping the first data to obtain a second data set. After responding to a write data request sent by the stream processing system, the master service node can send service node information to the stream processing system, where the service node information includes multiple slave service nodes corresponding to the multiple task managers. In other words, the multiple task managers can send write data requests to the master service node, and then receive slave service node information sent by the master service node to send the second data set to the corresponding slave service nodes.

[0069] S303: Multiple slave service nodes receive multiple second data sets sent by the stream processing system, merge and store them.

[0070] After the stream processing system receives the service node information sent by the master service node, it can send multiple second data sets to multiple slave service nodes according to the multiple slave service nodes included in the service node information. The multiple slave service nodes correspond one-to-one to the multiple second data sets.

[0071] Based on the above embodiments, it can be seen that the task manager in the stream processing system can process the data group to obtain the corresponding second data set. Optionally, the task manager can send the obtained second data set to the corresponding slave service node. After the slave service node receives the second data set sent by the corresponding task manager, it can store the second data set.

[0072] In one possible implementation, after the stream processing system sends the multiple second data sets to the multiple slave service nodes, the master node may further receive a submission status request from the stream processing system, where the status indicates whether the stream processing system successfully sent the multiple second data sets. After granting the submission status request from the stream processing system, the master node may obtain a status set sent by the stream processing system, where the status set includes the status of the multiple task managers sending the multiple second data sets to the multiple slave service nodes.

[0073] When the slave service node stores data in the second data set, it can determine whether the size of the data stored in the slave service node is greater than a preset threshold, that is, exceeds the allowed storage size. In this case, the slave service node can send the stored data to the storage system for storage, and the slave service node can clear the stored data to store new data later.

[0074] In the above embodiment, after the stream processing system receives data for processing, it can send the data to the cache service system without waiting for a fixed interval, thereby achieving data visibility and reducing the latency of data visibility.

[0075] Based on the above method embodiment, the present application embodiment provides a streaming data processing device based on a data lake. Figure 4 , Figure 4 A schematic diagram of a data lake-based streaming data processing device provided in an embodiment of the present application.

[0076] The apparatus 400 can be applied to a stream processing system, and includes:

[0077] An acquisition unit 401 is configured to acquire a first data set from a data source;

[0078] A processing unit 402 is configured to perform parallel processing on the first data set to obtain multiple second data sets;

[0079] The first sending unit 403 is configured to send the multiple second data sets to the cache service system, so that the cache service system stores the multiple second data sets.

[0080] In one possible implementation, the stream processing system includes multiple task managers, and the processing unit 402 is specifically used to divide the first data set into multiple data groups based on data processing rules; use multiple task managers to process the multiple data groups to obtain the multiple second data sets, and the multiple task managers, the multiple data groups and the multiple second data sets correspond one to one.

[0081] In one possible implementation, the cache service system includes a master service node and multiple slave service nodes, and the first sending unit 403 is specifically used to receive service node information sent by the master service node, where the service node information includes multiple slave service nodes corresponding to the multiple task managers; for any task manager among the multiple task managers, the task manager is used to send a second data set to the slave service node corresponding to the task manager, where the second data set corresponds to the task manager.

[0082] In a possible implementation, the acquiring unit 401 is further configured to acquire a status of sending the second data set;

[0083] The apparatus 400 further includes: a receiving unit; the receiving unit is configured to, when the status is a sending failure, again receive new service node information sent by the primary service node to the task manager, wherein the new service node information includes a new secondary service node corresponding to the task manager;

[0084] The first sending unit 403 is further configured to send the second data set to the new slave service node by using the task manager.

[0085] In a possible implementation, the acquiring unit 401 is further configured to acquire a status set of sending multiple second data sets; and send the status set to the cache service system.

[0086] Based on the above method embodiment and device embodiment, the present application embodiment also provides a streaming data processing device based on a data lake. Figure 5 , Figure 5 A schematic diagram of another data lake-based streaming data processing device provided in an embodiment of the present application.

[0087] The device 500 can be applied to a cache service system, and includes:

[0088] A first receiving unit 501 is configured to receive a write data request sent by a stream processing system, wherein the stream processing system includes a plurality of task managers;

[0089] A second sending unit 502 is configured to send service node information to the stream processing system, where the service node information includes multiple slave service nodes corresponding to the multiple task managers;

[0090] The second receiving unit 503 is configured to receive and merge multiple second data sets sent by the stream processing system for storage, wherein the multiple slave service nodes correspond to the multiple second data sets in a one-to-one manner.

[0091] In a possible implementation, the second receiving unit 503 is specifically configured to receive and store the second data set sent by the task manager for any slave service node among the multiple slave service nodes, and the slave service node, the task manager and the second data set correspond one to one.

[0092] In a possible implementation, the first receiving unit 501 is further configured to receive a submission status request sent by the stream processing system;

[0093] The apparatus 500 further includes: an acquisition unit; the acquisition unit is configured to acquire a state set sent by the stream processing system, wherein the state set includes states of the multiple task managers sending multiple second data sets to the multiple slave service nodes.

[0094] In one possible implementation, the device 500 further includes: a judgment unit; the judgment unit is used to judge whether the size of the data stored in the slave service node is greater than a preset threshold; if so, the stored data is sent to the storage system and the stored data is cleared.

[0095] The beneficial effects of the data lake-based streaming data processing device provided in the embodiment of the present application can be found in the above method embodiment and will not be repeated here.

[0096] It should be noted that the specific implementation of each unit in this embodiment can refer to the relevant description in the above method embodiment. The division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional units in the embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. For example, in the above embodiment, the processing unit and the sending unit can be the same unit or different units. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0097] See also Figure 6 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing an embodiment of the present application. The terminal device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0098] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0099] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0100] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present application are performed.

[0101] The electronic device provided in the embodiment of the present application and the method provided in the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0102] An embodiment of the present application provides a computer storage medium having a computer program stored thereon, which implements the method provided in the above embodiment when the program is executed by a processor.

[0103] It should be noted that the computer-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0104] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0105] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0106] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the method.

[0107] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0109] The units involved in the embodiments described in this application may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0110] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0111] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] According to one or more embodiments of the present application, a data lake-based streaming data processing method is provided. The method is applied to a stream processing system and includes:

[0113] Obtaining a first data set from a data source;

[0114] performing parallel processing on the first data set to obtain multiple second data sets;

[0115] The multiple second data sets are sent to a cache service system so that the cache service system stores the multiple second data sets.

[0116] According to one or more embodiments of the present application, the stream processing system includes multiple task managers, and the parallel processing of the first data set to obtain multiple second data sets includes:

[0117] dividing the first data set into a plurality of data groups based on a data processing rule;

[0118] The plurality of data groups are processed by using a plurality of task managers to obtain the plurality of second data sets, wherein the plurality of task managers, the plurality of data groups and the plurality of second data sets correspond to each other one by one.

[0119] According to one or more embodiments of the present application, the cache service system includes a master service node and multiple slave service nodes, and sending the multiple second data sets to the cache service system includes:

[0120] receiving service node information sent by the master service node, wherein the service node information includes a plurality of slave service nodes corresponding to the plurality of task managers;

[0121] For any task manager among the multiple task managers, the task manager is used to send a second data set to a slave service node corresponding to the task manager, where the second data set corresponds to the task manager.

[0122] According to one or more embodiments of the present application, the method further includes:

[0123] Obtaining a status of sending the second data set;

[0124] When the status is sending failure, receiving again new service node information sent by the master service node to the task manager, the new service node information including a new slave service node corresponding to the task manager;

[0125] The second data set is sent to the new slave service node using the task manager.

[0126] According to one or more embodiments of the present application, the method further includes:

[0127] Acquire a status set of sending multiple second data sets;

[0128] The state set is sent to the cache service system.

[0129] According to one or more embodiments of the present application, a data lake-based streaming data processing method is provided. The method is applied to a cache service system, wherein the cache service system includes a master service node and multiple slave service nodes. The method includes:

[0130] The primary service node receives a write data request sent by a stream processing system, where the stream processing system includes a plurality of task managers;

[0131] The master service node sends service node information to the stream processing system, where the service node information includes multiple slave service nodes corresponding to the multiple task managers;

[0132] A plurality of slave service nodes receive a plurality of second data sets sent by the stream processing system and merge and store the received data sets. The plurality of slave service nodes correspond to the plurality of second data sets in a one-to-one manner.

[0133] According to one or more embodiments of the present application, the multiple slave service nodes receiving the multiple second data sets sent by the stream processing system and merging and storing the data includes:

[0134] For any slave service node among the multiple slave service nodes, the slave service node receives and stores the second data set sent by the task manager, and the slave service node, the task manager and the second data set correspond one to one.

[0135] According to one or more embodiments of the present application, the method further includes:

[0136] The master node receives a submission status request sent by the stream processing system;

[0137] The master node obtains a state set sent by the stream processing system, where the state set includes states where the multiple task managers send multiple second data sets to the multiple slave service nodes.

[0138] According to one or more embodiments of the present application, the method further includes:

[0139] The slave service node determines whether the size of the stored data is greater than a preset threshold, and if so, sends the stored data to the storage system and clears the stored data.

[0140] According to one or more embodiments of the present application, a data lake-based streaming data processing device is provided. The device is applied to a stream processing system and includes:

[0141] an acquiring unit, configured to acquire a first data set from a data source;

[0142] a processing unit, configured to perform parallel processing on the first data set to obtain a plurality of second data sets;

[0143] The first sending unit is configured to send the plurality of second data sets to a cache service system so that the cache service system stores the plurality of second data sets.

[0144] In one or more embodiments of the present application, the stream processing system includes multiple task managers, and the processing unit is specifically used to divide the first data set into multiple data groups based on data processing rules; use multiple task managers to process the multiple data groups to obtain the multiple second data sets, and the multiple task managers, the multiple data groups and the multiple second data sets correspond one to one.

[0145] In one or more embodiments of the present application, the cache service system includes a master service node and multiple slave service nodes, and the first sending unit is specifically used to receive service node information sent by the master service node, and the service node information includes multiple slave service nodes corresponding to the multiple task managers; for any task manager among the multiple task managers, the task manager is used to send a second data set to the slave service node corresponding to the task manager, and the second data set corresponds to the task manager.

[0146] In one or more embodiments of the present application, the acquiring unit is further configured to acquire a status of sending the second data set;

[0147] The apparatus further includes: a receiving unit; the receiving unit is configured to, when the status is a sending failure, again receive new service node information sent by the primary service node to the task manager, wherein the new service node information includes a new secondary service node corresponding to the task manager;

[0148] The first sending unit is further configured to send the second data set to the new slave service node by using the task manager.

[0149] In one or more embodiments of the present application, the acquiring unit is further configured to acquire a status set of sending multiple second data sets; and send the status set to the cache service system.

[0150] According to one or more embodiments of the present application, a data lake-based streaming data processing device is provided, which is applied to a cache service system and includes:

[0151] A first receiving unit, configured to receive a write data request sent by a stream processing system, wherein the stream processing system includes a plurality of task managers;

[0152] A second sending unit, configured to send service node information to the stream processing system, wherein the service node information includes a plurality of slave service nodes corresponding to the plurality of task managers;

[0153] The second receiving unit is configured to receive and merge multiple second data sets sent by the stream processing system for storage, wherein the multiple slave service nodes correspond to the multiple second data sets in a one-to-one manner.

[0154] In one or more embodiments of the present application, the second receiving unit is specifically used to receive and store the second data set sent by the task manager for any slave service node among the multiple slave service nodes, and the slave service node, the task manager and the second data set correspond one to one.

[0155] In one or more embodiments of the present application, the first receiving unit is further configured to receive a submission status request sent by the stream processing system;

[0156] The apparatus further includes: an acquisition unit; the acquisition unit is configured to acquire a state set sent by the stream processing system, wherein the state set includes states of the multiple task managers sending multiple second data sets to the multiple slave service nodes.

[0157] In one or more embodiments of the present application, the device further includes: a judgment unit; the judgment unit is used to judge whether the size of the data stored in the slave service node is greater than a preset threshold; if so, the stored data is sent to the storage system and the stored data is cleared.

[0158] According to one or more embodiments of the present application, an electronic device is provided, the device including: a processor and a memory;

[0159] The memory is used to store instructions or computer programs;

[0160] The processor is configured to execute the instructions or computer program in the memory so that the electronic device executes the data lake-based streaming data processing method.

[0161] According to one or more embodiments of the present application, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed on a device, the device executes the data lake-based streaming data processing method.

[0162] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0163] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0164] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0165] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0166] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A streaming data processing method based on a data lake, characterized in that: The method is applied to a stream processing system including a plurality of task managers, and the method includes: Obtaining a first data set from a data source; Processing the first data set in parallel to obtain multiple second data sets, where data in each second data set corresponds to the same data processing rule, where the data processing rule is key value information, data structure type, or data source; sending the plurality of second data sets to a cache service system so that the cache service system stores the plurality of second data sets, The performing parallel processing on the first data set to obtain multiple second data sets includes: dividing the first data set into a plurality of data groups based on the data processing rule; The plurality of data groups are processed by using a plurality of task managers to obtain the plurality of second data sets, wherein the plurality of task managers, the plurality of data groups and the plurality of second data sets correspond to each other one by one.

2. The method according to claim 1, characterized in that The cache service system includes a master service node and multiple slave service nodes, and sending the multiple second data sets to the cache service system includes: receiving service node information sent by the master service node, wherein the service node information includes a plurality of slave service nodes corresponding to the plurality of task managers; For any task manager among the multiple task managers, the task manager is used to send a second data set to a slave service node corresponding to the task manager, where the second data set corresponds to the task manager.

3. The method according to claim 2, characterized in that The method further comprises: Obtaining a status of sending the second data set; When the status is sending failure, receiving again new service node information sent by the master service node to the task manager, the new service node information including a new slave service node corresponding to the task manager; The second data set is sent to the new slave service node using the task manager.

4. The method according to claim 1, wherein The method further comprises: Acquire a status set of sending multiple second data sets; The state set is sent to the cache service system.

5. A streaming data processing method based on a data lake, characterized in that: The method is applied to a cache service system, the cache service system including a master service node and multiple slave service nodes, and the method includes: The primary service node receives a write data request sent by a stream processing system, where the stream processing system includes a plurality of task managers; The master service node sends service node information to the stream processing system, where the service node information includes multiple slave service nodes corresponding to the multiple task managers; Multiple slave service nodes receive multiple second data sets sent by the stream processing system and merge them for storage. The multiple slave service nodes correspond one-to-one to the multiple second data sets. The data in each second data set corresponds to the same data processing rule, which is key value information, data structure type or data source.

6. The method according to claim 5, characterized in that The plurality of slave service nodes receiving the plurality of second data sets sent by the stream processing system, merging and storing the second data sets, comprises: For any slave service node among the multiple slave service nodes, the slave service node receives and stores the second data set sent by the task manager, and the slave service node, the task manager and the second data set correspond one to one.

7. The method according to claim 6, characterized in that The method further comprises: The primary service node receives a submission status request sent by the stream processing system; The master service node obtains a state set sent by the stream processing system, where the state set includes states where the multiple task managers send multiple second data sets to the multiple slave service nodes.

8. The method according to claim 5, characterized in that The method further comprises: The slave service node determines whether the size of the stored data is greater than a preset threshold, and if so, sends the stored data to the storage system and clears the stored data.

9. A streaming data processing device based on a data lake, characterized in that: The device is applied to a stream processing system including a plurality of task managers, and includes: an acquiring unit, configured to acquire a first data set from a data source; a processing unit configured to perform parallel processing on the first data set to obtain a plurality of second data sets, wherein the data in each second data set corresponds to the same data processing rule, wherein the data processing rule is key value information, data structure type, or data source; A first sending unit is configured to send the plurality of second data sets to a cache service system so that the cache service system stores the plurality of second data sets. The performing parallel processing on the first data set to obtain multiple second data sets includes: dividing the first data set into a plurality of data groups based on the data processing rule; The plurality of data groups are processed by using a plurality of task managers to obtain the plurality of second data sets, wherein the plurality of task managers, the plurality of data groups and the plurality of second data sets correspond to each other one by one.

10. A streaming data processing device based on a data lake, characterized in that: The device is applied to a cache service system, and includes: A first receiving unit, configured to receive a write data request sent by a stream processing system, wherein the stream processing system includes a plurality of task managers; A second sending unit, configured to send service node information to the stream processing system, wherein the service node information includes a plurality of slave service nodes corresponding to the plurality of task managers; The second receiving unit is used to receive multiple second data sets sent by the stream processing system and merge them for storage, the multiple slave service nodes and the multiple second data sets correspond one to one, and the data in each second data set corresponds to the same data processing rule, which is key value information, data structure type or data source.

11. An electronic device, characterized in that: The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory, so that the electronic device executes the method according to any one of claims 1 to 4 or 5 to 8.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a device, the device is caused to execute the method according to any one of claims 1 to 4 or 5 to 8.

Images