Data processing method, system and non-transitory computer readable storage medium

Abstract

The present disclosure provides a data processing method, system and non-transient computer-readable storage medium, which relate to the field of computer technology. The data processing method includes: writing the stream data received from the client into the message middleware; performing real-time calculation on the stream data through the structured stream processing engine to obtain the real-time calculation results; writing the real-time calculation results into the data lake and the online analytical processing database respectively by updating or inserting Upsert; receiving and responding to offline data requests through the data lake, and receiving and responding to real-time data requests through the online analytical processing database. The present disclosure implements a stream-batch integrated data processing method, which can not only save computing resources and storage resources, but also ensure the consistency of offline data requests and real-time data requests in response results, thereby providing convenience for the development and maintenance of data processing systems.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, system and non-transitory computer-readable storage medium. Background Art

[0002] Data processing systems using Lambda architecture usually include a real-time subsystem and an offline subsystem. These two subsystems use different computing logic and storage resources.

[0003] By initiating query requests to these two subsystems respectively through different query methods, different business timeliness requirements can be met respectively. Summary of the invention

[0004] A technical problem solved by the present disclosure is how to implement a stream-batch integrated data processing method.

[0005] According to one aspect of the present disclosure, a data processing method is provided, including: writing stream data received from a client into a message middleware; performing real-time calculations on the stream data through a structured stream processing engine to obtain real-time calculation results; writing the real-time calculation results into a data lake and an online analytical processing database respectively in an update or insert Upsert manner; receiving and responding to offline data requests through the data lake, and receiving and responding to real-time data requests through the online analytical processing database.

[0006] In some embodiments, performing real-time calculations on stream data through a structured stream processing engine includes: reading stream data from a message middleware through a Spark structured stream processing engine and grouping the stream data according to a session identifier to obtain stream data for each group session; caching the stream data for each group session and executing business calculation logic on the stream data for each group session through the state operator flatMapGroupsWithState of the Spark structured stream processing engine to obtain statistical fields for each group session.

[0007] In some embodiments, executing business calculation logic on the flow data of each group session to obtain the statistical fields of each group session includes: sorting each flow data in any group session; obtaining business indicator parameters of each flow data in any group session; and determining the statistical fields of any group session based on the business indicator parameters.

[0008] In some embodiments, writing the real-time calculation results into the data lake and the online analytical processing database respectively by update or Upsert includes: when the real-time calculation results change, writing the real-time calculation results into the data lake and the online analytical processing database respectively by update or Upsert.

[0009] In some embodiments, the interval duration for real-time calculation of streaming data by the structured stream processing engine is greater than a preset duration.

[0010] In some embodiments, the data processing method further includes: setting a session completed partition and a session uncompleted partition on the data lake; wherein the real-time calculation result is written to the session uncompleted partition of the data lake in an Upsert manner.

[0011] In some embodiments, writing the stream data received from the client to the message middleware includes: receiving the stream data from the client through an HTTP receiver; and writing the stream data to the Kafka message middleware through the HTTP receiver.

[0012] According to another aspect of the present disclosure, a data processing system is provided, including: an HTTP receiver, configured to write streaming data received from a client into a message middleware; the message middleware, configured to store the streaming data; a Spark structured stream processing engine, configured to perform real-time calculations on the streaming data, obtain real-time calculation results, and configured to write the real-time calculation results into a data lake and an online analytical processing database respectively in an update or insert Upsert manner; the data lake, configured to receive and respond to offline data requests; and the online analytical processing database, configured to receive and respond to real-time data requests.

[0013] In some embodiments, the Spark structured streaming processing engine is configured to: read stream data from the message middleware and group the stream data according to the session identifier to obtain the stream data of each group session; through the state operator flatMapGroupsWithState, cache the stream data of each group session and execute business calculation logic on the stream data of each group session to obtain the statistical fields of each group session.

[0014] In some embodiments, the Spark structured stream processing engine is configured to: sort each stream data in any group session; obtain business indicator parameters of each stream data in any group session; and determine the statistical fields of any group session based on the business indicator parameters.

[0015] In some embodiments, the Spark structured stream processing engine is configured to: when the real-time calculation results change, write the real-time calculation results into the data lake and the online analytical processing database respectively by updating or inserting Upsert.

[0016] In some embodiments, the interval duration for real-time calculation of streaming data by the structured stream processing engine is greater than a preset duration.

[0017] In some embodiments, a session completed partition and a session uncompleted partition are set on the data lake; the Spark structured stream processing engine is configured to write the real-time calculation results into the session uncompleted partition of the data lake in an Upsert manner.

[0018] In some embodiments, the message middleware is Kafka message middleware; the HTTP receiver is configured to: receive stream data from the client; and write the stream data to the Kafka message middleware.

[0019] According to another aspect of the present disclosure, a data processing system is provided, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the aforementioned data processing method based on instructions stored in the memory.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the aforementioned data processing method is implemented.

[0021] The present disclosure implements a stream-batch integrated data processing method, which can not only save computing resources and storage resources, but also ensure the consistency of response results between offline data requests and real-time data requests, thereby facilitating the development and maintenance of data processing systems.

[0022] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 A flowchart of a data processing method according to some embodiments of the present disclosure is shown.

[0025] Figure 2 A schematic diagram of the process of performing real-time calculations on streaming data through a structured stream processing engine is shown.

[0026] Figure 3 A schematic diagram of the structure of a data processing system according to some embodiments of the present disclosure is shown.

[0027] Figure 4 A schematic diagram showing the structure of a data processing system according to some other embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0029] Research has found that data processing systems that use Lambda architecture waste computing and storage resources, and are more complex to develop and maintain. For example, in order to modify a technical indicator, it is necessary to modify the calculation logic in the two subsystems separately. At the same time, due to the different calculation logics in the two subsystems, there is a high probability that there will be differences between the offline response results and the real-time response results, requiring subsequent comparison of the result data.

[0030] If the open source stream processing framework Flink or the real-time computing framework Spark Structured Streaming is used to implement the stream-batch integration of the Append mode, there are certain defects. For example, the use of Flink or Spark Structured Streaming requires the calculation results to be output to the HDFS (Hadoop Distributed File System) and OLAP (Online Analytical Processing) system for storage, but HDFS does not support Upsert (update or insert) operations but only supports Append (insert at the end) operations, so it is necessary to ensure the correctness of the written data before writing, which leads to a large delay, affecting timeliness and user experience. For another example, HDFS writing is not atomic, and when the task fails, it is easy to cause dirty data, and the processing is relatively complicated. In view of this, the present disclosure uses the open source Delta Lake data lake technology to implement a stream-batch integrated data processing method.

[0031] First combine Figure 1 Some embodiments of the data processing method of the present disclosure are described.

[0032] Figure 1 FIG. 1 is a flow chart showing a data processing method according to some embodiments of the present disclosure. Figure 1 As shown, the method includes steps S101 to S104.

[0033] In step S101 , the stream data received from the client is written into the message middleware.

[0034] For example, streaming data is received from a client via an HTTP receiver. The client may be an application or web page on a personal computer or mobile device, and streaming data may be embedded point data and user browsing data. Then, the streaming data may be written to the Kafka message middleware via the HTTP receiver.

[0035] In step S102, the stream data is calculated in real time by the structured stream processing engine to obtain real-time calculation results.

[0036] For example, the stream data stored in the Kafka message middleware can be read through the structured stream processing engine Spark Structured Streaming, and the stream data can be calculated in real time. Those skilled in the art will understand that there is a certain interval between each real-time calculation of the stream data. That is to say, under the premise of satisfying the correctness of the data, in order to improve the timeliness of the data, a real-time calculation is triggered at a certain interval. The interval can be set to be greater than the preset duration, for example, set to 1 minute, so as to take into account the timeliness of the data and the throughput of the data, and reduce the negative impact on the Delta Lake and OLAP systems.

[0037] In step S103, the real-time calculation results are written into the data lake and the online analytical processing database respectively by updating or inserting Upsert.

[0038] For example, the structured stream processing engine can use the "Batch Result Upsert (batch result update or insert)" module to write the real-time calculation results to the Delta Lake data lake and the online analytical processing database OLAP system in the form of update or insert Upsert.

[0039] In some embodiments, in order to further improve the efficiency and performance of data processing, it can be determined whether there is a change between each real-time calculation result and the last real-time calculation result. In the case where the real-time calculation result changes, the real-time calculation result is written to the data lake and the online analytical processing database respectively in an update or insert Upsert manner. Among them, Upsert is a term in database operations, which means that if the primary key exists, the data is updated in Update mode, and if the primary key does not exist, the data is inserted in Insert mode. If the real-time calculation result does not change, there is no need to write the real-time calculation result to the Delta Lake data lake or the online analytical processing database OLAP system. After the session ends, the final real-time calculation result can be written to the Delta Lake and OLAP system again in Upsert mode to correct the real-time calculation result of the intermediate state written previously.

[0040] In some embodiments, in order to further improve the efficiency and performance of the Upsert operation, a session completed partition and a session uncompleted partition can be set on the Delta Lake data lake. The real-time calculation results are written to the session uncompleted partition of the data lake in an Upsert manner, thereby greatly reducing the amount of data that needs to be scanned for the Upsert operation.

[0041] In step S104, offline data requests are received and responded to through the data lake, and real-time data requests are received and responded to through the online analytical processing database.

[0042] The data in the Delta Lake data lake is mainly used for offline computing and analysis, so the Delta Lake data lake supports offline data query requests and data analysis requests, such as query requests from engines such as Hive, Spark, and Presto. The OLAP database supports multi-dimensional real-time data query requests, such as multi-dimensional real-time data query requests for web pages. Those skilled in the art can understand that the OLAP system here can specifically use Clickhouse, Doris, etc., as long as the OLAP system supports Upsert operations.

[0043] This embodiment implements a data processing method that integrates stream and batch processing through a data lake, which can overcome the shortcomings of the Lambda architecture. On the premise of meeting multi-dimensional data query requests, it can not only save computing resources and storage resources, but also ensure the consistency of offline data requests and real-time data requests in response results, thereby facilitating the development and maintenance of data processing systems.

[0044] At the same time, with the help of the data lake's support for Upsert, this embodiment continuously uses more accurate real-time calculation results to correct the real-time calculation results that have been written previously, thereby taking into account both data accuracy and timeliness; with the help of the data lake's support for atomicity, this embodiment will not generate dirty data after a task write fails; with the help of the data lake's support for multiple versions, the data in this embodiment can be rolled back to a previous version at any time, thereby improving data security.

[0045] In addition, writing real-time calculation results into the OLAP system in batches can take into account the fault tolerance and reliability of the data and the load balancing between the nodes in the OLAP system.

[0046] Combine the following Figure 2 Some embodiments of performing real-time computation on streaming data through a structured streaming processing engine are described.

[0047] Figure 2 FIG. 1 shows a flow chart of real-time computing of stream data through a structured stream processing engine. Figure 2As shown, this embodiment includes steps S2021 to S2023.

[0048] In step S2021, the Spark structured stream processing engine is used to read stream data from the message middleware and group the stream data according to the session identifier Session Id to obtain the stream data of each grouped session.

[0049] In step S2022, the state operator flatMapGroupsWithState of the Spark structured stream processing engine is used to cache the stream data of each group session and execute business calculation logic on the stream data of each group session to obtain the statistical fields of each group session.

[0050] Since Spark Structured Streaming does not have a built-in concept of Session Window, you can use flatMapGroupsWithState to simulate Session Window to cache all messages in the session and further execute business calculation logic. The following example uses the statistical field of any grouped session to illustrate how to execute business calculation logic.

[0051] First, each stream data in any packet session is sorted, for example, the stream data can be sorted and traversed according to the message sequence number and request time.

[0052] Then, the business indicator parameters of each flow data in any group session are obtained, wherein the business indicator parameters may specifically include the geographical information of the client, network operator information, page dwell time, access depth, the first page visited in the session, the last page visited in the session, the client's operating system and version number, etc.

[0053] Finally, the statistical fields of any group session are determined based on these business indicator parameters. For example, the residence time of the current page in the group session can be determined by subtracting the request time of the current page from the request time of the next page in the group session.

[0054] Those skilled in the art will understand that Spark Structured Streaming supports exactly-once semantics and provides a checkpoint function. Combined with Kafka's ability to reset consumption sites, Delta Lake's atomicity, and OLAP's Upsert semantics, it meets the end-to-end exactly-once feature, ensuring that messages are not lost or duplicated.

[0055] Combine the following Figure 3 Some embodiments of the disclosed data processing system are described.

[0056] Figure 3 FIG. 1 is a schematic diagram showing the structure of a data processing system in some embodiments of the present disclosure. Figure 3 As shown, the data processing system 30 includes: an HTTP receiver 301, configured to write the streaming data received from the client into the message middleware; a message middleware 302, configured to store the streaming data; a Spark structured stream processing engine 303, configured to perform real-time calculations on the streaming data, obtain real-time calculation results, and configured to write the real-time calculation results into the data lake and the online analytical processing database respectively in an update or insert Upsert manner; a data lake 304, configured to receive and respond to offline data requests; and an online analytical processing database 305, configured to receive and respond to real-time data requests.

[0057] In some embodiments, the Spark structured stream processing engine 303 is configured to: read stream data from the message middleware and group the stream data according to the session identifier to obtain the stream data of each group session; through the state operator flatMapGroupsWithState, cache the stream data of each group session and execute business calculation logic on the stream data of each group session to obtain the statistical fields of each group session.

[0058] In some embodiments, the Spark structured stream processing engine 303 is configured to: sort each stream data in any group session; obtain business indicator parameters of each stream data in any group session; and determine the statistical fields of any group session based on the business indicator parameters.

[0059] In some embodiments, the Spark structured stream processing engine 303 is configured to: when the real-time calculation results change, write the real-time calculation results into the data lake and the online analytical processing database respectively by updating or inserting Upsert.

[0060] In some embodiments, the interval duration for real-time calculation of streaming data by the structured stream processing engine 303 is greater than a preset duration.

[0061] In some embodiments, a session completed partition and a session uncompleted partition are set on the data lake; the Spark structured stream processing engine 303 is configured to write the real-time calculation results into the session uncompleted partition of the data lake in an Upsert manner.

[0062] In some embodiments, the message middleware 303 is Kafka message middleware; the HTTP receiver 301 is configured to: receive stream data from the client; and write the stream data into the Kafka message middleware.

[0063] This embodiment implements a stream-batch integrated data processing system through a data lake, which can overcome the shortcomings of the Lambda architecture. On the premise of meeting multi-dimensional data query requests, it can not only save computing resources and storage resources, but also ensure the consistency of offline data requests and real-time data requests in response results, thereby facilitating the development and maintenance of data processing systems.

[0064] At the same time, with the help of the data lake's support for Upsert, this embodiment continuously uses more accurate real-time calculation results to correct the real-time calculation results that have been written previously, thereby taking into account both data accuracy and timeliness; with the help of the data lake's support for atomicity, this embodiment will not generate dirty data after a task write fails; with the help of the data lake's support for multiple versions, the data in this embodiment can be rolled back to a previous version at any time, thereby improving data security.

[0065] In addition, this embodiment writes the real-time calculation results into the OLAP system in batches, which can take into account the fault tolerance and reliability of the data and the load balancing among the nodes in the OLAP system.

[0066] In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0067] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0068] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0069] Combine the following Figure 4 Other embodiments of the disclosed data processing system are described.

[0070] Figure 4FIG. 2 shows a schematic diagram of a data processing system in accordance with another embodiment of the present disclosure. Figure 4 As shown, the data processing system 40 includes: a memory 410 and a processor 420 coupled to the memory 410 , and the processor 420 is configured to execute the data processing method in any of the aforementioned embodiments based on instructions stored in the memory 410 .

[0071] The memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0072] The data processing system 40 may also include an input / output interface 430, a network interface 440, a storage interface 450, etc. These interfaces 430, 440, 450, the memory 410, and the processor 420 may be connected, for example, via a bus 460. The input / output interface 430 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 440 provides a connection interface for various networked devices. The storage interface 450 provides a connection interface for external storage devices such as an SD card and a USB flash drive.

[0073] The present disclosure also includes a non-transitory computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the data processing method in any of the aforementioned embodiments.

[0074] If the aforementioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0075] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0078] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0079] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0080] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Write the stream data received from the client to the message middleware; The structured stream processing engine is used to perform real-time calculations on the stream data to obtain real-time calculation results, including: reading the stream data from the message middleware through the Spark structured stream processing engine and grouping the stream data according to the session identifier to obtain the stream data of each group session; using the state operator flatMapGroupsWithState of the Spark structured stream processing engine, caching the stream data of each group session and executing business calculation logic on the stream data of each group session to obtain the statistical fields of each group session; Write the real-time calculation results to the data lake and online analytical processing database respectively by updating or inserting Upsert; Receive and respond to offline data requests through the data lake, and receive and respond to real-time data requests through the online analytical processing database.

2. The data processing method according to claim 1, wherein: The step of executing the service calculation logic on the flow data of each packet session to obtain the statistical fields of each packet session includes: Sort the data of each flow in any packet session; Obtaining service indicator parameters of each flow data in any packet session; According to the service indicator parameter, a statistical field of any packet session is determined.

3. The data processing method according to claim 1, wherein: Writing the real-time calculation results into the data lake and the online analytical processing database respectively by updating or inserting Upsert includes: When the real-time calculation results change, they are written to the data lake and the online analytical processing database respectively by updating or inserting Upsert.

4. The data processing method according to claim 1, wherein: The interval duration for real-time calculation of streaming data by the structured stream processing engine is longer than the preset duration.

5. The data processing method according to claim 1, further comprising: Set up session completed partitions and session incomplete partitions on the data lake; Among them, the session that writes the real-time calculation results to the data lake in Upsert mode is not completed.

6. The data processing method according to claim 1, wherein: Writing the stream data received from the client into the message middleware comprises: Receive streaming data from the client via HTTP receiver; The stream data is written to the Kafka message middleware through the HTTP receiver.

7. A data processing system comprising: An HTTP receiver configured to write stream data received from a client into a message middleware; Message middleware, configured to store streaming data; The Spark structured stream processing engine is configured to perform real-time calculations on stream data to obtain real-time calculation results, including: reading stream data from the message middleware and grouping the stream data according to the session identifier to obtain the stream data of each group session; caching the stream data of each group session and executing business calculation logic on the stream data of each group session through the state operator flatMapGroupsWithState to obtain the statistical fields of each group session; the Spark structured stream processing engine is also configured to write the real-time calculation results to the data lake and the online analytical processing database respectively in the form of update or insert Upsert; Data lake, configured to receive and respond to offline data requests; An OLAP database is configured to receive and respond to real-time data requests.

8. The data processing system according to claim 7, wherein: The Spark structured streaming processing engine is configured as follows: Sort the data of each flow in any packet session; Obtaining service indicator parameters of each flow data in any packet session; According to the service indicator parameter, a statistical field of any packet session is determined.

9. The data processing system according to claim 7, wherein: The Spark structured streaming processing engine is configured as follows: When the real-time calculation results change, they are written to the data lake and the online analytical processing database respectively by updating or inserting Upsert.

10. The data processing system according to claim 7, wherein: The interval duration for real-time calculation of streaming data by the structured stream processing engine is longer than the preset duration.

11. The data processing system according to claim 7, wherein: The data lake is provided with a session completed partition and a session incomplete partition; The Spark structured stream processing engine is configured to write the real-time calculation results into the unfinished partitions of the session of the data lake in an Upsert manner.

12. The data processing system according to claim 7, wherein: The message middleware is Kafka message middleware; The HTTP receiver is configured to: receive stream data from a client; and write the stream data into a Kafka message middleware.

13. A data processing system comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the data processing method according to any one of claims 1 to 6 based on instructions stored in the memory.

14. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

Images