Multi-source data stream processing method and device, electronic equipment and medium

Abstract

This application relates to the field of data processing technology, and discloses a method, apparatus, electronic device, and medium for processing multi-source data streams. The method includes: determining a delay correction value based on the historical delay data of multiple data sources; monitoring the consumption progress of data from each data source and obtaining the minimum event time among the event times of currently consumed data from each data source; the event time refers to the timestamp of data generation in the corresponding data source; determining the processing trigger time point based on the minimum event time and the delay correction value; and analyzing and processing data from each data source whose event time is less than or equal to the processing trigger time point. This application can improve the accuracy and timeliness of multi-source data stream processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to multi-source data stream processing methods, apparatus, electronic devices, and media. Background Technology

[0002] With the advent of the big data era, enterprises have placed higher demands on the timeliness and accuracy of data processing, and an increasing number of application scenarios require the analysis and processing of data from multiple sources. However, due to factors such as unstable network transmission, asynchronous device clocks, and inconsistent data acquisition frequencies, the problem of untimely data processing is prevalent in multi-source data streams. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and medium for processing multi-source data streams to solve the problem of untimely data processing that is common in multi-source data streams.

[0004] In a first aspect, this application provides a method for processing multi-source data streams, the method comprising: Determine the delay correction value based on the historical delay data from multiple data sources; Monitor the consumption progress of data from each data source and obtain the minimum event time among the event times of the currently consumed data from each data source; the event time refers to the timestamp of data generation in the corresponding data source. The processing trigger time point is determined based on the minimum event time and the delay correction value; Data from various data sources whose event times are less than or equal to the processing trigger time are analyzed and processed.

[0005] The method provided in this application determines a latency correction value based on historical latency data from multiple data sources, monitors the consumption progress of each data source in real time to obtain the minimum event time, and then determines the processing trigger time point based on the minimum event time and the latency correction value. This enables unified analysis and processing of data whose event time is less than or equal to the processing trigger time point. Starting from the temporal characteristics and latency patterns of multi-source data streams, this solution dynamically adapts to out-of-order and latency issues caused by network fluctuations, asynchronous device clocks, and inconsistent acquisition frequencies. It avoids data loss due to premature triggering and untimely processing due to late triggering, significantly improving the accuracy, timeliness, and stability of multi-source data stream processing and ensuring the reliability of calculation results.

[0006] In one possible implementation, determining the processing trigger time point based on the minimum event time and the delay correction value includes: Calculate the difference between the minimum event time and the delay correction value, and use the difference as the processing trigger time point.

[0007] The method provided in this application calculates the difference between the minimum event time and the delay correction value, and uses this difference as the processing trigger time. This allows the trigger time to accurately reflect the actual progress and typical latency level of the slowest data source in a multi-source data stream. It enables processing at the minimum effective trigger time while ensuring coverage of the vast majority of delayed data. This avoids missing data due to triggering too early, and also avoids overall processing delays due to triggering too late, ensuring that the processing timing is highly consistent with the actual state of the data stream and effectively improving the timeliness of analysis and processing.

[0008] In one possible implementation, after obtaining the minimum event time among the event times of the currently consumed data from each data source, the following is also included: Determine whether there are any consumption anomalies based on the event time of the currently consumed data from each data source; In the event of abnormal consumption, the step of determining the processing trigger time point based on the minimum event time and the delay correction value will not be executed; or... In the absence of any consumption anomalies, the step of determining the processing trigger time point based on the minimum event time and the delay correction value is executed.

[0009] The method provided in this application adds a judgment on whether there is an anomaly in consumption after obtaining the minimum event time, and selectively executes the processing trigger time point determination step based on the judgment result. It can proactively pause triggering when there is a significant anomaly in data source consumption, avoiding the continuation of invalid processing in abnormal scenarios such as data loss, progress imbalance, and transmission interruption. This improves the robustness and fault tolerance of the overall process, prevents calculation distortion and result errors, and ensures that the analysis and processing stage is only entered when the consumption status of each data source is normal and the timing is reliable. This makes the analysis and processing process more stable and the results more reliable, effectively improving the reliability of multi-source data stream analysis and processing in complex network environments.

[0010] In one possible implementation, determining whether there are consumption anomalies based on the event time of the currently consumed data from each data source includes: Calculate the time difference between any two event times for the currently consumed data from each data source; If all time differences are less than the first preset threshold, then it is determined that there is no abnormal consumption situation; or, If all time differences are greater than or equal to the first preset threshold, then it is determined that there is no abnormal consumption situation; or, If some time differences among all time differences are greater than or equal to a first preset threshold, then an abnormal consumption situation is determined to exist; "some time differences" means that there are at least one, but not all, time differences.

[0011] The method provided in this application calculates the pairwise time differences between the event times of currently consumed data from each data source and uses a first preset threshold to determine whether some time differences exceed the limit. This method can accurately identify anomalies such as progress imbalances between multi-source data streams and blockages in individual data sources. Based on multi-source collaborative comparison logic, this application determines the existence of consumption anomalies only when at least one, but not all, time differences are greater than or equal to the first preset threshold, thus improving the accuracy of consumption anomaly identification.

[0012] In one possible implementation, the method further includes: The steps are to periodically obtain the minimum event time among the event times of the currently consumed data from each data source at preset time intervals; and to determine the processing trigger time point based on the minimum event time and the delay correction value.

[0013] The method provided in this application periodically acquires the minimum event time and updates the processing trigger time point at preset time intervals, enabling multi-source data stream processing to proceed continuously, automatically, and stably without manual intervention or external triggering. The periodic mechanism ensures real-time synchronization between the processing flow and the data stream, continuously capturing the latest consumption progress and latency changes, dynamically updating the processing trigger time point, and gradually advancing the process. This avoids data accumulation and processing gaps, improving the continuity and timeliness of analysis and processing.

[0014] In one possible implementation, determining the delay correction value based on the historical delay data of multiple data sources includes: Obtain the event time of data corresponding to multiple data sources received by the processing terminal within a historical time range, as well as the reception time of each data arriving at the processing terminal; Calculate the difference between the event time and the reception time for each data point, and use the difference as the delay duration for the corresponding data. A typical delay duration is determined from all delay durations, and the typical delay duration is used as the delay correction value; the typical delay duration is the delay duration that occurs most frequently among all delay durations.

[0015] The method provided in this application collects the event times and arrival times at the processing end from various data sources within a historical time range, calculates the delay duration, and uses the most frequently occurring typical delay duration as the delay correction value. This ensures that the delay correction value accurately reflects the historical delay patterns of multi-source data streams. The delay correction value is derived from historical data statistics, possessing high representativeness and applicability. It can adapt to the normal delay levels of the vast majority of data, improving the rationality and accuracy of processing trigger times, and effectively mitigating the inaccuracy in calculations caused by out-of-order data and delayed arrival.

[0016] In one possible implementation, the analysis and processing of data from various data sources whose event times are less than or equal to the processing trigger time includes: Get the preset time window; The processing trigger time is taken as the time corresponding to the end boundary of the preset time window, and the data corresponding to each data source covered by the preset time window is analyzed and processed.

[0017] The method provided in this application obtains a preset time window and uses the processing trigger time point as the end boundary of the window. It then performs unified analysis and processing of multi-source data within the preset time window's coverage area, ensuring a clear data processing scope, well-defined boundaries, and controllable timing. This method can segment continuous streaming data into standardized time segments, enabling batch-based, ordered, and repeatable aggregation calculations, guaranteeing data integrity and timing alignment within the same preset time window. Simultaneously, using the trigger time point as the boundary ensures a reasonable window closure timing, preventing data loss and delays, significantly improving the standardization and accuracy of real-time statistics and analysis of multi-source data streams.

[0018] Secondly, this application provides a multi-source data stream processing apparatus, the apparatus comprising: The first processing module is used to determine the delay correction value based on the historical delay data of data from multiple data sources. The second processing module is used to monitor the consumption progress of data corresponding to each data source and obtain the minimum event time among the event times of the currently consumed data from each data source; the event time refers to the timestamp of data generation in the corresponding data source. The third processing module is used to determine the processing trigger time point based on the minimum event time and the delay correction value; The fourth processing module is used to analyze and process data from various data sources whose event time is less than or equal to the processing trigger time.

[0019] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the multi-source data stream processing method described in the first aspect or any corresponding embodiment.

[0020] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the multi-source data stream processing method described in the first aspect or any corresponding embodiment. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 2 This is a flowchart of a multi-source data stream processing method according to an embodiment of this application; Figure 3 This is a structural block diagram of a multi-source data stream processing apparatus according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] As one optional application scenario in the embodiments of this application, such as Figure 1 As shown, the multi-source data stream processing system may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0026] Specifically, the terminal device can be a mobile phone. Server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranets, local area networks, wide area networks, mobile communication networks, and combinations thereof.

[0027] According to an embodiment of this application, a multi-source data stream processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] This embodiment provides a multi-source data stream processing method. Figure 2 This is a flowchart of a multi-source data stream processing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: S201: Determine the delay correction value based on the historical delay data of the corresponding data from multiple data sources.

[0029] In this embodiment of the application, the data source can be a data stream generated by different application modules within the generating end. For example, the application can be a travel service application, a real-time monitoring application, or a data collection application, and the data source can be order data, location data, status data, behavior data, etc.

[0030] The data generation end can be user terminal devices such as mobile phones, computers, and vehicle terminals, while the data processing end is the backend server. The generation end and the processing end communicate and connect through a network.

[0031] During the transmission of data from the generation end to the processing end over the network, factors such as network fluctuations, transmission congestion, asynchronous device clocks, and inconsistent data acquisition frequencies can cause data delays. This can prevent the processing end from obtaining complete data in a timely manner. Especially in multi-source parallel data stream scenarios, the latency levels of different data sources are inconsistent, which can easily lead to data out-of-order processing, delayed partial data, inaccurate calculation results, or untimely processing. To address the aforementioned problems of delay, out-of-order processing, and unreasonable processing timing in multi-source data streams, this application provides a multi-source data stream processing method.

[0032] In this embodiment, considering that data is generated at the generator end with an event time, and there is a delay in its arrival at the processing end after network transmission, there is a deviation between the event time and the actual reception time of the data at the processing end. If processing is triggered directly according to the minimum event time, data loss is very likely, and the data intended for analysis and processing may not have arrived at the processing end. Therefore, this application needs to determine a delay correction value. By analyzing historical delay data, a delay correction value that conforms to the actual transmission pattern can be obtained, thus avoiding data loss.

[0033] In this embodiment, multiple data sources are data streams generated and transmitted in parallel. For example, these could be order data sources, location data sources, vehicle departure / receiver status data sources, and equipment operation data sources. By performing aggregation analysis, time-series alignment, and unified calculation based on the data from these multiple data sources, real-time and accurate business statistics or status judgment results can be obtained.

[0034] This embodiment determines the delay correction value based on historical delay data, so that the delay correction value can fit the actual transmission characteristics of multi-source data streams, thereby improving the rationality and reliability of the analyzed and processed data from the source.

[0035] S202: Monitor the consumption progress of data corresponding to each data source and obtain the minimum event time among the event times of the currently consumed data from each data source.

[0036] In this embodiment, the event time refers to the timestamp of data generation in the corresponding data source. For example, the generation time of order data when a user places an order, and the time when location data is collected, can be the system clock time when the processing end receives the data, which is the receiving time on the processing end side.

[0037] In this embodiment of the application, as an example, a corresponding message queue can be set up for each data source, and the data sent by the generating end can be stored in the message queue for processing. Consuming data refers to reading the received data from the message queue and loading the read data into the processing end's memory or cache as data to be analyzed and processed. Currently consumed data refers to the latest data that has been successfully read and loaded by the processing end at the most recent time.

[0038] In this embodiment, monitoring the consumption progress of data corresponding to each data source can be achieved by listening to the consumption position of the message queue, reading consumption records, or periodically querying the consumption status. Specifically, this includes: obtaining the current consumption position of the message queue corresponding to each data source in real time, recording the latest event time of the consumed data, thereby determining the real-time consumption progress.

[0039] In this embodiment, the minimum event time refers to the smallest and earliest event time among the event times of the currently consumed data from all data sources, representing the event time of the data consumed by the data source with the slowest consumption progress. This application considers that multi-source data streams need to maintain temporal consistency; only by using the slowest consumption progress as a benchmark can it be guaranteed that data from all data sources can be obtained within the same time (segment). Therefore, it is necessary to obtain the minimum event time from the event times of the currently consumed data.

[0040] This embodiment monitors the consumption progress of each data source and determines the minimum event time. It uses the slowest consuming data source as the overall benchmark to ensure the timing consistency of multi-source data processing and avoid errors in results due to lag in the consumption of some data sources.

[0041] In this application, the specific data, the corresponding event time, and the reception time are all encapsulated into a data packet of a uniform format for transmission and storage. The data packet may also include a data source identifier to distinguish data from different sources. For example, if the data source is an order data source, the data is order creation information, and the event time is the timestamp of the order creation, this data is encapsulated into a data packet and sent to the processing end. When the processing end receives the data packet, it adds the current reception time and stores the data packet containing the data source identifier, event time, data content, and reception time into the message queue of the corresponding data source, awaiting consumption by the processing end.

[0042] S203: Determine the processing trigger time based on the minimum event time and the delay correction value.

[0043] In this embodiment, S203, determining the processing trigger time point based on the minimum event time and the delay correction value, specifically includes: calculating the difference between the minimum event time and the delay correction value, and using the difference as the processing trigger time point. The processing trigger time point is also a timestamp.

[0044] This embodiment calculates the difference between the minimum event time and the delay correction value, and uses this difference as the processing trigger time. This ensures that the trigger time accurately reflects the actual progress and typical latency level of the slowest data source in a multi-source data stream. It allows processing to be performed at the minimum effective trigger time while ensuring coverage of the vast majority of delayed data. This avoids missing data due to triggering too early, and also avoids overall processing delays due to triggering too late, ensuring that the processing timing is highly consistent with the actual state of the data stream and effectively improving the timeliness of analysis and processing.

[0045] S204: Analyze and process data from various data sources whose event times are less than or equal to the processing trigger time.

[0046] In this embodiment, a preset time length can be determined based on the business characteristics and statistical requirements of the data source. The difference between the processing trigger time and the preset time length is calculated to obtain the start time of the time window. Data (including the data corresponding to the start time and the processing trigger time) within the range of the event time from the start time to the processing trigger time can be obtained for each data source. The data within the above time range is then analyzed and processed. The specific analysis and processing are determined based on the business type and calculation objectives of the data source. For example, when the data source is an order data source, a location data source, or a vehicle arrival / departure status data source, the analysis and processing of the above multi-source data includes time sequence alignment, disorder correction, aggregation statistics, duration calculation, and status judgment. The analysis and processing results can include driver online time, order validity time, equipment operating status, and real-time business statistical indicators.

[0047] This embodiment uses the processing trigger time point as the cutoff boundary to uniformly analyze and process data whose event time is less than or equal to that time point. This ensures that the data involved in the calculation is complete, sequential, and without omissions, effectively solving the problem of inaccurate calculations caused by out-of-order and delayed arrival of multi-source data streams, and improving the accuracy and reliability of the analysis and processing results.

[0048] The method provided in this application determines a latency correction value based on historical latency data from multiple data sources, monitors the consumption progress of each data source in real time to obtain the minimum event time, and then determines the processing trigger time point based on the minimum event time and the latency correction value. This enables unified analysis and processing of data whose event time is less than or equal to the processing trigger time point. Starting from the temporal characteristics and latency patterns of multi-source data streams, this solution dynamically adapts to out-of-order and latency issues caused by network fluctuations, asynchronous device clocks, and inconsistent acquisition frequencies. It avoids data loss due to premature triggering and untimely processing due to late triggering, significantly improving the accuracy, timeliness, and stability of multi-source data stream processing and ensuring the reliability of calculation results.

[0049] In one possible implementation, S201 determines the delay correction value based on the historical delay data of data from multiple data sources, including: Obtain the event time of data from multiple data sources received by the processing end within a historical time range, as well as the reception time of each data point at the processing end.

[0050] Calculate the difference between the event time and the reception time for each data point, and use the difference as the delay duration for the corresponding data.

[0051] The typical delay duration is determined from all delay durations and used as the delay correction value.

[0052] In this application embodiment, as an example, the typical delay duration can be the delay duration that occurs most frequently among all delay durations. As another example, the typical delay duration can also be the average or median of all delay durations, or the average of the top N most frequent delay durations, etc., to adapt to the balance requirements of out-of-order tolerance and real-time processing in different business scenarios.

[0053] In this application embodiment, the historical time range can be a relatively long statistical period such as the most recent 30 days, 60 days, or 90 days. It can obtain the event time of data corresponding to multiple data sources within the historical time range, as well as the reception time of each data arriving at the processing end.

[0054] The method provided in this application collects the event times and arrival times at the processing end from various data sources within a historical time range, calculates the delay duration, and uses the most frequently occurring typical delay duration as the delay correction value. This ensures that the delay correction value accurately reflects the historical delay patterns of multi-source data streams. The delay correction value is derived from historical data statistics, possessing high representativeness and applicability. It can adapt to the normal delay levels of the vast majority of data, improving the rationality and accuracy of processing trigger times, and effectively mitigating the inaccuracy in calculations caused by out-of-order data and delayed arrival.

[0055] In one possible implementation, after obtaining the minimum event time among the event times of the currently consumed data from each data source, the multi-source data stream processing method may also include Sa1, Sa2, or Sa1, Sa3.

[0056] Sa1: Determine whether there are any consumption anomalies based on the event time of the currently consumed data from each data source.

[0057] In this embodiment of the application, abnormal consumption refers to a situation where the consumption progress of multiple data sources is severely unbalanced or the consumption of individual data sources is too lagging, which can lead to distortion of timing judgment.

[0058] In this application embodiment, as an example, determining whether there is a consumption anomaly based on the event time of the currently consumed data from each data source may include: Calculate the time difference between every two event times for the currently consumed data from each data source.

[0059] If all time differences are less than the first preset threshold, it is determined that there are no abnormal consumption situations. All time differences being less than the first preset threshold indicates that the overall consumption progress of each data source is balanced and consistent. At this point, the multi-source data time sequence is reliable, and subsequent processing can be triggered normally.

[0060] If all time differences are greater than or equal to the first preset threshold, it is determined that there is no consumption anomaly. All time differences being greater than or equal to the first preset threshold indicates that the overall synchronization of each data source is lagging, but there is no progress imbalance between data sources. Although overall consumption is slow, it still maintains temporal consistency and will not cause local data distortion.

[0061] If some time differences among all time differences are greater than or equal to a first preset threshold, then an abnormal consumption situation is determined to exist. "Some time differences" means that at least one, but not all, time differences exist. The existence of some time differences greater than or equal to the first preset threshold indicates that some data sources are consuming normally, while others are severely lagging, resulting in an imbalance in the progress of multiple sources. This can lead to timing chaos and unreliable calculation results, thus requiring a determination of an abnormal consumption situation.

[0062] In this embodiment of the application, the first preset threshold can be determined based on the service latency tolerance, the normal latency level of the data source and the network fluctuation. For example, the first preset threshold can be 30 seconds, 1 minute, 3 minutes, 5 minutes, etc.

[0063] In this application embodiment, as another example, determining whether there is a consumption anomaly based on the event time of the currently consumed data from each data source may further include: Calculate the time difference between the event time of the currently consumed data from each data source and the current time at the processing end. If all time differences are less than a second preset threshold, it is determined that there is no consumption anomaly. If all time differences are greater than or equal to the second preset threshold, it is determined that there is no consumption anomaly. If some time differences are greater than or equal to the first preset threshold, it is determined that there is a consumption anomaly. "Some time differences" means that at least one, but not all, of the time differences exist.

[0064] This embodiment can quickly identify message anomalies caused by severe delays in the consumption of some data sources by directly comparing the event time with the current time of the processing end, thereby improving the efficiency of anomaly detection.

[0065] This embodiment calculates the pairwise time differences between the event times of currently consumed data from each data source and uses a first preset threshold to determine whether any time differences exceed the limit. This allows for accurate identification of anomalies such as progress imbalances, excessive local delays, and blockages in individual data sources among multi-source data streams. Based on multi-source collaborative comparison logic, this embodiment determines consumption anomalies only when at least one, but not all, time differences are greater than or equal to the first preset threshold, thus improving the accuracy of consumption anomaly identification.

[0066] Sa2: In the event of an abnormal consumption situation, determine not to execute the step of determining the processing trigger time point based on the minimum event time and the delay correction value.

[0067] In this embodiment of the application, if it is determined that the step of determining the processing trigger time point based on the minimum event time and the delay correction value will not be executed, the event time of the currently consumed data of each data source will still be continuously obtained, and the step of Sa1 determining whether there is a consumption anomaly based on the event time of the currently consumed data of each data source will be executed until it is determined that there is no consumption anomaly.

[0068] Alternatively, Sa3: In the absence of consumption anomalies, determine the steps to execute based on the minimum event time and delay correction value to determine the processing trigger time.

[0069] The method provided in this application adds a judgment on whether there is an anomaly in consumption after obtaining the minimum event time, and selectively executes the processing trigger time point determination step based on the judgment result. It can proactively pause triggering when there is a significant anomaly in data source consumption, avoiding the continuation of invalid processing in abnormal scenarios such as data loss, progress imbalance, and transmission interruption. This improves the robustness and fault tolerance of the overall process, prevents calculation distortion and result errors, and ensures that the analysis and processing stage is only entered when the consumption status of each data source is normal and the timing is reliable. This makes the analysis and processing process more stable and the results more reliable, effectively improving the reliability of multi-source data stream analysis and processing in complex network environments.

[0070] In one possible implementation, S204 analyzes and processes data from various data sources whose event times are less than or equal to the processing trigger time, including: Get the preset time window.

[0071] The processing trigger time is taken as the end boundary of the preset time window, and the data corresponding to each data source covered by the preset time window is analyzed and processed.

[0072] In this embodiment, a preset time length can be determined based on the business characteristics and statistical requirements of the data source. The difference between the processing trigger time and the preset time length is calculated to obtain the start time of the preset time window. The preset time window is a time interval with a fixed duration, determined by a start boundary and an end boundary. The start time is the time corresponding to the start boundary of the preset time window, and the processing trigger time is the time corresponding to the end boundary of the preset time window.

[0073] The method provided in this application obtains a preset time window and uses the processing trigger time point as the end boundary of the window. It then performs unified analysis and processing of multi-source data within the preset time window's coverage area, ensuring a clear data processing scope, well-defined boundaries, and controllable timing. This method can segment continuous streaming data into standardized time segments, enabling batch-based, ordered, and repeatable aggregation calculations, guaranteeing data integrity and timing alignment within the same preset time window. Simultaneously, using the trigger time point as the boundary ensures a reasonable window closure timing, preventing data loss and delays, significantly improving the standardization and accuracy of real-time statistics and analysis of multi-source data streams.

[0074] In one possible implementation, the multi-source data stream processing method further includes: The process periodically retrieves the minimum event time from the currently consumed data of each data source at preset time intervals. Based on the minimum event time and the delay correction value, it determines the processing trigger point.

[0075] In this embodiment, the preset time interval can be set and dynamically adjusted according to actual business needs, data generation frequency, and real-time processing requirements. The preset time interval is usually set to a short duration in the range of seconds or minutes, such as 1 second, 5 seconds, 10 seconds, 30 seconds, or 1 minute, to ensure that while taking into account the processing pressure on the processing end, changes in the consumption progress of each data source can be detected in a timely manner, and the processing trigger time point can be quickly updated, thereby improving the timeliness and continuity of data analysis and processing.

[0076] In this application embodiment, in addition to the periodic execution method, a real-time execution method can also be adopted to continuously obtain the minimum event time among the event times of the currently consumed data from each data source, and determine the processing trigger time point in real time based on the minimum event time and the delay correction value, so as to further reduce the processing delay and meet the usage requirements of ultra-high real-time business scenarios.

[0077] The method provided in this application periodically acquires the minimum event time and updates the processing trigger time point at preset time intervals, enabling multi-source data stream processing to proceed continuously, automatically, and stably without manual intervention or external triggering. The periodic mechanism ensures real-time synchronization between the processing flow and the data stream, continuously capturing the latest consumption progress and latency changes, allowing the processing trigger time point to be dynamically updated and progressively advanced, avoiding data accumulation and processing gaps, and improving the continuity and timeliness of analysis and processing. This application can dynamically determine the processing trigger time point based on the real-time consumption progress and historical latency characteristics of multi-source data streams. Compared to fixed processing trigger time points, such as fixed system time triggers, fixed period triggers, or fixed duration delay triggers, the dynamic determination of the processing trigger time point can adaptively match the actual consumption status of each data source, adapting to dynamically changing network latency and data source fluctuations. This effectively avoids data omissions, calculation distortions, or processing delays caused by fixed trigger mechanisms, significantly improving the accuracy and timeliness of multi-source data stream processing.

[0078] Traditional processing methods typically employ fixed-time-window triggering for analysis. When some data within the window arrives late due to network fluctuations, it can lead to missing or incorrect calculation results. Furthermore, fixed-trigger mechanisms are prone to delays in the advancement of computational markers, significantly reducing the timeliness of data processing and failing to meet the demands for high-precision, low-latency real-time computation.

[0079] To address this issue, this application effectively resolves the problems of inaccurate calculations and untimely processing caused by out-of-order and delayed arrival of multi-source data streams by adaptively adjusting the processing trigger time point through real-time sensing of the consumption progress of multi-source data streams. This application pre-emptively investigates the historical latency of the data sources to determine a latency correction value that conforms to actual transmission patterns. It monitors the consumption progress of multi-source data in real time, using the minimum event time of consuming the slowest data source as the overall benchmark. The processing trigger time point is dynamically determined by subtracting the latency correction value from the minimum event time, minimizing processing latency while avoiding data loss.

[0080] In practical applications, this application can be adapted to various typical business scenarios. Under normal consumption scenarios, the processing trigger time point is determined directly based on the minimum event time and latency correction value of each data source. Under scenarios where there is a backlog of overall data source consumption, the processing trigger time point is still determined using the calculation method for the processing trigger time point under normal consumption scenarios. Under abnormal consumption scenarios where individual data sources have no messages for a long time, the calculation of the processing trigger time point is stopped. Through the above multi-scenario adaptive processing, this application can achieve dynamic out-of-order tolerance and precise latency control in a multi-source heterogeneous data stream environment, significantly improving the accuracy, stability, and timeliness of real-time calculations. It is especially suitable for business scenarios that require high-precision, low-latency time-series analysis, such as driver online time and order validity period.

[0081] This application also provides a multi-source data stream processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0082] This application provides a multi-source data stream processing device, such as... Figure 3 As shown, it includes: The first processing module 301 is used to determine the delay correction value based on the historical delay of data corresponding to multiple data sources.

[0083] The second processing module 302 is used to monitor the consumption progress of data from each data source and obtain the minimum event time among the event times of the currently consumed data from each data source. The event time refers to the timestamp of data generation in the corresponding data source.

[0084] The third processing module 303 is used to determine the processing trigger time point based on the minimum event time and the delay correction value.

[0085] The fourth processing module 304 is used to analyze and process data from various data sources whose event time is less than or equal to the processing trigger time.

[0086] In one possible implementation, the first processing module 301 is specifically used to acquire the event times of data corresponding to multiple data sources received by the processing end within a historical time range, as well as the reception time of each data arriving at the processing end. The difference between the event time and reception time of each data point is calculated, and this difference is used as the delay duration for the corresponding data. A typical delay duration is determined from all delay durations and used as a delay correction value. The typical delay duration is the delay duration that occurs most frequently among all delay durations.

[0087] In one possible implementation, the third processing module 303 is specifically used to calculate the difference between the minimum event time and the delay correction value, and use the difference as the processing trigger time point.

[0088] In one possible implementation, the fourth processing module 304 is specifically used to obtain a preset time window. The processing trigger time point is taken as the end boundary of the preset time window, and the data corresponding to each data source covered by the preset time window is analyzed and processed.

[0089] In one possible implementation, the multi-source data stream processing device further includes: a fifth processing module, comprising: The first processing unit is used to determine whether there is a consumption anomaly based on the minimum event time among the event times of the currently consumed data from each data source after obtaining the minimum event time among the event times of the currently consumed data from each data source.

[0090] The second processing unit is used to determine, when there is a consumption anomaly, not to execute the step of determining the processing trigger time point based on the minimum event time and the delay correction value.

[0091] The third processing unit is used to determine the step of determining the processing trigger time point based on the minimum event time and the delay correction value when there is no consumption anomaly.

[0092] In one possible implementation, the first processing unit is specifically used to calculate the time difference between every two event times of the currently consumed data from each data source.

[0093] If all time differences are less than the first preset threshold, then it is determined that there is no abnormal consumption situation.

[0094] If all time differences are greater than or equal to the first preset threshold, then it is determined that there is no abnormal consumption situation.

[0095] If some time differences among all time differences are greater than or equal to a first preset threshold, then an abnormal consumption situation is determined to exist. "Some time differences" means that at least one, but not all, of the time differences exist.

[0096] In one possible implementation, the multi-source data stream processing device further includes: a sixth processing module, configured to periodically execute the step of acquiring the minimum event time among the event times of currently consumed data from each data source at preset time intervals, and determining the processing trigger time point based on the minimum event time and a delay correction value.

[0097] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. See below for details. Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from memory 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of the electronic device. The processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0098] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0099] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from memory 408, or installed from ROM 402. When the computer program is executed by processor 401, it performs the functions defined in the multi-source data stream processing method of embodiments of this application.

[0100] Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0101] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc. Further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessors, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the multi-source data stream processing method shown in the above embodiments is implemented.

[0102] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0103] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for processing multiple data streams, the method comprising: receiving a plurality of data streams; and processing the plurality of data streams in parallel. The method includes: Determine the delay correction value based on the historical delay data from multiple data sources; Monitor the consumption progress of data from each data source and obtain the minimum event time among the event times of the currently consumed data from each data source; the event time refers to the timestamp of data generation in the corresponding data source. The processing trigger time point is determined based on the minimum event time and the delay correction value; Data from various data sources whose event times are less than or equal to the processing trigger time are analyzed and processed.

2. The method of claim 1, wherein, The step of determining the processing trigger time point based on the minimum event time and the delay correction value includes: Calculate the difference between the minimum event time and the delay correction value, and use the difference as the processing trigger time point.

3. The method according to claim 1, characterized in that, After obtaining the minimum event time among the event times of currently consumed data from each data source, the following is also included: Determine whether there are any consumption anomalies based on the event time of the currently consumed data from each data source; In the event of abnormal consumption, the step of determining the processing trigger time point based on the minimum event time and the delay correction value will not be executed; or... In the absence of any consumption anomalies, the step of determining the processing trigger time point based on the minimum event time and the delay correction value is executed.

4. The method according to claim 3, characterized in that, The process of determining whether there are consumption anomalies based on the event time of currently consumed data from each data source includes: Calculate the time difference between any two event times for the currently consumed data from each data source; If all time differences are less than the first preset threshold, then it is determined that there is no abnormal consumption situation; or, If all time differences are greater than or equal to the first preset threshold, then it is determined that there is no abnormal consumption situation; or, If some time differences among all time differences are greater than or equal to a first preset threshold, then an abnormal consumption situation is determined to exist; "some time differences" means that there are at least one, but not all, time differences.

5. The method according to claim 1, characterized in that, The method further includes: The steps are to periodically obtain the minimum event time among the event times of the currently consumed data from each data source at preset time intervals; and to determine the processing trigger time point based on the minimum event time and the delay correction value.

6. The method according to claim 1, characterized in that, The step of determining the delay correction value based on the historical delay data of multiple data sources includes: Obtain the event time of data corresponding to multiple data sources received by the processing terminal within a historical time range, as well as the reception time of each data arriving at the processing terminal; Calculate the difference between the event time and the reception time for each data point, and use the difference as the delay duration for the corresponding data. A typical delay duration is determined from all delay durations, and the typical delay duration is used as the delay correction value; the typical delay duration is the delay duration that occurs most frequently among all delay durations.

7. The method according to claim 1, characterized in that, The analysis and processing of data from various data sources whose event times are less than or equal to the processing trigger time includes: Get the preset time window; The processing trigger time is taken as the time corresponding to the end boundary of the preset time window, and the data corresponding to each data source covered by the preset time window is analyzed and processed.

8. A multi-source data stream processing device, characterized in that, The device includes: The first processing module is used to determine the delay correction value based on the historical delay data of data from multiple data sources. The second processing module is used to monitor the consumption progress of data corresponding to each data source and obtain the minimum event time among the event times of the currently consumed data from each data source; the event time refers to the timestamp of data generation in the corresponding data source. The third processing module is used to determine the processing trigger time point based on the minimum event time and the delay correction value; The fourth processing module is used to analyze and process data from various data sources whose event time is less than or equal to the processing trigger time.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the steps of the method according to any one of claims 1 to 7.

Images