An event-triggered stream data timing consistency guarantee method and system

Abstract

The application provides an event-triggered stream data timing consistency guarantee method and system, and belongs to the technical field of data processing. The method specifically comprises the following steps: determining an active dismounting method of a trigger event of a business type according to a sorting result of a sending time of message data of the trigger event of the associated business data; performing control processing of the trigger event of each business type by using the active dismounting method; and determining whether an event trigger mechanism of stream data processing of the business type needs to be updated according to the message data of the trigger event of each business type, so as to reduce the data amount in the message queue.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method and system for ensuring the temporal consistency of streaming data based on event triggering. Background Technology

[0002] In traditional streaming computing architectures, the asynchronous nature of data writing to the storage system and computing tasks can lead to computing tasks being executed before the data is fully visible, resulting in inconsistent results. For example, in real-time inventory statistics in e-commerce, if the write operation to deduct inventory has not yet been completed, but the computing task has already generated a report based on the old data, it will inevitably lead to the risk of overselling. Therefore, how to ensure the consistency of data processing in streaming computing architectures has become an urgent technical problem to be solved.

[0003] Meanwhile, when processing data of the same business type, it is often necessary to trigger multiple types of business data. For example, after processing a consumption, it may be necessary to write the consumption amount, merchant type, and overdue amount to multiple databases at the same time. However, the write latency of different databases is different. This means that if message delivery processing is performed after different types of business data are written, it will inevitably lead to an excessive amount of messages, resulting in excessively high data processing latency.

[0004] To address the aforementioned technical issues, this application provides a method and system for ensuring the temporal consistency of streaming data based on event triggering. Summary of the Invention

[0005] To achieve the objectives of this invention, the following technical solution is adopted:

[0006] Specifically, this application provides a method for ensuring the temporal consistency of streaming data based on event triggering, which includes:

[0007] S1 determines the associated business data based on the business type, determines the write latency type of the associated business data based on the queue data of the written associated business data, and proceeds to the next step when the event triggering mechanism of the business type needs to be optimized based on the deviation of the write latency type of the associated business data.

[0008] S2 determines the active delivery method for the trigger events of the business type based on the sorting result of the sending time of the message data of the trigger events of the associated business data, and uses the active delivery method to control the trigger events of each business type. Based on the message data of the trigger events of each business type, S2 determines whether the event triggering mechanism of the streaming data processing of the business type needs to be updated.

[0009] The beneficial effects of this invention are as follows:

[0010] Based on the sorting results of the message sending time of the trigger events of related business data, the active delivery method of the trigger events of business types is determined. This ensures that the active delivery of trigger events is only carried out after the related business data with a long delivery delay has been stored. This avoids the technical problem of excessive delay in obtaining and processing trigger events of different business types due to the large number of messages.

[0011] Based on the message data of the trigger events for each business type, it is determined whether the event triggering mechanism of the streaming data processing for each business type needs to be updated. This enables the update processing of the event triggering mechanism for streaming data processing for some business types when the amount of message data is small. This ensures the reliability of the update monitoring of the associated business data of the proactively delegated processing for some business types, while also ensuring the reliability of data processing for different business types.

[0012] Furthermore, the associated business data refers to the business data that the business type needs to combine when performing data processing. For example, when performing credit risk identification and processing, it is necessary to combine user consumption data, personal information, credit data, and account balance, etc. All of the above business data belong to the associated business data of the business type.

[0013] Furthermore, the queue data for writing the associated business data is determined based on the processing delay of the queue on the server port where the associated business data is written.

[0014] Furthermore, the method for determining the write latency type of the associated business data is as follows:

[0015] Based on the queue data of the related business data being written, determine the processing delay of the queue of the server port where the related business data is written;

[0016] Based on the processing latency, determine the average write latency of the associated business data in history;

[0017] The write latency type of the associated business data is determined using the average write latency.

[0018] Furthermore, the control and processing of trigger events for various business types using the aforementioned proactive decentralization method specifically includes:

[0019] In each business type, the proactive delivery method is used to control and process the trigger events of the business type. When a trigger event is obtained, the business type proactively obtains the update results of the trigger events of other related business data, and grants the permission to proactively deliver the trigger events of related business data that have not yet been stored. This process continues until all the trigger events of related business data are obtained, and then the data processing of the related business data of the business type is performed.

[0020] Furthermore, determining whether the event triggering mechanism for streaming data processing of the aforementioned business type needs to be updated specifically includes:

[0021] Based on the message data of trigger events for each business type, determine the message data in different time periods;

[0022] Based on the decentralization processing method for the aforementioned business type, the monitoring data for delayed decentralization data in the aforementioned business type is determined;

[0023] Based on the monitoring data of delayed data delivery in the business type and the message data in different time periods, determine whether the event triggering mechanism of streaming data processing for the business type needs to be updated.

[0024] In a second aspect, the present invention provides a computer system comprising: a memory and a processor connected in communication, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the above-described event-triggered streaming data timing consistency guarantee method when running the computer program.

[0025] Other features and advantages will be set forth in the following description, and the objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0027] The above and other features and advantages of the present invention will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart of an event-triggered streaming data timing consistency guarantee method;

[0029] Figure 2 This is a flowchart illustrating the method for determining the write delay type of associated business data;

[0030] Figure 3This is a flowchart illustrating the process that needs to be optimized to determine the event triggering mechanism for a given business type.

[0031] Figure 4 This is a flowchart illustrating the method for determining the proactive delivery method of a business-type trigger event;

[0032] Figure 5 It is a framework diagram of a computer system. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0034] In this application, when there is a delay in storing certain business data associated with a business type, the permission control process for actively releasing messages is performed after storage is completed, thereby reducing the number of messages that trigger events and improving the efficiency and reliability of data processing.

[0035] Example 1

[0036] like Figure 1 As shown, this application provides an event-triggered streaming data timing consistency guarantee method, specifically including:

[0037] S1 determines the associated business data based on the business type, determines the write latency type of the associated business data based on the queue data of the written associated business data, and proceeds to the next step when the event triggering mechanism of the business type needs to be optimized based on the deviation of the write latency type of the associated business data.

[0038] Furthermore, the associated business data refers to the business data that the business type needs to combine when performing data processing. For example, when performing credit risk identification and processing, it is necessary to combine user consumption data, personal information, credit data, and account balance, etc. All of the above business data belong to the associated business data of the business type.

[0039] Furthermore, the queue data for writing the associated business data is determined based on the processing delay of the queue on the server port where the associated business data is written.

[0040] Specifically, such as Figure 2 As shown, the method for determining the write latency type of the associated business data is as follows:

[0041] Based on the queue data of the related business data being written, determine the processing delay of the queue of the server port where the related business data is written;

[0042] Based on the processing latency, determine the average write latency of the associated business data in history;

[0043] The write latency type of the associated business data is determined using the average write latency.

[0044] It should be noted that the average write latency is the average write time of the associated business data to the server, that is, the average write time from the start of sorting in the write queue to successful write in history.

[0045] It is understood that the average write latency is used to determine the write latency type of the associated business data, specifically including:

[0046] When the average write latency is greater than a preset duration threshold, the write latency type of the associated business data is determined to be an abnormal latency type.

[0047] When the average write latency is not greater than a preset duration threshold, the write latency type of the associated business data is determined to be a normal latency type.

[0048] In one possible specific embodiment, if the average write latency is greater than 2 minutes, the write latency type of the associated business data is determined to be an abnormal latency type.

[0049] Optionally, the method for determining the write delay type of the associated business data is as follows:

[0050] Based on the queue data of the related business data being written, determine the processing delay of the queue of the server port where the related business data is written;

[0051] Based on the processing delay, determine the number of times the write delay of the associated business data in history is greater than a preset duration threshold, and use this as the number of delayed writes.

[0052] The write delay type of the associated business data is determined by using the number of delayed writes.

[0053] It is understood that when the number of delayed writes exceeds the preset write count threshold, the write delay type of the associated business data is determined to be an abnormal delay type; otherwise, it is a normal delay type.

[0054] Specifically, such as Figure 3As shown, the event triggering mechanism for determining the business type needs to be optimized, specifically including:

[0055] Based on the deviation of the write latency type of the associated business data, determine the associated business data with abnormal latency type in the same business event;

[0056] Based on the associated business data of abnormal delay types, determine the deviation between the average write duration of the associated business data of abnormal delay types;

[0057] Based on the deviation between the average write time of associated business data of abnormal latency type, determine whether the event triggering mechanism of the business type needs to be optimized.

[0058] It should be noted that the associated business data of the abnormal delay type in the business event refers to the associated business data of the abnormal delay type that needs to be written after the same event occurs.

[0059] It is understandable that, based on the deviation between the average write time of associated business data of abnormal latency types, it is determined whether the event triggering mechanism of the business type needs to be optimized, specifically including:

[0060] When the number of associated business data of the abnormal delay type in the same business event does not meet the requirements, since there are a large number of associated business data of the abnormal delay type, if the message of the triggering event is generated in real time after updating different associated business data, there may be duplicate data processing due to the inconsistent write latency of different associated business data. Therefore, it is determined that the event triggering mechanism of the business type needs to be optimized.

[0061] Additionally, it is understood that when the number of associated business data of abnormal delay type in the same business event meets the requirements, it is also necessary to determine the deviation between the average write time of associated business data of abnormal delay type. When the deviation between the average write time of associated business data of abnormal delay type is greater than the preset time threshold, it is determined that the event triggering mechanism of the business type needs to be optimized.

[0062] In one possible specific embodiment, when the number of associated business data of the abnormal delay type in the same business event is no more than 2, it is determined that the number of associated business data of the abnormal delay type in the same business event meets the requirements. When the deviation between the average writing time of associated business data of the abnormal delay type is greater than 1 minute, it is determined that the event triggering mechanism of the business type needs to be optimized.

[0063] S2 determines the active delivery method for the trigger events of the business type based on the sorting result of the sending time of the message data of the trigger events of the associated business data, and uses the active delivery method to control the trigger events of each business type. Based on the message data of the trigger events of each business type, S2 determines whether the event triggering mechanism of the streaming data processing of the business type needs to be updated.

[0064] Specifically, such as Figure 4 As shown, the method for determining the proactive delivery method of the triggering event for the aforementioned business type is as follows:

[0065] Based on the sorting results of the sending time of the message data of the triggering event of the associated business data in the same business event, the associated business data with the longest sending time of the triggering event message data in different processing processes in the same business event is determined and it is used as the delayed business data.

[0066] Based on the distribution of delayed service data in different processing processes, the associated service data is determined as the processing process for delayed service data;

[0067] Based on the processing data of each associated business data as delayed business data, determine the proactive delivery method of the triggering event for the business type.

[0068] It is understandable that when the associated business data meets the requirements as the processing data of delayed business data, since its processing delay is short, it is determined that after the associated business data is stored on the server, the message data of its triggering event will no longer be actively distributed.

[0069] Additionally, it can be understood that when the associated business data does not meet the requirements of the processing process data for delayed business data, the associated business data that does not meet the requirements of the processing process data for delayed business data will be treated as delayed delivery data. When there is only one delayed delivery data, the active delivery method for the triggering event of the business type is to actively deliver the message data of the triggering event of the delayed delivery data in the business event.

[0070] In one possible specific embodiment, if the number of processing steps of the associated business data as delayed business data accounts for more than 0.1% of the processing steps in a business event, then the associated business data is determined to be delayed delivery data.

[0071] Additionally, it can be understood that when there are multiple delayed data releases, the proportion of the number of delayed processing processes corresponding to the delayed data releases in the processing processes of the business event is taken as the release delay proportion. If the release delay proportion of the delayed data release with the largest release delay proportion is greater than the preset delay proportion threshold, then the active release method for the trigger event of the business type is determined to be that the message data of the trigger event of the delayed data release with the largest release delay proportion is actively released in the business event.

[0072] In a possible specific embodiment, if the proportion of the delayed data with the largest proportion of delayed data is not less than 0.7, then the method for determining the active delivery of the triggering event of the service type is to actively deliver the message data of the triggering event of the delayed data with the largest proportion of delayed data in the service event.

[0073] It should also be noted that if the percentage of delayed data with the largest percentage of delayed data is not greater than the preset delay percentage threshold, then the active delivery method for the triggering event of the business type is to actively deliver the message data of the triggering event of the preset number of delayed data in the business event.

[0074] It should be noted that the preset number of delayed data is based on the percentage of delayed data, and the preset number of delayed data with the largest percentage of delayed data is selected as the delayed data for active delivery of the message data that triggers the event in the business event. In one possible embodiment, the preset number is 2.

[0075] Furthermore, the control and processing of trigger events for various business types using the aforementioned proactive decentralization method specifically includes:

[0076] In each business type, the proactive delivery method is used to control and process the trigger events of the business type. When a trigger event is obtained, the business type proactively obtains the update results of the trigger events of other related business data, and grants the permission to proactively deliver the trigger events of related business data that have not yet been stored. This process continues until all the trigger events of related business data are obtained, and then the data processing of the related business data of the business type is performed.

[0077] Optionally, determining whether the event triggering mechanism for streaming data processing of the aforementioned business type needs to be updated includes:

[0078] Based on the message data of trigger events for each business type, determine the message data in different time periods;

[0079] Based on the decentralization processing method of the aforementioned business type, determine the number of associated business data that have not yet been fully stored after the business type receives the proactive decentralization message;

[0080] Based on the number of related business data that have not yet been stored after the business type receives the proactive delivery message, and the message data in different time periods, determine whether the event triggering mechanism of the streaming data processing of the business type needs to be updated.

[0081] It should be noted that when the amount of message data in different time periods does not meet the requirements, since there are many messages in different time periods, updating the data will inevitably lead to an even larger amount of message data. Therefore, the event triggering mechanism for streaming data processing of different business types does not need to be updated.

[0082] Furthermore, if the amount of message data in different time periods meets the requirements, the number of related business data that have not yet been stored after the business type receives the actively distributed message is determined. When the number of related business data that have not yet been stored meets the requirements in different data processing processes after receiving the actively distributed message, for example, when they are all less than 3, it is determined that the event triggering mechanism of the streaming data processing of the business type does not need to be updated.

[0083] Additionally, it is understandable that when the number of related business data that has not yet been stored does not meet the requirements in different data processing processes after receiving the proactive delivery message, the related business data with the largest proportion of delayed delivery, excluding the delayed delivery data processed by proactive delivery, will also be used as the message data that triggers the event in the business event and processed by proactive delivery, thereby reducing the delivery processing of trigger event messages of clustered related business data.

[0084] Example 2

[0085] Secondly, such as Figure 5 As shown, the present invention provides a computer system, including: a memory and a processor connected in communication, and a computer program stored on the memory and capable of running on the processor. When the processor runs the computer program, it executes the above-described event-triggered streaming data timing consistency guarantee method.

[0086] Optionally, the method for determining the proactive delivery method of the triggering event for the aforementioned business type is as follows:

[0087] Based on the sorting results of the sending time of the message data of the triggering event of the associated business data in the same business event, the associated business data with the longest sending time of the triggering event message data in different processing processes in the same business event is determined and it is used as the delayed business data.

[0088] Based on the distribution of delayed service data in different processing processes, the associated service data is determined as the processing process for delayed service data;

[0089] Based on the processing data of each associated business data as delayed business data, the proportion of delayed delivery of associated business data is determined, and the active delivery method of the triggering event of the business type is determined based on the delayed delivery proportion.

[0090] It is understandable that when the associated business data meets the requirements as the processing data of delayed business data, since its processing delay is short, it is determined that after the associated business data is stored on the server, the message data of its triggering event will no longer be actively distributed.

[0091] Additionally, it can be understood that when the processing data of the associated business data as delayed business data does not meet the requirements, the method for actively releasing the triggering event of the business type is to actively release the message data of the triggering event of a preset number of associated business data in the business event.

[0092] It should be noted that the preset number of associated business data is selected based on the percentage of the delivery delay, and the preset number of associated business data with the largest percentage of the delivery delay is selected as the associated business data for actively delivery processing as the message data that triggers the event in the business event.

[0093] Example 3

[0094] Furthermore, determining whether the event triggering mechanism for streaming data processing of the aforementioned business type needs to be updated specifically includes:

[0095] Based on the message data of trigger events for each business type, determine the message data in different time periods;

[0096] Based on the decentralization processing method for the aforementioned business type, the monitoring data for delayed decentralization data in the aforementioned business type is determined;

[0097] Based on the monitoring data of delayed data delivery in the business type and the message data in different time periods, determine whether the event triggering mechanism of streaming data processing for the business type needs to be updated.

[0098] It should be noted that when the amount of message data in different time periods does not meet the requirements, since there are many messages in different time periods, updating the data will inevitably lead to an even larger amount of message data. Therefore, the event triggering mechanism for streaming data processing of different business types does not need to be updated.

[0099] In one possible embodiment, if message data exists for different service types in different time periods, it is determined that the amount of message data in different time periods does not meet the requirements, for example, the time period is 10 seconds.

[0100] Furthermore, when the amount of message data in different time periods meets the requirements, the monitoring data of delayed delivery data in the business type is determined. If all delayed delivery data of the business type can be actively delivered as messages, then the event triggering mechanism of the streaming data processing of the business type does not need to be updated.

[0101] Additionally, it should be noted that if not all delayed delivery data of the aforementioned business type can be actively delivered for message processing, the sum of the delayed delivery ratios of the associated business data of the actively delivered delayed delivery data is obtained. When the sum of the delayed delivery ratios is less than a preset delayed delivery ratio threshold, the event triggering mechanism of the streaming data processing of the aforementioned business type is updated using the basic update scheme.

[0102] In one possible specific embodiment, when the sum of the delayed delivery ratios is less than 0.6, it is determined that the event triggering mechanism for the streaming data processing of the business type will be updated using the basic update scheme. That is, delayed delivery data for active delivery processing of new trigger event message data will be added. Specifically, based on the delayed delivery ratio of the delayed delivery data as a constraint, the delayed delivery data will be combined to obtain the combination of delayed delivery data with the smallest number of delayed delivery ratios that are not less than a preset delayed delivery ratio threshold. This combination will be used as the associated business data for active delivery processing of the message data that triggers the event in the business event.

[0103] Furthermore, when the sum of the delayed delivery ratios is less than a preset delayed delivery ratio threshold, the number of related business data that have not yet been stored after the business type receives the proactive delivery message is determined. When the number of related business data that have not yet been stored meets the requirements in different data processing processes after receiving the proactive delivery message, such as being less than 3, the event triggering mechanism of the streaming data processing of the business type is determined to be unnecessary to be updated.

[0104] Additionally, it is understandable that when the number of related business data that has not yet been stored does not meet the requirements in different data processing processes after receiving the proactive delivery message, the related business data with the largest proportion of delayed delivery, excluding the delayed delivery data processed by proactive delivery, will also be used as the message data that triggers the event in the business event and processed by proactive delivery, thereby reducing the delivery processing of trigger event messages of clustered related business data.

[0105] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0106] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0107] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. An event-triggered flow data timing consistency guarantee method, characterized in that, Specifically comprising: Determine the associated business data of the business type based on the business type, determine the write delay type of the associated business data based on the queue data of the write of the associated business data, and determine whether the event triggering mechanism of the business type needs to be optimized when the event triggering mechanism of the business type needs to be optimized according to the deviation of the write delay type of the associated business data, and enter the next step; Determine the active offloading method of the triggering event of the business type according to the sorting result of the sending time of the message data of the triggering event of the associated business data, control the triggering event of each business type by using the active offloading method, and determine whether the event triggering mechanism of the stream data processing of the business type needs to be updated according to the message data of the triggering event of each business type. The method for determining the write delay type of the associated business data is: Determine the processing delay of the queue of the server port of the write of the associated business data based on the queue data of the write of the associated business data; Determine the average write delay of the associated business data in history based on the processing delay; Determine the write delay type of the associated business data by using the average write delay; The method for determining the active offloading method of the triggering event of the business type is: Determine the associated business data with the longest sending time of the message data of the triggering event in different processing processes in the same business event as the delay business data based on the sorting result of the sending time of the message data of the triggering event of the associated business data in the same business event; Determine the processing process of the associated business data as the delay business data according to the distribution of the delay business data in different processing processes; Determine the active offloading method of the triggering event of the business type according to the processing process data of each associated business data as the delay business data; Determine whether the event triggering mechanism of the stream data processing of the business type needs to be updated, specifically comprising: Determine the message data in different time periods based on the message data of the triggering event of each business type; Determine the monitoring data of the delay offloading data in the business type based on the offloading processing method of the business type; Determine whether the event triggering mechanism of the stream data processing of the business type needs to be updated according to the monitoring data of the delay offloading data in the business type and the message data in different time periods.

2. The event-triggered flow data timing consistency guarantee method according to claim 1, wherein, The associated business data is the business data that needs to be combined when the business type is processing data.

3. The event-triggered flow data timing consistency guarantee method according to claim 1, wherein, The queue data of the write of the associated business data is determined according to the processing delay of the queue of the server port of the write of the associated business data.

4. The event-triggered flow data timing consistency guarantee method according to claim 1, wherein, The average write delay is the average write time of the associated business data when writing to the server, that is, the average write time from the start of sorting in the write queue to the success of writing.

5. The event-triggered flow data timing consistency guarantee method according to claim 4, wherein, Determine the write delay type of the associated business data by using the average write delay, specifically comprising: When the average write delay is greater than a preset time threshold, it is determined that the write delay type of the associated service data is an abnormal delay type; When the average write delay is not greater than the preset time threshold, it is determined that the write delay type of the associated service data is a normal delay type.

6. The event-triggered flow data timing consistency guarantee method according to claim 1, wherein, The control processing of the trigger event of each service type is performed by using the active release method, and specifically includes: In each service type, the control processing of the trigger event of the service type is performed by using the active release method, and when the trigger event is obtained, the service type actively acquires the update result of the trigger event of other associated service data, and opens the permission of the active release message of the trigger event of the associated service data which has not been stored, until all the trigger events of the associated service data are obtained, and then the data processing of the associated service data of the service type is performed.

7. A computer system comprising: The memory and the processor connected by communication, and the computer program stored on the memory and capable of running on the processor, characterized in that the processor executes the computer program to perform the event-triggered stream data time sequence consistency guarantee method of any one of claims 1-6.

Images