Data processing method and device and related equipment

Abstract

The invention provides a data processing method, which comprises the following steps of: acquiring first event data, creating a first trigger for indicating first window withdrawing time for the first event data, and executing data calculation based on the event data in a first time window to obtain a first calculation result for updating a database and outputting the first calculation result; wherein the event data in the first time window comprises the first event data. When the first window withdrawing time indicated by the first trigger is reached, data calculation is executed based on the event data in the second time window, a second calculation result used for updating the database is obtained and output, and the event data in the second time window does not comprise the first event data. Therefore, when the first event data expires, the event data in the second time window is triggered to update the calculation result in the database, so that the accuracy of reading the calculation result from the database is improved. In addition, the invention also provides a corresponding data processing device and related equipment.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 10, 2023, with application number 202311308663.5 and invention name “Data processing methods, devices and related equipment”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of data processing technology, and in particular to a data processing method, device and related equipment. Background Art

[0003] In real business scenarios, it is often necessary to generate calculation results based on the data in the latest time window. For example, in a bank risk control scenario, the anti-fraud system can count the number of transactions or transaction amounts of users in the last hour (i.e., the time window) to identify whether the user's transaction behavior is abnormal. For example, when the number of transactions of a user in the last hour is too high or the transaction amount is too large, the anti-fraud system can determine that the user's transaction behavior is abnormal, and can further intercept the user's subsequent transaction behavior.

[0004] At present, the calculation process for the event data in the latest time window (that is, the data related to the event that occurred) is usually triggered when there is an event drive, and the generated calculation results are saved in the database. For example, every time a user completes a transaction of an amount, it can trigger the aggregation calculation of the transaction data generated by the user in the last hour, and save the aggregation results. However, when no event data is generated, the calculation results saved in the database will continue to remain unchanged, which makes the calculation results saved in the database prone to expiration, that is, the calculation results read from the database are not the real calculation results corresponding to the event data in the most recent time window, so the problem of errors in the read calculation results often occurs. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a data processing method to improve the accuracy of calculation results read from a database. The present application also provides a corresponding data processing device, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0006] In the first aspect, an embodiment of the present application provides a data processing method, which can be executed by a corresponding data processing device. Specifically, the data processing device obtains first event data, which can be, for example, event data generated by an application during operation, such as transaction event data. Then, in response to obtaining the first event data, the data processing device creates a first trigger for the first event data, and the first trigger is used to indicate the first window withdrawal time corresponding to the first event data, that is, to indicate the expiration time of the first event data. In addition, in response to obtaining the first event data, the data processing device performs data calculation based on the event data in the first time window, obtains a first calculation result, and outputs the first calculation result, wherein the event data in the first time window includes the first event data, and the first calculation result is used to update the calculation result in the database. That is, when there is new event data, the data processing device will be triggered to perform window calculation and update the database. When the first exit window time indicated by the first trigger is reached, indicating that the first event data has expired, the data processing device will perform data calculation based on the event data in the second time window, obtain a second calculation result, and output the second calculation result. The event data in the second time window does not include the expired first event data, and the starting time of the second time window is after the starting time of the first time window. The second calculation result is used to update the calculation result in the database.

[0007] In this way, the data processing device not only triggers the use of event data in the first time window to update the calculation results stored in the database when the first event data is obtained, but also triggers the use of event data in the second time window (that is, the new time window) to update the calculation results stored in the database when the first event data expires (that is, the first exit window time corresponding to the first event data is reached). This allows the calculation results stored in the database to be dynamically updated as the time window passes, avoiding the expiration of the calculation results stored in the database, thereby effectively improving the accuracy of the calculation results read from the database. In addition, the data processing device consumes less resources in the process of updating the calculation results in the database, such as less computing resources and storage resources, and the reliability and performance of the calculation results in the updated database are higher.

[0008] In a possible implementation, the data processing device may also acquire the second event data and the third event data, and in response to acquiring the second event data, create a second trigger for the second event data, the second trigger being used to indicate the second window exit time corresponding to the second event data; and the data processing device may also, in response to acquiring the third event data, create a third trigger for the third event data, the third trigger being used to indicate the third window exit time corresponding to the third event data, the time interval between the third window exit time and the second window exit time being less than the delayed window exit time corresponding to the second event data, and the third window exit time being later than the second window exit time. In actual application, the data processing device may also trigger the execution of the data calculation process based on the acquired second event data and the third event data, respectively. In this way, the data processing device may first determine the delay time period according to the second window exit time and the delayed window exit time, and then, when the end time of the delay time period is reached, the data processing device performs data calculation based on the event data in the third time window, and the event data in the third time window does not include the second event data and the third event data. In this way, when multiple pieces of event data expire within a short period of time, the data processing device can be triggered to perform a data calculation process only once, thereby reducing resource consumption of the data processing device.

[0009] In a possible implementation, the second event data and the third event data are stored in the target storage area. Then, when the data processing device performs data calculation based on the event data in the third time window, it can specifically first obtain the original event data stored in the target storage area, and remove the second event data and the third event data from the original event data to obtain the event data in the third time window, so that the data processing device can perform data calculation based on the event data in the third time window to obtain a third calculation result, which is used to update the calculation result in the database. In this way, the data processing device can calculate the correct calculation result by removing the expired event data, so that after updating the database with the correct calculation result, the accuracy of the calculation result read from the database can be effectively improved.

[0010] In a possible implementation, the second event data and the third event data include the same primary key, and the target storage area stores event data including different primary keys, so that when the data processing device obtains the original event data stored in the target storage area, it can specifically obtain the original event data stored in the target storage area according to the primary key. In this way, the data processing device can perform data calculation for event data with the same primary key each time, and event data with different primary keys will not participate in the same data calculation process, so as to meet the differentiated needs of different businesses in actual application scenarios, that is, event data for different businesses may not participate in data calculation.

[0011] In a possible implementation, the data processing device may also obtain fourth event data, and in response to obtaining the fourth event data, create a fourth trigger for the fourth event data, the fourth trigger is used to indicate the fourth window-out time corresponding to the fourth event data, the time interval between the fourth window-out time and the second window-out time is greater than the delayed window-out time corresponding to the second event data, and the fourth window-out time is later than the second window-out time; thus, after performing data calculation based on the event data in the third time window, when the fourth window-out time indicated by the fourth trigger is reached, data calculation is performed based on the event data in the fourth time window, and the event data in the fourth time window does not include the second event data, the third event data, and the fourth event data. In this way, when the expiration time interval between two event data is long, the data processing device can perform a data calculation process respectively when the two event data expire, so as to realize the dynamic update of the calculation results in the database and improve the accuracy of the calculation results read from the database.

[0012] In a possible implementation, after obtaining the third calculation result, the data processing device may delete the second event data and the third event data in the target storage area, thereby releasing storage resources occupied by expired event data and improving resource utilization.

[0013] In a possible implementation, the data processing device may also output a configuration interface, such as outputting the configuration interface to a client so that the client can present the configuration interface to the user, so that the data processing device can respond to the user's configuration operation on the configuration interface for the delayed window exit duration, and obtain the delayed window exit duration corresponding to the second event data. In this way, the output processing device can support the user's customized configuration of the delayed window exit duration, thereby meeting the differentiated needs of different users for the delayed window exit duration, and improving the flexibility of the data processing device in calculating the event data.

[0014] In a second aspect, the present application provides a data processing device, which includes: an interaction module, used to obtain first event data; a creation module, used to create a first trigger for the first event data in response to obtaining the first event data, the first trigger being used to indicate a first exit window time corresponding to the first event data; a processing module, used to perform data calculation based on the event data in a first time window in response to obtaining the first event data, to obtain a first calculation result, the event data in the first time window including the first event data, and the first calculation result being used to update the calculation result in a database; the interaction module is also used to output the first calculation result; the processing module is also used to perform data calculation based on the event data in a second time window when the first exit window time indicated by the first trigger is reached, to obtain a second calculation result, the event data in the second time window does not include the first event data, the start time of the second time window is after the start time of the first time window, and the second calculation result is used to update the calculation result in the database; the interaction module is also used to output the first calculation result.

[0015] In one possible implementation, the interaction module is also used to obtain second event data before performing data calculation based on the event data in the second time window; the creation module is also used to create a second trigger for the second event data in response to obtaining the second event data, and the second trigger is used to indicate a second window exit time corresponding to the second event data; in response to obtaining the third event data, a third trigger is created for the third event data, and the third trigger is used to indicate a third window exit time corresponding to the third event data, and the time interval between the third window exit time and the second window exit time is less than the delayed window exit time corresponding to the second event data, and the third window exit time is later than the second window exit time; then, the processing module is also used to determine a delay time period according to the second window exit time and the delayed window exit time; when the end time of the delay time period is reached, data calculation is performed based on the event data in the third time window, and the event data in the third time window does not include the second event data and the third event data.

[0016] In one possible implementation, the second event data and the third event data are stored in a target storage area; then, the processing module is used to: obtain the original event data stored in the target storage area; remove the second event data and the third event data from the original event data to obtain the event data within a third time window; perform data calculation based on the event data within the third time window to obtain a third calculation result, and the third calculation result is used to update the calculation result in the database.

[0017] In a possible implementation, the second event data and the third event data include the same primary key, and the target storage area stores event data including different primary keys; the processing module is used to obtain the original event data stored in the target storage area according to the primary key.

[0018] In one possible implementation, the interaction module is also used to obtain fourth event data; the creation module is also used to create a fourth trigger for the fourth event data in response to obtaining the fourth event data, the fourth trigger being used to indicate a fourth window-outtime time corresponding to the fourth event data, the time interval between the fourth window-outtime time and the second window-outtime time being greater than the delayed window-outtime duration corresponding to the second event data, and the fourth window-outtime time being later than the second window-outtime time; the processing module is also used to perform data calculation based on the event data in the fourth time window after performing data calculation based on the event data in the third time window, when the fourth window-outtime time indicated by the fourth trigger is reached, based on the event data in the fourth time window, the event data in the fourth time window does not include the second event data, the third event data and the fourth event data.

[0019] In a possible implementation, the device further includes: a storage module, configured to delete the second event data and the third event data in the target storage area after obtaining the third calculation result.

[0020] In a possible implementation, the interaction module is further used to: output a configuration interface; and in response to a user's configuration operation on the configuration interface for a delayed window exiting time length, obtain a delayed window exiting time length corresponding to the second event data.

[0021] The data processing device provided in the second aspect corresponds to the data processing method provided in the first aspect. Therefore, the technical effects of the data processing device provided in the second aspect can be referred to the relevant description of the technical effects of the above-mentioned first aspect or any implementation method of the first aspect, and will not be elaborated on here.

[0022] In a third aspect, the present application provides a computing device cluster, the computing device cluster includes at least one computing device, the at least one computing device includes at least one processor and at least one memory; the at least one memory is used to store instructions, and the at least one processor executes the instructions stored in the at least one memory, so that the computing device cluster executes the data processing method in the above-mentioned first aspect or any possible implementation of the first aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The at least one computing device may also include a bus. The processor is connected to the memory via a bus. The memory may include a readable memory and a random access memory.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on at least one computing device, the at least one computing device executes the data processing method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0024] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on at least one computing device, enables the at least one computing device to execute the data processing method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0025] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0027] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided by the present application;

[0028] Figure 2 A schematic diagram of the number of event data within multiple time windows provided for this application;

[0029] Figure 3 A flowchart of a data processing method provided in this application;

[0030] Figure 4 A schematic diagram of an exemplary table structure for defining consumption of Kafka provided in this application;

[0031] Figure 5 A schematic diagram of an exemplary definition of writing data to redis provided in this application;

[0032] Figure 6 A flowchart of another data processing method provided for this application;

[0033] Figure 7 A schematic diagram of an exemplary configuration interface provided by this application;

[0034] Figure 8 A schematic diagram of an exemplary SQL job definition provided by this application;

[0035] Fig. 9 A schematic diagram of the structure of a computing device provided for this application;

[0036] Fig.10 A schematic diagram of the structure of a computing device cluster provided in this application. DETAILED DESCRIPTION

[0037] The scheme in the embodiments provided in this application will be described below in conjunction with the drawings in this application.

[0038] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0039] See also Figure 1 , which is a schematic diagram of the structure of an exemplary data processing system 10. Figure 1 As shown, the data processing system 10 includes an application 101, a message management device 102, a data processing device 200, and a database 103, and the application 101, the message management device 102, the data processing device 200, and the database 103 can communicate with each other through a network. Furthermore, the data processing system 10 can also provide a client to the outside, such as Figure 1 The client 104 is shown. The client 104 is used to interact with the user, and may be an application running on a user-side device, or may be a web browser provided by the data processing system 10 to the outside.

[0040] Application 101 may be run on a user-side device, or may be run on a server in a network, such as an application server. Furthermore, during operation, application 101 may generate an event and generate data related to the event (hereinafter referred to as event data). For example, application 101 may be a transaction software, so application 101 may generate a transaction event based on a transaction operation performed by a user, and further generate data related to the transaction event, such as transaction amount, transaction time, account numbers of both parties to the transaction, user identities, transaction order numbers, and other data.

[0041] The message management device 102 is used to cache the event data generated by the application 101 and provide the event data for consumption by the data processing device 200. For example, the message management device 102 can be configured with one or more queues, so that the application 101 can write the generated event data into the queue, and accordingly, the event data written into the queue can be accessed by the data processing device 200.

[0042] The data processing device 200 is used to obtain event data from the message management device 102, and perform data calculation according to the event data within the time window to obtain corresponding calculation results. The event data within the time window refers to the event data obtained by the data processing device 200 within the time window, or the event data that has not expired within the time window.

[0043] The database 103 is used to store the calculation results generated by the data processing device 200. In actual application, the calculation results stored in the database 103 can be read by other devices or applications.

[0044] When there is an event-driven situation, that is, when new event data is cached in the message management device 102, the data processing device 200 can generate a calculation result based on the event data belonging to the current time window. This makes it possible that when the time window changes over time, the existence duration of some / all event data read by the data processing device 200 exceeds the duration of the time window, that is, the read event data expires. At this time, the calculation result stored in the database 103 remains unchanged, but the calculation result is not the actual calculation result corresponding to the event data in the current time window.

[0045] Take the time window size of 20 minutes as an example. Figure 2 As shown, multiple time windows can be divided between 1:10 and 1:32, including: [1:10-1:30], [1:11-1:31], [1:12-1:32], etc. Assuming that the data calculation process performed by the data processing device 200 based on the event data in each time window is specifically to count the number of event data, then, in the time window [1:10-1:30], the data processing device 200 can count the number of event data in the time window, in the time window [1:11-1:31], the data processing device 200 can count the number of event data in the time window, in the time window [1:12-1:32], the data processing device 200 can count the number of event data in the time window. Since a new event data is generated at 1:30, the data processing device 200 counts the number of event data as 13 based on the event data in the time window [1:10-1:30], and saves 13 in the database 103. At this time, the number read from the database 103 at 1:30 is 13.

[0046] However, after 1:30, since no new event data is generated, in the absence of event drive, if the data processing device 200 is not triggered to update the database 103, the number of event data stored in the database 103 remains 13, that is, at 1:31 and 1:32, the number of events read from the database 103 is 13. However, in fact, the number of events in the time window [1:11-1:31] and the time window [1:12-1:32] are 11 and 9 respectively, that is, the number of events read from the database 103 is not the actual number of event data in the current time window. As shown in Table 1 below, the data read from the database 103 often has errors.

[0047] Table 1

[0048] Time window The real number Number of reads [1:10 to 1:30] 13 13 [1:11 to 1:31] 11 13 [1:12 to 1:32] 9 13

[0049] Based on this, the data processing device 200 provided in the present application can automatically trigger the process of performing data calculation according to the event data in the current time window in the absence of event-driven operation, so as to achieve timely update of the database 103.

[0050] When implementing it specifically, Figure 1As shown, the data processing device 200 may include an interaction module 201, a creation module 202, and a processing module 203. Further, the data processing device 200 may also include a storage module 204. The interaction module 201 is used to obtain event data 1 from the message management device 101, and the obtained event data 1 may be cached in the storage module 204. In addition, the interaction module 201 may notify the creation module 202 to create a trigger 1 for the event data 1, and the trigger 1 is used to indicate the exit window time corresponding to the event data 1 (when the exit window time is reached, it indicates that the event data 1 is expired), and provide the event data 1 to the processing module 203. In the case of obtaining new event data 1, the processing module 203 may access the event data (including the event data 1) obtained in the time window 1 from the storage module 204, and perform data calculation according to the event data in the time window 1, such as counting the number of event data in the time window 1, so as to obtain the calculation result 1, and at this time, the calculation result in the database 103 may be updated to the calculation result 1. When the window exit time indicated by trigger 1 is reached, the creation module 202 can trigger the processing module 203 to perform data calculation based on the event data in time window 2 to obtain calculation result 2. At this time, the calculation result in the database 103 can be updated to calculation result 2, specifically, the calculation result 2 can be sent to the database 103 through the interaction module 201. Among them, the starting time of time window 2 is after the starting time of time window 1, that is, time window 2 is later than event window 1, and the event data in time window 2 does not include event data 1.

[0051] In this way, the data processing device 200 not only triggers the use of event data within time window 1 to update the calculation results saved in the database 103 when event data 1 is obtained, but also triggers the use of event data within time window 2 to update the calculation results saved in the database 103 when event data 1 expires (that is, when the exit window time corresponding to event data 1 is reached). This allows the calculation results saved in the database 103 to be dynamically updated as the time window passes, thereby avoiding the expiration of the calculation results saved in the database 103, thereby effectively improving the accuracy of the calculation results read from the database 103.

[0052] Furthermore, the data processing device 200 consumes less computing resources and storage resources in the process of updating the calculation results in the database 103. Moreover, the created trigger can timely trigger the data processing device 200 to update the calculation results when the event data expires, thereby effectively ensuring the reliability, accuracy and performance of the calculation results in the updated database 103.

[0053] Still Figure 2The time window shown is used as an example for explanation. The data processing device 200 can create trigger 1 for the two newly generated event data at 1:10, and the window exit time indicated by the trigger 1 is 1:31. At 1:11, trigger 2 is created for the two newly generated event data, and the window exit time indicated by the trigger 2 is 1:32. In this way, when it reaches 1:31, trigger 1 can trigger the data processing device 200 to update the calculation results stored in the database 103. At this time, the data processing device 200 can specifically count the number of event data as 11 according to the event data in the time window [1:11 to 1:31], and update the number stored in the database 103 from 13 to 11. At this time, the number read from the database 103 at 1:31 is 11. Similarly, when it reaches 1:32, trigger 2 can trigger the data processing device 200 to count the number of event data in the time window [1:12 to 1:32] as 9, and update the number stored in the database 103 from 11 to 9. At this time, the number read from the database 103 at 1:32 is 9. As shown in Table 2 below, the data read from the database 103 may be correct data.

[0054] Table 1

[0055] Time window The real number Number of reads [1:10 to 1:30] 13 13 [1:11 to 1:31] 11 11 [1:12 to 1:32] 9 9

[0056] It is worth noting that Figure 1 The data processing system 10 shown is only used as an exemplary illustration and is not intended to limit the specific implementation of the data processing system. Figure 1 In the data processing system 10 shown, the data processing device 200 can be deployed in the cloud, such as in a public cloud, a hybrid cloud, etc., and can provide cloud services for calculating data within a time window. The client 104 can be deployed on the user side to facilitate the user to interact with the data processing system 10 through the client 104, etc.; in other possible data processing systems, the client 104 may not be included. In this case, the data processing device 200 can be deployed on the user side, so that the data processing device 200 can provide the user with a localized service for calculating data within a time window; at the same time, the data processing device 200 can present a configuration interface to the user, so that the user can interact with the data processing device 200 through the configuration interface. Alternatively, in other possible data processing systems, the data processing system 10 can perform corresponding calculations for event data provided by multiple applications. Alternatively, the data processing system may include other types of devices, such as the database 103 can also be connected to other devices to support other devices to access the calculation results stored in the database 103, etc. This application does not limit the specific architecture of the data processing system.

[0057] In addition, the above data processing device 200 may be implemented by software or hardware. As an example, the implementation of the data processing device 200 is described below.

[0058] As an example of a software functional unit, the data processing device 200 may include code running on an instance, which may be at least one of a host, a virtual machine, a container, a thread, and a process. In addition, the above instance may be one or more. Exemplarily, the data processing device 200 may be a stream processing engine that can process and analyze real-time data streams based on an event-driven mode, and can trigger the processing of event data without event-driven by loading a corresponding software development kit (SDK). Alternatively, the data processing device 200 may include code running on multiple hosts / virtual machines / containers. It should be noted that multiple hosts / virtual machines / containers for running the code may be distributed in the same region or in different regions. Further, multiple hosts / virtual machines / containers for running the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Among them, a region may generally include multiple AZs.

[0059] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0060] The data processing device 200 is taken as an example of a hardware functional unit, and the data processing device 200 may include at least one computing device, such as a cloud server, etc. Alternatively, the data processing device 200 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The above-mentioned PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU) or any combination thereof.

[0061] When the data processing apparatus 200 may include multiple computing devices, the multiple computing devices may be distributed in the same region or in different regions. The multiple computing devices included in the data processing apparatus 200 may be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the data processing apparatus 200 may be distributed in the same VPC or in multiple VPCs. The multiple computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0062] It should be noted that the method steps executed by each module in the above-mentioned data processing device 200 are only used as an exemplary description and are not intended to be limiting. For example, in other implementation examples, the interaction module 201 in the data processing device 200 can be used to execute the method steps executed by any module in the above-mentioned data processing device 200, the creation module 202 can be used to execute the method steps executed by any module in the above-mentioned data processing device 200, the processing module 203 can be used to execute the method steps executed by any module in the above-mentioned data processing device 200, and the storage module 204 can be used to execute the method steps executed by any module in the above-mentioned data processing device 200. The steps that the interaction module 201, the creation module 202, the processing module 203, and the storage module 204 are responsible for implementing can be specified as needed, and the interaction module 201, the creation module 202, the processing module 203, and the storage module 204 respectively implement different method steps in the above-mentioned data processing process to realize all the functions of the data processing device 200.

[0063] To facilitate understanding, various non-limiting specific implementations of the process of processing event data are described in detail below.

[0064] See also Figure 3 , is a flow chart of a data processing method in an embodiment of the present application. The method can be applied to the above Figure 1 The data processing system 10 shown in FIG. 1 may also be applied to other applicable data processing systems. Figure 1 Taking the data processing system 10 as an example, the process of the modules in the data processing device 200 processing event data and updating the calculation results in the database 103 is introduced.

[0065] like Figure 3 As shown, the data processing method may specifically include:

[0066] S301: The application 101 generates first event data, and sends the first event data to the message management device 102 for caching.

[0067] In actual application, one or more applications in the data processing system 10 may generate events and generate data related to the events (i.e., event data) during operation. For example, financial software may generate transaction events and details related to the transaction events in response to user operations during operation, including transaction amount, transaction time, account numbers of both parties to the transaction, user identities, transaction order numbers, and other data.

[0068] In this embodiment, taking the case where the application 101 generates the first event data as an example, the application 101 may send the first event data to the message management device 102 for caching, such as writing the first event data into a queue in the message management device 102. In an actual application scenario, the application 101 may generate a data stream within a period of time, and the data stream includes multiple event data, so that the application 101 may write all the multiple event data into the message management device 102.

[0069] S302 : the interaction module 201 obtains first event data from the message management device 102 .

[0070] In a possible implementation, the interaction module 201 may periodically poll the message management device 102 to determine whether event data is cached in the message management device 102, so that after the application 101 writes the first event data to the message management device 102, the interaction module 201 may access and obtain the first event data.

[0071] For example, the message management device 102 may include a Kafka system (a high-throughput distributed publish-subscribe message system), which can support the data processing device 200 to consume event data stored in the Kafka system. Figure 4 The definition shown is written into the Kafka system for caching.

[0072] In other embodiments, the interaction module 201 may also obtain the first event data based on other methods, such as the message management device 102 actively sending the first event data to the interaction module 201.

[0073] Then, the interaction module 201 may notify the creation module 202 to obtain the first event data, and may provide the first event data to the processing module 203 .

[0074] S303: In response to acquiring the first event data, the creation module 202 creates a first trigger for the first event data, where the first trigger is used to indicate a first window exit time corresponding to the first event data.

[0075] Exemplarily, the first trigger created may include, for example, a timer, which can start timing after the first trigger is created, and when the timing duration of the timer reaches a preset duration, the first trigger can trigger the processing module 203 to recalculate the event data in the time window. The preset duration may be the duration of a single time window, such as 20 minutes. Then, when the timing duration of the timer reaches 20 minutes, the processing module 203 may be triggered to recalculate the event data. Accordingly, the moment when the timing duration of the timer reaches 20 minutes is the first window exit time corresponding to the first event data, that is, the expiration time of the first event data.

[0076] The time window refers to a fixed time period, and the time period changes with the passage of time, such as Figure 2 As shown, multiple time windows change over time, etc. In practical applications, the time window can be a period of time before the current moment (such as the current time of the data processing system 10).

[0077] In one possible implementation, the creation module 202 may provide a trigger service, so that after acquiring the first event data, the interaction module 201 may request the trigger service in the creation module 202 to register a trigger for the first event data, so that the trigger service may create a corresponding first trigger based on the request.

[0078] S304: In response to obtaining the first event data, the processing module 203 performs data calculation based on the event data in the first time window to obtain a first calculation result, where the event data in the first time window includes the first event data. The obtained first calculation result is used to update the calculation result in the database 103.

[0079] Generally, the data processing device 200 can trigger the calculation of event data in an event-driven manner. Therefore, after the interaction module 201 provides the first event data to the processing module 203, the processing module 203 can obtain the event data within the first time window, and the first time window can be a time window with the current time as the end time. The event data obtained within the first time window is the event data that has not expired within the time period indicated by the first time window, or is called the event data obtained by the interaction module 201 within the time period indicated by the first time window.

[0080] Exemplarily, each time after acquiring event data, the interaction module 201 may save the event data to the storage module 204 , so that the processing module 203 may access the event data within the first time window from the storage module 204 .

[0081] Then, the processing module 203 can perform data calculation on the event data in the first time window to obtain a corresponding first calculation result. The data calculation performed may, for example, be a quantitative count of the event data in the first time window, and the value indicating the number of event data obtained is the first calculation result; or, the data calculation may be a summation of the event data in the first time window, such as calculating the sum of the transaction amounts in each event data, and the sum obtained is the first calculation result; or, the data calculation may be a calculation of the average value / median value of the event data in the first time window, and the average value / median value obtained is the first calculation result. This embodiment does not limit the specific implementation method of the data calculation.

[0082] After obtaining the first calculation result through calculation, the processing module 203 provides the first calculation result to the interaction module 201 .

[0083] S305 : the interaction module 201 writes the first calculation result into the database 103 , so as to update the calculation result stored in the database 103 .

[0084] Exemplarily, the database 103 may store the first calculation result (and other calculation results) provided by the interaction module 201 in the form of key-value pairs. In this case, the database 103 may be, for example, a Redis database. Figure 5 According to the definition method shown, the first calculation result is written into the database 103.

[0085] At this time, the calculation result read from the database 103 is the first calculation result.

[0086] S306: When the first exit window time indicated by the first trigger is reached, the processing module 203 performs data calculation based on the event data in the second time window to obtain a second calculation result, wherein the event data in the second time window does not include the first event data, and the starting time of the second time window is after the starting time of the first time window, and the obtained second calculation result is used to update the calculation result in the database 103.

[0087] It is understandable that as time goes by, the existence time of the first event data exceeds the time of a single time window, that is, the first event data expires. At this time, the calculation result stored in the database 103 is still calculated based on the expired first event data, which easily leads to the problem that the calculation result read from the database 103 has errors. Therefore, in this embodiment, when the first event data expires, the processing module 203 can recalculate the data based on the non-expired event data.

[0088] Exemplarily, when the timing duration of the timer included in the first trigger reaches the duration of the time window, representing the current moment as the expiration time of the first event data, that is, the first exit window time, the creation module 202 can trigger the processing module 203 to perform data calculation on the event data by calling the corresponding function or application programming interface (API) interface.

[0089] Under the triggering of the creation module 202, the processing module 203 obtains the event data within the second time window. The second time window can be a time window with the current moment as the end moment. The event data within the second time window obtained is the event data that has not expired within the time period indicated by the second time window. Since the first event data has expired at the current moment, the event data within the second time window obtained by the processing module 203 does not include the first event data.

[0090] In a possible implementation, the processing module 203 can access the target storage area in the storage module 204 to obtain the original event data, which includes the first event data. Then, the processing module 203 can remove the expired first event data from the original event data, and the remaining event data are all unexpired data, that is, event data located in the second time window, so that the processing module 203 can perform data calculation based on the event data in the second time window to obtain a new calculation result, which is referred to as the second calculation result in this embodiment. Among them, the data calculation process performed according to the event data in the second time window is similar to the data calculation process performed according to the event data in the first time window, and will not be repeated here.

[0091] After obtaining the second calculation result through calculation, the processing module 203 provides the second calculation result to the interaction module 201 .

[0092] S307 : the interaction module 201 writes the second calculation result into the database 103 , so as to update the calculation result stored in the database 103 .

[0093] At this time, the calculation result read from the database 103 is the second calculation result.

[0094] In this way, when the first event data expires, in the absence of event driving, the processing module 203 will also calculate the event data within the second time window to update the calculation results saved in the database 103. This allows the calculation results saved in the database 103 to be dynamically updated as the time window passes, thereby avoiding the expiration of the calculation results saved in the database 103, thereby effectively improving the accuracy of the calculation results read from the database 103.

[0095] Furthermore, after the first event data expires, the processing module 203 may instruct the storage module 204 to delete the first event data in the target storage area to release storage resources occupied by the first event data and reduce resource waste.

[0096] In actual application, the creation module 202 can create triggers for event data to indicate the window exit event, and can also create other types of triggers. For example, the creation module 202 can also create a trigger for indicating the upper limit of the number of event data. When the number of event data acquired within a preset time period exceeds a preset number, the trigger can trigger the processing module 203 to calculate the event data within the current time window, thereby avoiding data calculation omissions due to acquiring a large number of event data in a short time.

[0097] The process of processing event data described in this embodiment is mainly explained by taking the example of the data processing device 200 obtaining the first event data and triggering the execution of calculation of event data when the first event data expires. When the data processing device 200 obtains the second event data (and other event data), and the second event data (and other event data) expires, the data processing device 200 can refer to the above process to execute the process of calculating the event data in the current time window, and use the obtained calculation results to update the database 103, so as to ensure the accuracy of the calculation results read from the database 103.

[0098] In the above Figure 3 In the data processing method shown, when each event data expires, the data processing device 200 can be triggered to recalculate the event data in the current time window. In actual application scenarios, when multiple event data expire in a short period of time, the data processing device 200 may frequently execute the data calculation process, thereby generating more resource consumption, including more computing resources. Based on this, in other possible embodiments, when multiple event data expire in a short period of time, the data processing device 200 can be triggered to execute the data calculation process only once, thereby reducing the resource consumption of the data processing device 200. The following is combined with Figure 6 This is explained in detail.

[0099] See also Figure 6 , shows another data processing flow chart. For ease of description, Figure 6 In the illustrated embodiment, the case where two pieces of event data expire within a short period of time is used as an example for explanation. When a larger number of event data expire, similar processing can be used.

[0100] like Figure 6 As shown, the method may specifically include:

[0101] S601: The application 101 generates event data 1 and event data 2, and sends the event data 1 and event data 2 to the message management device 102 for caching.

[0102] S602 : The interaction module 201 obtains event data 1 from the message management device 102 .

[0103] S603: In response to acquiring event data 1, the creation module 202 creates trigger 1 for event data 1, where trigger 1 is used to indicate window exit time 1 corresponding to event data 1.

[0104] S604 : In response to acquiring event data 1 , the processing module 203 performs data calculation based on the event data in time window 1 to obtain calculation result 1 , where the event data in time window 1 includes event data 1 .

[0105] S605 : the interaction module 201 writes the calculation result 1 into the database 103 , so as to update the calculation result stored in the database 103 .

[0106] At this time, the calculation result read from the database 103 is the calculation result 1.

[0107] The specific implementation of steps S601 to S605 is similar to the implementation of steps S301 to S305 in the above embodiment, and the relevant descriptions of the above embodiment may be referred to, and will not be repeated here.

[0108] S606 : The interaction module 201 obtains event data 2 from the message management device 102 .

[0109] Among them, the time interval between the moment when the interaction module 201 obtains event data 2 and the moment when event data 1 is obtained can be less than a threshold value, and the threshold value can be, for example, the delayed window withdrawal time length described below. In this way, the time interval between the moment when subsequent event data 1 expires and the moment when event data 2 expires can also be less than the threshold value, that is, the expiration time of event data 1 and event data 2 is close.

[0110] S607 : In response to acquiring the event data 2 , the creation module 202 creates a trigger 2 for the event data 2 , where the trigger 2 is used to indicate the window exit time 2 corresponding to the event data 2 .

[0111] S608: In response to acquiring event data 2, the processing module 203 performs data calculation based on the event data in time window 2 to obtain calculation result 2. The event data in time window 2 includes event data 1 and event data 2.

[0112] S609 : the interaction module 201 writes the calculation result 2 into the database 103 , so as to update the calculation result stored in the database 103 .

[0113] At this time, the calculation result read from the database 103 is the calculation result 2.

[0114] In this embodiment, each time the data processing device 200 acquires new event data, it can trigger a process of performing data calculation on the event data within the time window, so as to obtain corresponding calculation results and update the database 103.

[0115] The specific implementation of steps S606 to S609 is similar to the implementation of steps S302 to S305 in the above embodiment, and the relevant descriptions of the above embodiment may be referred to, and will not be repeated here.

[0116] S610: When the window exiting time 1 indicated by trigger 1 is reached, the processing module 203 determines a delay time period according to the window exiting time 1 and the delayed window exiting time length.

[0117] S611: When the exit window time 2 indicated by trigger 2 is within the delay time period, the processing module 203 performs data calculation based on the event data in the time window 3 when the end moment of the delay time period is reached to obtain calculation result 3. The event data in the time window 3 does not include event data 1 and event data 2.

[0118] In this embodiment, as time goes by, event data 1 and event data 2 will expire one after another. When the window exit time 1 indicated by trigger 1 is reached, creation module 202 can trigger processing module 203 to perform the data calculation process. At this time, since there may be other event data (that is, event data 2 in this embodiment) that are also about to expire, processing module 203 can delay the data calculation process for a period of time.

[0119] In specific implementation, the processing module 203 can first determine the delay time period according to the window withdrawal time 1 corresponding to the event data 1 and the delayed window withdrawal duration, that is, determine the time to delay the execution of the data calculation process (the end time of the delay time period). In this way, when the window withdrawal time indicated by the trigger 2 corresponding to the event data 2 (and the trigger corresponding to other event data) is also within the delay time period, the processing module 203 may not respond to the trigger indication sent by the creation module 202 for the event data 2 (or other event data), and will not start the data calculation process until the end time of the delay time period is reached.

[0120] Exemplarily, the processing module 203 can access the target storage area of ​​the storage module 204 to obtain the original event data, which includes event data 1 and event data 2, and can also include other data obtained after obtaining event data 2. Then, the processing module 203 can remove event data 1 and event data 2 from the original event data to obtain the event data within the time window 3 (that is, the original event data after removal), and perform data calculation based on the event data within the time window 3 to obtain calculation result 3.

[0121] In actual application scenarios, the data processing device 200 can perform data calculations for different types of event data of the application 101, or can perform data calculations for event data generated by different applications. At this time, the data processing device 200 can perform data calculations for event data with the same primary key each time, and event data with different primary keys do not participate in the same data calculation process. For example, transaction-type event data A1 and event data A2 generated by application 101 can have the same primary key "transaction amount", and query-type event data B1 and event data B2 generated by application 101 can have the same primary key "query location". Accordingly, the data processing device 200 will perform a data calculation process of type A for event data A1 and event data A2 with the same primary key, and perform a data calculation process of type B for event data B1 and event data B2 with the same primary key.

[0122] Therefore, the processing module 203 can access the target storage area according to the primary key included in the event data 1 to obtain the original event data with the same primary key, and remove event data 1 and event data 2 from the original event data with the same primary key, and generate calculation result 3 based on the remaining event data.

[0123] S612 : the interaction module 201 writes the calculation result 3 into the database 103 , so as to update the calculation result stored in the database 103 .

[0124] At this time, the calculation result read from the database 103 is calculation result 3.

[0125] In this way, during the delay time period, although event data 1 and event data 2 have both expired, the processing module 203 can only execute the data calculation process once at the end of the delay time period. This can avoid the situation where the processing module 203 executes the data calculation process multiple times when multiple event data expire. This can effectively reduce the number of times the data processing device 200 frequently executes the data calculation process in a short period of time, thereby avoiding more resource consumption.

[0126] Furthermore, after the event data 1 and the event data 2 expire, the processing module 203 may also instruct the storage module 204 to delete the event data 1 and the event data 2 in the target storage area to release the storage resources occupied by the two event data.

[0127] It can be understood that the above is an exemplary explanation using two event data as an example. If there is still event data 3, then when the expiration time of event data 3 is also within the delay time period, the trigger corresponding to the event data 3 will not trigger the processing module 203 to perform the data calculation process. When the expiration time of event data 3 exceeds the delay time period, that is, the expiration time of event data 3 is later than the end time of the delay time period, when event data 3 expires, the trigger corresponding to event data 3 can trigger the processing module 203 to perform the data calculation process again. Specifically, assuming that the interaction module 201 also obtains the event data 3 generated by the application 101 from the message management device 102, the creation module 202 can respond to the acquisition of event data 3 and create a trigger 3 for event data 3. The trigger 3 is used to indicate the window exit time 3 corresponding to event data 3. It is assumed that the time interval between the window exit time 3 and the window exit time 1 is greater than the delayed window exit time corresponding to event data 1, and the window exit time 3 is later than the window exit time 1. In response to acquiring event data 3, processing module 203 performs data calculation based on the event data in time window 4 to obtain calculation result 4, and the event data in time window 4 includes event data 3. When the end time of the delay time period is reached, the window withdrawal time 3 has not yet been reached, then, processing module 203 performs data calculation based on the event data in the above-mentioned time window 3 to obtain calculation result 3. Then, when the window withdrawal time 3 is reached, processor 203 performs data calculation based on the event data in time window 5 to obtain calculation result 4. At this time, the event data in time window 5 does not include event data 1, event data 2 and event data 3. In this way, when event data 1, event data 2 and event data 3 expire, processing module 203 will be triggered to perform two data calculation processes.

[0128] In actual application, the data processing device 200 can also support the user to configure the window exit function and the delayed window exit function. Specifically, the data processing device 200 can output a configuration interface to the client 104, for example, Figure 7 The configuration interface shown is used to prompt the user whether to enable the window exit function and the delayed window exit function. The window exit function refers to triggering the processing module 203 to perform a data calculation process when event data expires. The delayed window exit function refers to triggering the processing module 203 to perform a data calculation process only once when multiple event data expire in a short period of time. When the user configures to enable the delayed window exit function on the configuration interface, such as when the user Figure 7 By checking the "Delayed Window Exit" button on the configuration interface shown, you can further configure the delayed window exit time on the configuration interface, such as configuring the delayed window exit time to 1 second.

[0129] In this way, the client 104 provides the configuration information (including the value of the delayed window exit function and the delayed window exit duration) to the data processing device 200 based on the operation performed by the user, so that the data processing device 200 can perform the corresponding data calculation process based on the configuration information when the event data expires. For example, the data processing device 200 can define a corresponding job, such as Figure 8 The SQL job shown can enable the window retreat function, delay the window retreat function, merge event data (or it can be called traffic optimization), etc. through annotations and other methods.

[0130] Exemplarily, after receiving the configuration information, the data processing device 200 can set the value of the first field, wherein different values ​​of the first field can respectively indicate that the window back function is not turned on, the window back function is turned on but the delayed window back function is not turned on, and the delayed window back function is turned on. For example, the first field may be a field named "over.window.interval". And when the value of this field is -1, it is used to indicate that the window back function is not turned on; when the value of this field is 0, it is used to indicate that the window back function is turned on but the delayed window back function is not turned on; when the value of this field is greater than 0, it is used to indicate that the delayed window back function is turned on, and the value of this field indicates the delayed window back duration.

[0131] Furthermore, when the value of the first field is greater than 0, that is, when the user instructs to turn on the delayed window exit function, the data processing device 200 can also set the value of the second field according to the configuration information, wherein the value of the second field is used to indicate whether to merge the event data to trigger the execution data calculation. For example, the second field can be a field named "over.window.interval.last.rowdata". And when the value of this field is "true", it is used to indicate that the merged event data triggers the execution data calculation. When there are multiple event data that expire within the delay time period, the execution data calculation is triggered only at the end of the delay time period. When the value of this field is "false", it is used to indicate that each expired event data triggers the execution data calculation, that is, each event data can be delayed and triggered separately when it expires.

[0132] It is worth noting that Figure 6The embodiment shown is only used as an implementation example and is not intended to be limited. For example, in other possible embodiments, when the window-back time 1 indicated by trigger 1 is reached, the creation module 202 may also determine the delay time period according to the window-back time 1 and the delayed window-back duration, and temporarily not trigger the processing module 203 to perform data calculation. In the delay time period, when there is event data 2 that expires, that is, when the window-back time 1 corresponding to trigger 2 is reached, the creation module 202 may also not trigger the processing module 203 to perform data calculation, until the end time of the delay time period is reached, the processing module 203 will be triggered to perform data calculation. That is, when multiple event data expire, the creation module 202 will only send a trigger indication to the processing module 203 once, so that the processing module 203 performs a data calculation process. Alternatively, after the user configures the delayed window-back duration, the processing module 203 can determine the time indicated by the window-back time 1 according to the fixed duration of the window-back time and the delayed window-back duration, that is, the end time of the above-mentioned delay time period.

[0133] Above Figures 1 to 8 In the illustrated embodiment, the data processing device 200 may be software configured on a computing device or a computing device cluster, and by running the software on the computing device or the computing device cluster, the computing device or the computing device cluster may implement the functions of the above-mentioned data processing device 200. The data processing device 200 is described in detail below from the perspective of hardware device implementation.

[0134] Fig. 9 The schematic diagram of the structure of a computing device is shown. The data processing device 200 can be deployed on the computing device. The computing device can be a computing device in a cloud environment (such as a server) or a computing device in an edge environment, and can be used to implement the above Figure 3 , Figure 6 The functions of the data processing device 200 in the illustrated embodiment.

[0135] like Fig. 9 As shown, computing device 900 includes processor 910, memory 920, communication interface 930 and bus 940. Processor 910, memory 920 and communication interface 930 communicate through bus 940. Bus 940 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9The communication interface 930 is used for communicating with the outside, such as obtaining event data, outputting calculation results, etc.

[0136] Among them, the processor 910 can be a central processing unit (CPU), an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits. The processor 910 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, the functions of each device in the data processing device 200 can be completed by the hardware integrated logic circuit or software instructions in the processor 910. The processor 910 can also be a general processor, a digital signal process (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. Among them, the general processor can be a microprocessor or the processor can also be any conventional processor, etc., and the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 920 , and the processor 910 reads the information in the memory 920 and completes part or all of the functions in the data processing device 200 in combination with its hardware.

[0137] The memory 920 may include a volatile memory, such as a random access memory (RAM), or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD, or a SSD.

[0138] The memory 920 stores executable codes, and the processor 910 executes the executable codes to execute the method executed by the aforementioned data processing device 200 .

[0139] Specifically, in implementing Figure 3 , Figure 6 In the case of the embodiment shown, and Figure 3 , Figure 6 In the case where the data processing device 200 described in the illustrated embodiment is implemented by software, the execution Figure 3 , Figure 6 The software or program code required for the functions of the data processing device 200 is stored in the memory 920. The interaction between the data processing device 200 and other devices is realized through the communication interface 930. The processor is used to execute the instructions in the memory 920 to implement the method executed by the data processing device 200.

[0140] Fig.10 A schematic diagram of the structure of a computing device cluster is shown. Fig.10 The computing device cluster 100 shown includes multiple computing devices, and the data processing apparatus 200 can be deployed on multiple computing devices in the computing device cluster 100 in a distributed manner. Fig.10 As shown, the computing device cluster 100 includes multiple computing devices 1000 , each computing device 1000 includes a memory 1020 , a processor 1010 , a communication interface 1030 and a bus 1040 , wherein the memory 1020 , the processor 1010 , and the communication interface 1030 are communicatively connected to each other via the bus 1040 .

[0141] The processor 1010 may be a CPU, a GPU, an ASIC or one or more integrated circuits. The processor 1010 may also be an integrated circuit chip having signal processing capabilities. In the implementation process, some functions of the data processing device 200 may be completed by the hardware integrated logic circuit or software instructions in the processor 1010. The processor 1010 may also be a DSP, an FPGA, a general processor, other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and may implement or execute some of the methods, steps and logic block diagrams disclosed in the embodiments of the present application. Among them, the general processor may be a microprocessor or the processor may also be any conventional processor, etc., and the steps of the method disclosed in the embodiments of the present application may be directly embodied as a hardware decoding processor for execution, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1020. In each computing device 1000, the processor 1010 reads the information in the memory 1020 and can complete part of the functions of the data processing device 200 in combination with its hardware.

[0142] The memory 1020 may include ROM, RAM, static storage device, dynamic storage device, hard disk (e.g., SSD, HDD), etc. The memory 1020 may store program codes, for example, part or all of the program codes for implementing the interaction module 201, part or all of the program codes for implementing the creation module 202, part or all of the program codes for implementing the processing module 203, etc. For each computing device 1000, when the program codes stored in the memory 1020 are executed by the processor 1010, the processor 1010 executes part of the method executed by the data processing device 200 based on the communication interface 1030. The memory 1020 may also store data, for example: intermediate data or result data generated by the processor 1010 during the execution process, for example, the above-mentioned calculation results, etc.

[0143] The communication interface 1003 in each computing device 1000 is used for external communication, such as interacting with other computing devices 1000 .

[0144] The bus 1040 may be a peripheral component interconnect standard bus or an extended industry standard architecture bus. For ease of representation, Fig.10 The bus 1040 in each computing device 1000 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0145] The plurality of computing devices 1000 establish communication paths through a communication network to implement the functions of the data processing apparatus 200. Any computing device may be a computing device (eg, a server) in a cloud environment, or a computing device in an edge environment.

[0146] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on one or more computing devices, the one or more computing devices execute the method executed by the data processing device 200 of the above embodiment.

[0147] In addition, the embodiment of the present application further provides a computer program product, and when the computer program product is executed by one or more computing devices, the one or more computing devices execute any of the aforementioned data processing methods. The computer program product may be a software installation package, and when any of the aforementioned data processing methods is required, the computer program product may be downloaded and executed on a computer.

[0148] It should also be noted that the system embodiments described above are merely schematic, wherein the devices described as separate components may or may not be physically separated, and the components displayed as devices may or may not be physical devices, that is, they may be located in one place, or they may be distributed on multiple network devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiments provided by the present application, the connection relationship between the devices indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0149] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0150] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0151] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining first event data; In response to acquiring the first event data, creating a first trigger for the first event data, the first trigger being used to indicate a first window exit time corresponding to the first event data; In response to acquiring the first event data, performing data calculation based on the event data in a first time window to obtain a first calculation result, and outputting the first calculation result, the event data in the first time window including the first event data, the first calculation result being used to update the calculation result in the database; When the first exit window time indicated by the first trigger is reached, data calculation is performed based on the event data in the second time window to obtain a second calculation result, and the second calculation result is output. The event data in the second time window does not include the first event data. The starting time of the second time window is after the starting time of the first time window. The second calculation result is used to update the calculation results in the database.

2. The method according to claim 1, characterized in that The method further comprises: Acquire second event data and third event data; In response to acquiring the second event data, creating a second trigger for the second event data, where the second trigger is used to indicate a second window exit time corresponding to the second event data; In response to acquiring the third event data, creating a third trigger for the third event data, the third trigger being used to indicate a third window exit time corresponding to the third event data, the time interval between the third window exit time and the second window exit time being less than the delayed window exit time corresponding to the second event data, and the third window exit time being later than the second window exit time; Determine a delay time period according to the second window exit time and the delayed window exit time; When the end time of the delay period is reached, data calculation is performed based on the event data in the third time window, and the event data in the third time window does not include the second event data and the third event data.

3. The method according to claim 2, characterized in that The second event data and the third event data are stored in a target storage area; Then, performing data calculation based on the event data in the third time window includes: Acquire the original event data stored in the target storage area; Eliminate the second event data and the third event data from the original event data to obtain event data within the third time window; Data calculation is performed based on the event data in the third time window to obtain a third calculation result, and the third calculation result is used to update the calculation results in the database.

4. The method according to claim 3, characterized in that The second event data and the third event data include the same primary key, and event data including different primary keys are stored in the target storage area; The obtaining of the original event data stored in the target storage area includes: According to the primary key, the original event data stored in the target storage area is acquired.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: Obtaining fourth event data; In response to acquiring the fourth event data, creating a fourth trigger for the fourth event data, the fourth trigger being used to indicate a fourth window-outgoing time corresponding to the fourth event data, a time interval between the fourth window-outgoing time and the second window-outgoing time being greater than a delayed window-outgoing time corresponding to the second event data, and the fourth window-outgoing time being later than the second window-outgoing time; After performing data calculation based on the event data in the third time window, when the fourth window exit time indicated by the fourth trigger is reached, data calculation is performed based on the event data in the fourth time window, and the event data in the fourth time window does not include the second event data, the third event data and the fourth event data.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: After obtaining the third calculation result, the second event data and the third event data in the target storage area are deleted.

7. The method according to any one of claims 2 to 6, characterized in that: The method further comprises: Output configuration interface; In response to the user's configuration operation on the configuration interface for the delayed window exiting time length, the delayed window exiting time length corresponding to the second event data is obtained.

8. A data processing device, characterized in that: The device comprises: An interaction module, used for acquiring first event data; a creation module, configured to create a first trigger for the first event data in response to acquiring the first event data, wherein the first trigger is used to indicate a first window exit time corresponding to the first event data; a processing module, configured to, in response to acquiring the first event data, perform data calculation based on the event data in a first time window to obtain a first calculation result, the event data in the first time window including the first event data, and the first calculation result is used to update the calculation result in the database; The interaction module is further configured to output the first calculation result; The processing module is further configured to, when the first window exit time indicated by the first trigger is reached, perform data calculation based on the event data in a second time window to obtain a second calculation result, the event data in the second time window does not include the first event data, the start time of the second time window is after the start time of the first time window, and the second calculation result is used to update the calculation result in the database; The interaction module is further used to output the first calculation result.

9. The device according to claim 8, characterized in that The interaction module is further used to obtain the second event data and the third event data; The creation module is further configured to, in response to acquiring the second event data, create a second trigger for the second event data, wherein the second trigger is configured to indicate a second window exit time corresponding to the second event data; In response to acquiring the third event data, creating a third trigger for the third event data, the third trigger being used to indicate a third window exit time corresponding to the third event data, the time interval between the third window exit time and the second window exit time being less than the delayed window exit time corresponding to the second event data, and the third window exit time being later than the second window exit time; The processing module is further used for: Determine a delay time period according to the second window exit time and the delayed window exit time; When the end time of the delay period is reached, data calculation is performed based on the event data in the third time window, and the event data in the third time window does not include the second event data and the third event data.

10. The device according to claim 9, characterized in that The second event data and the third event data are stored in a target storage area; Then, the processing module is used to: Acquire the original event data stored in the target storage area; Eliminate the second event data and the third event data from the original event data to obtain event data within the third time window; Data calculation is performed based on the event data in the third time window to obtain a third calculation result, and the third calculation result is used to update the calculation results in the database.

11. The device according to claim 10, characterized in that The second event data and the third event data include the same primary key, and event data including different primary keys are stored in the target storage area; The processing module is used to obtain the original event data stored in the target storage area according to the primary key.

12. The device according to any one of claims 9 to 11, characterized in that The interaction module is further used to obtain fourth event data; The creation module is further configured to, in response to acquiring the fourth event data, create a fourth trigger for the fourth event data, the fourth trigger being configured to indicate a fourth window-outgoing time corresponding to the fourth event data, the time interval between the fourth window-outgoing time and the second window-outgoing time being greater than the delayed window-outgoing time corresponding to the second event data, and the fourth window-outgoing time being later than the second window-outgoing time; The processing module is also used to perform data calculation based on the event data in the third time window, and when the fourth exit time indicated by the fourth trigger is reached, perform data calculation based on the event data in the fourth time window, and the event data in the fourth time window does not include the second event data, the third event data and the fourth event data.

13. The device according to any one of claims 10 to 12, characterized in that The device also includes: The storage module is used to delete the second event data and the third event data in the target storage area after obtaining the third calculation result.

14. The device according to any one of claims 9 to 13, characterized in that The interaction module is further used for: Output configuration interface; In response to the user's configuration operation on the configuration interface for the delayed window exiting time length, the delayed window exiting time length corresponding to the second event data is obtained.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the computing device cluster executes the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on at least one computing device, enable the at least one computing device to perform the method according to any one of claims 1 to 7.

Images