Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the field of business data statistics, the method comprises the following steps: in the process of writing business data into an original data table from message middleware, continuously detecting the total quantity of the business data generated in a current time window in the original data table; judging whether the change condition of the total number meets a preset write-in completion condition or not; if the change condition of the total quantity meets a preset write-in completion condition, performing aggregation statistical processing on the business data generated in the current window period in the original data table; the method comprises the following steps: judging whether business data generated in a current time window is basically written into an original data table or not according to the change condition of the total quantity of the business data generated in the current time window in the original data table; and the business data generated in the current window period in the original data table is subjected to aggregation statistics processing, so that the reliability of aggregation statistics can be improved.

Description

Technical Field

[0001] The present invention relates to the field of business data statistics, and in particular to a data processing method, device, electronic equipment and storage medium. Background Art

[0002] In the related art, business data such as alarm data and log data can be collected by the front-end collection equipment, transmitted to the back-end through the message middleware, and aggregated and counted by the back-end to achieve the purpose of monitoring the alarm data and log data. In addition, users have certain requirements for the real-time nature of data, and need to count the group data of the original data in a short time after the original business data is generated. However, due to network fluctuations, message middleware anomalies, etc., the transmission of business data may have combinations, resulting in the original data generated in the current time window not entering the back-end storage space when the back-end performs aggregate statistics, which in turn causes the business data to not match the aggregated and grouped statistical data. Summary of the invention

[0003] The purpose of the present invention is to provide a data processing method, device, electronic device and storage medium, which can judge whether the business data generated in the current time window has been basically written into the original data table according to the change of the total amount of business data generated in the original data table in the current time window, and only when it is determined that the business data has been basically written into the original data table, the business data generated in the original data table in the current window period is aggregated and statistically processed, thereby improving the reliability of the aggregated statistics.

[0004] In order to solve the above technical problems, the present invention provides a data processing method, comprising:

[0005] In the process of writing the service data from the message middleware into the original data table, continuously detecting the total amount of service data generated in the current time window in the original data table;

[0006] Determine whether the change in the total quantity meets a preset write completion condition;

[0007] If the change in the total quantity meets the preset write completion condition, aggregate statistical processing is performed on the business data generated in the original data table during the current window period.

[0008] Optionally, it also includes:

[0009] In the process of writing the business data from the message middleware into the original data table, determining whether the current time has reached the latest aggregation time corresponding to the current time window;

[0010] If the current time reaches the latest aggregation time, aggregation statistics processing is performed on the business data generated in the current window period in the original data table.

[0011] Optionally, the determining whether the change in the total quantity satisfies a preset write completion condition includes:

[0012] Determining whether the total quantity increases;

[0013] If the total number does not increase, the no-change count is incremented by one; the initial value of the no-change count corresponding to the current time window is zero;

[0014] Determining whether the no-change count reaches a preset threshold;

[0015] If the no-change count reaches the preset threshold, it is determined that the change in the total number meets the preset write completion condition.

[0016] Optionally, before continuously detecting the total amount of business data generated in the current time window in the original data table, the method further includes:

[0017] Determine whether there is business data in the original data table;

[0018] If the service data is not stored in the original data table, waiting for a first preset time, and entering the step of determining whether the service data is stored in the original data table when the waiting is over;

[0019] If the business data is stored in the original data table, determining whether the message middleware is alive;

[0020] If the message middleware is not alive, wait for the first preset time, and enter the step of determining whether there is business data in the original data table when the waiting is over;

[0021] If the message middleware survives, the task identifier is set to the start state;

[0022] The continuously detecting in the original data table the total amount of business data generated in the current time window includes:

[0023] When it is determined that the task identifier is set to the start state, the total amount of business data generated in the current time window begins to be continuously detected in the original data table.

[0024] Optionally, when it is determined that the task identifier is set to the start state, starting to continuously detect the total amount of business data generated in the current time window in the original data table includes:

[0025] When it is determined that the task identifier is set to the start state, wait for a second preset time period, and start to continuously detect the total amount of business data generated in the current time window in the original data table when the waiting ends.

[0026] Optionally, the determining whether the message middleware is alive includes:

[0027] Whether the message middleware is alive is determined according to the heartbeat signal of the message middleware.

[0028] Optionally, after performing aggregate statistical processing on the service data generated in the original data table within the current window period, the method further includes:

[0029] When the aggregated statistical processing of the business data generated in the current window period is completed, the task identifier is adjusted to an end state.

[0030] The present invention also provides a data processing device, comprising:

[0031] A total quantity detection module is used to continuously detect the total quantity of business data generated in the current time window in the original data table during the process of writing the business data from the message middleware into the original data table;

[0032] A writing completion judgment module is used to judge whether the change of the total quantity meets the preset writing completion condition;

[0033] The aggregation statistics module is used to perform aggregation statistics processing on the business data generated in the original data table within the current window period if the change in the total quantity meets the preset writing completion condition.

[0034] The present invention also provides an electronic device, comprising:

[0035] Memory for storing computer programs;

[0036] A processor is used to implement the data processing method as described above when executing the computer program.

[0037] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the data processing method as described above is implemented.

[0038] The present invention provides a data processing method, comprising: in the process of writing business data from a message middleware into an original data table, continuously detecting the total amount of business data generated in a current time window in the original data table; judging whether a change in the total amount satisfies a preset write completion condition; if a change in the total amount satisfies the preset write completion condition, performing aggregate statistical processing on the business data generated in the original data table in the current window period.

[0039] It can be seen that in the process of writing business data from the message middleware to the original data table, the present invention can continuously detect the total number of business data generated in the current time window in the original data table, and then determine whether the change in the total number meets the preset write completion condition, that is, whether the business data generated in the current time window has been basically written into the original data table according to the change in the total number, such as when the total number does not change or the number of consecutive times that the total number does not change reaches a preset value, it is determined whether the business data generated in the current time window has been basically written into the original data table. Furthermore, if the change in the total number meets the preset write completion condition, the business data generated in the original data table during the current window period can be aggregated and statistically processed, so that the original data generated in the current time window does not enter the back-end storage space, resulting in the mismatch between the business data and the aggregated group statistics, thereby improving the reliability of data aggregation statistics. The present invention also provides a data processing device, an electronic device, and a storage medium, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;

[0042] Figure 2 A swimming lane schematic diagram of a data processing method provided by an embodiment of the present invention;

[0043] Figure 3 A structural block diagram of a data processing device provided by an embodiment of the present invention;

[0044] Figure 4 The present invention provides a structural block diagram of an electronic device. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In the related art, business data such as alarm data and log data can be collected by the front-end collection equipment, transmitted to the back-end through the message middleware, and aggregated and counted by the back-end to achieve the purpose of monitoring the alarm data and log data. In addition, users have certain requirements for the real-time nature of data, and need to count the group data of the original data in a short time after the original business data is generated. However, due to network fluctuations, message middleware anomalies, etc., the transmission of business data may have combinations, resulting in the original data generated in the current time window not entering the back-end storage space when the back-end performs aggregate statistics, which in turn causes the business data to not match the aggregated and grouped statistical data.

[0047] In view of this, in order to solve the technical problem of how to avoid the situation where business data is not written into the back-end storage space, which leads to the mismatch between business data and aggregated group statistics, the present invention can provide a data processing method, which can judge whether the business data generated in the current time window has been basically written into the original data table according to the change in the total number of business data generated in the original data table in the current time window, and only perform aggregated statistical processing on the business data generated in the original data table in the current window period when it is determined that the business data has been basically written into the original data table, thereby improving the reliability of aggregated statistics.

[0048] It should be noted that this embodiment does not limit the type of hardware device for executing this method, and it can be set according to actual application requirements, for example, it can be a personal computer, a server, etc.

[0049] For easier understanding, please refer to Figure 1 , Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention, the method may include:

[0050] S101 . In the process of writing business data from the message middleware into an original data table, continuously detecting the total amount of business data generated in the current time window in the original data table.

[0051] In this embodiment, the aggregate statistics of the service data are usually performed according to the time window, that is, each time a time window is reached, the service data generated in the time window is aggregated and statistically processed. This embodiment does not limit the specific length of the time window, for example, it can be 10 minutes, that is, every 10 minutes, the service data generated in the current 10 minutes is aggregated and statistically processed.

[0052] In order to avoid the situation where the business data generated by the backend in the current time window has not been completely written into the backend original data table, the present embodiment can first continuously detect the total amount of business data generated in the current time window in the original data table in the process of writing the business data from the message middleware to the original data table, so as to determine the growth change of the total amount. This is because even if it is affected by factors such as network fluctuations and message middleware blocking, the overall growth trend of the total amount should be gradually slowing down and eventually stop growing. When it is determined that the total amount has basically stopped growing, it can be determined that the business data generated in the current time window has been basically written into the backend original data table, and only a small amount of business data is still blocked in the message middleware, or there is no business data still blocked in the message middleware. Furthermore, performing aggregate statistical processing at this time can ensure that the business data and the aggregate statistical data basically match, thereby improving the reliability of aggregate statistics.

[0053] It should be noted that this embodiment is not limited to a specific message middleware, for example, it may be Kafka.

[0054] S102: Determine whether the change in the total quantity meets the preset write completion condition. If not, proceed to step S101; if satisfied, proceed to step S103.

[0055] In this embodiment, the preset write completion condition represents that the business data of the current time window has been basically written into the original data table. This embodiment does not limit the specific preset write completion condition, and it can be set according to the actual application requirements. For example, the preset write completion condition can be that the difference between the total quantities obtained by two adjacent detections is less than a preset value; the preset write completion condition can also be that the cumulative number of detections that detect that the difference between the adjacent total quantities is less than the preset value reaches a preset threshold;; the preset write completion condition can also be that the difference between the total quantities obtained by two adjacent detections is zero (that is, the total quantity does not increase); the preset write completion condition can also be that the cumulative number of detections that detect that the total quantity does not increase reaches a preset threshold. In this embodiment, in order to improve reliability, the preset write completion condition can be that the cumulative number of detections that detect that the total quantity does not increase reaches a preset threshold.

[0056] Based on this, the determining whether the change in the total number satisfies a preset write completion condition may include:

[0057] Step 11: Determine whether the total quantity increases.

[0058] It can be understood that step 11 is to determine whether the difference between the total quantities obtained from two adjacent detections is zero.

[0059] Step 12: If the total number does not increase, the no-change count is incremented by one; the initial value of the no-change count corresponding to the current time window is zero.

[0060] Step 13: Determine whether the no-change count reaches a preset threshold.

[0061] Step 14: If the no-change count reaches the preset threshold, it is determined that the change in the total number meets the preset write completion condition.

[0062] It should be noted that this embodiment does not limit the specific value of the preset threshold, which can be set according to actual application requirements. For example, it can be 3 times, that is, when it is determined that the cumulative number of detections without increasing the total number reaches 3 times, it is determined that the business data generated in the current time window has been basically written into the original data table of the back end.

[0063] S103: If the change in the total quantity meets the preset write completion condition, aggregate statistical processing is performed on the business data generated in the original data table during the current window period.

[0064] In this embodiment, after determining that the change in the total amount of business data generated within the current time window meets the above-mentioned preset write completion condition, the business data generated in the current window period in the original data table can be aggregated and statistically processed to ensure the reliability of the aggregated statistics.

[0065] Furthermore, when encountering serious network fluctuations or serious failures of the message middleware, the change in the above-mentioned total number may not always meet the preset completion writing conditions, which may lead to the problem that the aggregation statistical processing cannot be executed and affect the user experience. Therefore, for each time window, a corresponding latest aggregation time can also be set. When it is determined that the current time reaches the latest aggregation time, the business data generated in the original data table during the current window period can be forced to perform aggregation statistical processing. In this way, the timely execution of the aggregation statistical processing can be ensured. Of course, the aggregation statistical processing results obtained at this time may have large deviations. Therefore, for the aggregation statistical processing results that reach the latest aggregation time and force aggregation statistics, the corresponding prompt information can be output to remind relevant personnel to check the system failure in time.

[0066] Based on this, the method may further include:

[0067] Step 21: In the process of writing the business data from the message middleware into the original data table, it is determined whether the current time has reached the latest aggregation time corresponding to the current time window.

[0068] Step 22: If the current time reaches the latest aggregation time, aggregate statistical processing is performed on the business data generated in the current window period in the original data table.

[0069] Based on the above embodiment, in the process of writing business data from the message middleware to the original data table, the present invention can continuously detect the total amount of business data generated in the current time window in the original data table, and then determine whether the change in the total amount meets the preset write completion condition, that is, whether the business data generated in the current time window has been basically written into the original data table according to the change in the total amount, such as when the total amount does not change or the number of consecutive times that the total amount does not change reaches a preset value, it is determined whether the business data generated in the current time window has been basically written into the original data table. Furthermore, if the change in the total amount meets the preset write completion condition, the business data generated in the original data table during the current window period can be aggregated and statistically processed, so as to avoid the situation where the original data generated in the current time window does not enter the back-end storage space, resulting in the mismatch between the business data and the aggregated group statistics data, thereby improving the reliability of data aggregation statistics.

[0070] Based on the above embodiment, in order to improve the effectiveness of the detection of the total number of business data, this embodiment can also detect the storage of business data in the original data table and the working status of the message middleware before counting the total number, and only start counting the total number of business data when it is determined that the original data table has stored business data and the message middleware is working normally. Based on this, the method can also include:

[0071] S201, determining whether there is business data in the original data table.

[0072] S202: If the business data is not stored in the original data table, wait for a first preset time period, and enter the step of determining whether the business data is stored in the original data table when the waiting is over.

[0073] In this embodiment, first, it is possible to detect whether there is business data in the original data table. If there is no business data, the conditions for executing the aggregate statistics task are not met. At this time, a first preset time period may be waited, and when the waiting is over, it is determined again whether there is business data in the original data table.

[0074] It should be noted that this embodiment does not limit the specific value of the first preset time length, for example, it can be 5 minutes.

[0075] S203: If the business data is stored in the original data table, determine whether the message middleware is alive.

[0076] In this embodiment, if it is determined that there is business data in the original data table, it can be further determined whether the message middleware is alive. If the message middleware is not alive, it means that it cannot work normally and cannot move the business data from it to the original data table. At this time, you can wait for the first preset time, and judge again whether there is business data in the original data table at the end of the wait. If the message middleware is alive, it means that it can work normally. At this time, considering that there is business data in the original data table and the message middleware works normally, the conditions for executing the aggregation statistics task are met, so the statistics of the total number of business data can be started.

[0077] It should be noted that this embodiment does not limit how to determine whether the message middleware is alive. For example, the working status of the message middleware can be pulled; the heartbeat interface of the message middleware can also be checked, and whether the message middleware is alive can be determined based on the heartbeat signal of the message middleware (such as the heartbeat signal frequency).

[0078] Based on this, the determining whether the message middleware is alive may include:

[0079] Step 31: Determine whether the message middleware is alive according to the heartbeat signal of the message middleware.

[0080] S204, if the message middleware is not alive, waiting for the first preset time, and entering the step of determining whether there is business data in the original data table when the waiting is over;

[0081] S205: If the message middleware survives, setting the task identifier to a startup state;

[0082] S206: When it is determined that the task identifier is set to the start state, start to continuously detect the total amount of business data generated in the current time window in the original data table.

[0083] In this embodiment, only when it is determined that the task flag is set to the start state, the total amount of business data generated in the current time window is continuously detected in the original data table, thereby improving the statistical validity.

[0084] Furthermore, considering that the business data generated within the current time window requires a certain amount of time to be written from the message middleware to the original data table, and performing total business data statistics during this period of time is of little significance, when determining that the task identifier is set to the start state, you can further wait for a second preset time length, and at the end of the waiting period, start to continuously detect the total amount of business data generated within the current time window in the original data table.

[0085] Based on this, when it is determined that the task identifier is set to the start state, starting to continuously detect the total amount of business data generated in the current time window in the original data table may include:

[0086] Step 41: when it is determined that the task identifier is set to the start state, wait for a second preset time period, and when the waiting period ends, start to continuously detect the total amount of business data generated in the current time window in the original data table.

[0087] It should be noted that the present embodiment does not limit the specific value of the second preset time length, which can be set according to actual application requirements, for example, it can be 10 minutes.

[0088] S207: Determine whether the change in the total quantity meets a preset write completion condition.

[0089] S208. If the change in the total quantity meets the preset write completion condition, aggregate statistical processing is performed on the business data generated in the original data table during the current window period.

[0090] The description of steps S207 and S208 is consistent with the description of steps S102 and S103, and will not be repeated here.

[0091] S209: When the aggregated statistical processing of the business data generated in the current window period is completed, the task identifier is adjusted to an end state.

[0092] In this embodiment, when the aggregated statistical processing of the business data generated in the current window period is completed, the task identifier can be adjusted to the end state to avoid continuing the statistics.

[0093] Based on the above embodiment, the above data processing method is fully introduced below based on a specific schematic diagram. Figure 2 , Figure 2 A swimming lane diagram of a data processing method provided by an embodiment of the present invention. The overall description of the present invention is as follows:

[0094] Two scheduled tasks, the inspection task and the aggregation task, are used to control the timing of the aggregation group statistics, so as to perform the statistical operation within a certain delay time.

[0095] As shown in the figure above, the check of scheduled tasks has three functions. The first is to determine whether there is data in the original table space, so as to determine whether to perform subsequent aggregate statistical operations. The second is to check the heartbeat interface of the message middleware (Kafka), which also determines whether to perform subsequent statistical operations. The third is to determine whether to enable the aggregate scheduled task, which acts as a switch.

[0096] Aggregation scheduled tasks mainly focus on querying the original tablespace. The total number of the original tablespace is repeatedly queried according to the 10-minute time window, and the number of repetitions can be configured according to demand. In addition, the total number result of the previous query is compared with the total number result of the current query. At this time, if the comparison results are different, it means that there is still original data consumed from the message middleware during the 10-minute window period, and aggregate statistics cannot be performed. If the comparison results are equal and the configured number of comparisons is reached, the aggregate statistics operation is performed. At this time, the numerical deviation between the statistical data and the original data will be very small, as small as a single digit.

[0097] Of course, there are special cases. When the scheduled task is not executed on time due to various reasons, the aggregate scheduled task will first determine whether the current time is too far away from the time required for aggregation. If it exceeds the configured time interval, there is no need to perform subsequent comparison query result operations, and the statistics are directly aggregated.

[0098] The data processing device, electronic device, computer program product, and computer-readable storage medium provided in the embodiments of the present invention are introduced below. The data processing device, electronic device, computer program product, and computer-readable storage medium described below can be referenced to the data processing method described above.

[0099] Please refer to Figure 3 , Figure 3 A structural block diagram of a data processing device provided by an embodiment of the present invention, the device may include:

[0100] A total quantity detection module 301 is used to continuously detect the total quantity of business data generated in the current time window in the original data table during the process of writing the business data from the message middleware into the original data table;

[0101] A write completion judgment module 302 is used to judge whether the change of the total quantity meets a preset write completion condition;

[0102] The aggregation statistics module 303 is used to perform aggregation statistics processing on the business data generated in the original data table within the current window period if the change in the total quantity meets the preset write completion condition.

[0103] Optionally, the device may further include:

[0104] A timeout judgment module is used to judge whether the current time has reached the latest aggregation time corresponding to the current time window during the process of writing business data from the message middleware to the original data table; if the current time has reached the latest aggregation time, the business data generated in the original data table during the current window period is aggregated and statistically processed.

[0105] Optionally, the writing completion judgment module 302 includes:

[0106] A first judgment submodule is used to judge whether the total number increases; if the total number does not increase, the no-change count is incremented by one; the initial value of the no-change count corresponding to the current time window is zero;

[0107] The second judgment submodule is used to judge whether the no-change count reaches a preset threshold; if the no-change count reaches the preset threshold, it is determined that the change in the total number meets the preset write completion condition.

[0108] Optionally, the device may further include:

[0109] The original data table detection module is used to determine whether the business data is stored in the original data table; if the business data is not stored in the original data table, wait for a first preset time, and enter the step of determining whether the business data is stored in the original data table when the waiting is over;

[0110] A message middleware detection module, configured to determine whether the message middleware is alive if the service data is stored in the original data table; if the message middleware is not alive, wait for the first preset time period, and enter the step of determining whether the service data is stored in the original data table when the waiting is over;

[0111] An aggregation task startup module, used for setting the task identifier to a startup state if the message middleware is alive;

[0112] The total number detection module 301 includes:

[0113] The task start submodule is used to start continuously detecting the total amount of business data generated in the current time window in the original data table when determining that the task identifier is set to the start state.

[0114] Optionally, the task initiation submodule may be used to:

[0115] When it is determined that the task identifier is set to the start state, wait for a second preset time period, and start to continuously detect the total amount of business data generated in the current time window in the original data table when the waiting ends.

[0116] Optionally, the message middleware detection module may be used to:

[0117] Whether the message middleware is alive is determined according to the heartbeat signal of the message middleware.

[0118] Optionally, the device may further include:

[0119] The aggregation task ending module is used to adjust the task identifier to the ending state when the aggregation statistical processing of the business data generated in the current window period is completed.

[0120] Please refer to Figure 4 , Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention provides an electronic device 10, including a processor 11 and a memory 12; wherein the memory 12 is used to store a computer program; and the processor 11 is used to execute the data processing method provided in the aforementioned embodiment when executing the computer program.

[0121] For the specific process of the above data processing method, please refer to the corresponding content provided in the above embodiments, which will not be repeated here.

[0122] Furthermore, the memory 12 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0123] In addition, the electronic device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15 and a communication bus 16; wherein the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0124] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the data processing method described in the above embodiment is implemented.

[0125] Since the embodiments of the computer program product part correspond to the embodiments of the data processing method part, please refer to the description of the embodiments of the data processing method part for the embodiments of the computer program product part, and will not be repeated here.

[0126] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data processing method described in the above embodiment is implemented.

[0127] Since the embodiments of the computer-readable storage medium part correspond to the embodiments of the data processing method part, please refer to the description of the embodiments of the data processing method part for the embodiments of the storage medium part, which will not be repeated here.

[0128] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0129] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0130] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0131] The above is a detailed introduction to a data processing method, device, electronic device and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: In the process of writing the service data from the message middleware into the original data table, continuously detecting the total amount of service data generated in the current time window in the original data table; Determine whether the change in the total quantity meets a preset write completion condition; If the change in the total quantity meets the preset write completion condition, aggregate statistical processing is performed on the business data generated in the original data table during the current window period.

2. The data processing method according to claim 1, characterized in that: Also includes: In the process of writing the business data from the message middleware into the original data table, determining whether the current time has reached the latest aggregation time corresponding to the current time window; If the current time reaches the latest aggregation time, aggregation statistics processing is performed on the business data generated in the current window period in the original data table.

3. The data processing method according to claim 1, characterized in that: The determining whether the change in the total quantity satisfies a preset write completion condition includes: Determining whether the total quantity increases; If the total number does not increase, the no-change count is incremented by one; the initial value of the no-change count corresponding to the current time window is zero; Determining whether the no-change count reaches a preset threshold; If the no-change count reaches the preset threshold, it is determined that the change in the total number meets the preset write completion condition.

4. The data processing method according to any one of claims 1 to 3, characterized in that: Before continuously detecting the total amount of business data generated in the current time window in the original data table, the method further includes: Determine whether there is business data in the original data table; If the service data is not stored in the original data table, waiting for a first preset time, and entering the step of determining whether the service data is stored in the original data table when the waiting is over; If the business data is stored in the original data table, determining whether the message middleware is alive; If the message middleware is not alive, wait for the first preset time, and enter the step of determining whether there is business data in the original data table when the waiting is over; If the message middleware survives, the task identifier is set to the start state; The continuously detecting in the original data table the total amount of business data generated in the current time window includes: When it is determined that the task identifier is set to the start state, the total amount of business data generated in the current time window begins to be continuously detected in the original data table.

5. The data processing method according to claim 4, characterized in that: When it is determined that the task identifier is set to the start state, starting to continuously detect the total amount of business data generated in the current time window in the original data table includes: When it is determined that the task identifier is set to the start state, wait for a second preset time period, and start to continuously detect the total amount of business data generated in the current time window in the original data table when the waiting ends.

6. The data processing method according to claim 4, characterized in that: The determining whether the message middleware is alive includes: Whether the message middleware is alive is determined according to the heartbeat signal of the message middleware.

7. The data processing method according to claim 4, characterized in that: After performing aggregate statistical processing on the business data generated in the current window period in the original data table, the method further includes: When the aggregated statistical processing of the business data generated in the current window period is completed, the task identifier is adjusted to an end state.

8. A data processing device, characterized in that: include: A total quantity detection module is used to continuously detect the total quantity of business data generated in the current time window in the original data table during the process of writing the business data from the message middleware into the original data table; A writing completion judgment module is used to judge whether the change of the total quantity meets the preset writing completion condition; The aggregation statistics module is used to perform aggregation statistics processing on the business data generated in the original data table within the current window period if the change in the total quantity meets the preset writing completion condition.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images