Statistical data cleaning method, device, equipment and storage medium

By cross-storing and cleaning the statistical data of stateless objects between the main statistical table and the secondary statistical table, the problem of difficult to clean up stateless object data statistics is solved, and the accuracy of reducing operation complexity and retaining statistical data is achieved.

CN112000668BActive Publication Date: 2024-11-12GUANGZHOU BAIGUOYUAN INFORMATION TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202010808406.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-12
Publication Date
2024-11-12
Estimated Expiration
2040-08-12

AI Technical Summary

Technical Problem

The existing statistical data cleaning scheme is difficult to determine the timing of cleaning for stateless objects, which leads to data bloating or affects the accuracy of statistical items. Especially the statistical data cleaning method of stateless protocols such as UDP protocol has an impact on the use of statistical items. Issues of accuracy and service lag.

Method used

Using a combination of the main statistical table and the secondary statistical table, the statistical entries in the secondary statistical table are taken out when detecting the cleaning event, and the statistical item in the main statistical table is judged whether the statistical item exists, and the judgment results are cleared to avoid the entire statistics. Tables operate, reducing complexity and preserving real-time statistics.

Benefits of technology

It effectively reduces the complexity of statistical operations and the complexity of clear operations, avoids service lag, and reasonably retains instant statistics to avoid affecting the accuracy of the statistics in use.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112000668B_ABST
    Figure CN112000668B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a statistical data cleaning method, device, equipment and storage medium. The method is implemented based on a main statistical table and a secondary statistical table, and includes: detecting that a cleaning event of statistical data is triggered, wherein between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted in the main statistical table, the secondary statistical table remains unchanged, the first statistical entry in the secondary statistical table is taken out, wherein the first statistical entry contains a first statistical item and a first statistical value corresponding to the first statistical item, judging whether there is a statistical entry containing the first statistical item in the main statistical table, and performing corresponding clearing processing on the first statistical item according to the judgment result. The technical solution provided by the embodiment of the present invention can reduce the complexity of the cleaning operation, reasonably and effectively retain the real-time statistical data, and avoid affecting the accuracy of the statistical items being used.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to statistical data cleaning methods, devices, equipment and storage media. Background Art

[0002] Currently, for many devices or applications, background services often need to collect statistics on various types of data in order to perform targeted optimization on the devices or applications.

[0003] The data statistics process generally exists for a long time. As time goes by, some obsolete or no longer needed statistical resources need to be cleaned up in time to avoid data expansion.

[0004] For data statistics that rely on specific stateful objects (generally with independent storage capabilities and clear life cycles), it is generally only necessary to clean up related resources based on whether the object's life cycle has ended. For example, for a Transmission Control Protocol (TCP) connection, the statistics related to the connection can be cleaned up when the connection is disconnected.

[0005] For data statistics of stateless objects, which are opposite to stateful objects, it is difficult to determine the clearing time because there is no clear data life cycle. For example, the common User Datagram Protocol (UDP) is a stateless transport layer protocol. When the high-level application layer performs statistics on UDP (such as sending and receiving packets, etc.), it is naturally impossible to determine when to clean up the data based on the state of UDP. For data statistics of stateless objects, the existing statistical data cleaning methods generally include the following two methods: the first method is to directly clean up all statistical data in the entire statistical table when the set time is reached or the statistical data volume reaches the set threshold. This is a common practice used by the media background service when counting UDP sending and receiving packets. However, the method of deleting all will affect the accuracy of the statistical items being used; the second method is to add an access time field to the statistical item. When data is modified to the statistical item, the access time is set to the current time, and the entire statistical data table is scanned regularly to find the timed access time and delete the corresponding statistical data. However, scanning the hash table data at one time is prone to service jams.

[0006] Therefore, the existing statistical data cleaning solutions are still not perfect and need to be improved. Summary of the invention

[0007] The embodiments of the present invention provide a statistical data cleaning method, device, equipment and storage medium, which can.

[0008] In a first aspect, an embodiment of the present invention provides a statistical data cleaning method, which is implemented based on a main statistical table and a secondary statistical table. The method includes:

[0009] It is detected that a cleaning event of statistical data is triggered, wherein between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted into the main statistical table, and the secondary statistical table remains unchanged;

[0010] Taking out a first statistical entry from the secondary statistical table, wherein the first statistical entry includes a first statistical item and a first statistical value corresponding to the first statistical item;

[0011] It is determined whether there is a statistical entry containing the first statistical item in the main statistical table, and the first statistical entry is cleared accordingly according to the determination result.

[0012] In a second aspect, an embodiment of the present invention provides a statistical data cleaning device, which is implemented based on a main statistical table and a secondary statistical table, and the device includes:

[0013] A cleaning event trigger detection module is used to detect whether a cleaning event of statistical data is triggered, wherein between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted into the main statistical table, and the secondary statistical table remains unchanged;

[0014] A statistical entry fetching module, configured to fetch a first statistical entry from the secondary statistical table when detecting that a statistical data clearing event is triggered, wherein the first statistical entry includes a first statistical item and a first statistical value corresponding to the first statistical item;

[0015] The clearing processing module is used to determine whether there is a statistical entry containing the first statistical item in the main statistical table, and to perform corresponding clearing processing on the first statistical entry according to the determination result.

[0016] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a statistical data cleaning method as provided in an embodiment of the present invention is implemented.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a statistical data cleaning method as provided in an embodiment of the present invention.

[0018] The statistical data cleaning scheme provided in the embodiment of the present invention is implemented based on the main statistical table and the secondary statistical table. When it is detected that the cleaning event of the statistical data is triggered, the first statistical entry in the secondary statistical table is taken out, and it is determined whether there is a statistical entry containing the first statistical item in the main statistical table, and the first statistical entry is cleared accordingly according to the judgment result. Among them, between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted in the main statistical table, the secondary statistical table remains unchanged, and the first statistical entry contains the first statistical item and the first statistical value corresponding to the first statistical item. By adopting the above technical solution, a main statistical table and a sub-statistical table are used to store statistical data respectively. After a cleanup event is triggered and before the next cleanup event is triggered, the original data to be counted is counted in the main statistical table, while the sub-statistical table remains unchanged, that is, the statistical values ​​corresponding to each statistical item in the main statistical table are dynamically updated, which can be considered as real-time statistical data, while the sub-statistical table stores relatively old statistical data. Compared with the solution in the prior art that only a single complete statistical table exists, when performing statistical operations, it is possible to avoid operating the entire statistical table, thereby reducing the complexity of statistical operations. At the same time, according to the overlap of statistical items in the main and sub-statistical tables, only the old statistical data in the sub-statistical table is cleared, which not only reduces the complexity of the clearing operation, but also can reasonably and effectively retain the real-time statistical data to avoid affecting the accuracy of the statistical items being used. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a statistical data cleaning method provided by an embodiment of the present invention;

[0020] Figure 2 A flowchart of another statistical data cleaning method provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of a flow chart of another statistical data cleaning method provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of a statistical data cleaning process provided by an embodiment of the present invention;

[0023] Figure 5 A structural block diagram of a statistical data cleaning device provided by an embodiment of the present invention;

[0024] Figure 6 A structural block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0026] Figure 1 The present invention provides a flow chart of a statistical data cleaning method, which can be performed by a statistical data cleaning device, wherein the device can be implemented by software and / or hardware, and can generally be integrated into a computer device. Figure 1 As shown, the method includes:

[0027] Step 101: Detect that a cleanup event of statistical data is triggered, wherein between two adjacent detections of the cleanup event being triggered, the original data to be counted is counted into a main statistical table, and the secondary statistical table remains unchanged.

[0028] Exemplarily, a main statistical table and a secondary statistical table may be set, and the specific form of the statistical table is not limited, for example, it may be a hash table. In the main statistical table and the secondary statistical table, statistical entries may be stored, and a statistical entry may include a statistical item and a corresponding statistical value. Taking the hash table as an example, a statistical entry corresponds to a key-value pair, the statistical item is the key, and the statistical value is the value. The statistical items may include statistical items corresponding to stateless data, such as the number of packets received or sent based on the UDP protocol, and may also include statistical items corresponding to stateful data, such as the number of packets received or sent based on the TCP protocol. Of course, the statistical items may also include other items, which may be set according to actual needs.

[0029] Exemplarily, the triggering conditions of the statistical data cleanup event can be set according to actual needs. For example, it can be triggered periodically according to a fixed period, or it can be triggered at unequal intervals according to actual conditions (that is,), and the occurrence of other situations that require statistical data cleanup (such as the storage space corresponding to the main statistical table and the secondary statistical table reaching or exceeding the preset space threshold, etc.) can also be used as a triggering condition.

[0030] In the embodiment of the present invention, between two adjacent detections of the cleaning event being triggered (for the sake of convenience, this period of time will be referred to as the cleaning period hereinafter, and it is understandable that the cleaning period may be fixed or dynamically changing), the original data to be counted (such as the data acquired or generated in real time by the bottom layer, taking the number of packets sent based on the UDP protocol as an example, it may be that the bottom layer detects that a UDP data packet is currently sent) is counted in the main statistical table, and the secondary statistical table remains unchanged, that is, the statistical values ​​corresponding to each statistical item in the main statistical table are dynamically updated and can be considered as real-time data, while the secondary statistical table stores relatively old statistical data that are about to or may time out. In addition, since the statistical items in the main statistical table and the secondary statistical table are generally not completely consistent, when the original data to be counted is counted in the main statistical table, compared with the solution in the prior art where only a single complete statistical table exists, it is not necessary to traverse all statistical items, but only to search the statistical items corresponding to the original data to be counted in the main statistical table, thereby improving data statistical efficiency.

[0031] Step 102: Take out a first statistical entry from the secondary statistical table, wherein the first statistical entry includes a first statistical item and a first statistical value corresponding to the first statistical item.

[0032] Exemplarily, the first statistical item may include any one or more statistical items in the secondary statistical table, and the specific number is not limited. The statistical items may be taken out individually or in batches in sequence according to the order in which the statistical items are sorted in the secondary statistical table. The sorting rules of the statistical items in the secondary statistical table can be freely set. For example, the statistical items may be sorted in the order in which they are created, or they may be sorted according to the latest update time of the statistical values ​​corresponding to the statistical items. After the first statistical item is taken out from the secondary statistical table, the first statistical item no longer exists in the secondary statistical table, which is equivalent to deleting the first statistical item from the secondary statistical table. The taken out first statistical item may be immediately cleared or placed in a cache or other location to wait for subsequent clearing.

[0033] Step 103: determine whether there is a statistical entry containing the first statistical item in the main statistical table, and perform corresponding clearing processing on the first statistical entry according to the determination result.

[0034] Exemplarily, since new raw data to be counted is only counted in the main statistical table between two cleaning events being triggered, the overlap of statistical items in the main statistical table and the secondary statistical table can reflect the newly added raw data to be counted of a statistical item in the past period of time, for example, whether there is corresponding newly added raw data to be counted, whether the statistical item is a newly added statistical item, etc. When it is necessary to clear the first statistical entry containing the first statistical item, targeted clearing can be performed based on whether the first statistical item is included in the main statistical table.

[0035] The statistical data cleaning method provided in the embodiment of the present invention is implemented based on the main statistical table and the secondary statistical table. When it is detected that the cleaning event of the statistical data is triggered, the first statistical entry in the secondary statistical table is taken out, and it is determined whether there is a statistical entry containing the first statistical item in the main statistical table, and the first statistical entry is cleared accordingly according to the judgment result. Among them, between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted in the main statistical table, the secondary statistical table remains unchanged, and the first statistical entry contains the first statistical item and the first statistical value corresponding to the first statistical item. By adopting the above technical solution, the main statistical table and the sub-statistical table are used to store statistical data respectively. After one cleaning event is triggered and before the next cleaning event is triggered, the original data to be counted is counted in the main statistical table, and the sub-statistical table remains unchanged, that is, the statistical values ​​corresponding to each statistical item in the main statistical table are dynamically updated, which can be considered as real-time statistical data, while the sub-statistical table stores relatively old statistical data. Compared with the solution in the prior art that only a single complete statistical table exists, when performing statistical operations, it is possible to avoid operating the entire statistical table, reduce the complexity of statistical operations, and avoid causing service jams. At the same time, according to the overlap of statistical items in the main and sub-statistical tables, only the old statistical data in the sub-statistical table are cleared, which not only reduces the complexity of the clearing operation and avoids causing service jams, but also can reasonably and effectively retain real-time statistical data to avoid affecting the accuracy of the statistical items being used.

[0036] In some embodiments, the clearing of the first statistical item according to the judgment result includes: if it is judged that the statistical item containing the first statistical item does not exist in the main statistical table, the first statistical item is deleted. The advantage of such a setting is that when the currently retrieved statistical item does not exist in the main statistical table, it means that in the last cleaning cycle, the statistical item did not add any new raw data that needs to be counted, and the statistical item and the corresponding statistical value are relatively old and may no longer need to be saved, so the first statistical item can be directly deleted.

[0037] In some embodiments, the corresponding clearing process of the first statistical item according to the judgment result includes: if it is judged that there is a second statistical item containing the first statistical item in the main statistical table, the first statistical value is added to the second statistical value in the second statistical item to achieve statistical update of the first statistical item. The advantage of such a setting is that when there is a currently taken statistical item in the main statistical table, it means that in the last cleaning cycle, the statistical item has added raw data that needs to be counted, then the first statistical value in the secondary statistical table needs to be added to the second statistical value of the corresponding statistical item in the main statistical table, so as to ensure the real-time accuracy of the statistical value corresponding to the first statistical item, and at the same time, the effect of clearing the first statistical item in the secondary statistical table is achieved.

[0038] In some embodiments, after taking out the first statistical item from the secondary statistical table, the method further includes: taking out the remaining third statistical items from the secondary statistical table in turn, and performing relevant operations to determine whether there are statistical items containing the third statistical item in the main statistical table, until the number of statistical items taken out after the current cleaning event is triggered reaches a preset number threshold or the secondary statistical table is cleared, wherein the third statistical item contains a third statistical item and a third statistical value corresponding to the third statistical item. The advantage of such a setting is that by setting a preset number threshold, the statistical items in the secondary statistical table can be cleaned up in batches, that is, multiple rounds of cleaning, and the maximum number of single items (i.e., one statistical item) processed in each round of cleaning operation can be limited, thereby avoiding clearing all the statistical items in the secondary statistical table at once and avoiding program jams.

[0039] The number of the third statistical items is not limited and can be determined according to the size of the preset number threshold. Generally, assuming that the preset number threshold is N, the number of the third statistical items is N-1. For example, there are 100 statistical items in the secondary statistical table, among which the statistical item ranked first can be regarded as the first statistical item. If the preset number threshold is 10, the 9 statistical items ranked second to tenth can all be regarded as the third statistical items.

[0040] In some embodiments, the method may further include: if it is detected that after the fourth statistical entry in the secondary statistical table is taken out, the number of the statistical entries that have been taken out reaches the preset number threshold, then when the continuing cleaning instruction corresponding to this cleaning event is received, the next statistical entry of the fourth statistical entry is taken out, and the relevant operation of judging whether there is a statistical entry containing the next statistical item in the main statistical table is performed, wherein the next statistical item is included in the next statistical entry. The advantage of such a setting is that if the statistical entries that have been cleaned up in this round of cleaning operation have reached the upper limit, the cleaning operation of subsequent statistical entries can be continued when the continuing cleaning instruction corresponding to this cleaning event is received. It should be noted that the fourth statistical entry here can be considered as the last third statistical entry in the previous round of cleaning operation. For example, the statistical entry ranked tenth can be regarded as the fourth statistical entry, and the statistical entry ranked eleventh can be regarded as the next statistical entry of the fourth statistical entry. Among them, the generation interval of the continuing cleaning instruction can be set according to the actual situation, and generally can be set to a time sufficient to clean up the statistical entries of the preset number threshold. Optionally, the generation of the continuing cleaning instruction can be triggered by a clock signal.

[0041] It is understandable that multiple rounds of cleaning operations can be used to empty the secondary statistical table. After the secondary statistical table is emptied, the statistical items in the main statistical table can be put into the secondary statistical table to become older statistical data, while the main statistical table is empty and the statistics of the original data are continuously added.

[0042] In some embodiments, the method may further include: if it is detected that the secondary statistical table is cleared, swapping the identities of the primary statistical table and the secondary statistical table. The advantage of such a setting is that performance loss caused by data copying can be avoided.

[0043] In some embodiments, the detection of a statistical data cleanup event being triggered includes: determining that the cleanup event is triggered when the time interval between the current moment and the last time the statistical data cleanup event was triggered reaches a preset time threshold. The advantage of such a setting is that the triggering conditions of the cleanup event can be reasonably set, and the preset time threshold is used to limit the shortest timeout time, that is, after a cleanup cycle, it can be considered that the data in the secondary statistical table has existed for at least the preset time threshold, and it can be considered that the timeout time has been reached, so the cleanup process can be performed.

[0044] In some embodiments, it may also include: when it is determined that the frequency of the statistical operation of adding the raw data to be counted to the main statistical table within a preset time period is lower than a preset frequency threshold, when it is detected that the fifth statistical item in the fifth statistical entry in the main statistical table has the corresponding raw data to be counted, if there is a sixth statistical value corresponding to the fifth statistical item in the secondary statistical table, then the sixth statistical value is added to the fifth statistical value in the fifth statistical entry to achieve statistical update of the fifth statistical item. The advantage of such a setting is that the statistical operation of adding the raw data to be counted to the main statistical table can be considered as a write operation. If the write operation is relatively small, the statistical value synchronization operation of a single statistical item can be performed during the data input process, that is, during the process of adding the raw data to be counted to the main statistical table, which can ensure that the data synchronization process is faster and more efficient without causing program jams.

[0045] In some embodiments, the method may further include: when it is detected that a target statistical value acquisition event corresponding to a target statistical item is triggered, reading a first target statistical value corresponding to the target statistical item in the main statistical table, and reading a second target statistical value corresponding to the target statistical item in the secondary statistical table; calculating the sum of the first target statistical value and the second target statistical value to obtain the target statistical value. The advantage of such a setting is that the execution timing of the statistical value acquisition operation (which can be considered as a read operation) may be uncertain, so the target statistical item may exist in both the main statistical table and the secondary statistical table at the same time, and the corresponding statistical values ​​in the main statistical table and the secondary statistical table are read and merged at the same time, so that an accurate target statistical value can be obtained.

[0046] Figure 2 A flow chart of another statistical data cleaning method provided by an embodiment of the present invention, taking the primary and secondary statistical tables as hash tables as an example, Figure 2 As shown, the method may include:

[0047] Step 201, detect whether the time interval between the current moment and the last time a statistical data clearing event was triggered reaches a preset time threshold, if so, execute step 202; otherwise, repeat step 201.

[0048] Exemplarily, the preset duration threshold can be understood as a cleaning cycle, that is, a timeout period. A field can be added to record the timestamp when the last cleaning event (which can be considered as the last completion of data synchronization between the primary and secondary hash tables) was completed, for example, it can be recorded as a synchronization completion timestamp (or simply synchronization timestamp). In this step, it can be determined whether the time interval between the current moment and the synchronization completion timestamp reaches the preset duration threshold.

[0049] Wherein, between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted in the main statistical table, and the secondary statistical table remains unchanged.

[0050] Step 202: Take out the current statistical entry in the secondary statistical table, wherein the current statistical entry includes the current statistical item and a first statistical value corresponding to the current statistical item.

[0051] Exemplarily, when this step is performed for the first time, the current statistical entry may be the first statistical entry in the secondary statistical table. When this step is performed again, the current statistical entry may be the next statistical entry.

[0052] Step 203 , determine whether there is a statistical entry containing the current statistical item in the main statistical table, if so, execute step 204 ; otherwise, execute step 205 .

[0053] Step 204 : Add the first statistical value to the second statistical value corresponding to the current statistical item in the main statistical table to implement statistical update of the current statistical item, and then execute step 206 .

[0054] Step 205: Delete the current statistical entry and execute step 206.

[0055] Step 206 , determine whether the secondary statistical table is cleared, if so, return to step 209 ; otherwise, execute step 207 .

[0056] Step 207 , determine whether the number of statistical entries retrieved in this round of cleaning operation reaches a preset number threshold. If so, execute step 208 ; otherwise, return to execute step 202 .

[0057] Step 208 , determine whether a continue cleaning instruction corresponding to the current cleaning event is received, if so, return to step 202 ; otherwise, repeat step 208 .

[0058] The continue cleaning instruction may be generated periodically by a clock signal.

[0059] Step 209: swap the identities of the primary statistical table and the secondary statistical table, and return to step 201.

[0060] Optionally, in this step, the synchronization completion timestamp may be updated to the current timestamp.

[0061] Exemplarily, in the above process, if newly added raw data to be counted is detected, the raw data is counted in the corresponding statistical item in the main statistical table. If it is detected that the target statistical value acquisition event corresponding to the target statistical item is triggered, the first target statistical value corresponding to the target statistical item in the main statistical table is read, and the second target statistical value corresponding to the target statistical item in the secondary statistical table is read, and the sum of the first target statistical value and the second target statistical value is calculated to obtain the target statistical value.

[0062] The statistical data cleaning method provided by the embodiment of the present invention is implemented based on the main statistical table and the secondary statistical table, that is, a double buffer mechanism is used to store and synchronize data, one buffer stores real-time data, and the other stores data that is about to or has timed out. The entire processing and cleaning process is an operation with a complexity of O(n), but the idea of ​​block processing is used to process the data in blocks and time periods to avoid program jams caused by one-time processing, that is, the time is spread over multiple clock drive cycles. Since each round of data synchronization and cleaning process has a limited number of processes, jams caused by full data cleaning judgment can be avoided. When data is input, only the data in the main statistical table needs to be modified. When data is output, only the data in the main and secondary statistical tables need to be read and summed. Both reading and writing are operations with a complexity of O(1), which is suitable for situations where statistical performance requirements are very high. In addition, data transfer is performed by switching the identities of the main and secondary statistical tables instead of copying data, which can avoid performance loss caused by copying.

[0063] Figure 3 The flowchart of another statistical data cleaning method provided by the embodiment of the present invention is as follows, assuming that the main statistical table is a main hash table and the secondary statistical table is a secondary hash table. Figure 3 As shown, wait for the arrival of the clock signal. When the clock signal is detected, first determine whether the synchronization timestamp has timed out, that is, determine whether the time interval between the current moment and the synchronization timestamp reaches the preset time threshold. If so, perform this round of cleaning operations, otherwise continue to wait for the next clock signal. When performing this round of cleaning operations, set the current round count to 0, determine whether the current round count is less than the maximum number of processing in a single round (preset number threshold), if less than, continue to determine whether the secondary hash table is non-empty, if not empty, perform single-item data synchronization, that is, perform corresponding cleaning operations for the current statistical entry, then add one to the current round count, continue to determine whether the current round count is less than the maximum number of processing in a single round, if not less than, continue to determine whether the secondary hash table is non-empty, if not empty, wait for the next clock signal to perform the next round of cleaning operations, if not non-empty, that is, the secondary hash table is empty, exchange the identities of the primary and secondary hash tables, set the synchronization timestamp to the current timestamp, and continue to wait for the clock signal.

[0064] In order to better understand the embodiments of the present invention, the following examples are given. Figure 4A schematic diagram of a statistical data cleaning process provided by an embodiment of the present invention. Figure 4 As shown, the main statistical table is the main hash table, and the secondary statistical table is the secondary hash table. At the beginning, both the main hash table and the secondary hash table are empty, and the cleaning cycle is recorded as T. The operation cycle corresponding to each round of cleaning operation in a cleaning cycle is t. In the first cleaning cycle T1, 4 statistical items are counted in the main hash table, namely A, B, C and D, and the corresponding statistical values ​​are a1, b1, c1 and d1. After the cleaning event is triggered, since the secondary hash table is empty, the main and secondary hash tables exchange identities. In the second cleaning cycle T2, 2 statistical items are counted in the main hash table, namely A and C, and the corresponding statistical values ​​are a2 and c2. After the cleaning event is triggered (entering the third cleaning cycle T3), the statistical item A-a1 is taken out, and it is found that A exists in the main hash table, then a1 is accumulated to a2, and then the statistical item B-b1 is taken out, and it is found that B does not exist in the main hash table, that is, in the previous cleaning cycle T2, B does not have new raw data to be counted, so b1 becomes a relatively old statistical data and can be cleared. Assuming that the preset number threshold is 2, you need to wait for the continue cleaning instruction. After receiving the continue cleaning instruction, continue to clean the two entries C-c1 and D-d1, and the secondary hash table can be cleared after 2 t times. During the waiting process, if there is new raw data to be counted, it will be counted in the main hash table. If it is detected that the statistical value acquisition event for statistical item C is triggered, read c2 in the main hash table and c1 in the secondary hash table, and return the sum of the two as the final statistical value to the object that needs to read the statistical value.

[0065] Figure 5 This is a structural block diagram of a statistical data cleaning device provided by an embodiment of the present invention. The device can be implemented by software and / or hardware, and can generally be integrated in a computer device. The statistical data cleaning method can be executed to clean up the statistical data. Figure 5 As shown, the device is implemented based on the main statistical table and the auxiliary statistical table, including:

[0066] A cleaning event trigger detection module 501 is used to detect whether a cleaning event of statistical data is triggered, wherein between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted into the main statistical table, and the secondary statistical table remains unchanged;

[0067] A statistical entry fetching module 502, configured to fetch a first statistical entry from the secondary statistical table when it is detected that a statistical data clearing event is triggered, wherein the first statistical entry includes a first statistical item and a first statistical value corresponding to the first statistical item;

[0068] The clearing processing module 503 is used to determine whether there is a statistical entry containing the first statistical item in the main statistical table, and to perform corresponding clearing processing on the first statistical entry according to the determination result.

[0069] The statistical data cleaning device provided in the embodiment of the present invention is implemented based on the main statistical table and the sub-statistical table. When it is detected that the cleaning event of the statistical data is triggered, the first statistical entry in the sub-statistical table is taken out, and it is determined whether there is a statistical entry containing the first statistical item in the main statistical table, and the first statistical entry is cleared accordingly according to the judgment result. Among them, between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted in the main statistical table, the sub-statistical table remains unchanged, and the first statistical entry contains the first statistical item and the first statistical value corresponding to the first statistical item. By adopting the above technical solution, a main statistical table and a sub-statistical table are used to store statistical data respectively. After a cleanup event is triggered and before the next cleanup event is triggered, the original data to be counted is counted in the main statistical table, while the sub-statistical table remains unchanged, that is, the statistical values ​​corresponding to each statistical item in the main statistical table are dynamically updated, which can be considered as real-time statistical data, while the sub-statistical table stores relatively old statistical data. Compared with the solution in the prior art that only a single complete statistical table exists, when performing statistical operations, it is possible to avoid operating the entire statistical table, thereby reducing the complexity of statistical operations. At the same time, according to the overlap of statistical items in the main and sub-statistical tables, only the old statistical data in the sub-statistical table is cleared, which not only reduces the complexity of the clearing operation, but also can reasonably and effectively retain the real-time statistical data to avoid affecting the accuracy of the statistical items being used.

[0070] An embodiment of the present invention provides a computer device, in which the statistical data cleaning device provided by the embodiment of the present invention can be integrated. Figure 6 The computer device 600 includes a memory 601, a processor 602, and a computer program stored in the memory 601 and executable on the processor 602. When the processor 602 executes the computer program, the statistical data cleaning method provided in the embodiment of the present invention is implemented.

[0071] An embodiment of the present invention further provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to execute the statistical data cleaning method provided by the embodiment of the present invention.

[0072] The statistical data cleaning device, equipment and storage medium provided in the above embodiments can execute the statistical data cleaning method provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of executing the method. For technical details not described in detail in the above embodiments, please refer to the statistical data cleaning method provided in any embodiment of the present invention.

[0073] Note that the above are only preferred embodiments of the present invention. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the claims.

Claims

1. A statistical data cleaning method, characterized in that: Based on the main statistical table and the secondary statistical table, the method includes: It is detected that a cleaning event of statistical data is triggered, wherein between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted into the main statistical table, and the secondary statistical table remains unchanged; Taking out a first statistical entry from the secondary statistical table, wherein the first statistical entry includes a first statistical item and a first statistical value corresponding to the first statistical item; the first statistical entry includes any one or more statistical items in the secondary statistical table; after the first statistical entry is taken out from the secondary statistical table, the first statistical entry no longer exists in the secondary statistical table; Determine whether there is a statistical entry containing the first statistical item in the main statistical table, and perform corresponding clearing processing on the first statistical entry according to the determination result; After taking out the first statistical entry from the secondary statistical table, the method further includes: The remaining third statistical items in the secondary statistical table are sequentially taken out, and relevant operations of determining whether there are statistical items containing the third statistical items in the primary statistical table are performed, until the number of statistical items taken out after the current cleaning event is triggered reaches a preset number threshold or the secondary statistical table is cleared, wherein the third statistical items contain the third statistical items and the third statistical value corresponding to the third statistical items; When it is detected that a target statistical value acquisition event corresponding to a target statistical item is triggered, a first target statistical value corresponding to the target statistical item in the primary statistical table is read, and a second target statistical value corresponding to the target statistical item in the secondary statistical table is read; The sum of the first target statistical value and the second target statistical value is calculated to obtain the target statistical value.

2. The method according to claim 1, characterized in that The performing corresponding clearing processing on the first statistical item according to the judgment result includes: If it is determined that there is no statistical entry containing the first statistical item in the main statistical table, the first statistical entry is deleted.

3. The method according to claim 1, characterized in that The performing corresponding clearing processing on the first statistical item according to the judgment result includes: If it is determined that there is a second statistical entry containing the first statistical item in the main statistical table, the first statistical value is added to the second statistical value in the second statistical entry to implement statistical update of the first statistical item.

4. The method according to claim 1, characterized in that: Also includes: If it is detected that after the fourth statistical entry in the secondary statistical table is taken out, the number of the statistical entries that have been taken out reaches the preset number threshold, then when a continue cleaning instruction corresponding to this cleaning event is received, the next statistical entry of the fourth statistical entry is taken out, and relevant operations are performed to determine whether there is a statistical entry containing the next statistical item in the main statistical table, wherein the next statistical item is included in the next statistical entry.

5. The method according to claim 1, characterized in that Also includes: If it is detected that the secondary statistical table is cleared, the identities of the main statistical table and the secondary statistical table are swapped.

6. The method according to claim 1, characterized in that The detection of a statistical data cleanup event being triggered includes: When the time interval between the current moment and the last time a cleaning event of statistical data was triggered reaches a preset time threshold, it is determined that the cleaning event is triggered.

7. The method according to claim 1, characterized in that Also includes: When it is determined that the frequency of the statistical operation of counting the original data to be counted into the main statistical table within a preset time period is lower than a preset frequency threshold, when it is detected that the fifth statistical item in the fifth statistical entry in the main statistical table has corresponding original data to be counted, if there is a sixth statistical value corresponding to the fifth statistical item in the secondary statistical table, the sixth statistical value is added to the fifth statistical value in the fifth statistical entry to achieve statistical update of the fifth statistical item.

8. A statistical data cleaning device, characterized in that: Based on the main statistical table and the secondary statistical table, the device includes: A cleaning event trigger detection module is used to detect whether a cleaning event of statistical data is triggered, wherein between two adjacent detections of the cleaning event being triggered, the original data to be counted is counted into the main statistical table, and the secondary statistical table remains unchanged; A statistical entry fetching module is used to fetch a first statistical entry from the secondary statistical table when detecting that a statistical data cleanup event is triggered, wherein the first statistical entry includes a first statistical item and a first statistical value corresponding to the first statistical item; the first statistical item includes any one or more statistical items in the secondary statistical table; after the first statistical item is fetched from the secondary statistical table, the first statistical item no longer exists in the secondary statistical table; After taking out the first statistical item from the secondary statistical table, the method is further used to: sequentially take out the remaining third statistical items from the secondary statistical table, and perform relevant operations of determining whether there are statistical items containing the third statistical items in the primary statistical table, until the number of statistical items taken out after the current cleaning event is triggered reaches a preset number threshold or the secondary statistical table is cleared, wherein the third statistical items contain the third statistical items and the third statistical value corresponding to the third statistical items; a clearing processing module, used for judging whether there is a statistical entry containing the first statistical item in the main statistical table, and performing corresponding clearing processing on the first statistical entry according to the judgment result; The device is further configured to read a first target statistical value corresponding to the target statistical item in the main statistical table, and read a second target statistical value corresponding to the target statistical item in the secondary statistical table, when detecting that a target statistical value acquisition event corresponding to the target statistical item is triggered; The sum of the first target statistical value and the second target statistical value is calculated to obtain the target statistical value.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Data processing method and device

    CN105447059A

  • Processing method and device for rapid statistics

    CN110580307A

  • TCP session management method and device, storage medium and electronic device

    CN111405007A

  • Cache control method and device and electronic equipment

    CN111459852A