Log processing method, apparatus and device

Abstract

The application discloses a log processing method, device and equipment, and belongs to the technical field of log processing. The log processing method comprises the following steps: acquiring first log data uploaded by a business system, wherein the first log data comprises a first business label; determining a log data flow limiting index value of the first business label according to a hash table, wherein the hash table comprises at least one hash node, the hash node is established based on different business labels and a hash function, and the hash node is used for storing the corresponding relationship between the business label and the flow limiting index value; and in the case that the flow limiting index value meets a flow limiting condition, the first log data is prohibited from being stored into a cache space of a log system.

Description

Technical Field

[0001] This application belongs to the field of log processing technology, specifically relating to a log processing method, apparatus, and device. Background Technology

[0002] With the development of technology, electronic devices contain more and more applications, and the log data generated by these applications is also increasing. To facilitate subsequent searching and analysis, it is necessary to cache the log data.

[0003] However, the cache space is limited, so in subsequent use, the cache may run out, making it impossible to store log data, which will result in the loss of this part of the log data. Summary of the Invention

[0004] The purpose of this application is to provide a log processing method, apparatus, and device that can solve the problem of log loss caused by the exhaustion of the log system's cache.

[0005] In a first aspect, embodiments of this application provide a log processing method, including:

[0006] Obtain the first log data uploaded by the business system. The first log data includes the first business tag.

[0007] The log data rate limiting index value of the first business tag is determined based on the hash table. The hash table includes at least one hash node. The hash node is established based on different business tags and hash functions. The hash node is used to store the correspondence between business tags and rate limiting index values.

[0008] If the rate limiting metric value meets the rate limiting conditions, the first log data must not be stored in the log system's cache space.

[0009] Secondly, embodiments of this application provide a log processing apparatus, including:

[0010] The acquisition module is used to acquire the first log data uploaded by the business system. The first log data includes the first business tag.

[0011] The determination module is used to determine the log data rate limiting index value of the first business tag according to the hash table. The hash table includes at least one hash node. The hash node is established based on different business tags and hash functions. The hash node is used to store the correspondence between business tags and rate limiting index values.

[0012] The processing module is used to prevent the first log data from being stored in the log system's cache space when the rate limiting index value meets the rate limiting conditions.

[0013] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0015] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0016] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the method described in the first aspect.

[0017] This application embodiment obtains first log data uploaded by a business system. The first log data includes a first business tag. A rate-limiting index value for the log data of the first business tag is determined based on a hash table. The hash table includes at least one hash node, which is established based on different business tags and hash functions. The hash node stores the correspondence between business tags and rate-limiting index values. If the rate-limiting index value meets the rate-limiting conditions, the first log data is prohibited from being stored in the log system's cache space. In other words, after obtaining the first log data uploaded by the business system, this application embodiment can determine the rate-limiting flag value of the first business tag from the hash table. When the rate-limiting flag value meets the rate-limiting conditions, the first log data is prohibited from being stored in the log system's cache space. This avoids a situation where a large volume of log data output for a certain business type exhausts the log system's cache, leading to the inability to cache and loss of log data for other business types. This effectively improves the balance of log data caching across different business types. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a log processing method provided in an embodiment of this application;

[0019] Figure 2 A flowchart illustrating a log processing method provided in an embodiment of this application;

[0020] Figure 3 A schematic diagram illustrating a log processing procedure provided in an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of a log processing device provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] When developers need to analyze bugs in electronic devices, they need to rely on the logs cached by the logging system on the electronic devices to troubleshoot and analyze the problems.

[0027] With the development of technology, electronic devices support more and more services, and the logs generated by each service are also increasing. In this situation, how to save the logs of each service is a serious challenge for the log system.

[0028] Currently, there are two main approaches: First, by increasing the priority of the log thread, the problem of log loss can be mitigated; second, the cache space of the log system can be increased.

[0029] In the first approach, the increased priority of the log thread results in a longer CPU usage time, which in turn reduces the CPU usage of the user interface (UI) thread and packet sending / receiving thread. This can lead to screen lag, slow network response, and other issues, affecting the overall performance of the electronic device and reducing the user experience.

[0030] Regarding the second approach, increasing the cache space of the log system can only delay the time when the problem is exposed; it cannot fundamentally solve the problem of large log output volume exhausting the cache space and causing log loss.

[0031] In summary, none of the currently available log processing solutions can effectively solve the problem of log system cache exhaustion leading to log loss.

[0032] To address this issue, this application provides a log processing method, apparatus, and device that effectively solves the problem of log loss caused by the exhaustion of the log system's cache. The log processing method, apparatus, and device provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments and application scenarios.

[0033] It should be noted that the log processing scheme provided in this application embodiment can also be applied to web requests, etc. That is, it can not only limit the rate of logs, but also limit the rate of web requests or other content, and the processing process is similar.

[0034] Figure 1 This is a flowchart illustrating a log processing method provided in an embodiment of this application. This method can be applied to the logging system of an electronic device, which can be a ulog-based smart device such as a mobile phone, computer, wearable smart device, or other embedded device such as a router. Wearable smart devices can be, for example, smartwatches or smart bracelets.

[0035] like Figure 1 As shown, the log processing method may include the following steps:

[0036] S110. Obtain the first log data uploaded by the business system.

[0037] The first log data includes the first business tag.

[0038] S120. Determine the log data flow limit index value of the first business tag based on the hash table.

[0039] The hash table includes at least one hash node. The hash node is established based on different business tags and hash functions. The hash node is used to store the correspondence between business tags and rate limiting index values.

[0040] S130. If the rate limiting index value meets the rate limiting conditions, the first log data shall not be stored in the cache space of the log system.

[0041] This application embodiment obtains first log data uploaded by a business system. The first log data includes a first business tag. A rate-limiting index value for the log data of the first business tag is determined based on a hash table. The hash table includes at least one hash node, which is established based on different business tags and hash functions. The hash node stores the correspondence between business tags and rate-limiting index values. If the rate-limiting index value meets the rate-limiting conditions, the first log data is prohibited from being stored in the log system's cache space. In other words, after obtaining the first log data uploaded by the business system, this application embodiment can determine the rate-limiting flag value of the first business tag from the hash table. When the rate-limiting flag value meets the rate-limiting conditions, the first log data is prohibited from being stored in the log system's cache space. This avoids a situation where a large volume of log data output for a certain business type exhausts the log system's cache, leading to the inability to cache and loss of log data for other business types. This effectively improves the balance of log data caching across different business types.

[0042] The above steps are explained in detail below:

[0043] In S110, the business system is used to process different business processes. When processing different business processes, the business system can generate corresponding log data for each business process and upload the log data to the log system for caching, so that users can easily find and analyze it later.

[0044] The first business tag can be a string used to represent the business type of the first log data. Different business types can correspond to different business tags. Setting different business tags for different business types can avoid the loss of log data for other business types when caching the corresponding log data later.

[0045] In addition to the first business tag, the first log data can also include the specific log content.

[0046] For example, the logging system can obtain the first log data uploaded by the business system through the log management interface.

[0047] In S120, the hash table includes at least one hash node. One hash node corresponds to one first business label. For example, if the log system receives log data with three business labels, there will be three hash nodes. In actual application, the number of hash nodes can be updated in real time based on the log data received by the log system.

[0048] For example, hash nodes can be established based on business tags and hash functions. For instance, a hash operation can be performed on a business tag based on a hash function to obtain a hash result, and then a corresponding hash node can be established based on the hash result, and the hash result can be associated with and stored with the hash node.

[0049] This application does not limit the hash function, and may use string hash functions such as BKDRHash, APHash, DJBHash, JSHash, RSHash, SDBMHash, PJWHash, and ELFHash.

[0050] Using the string hash function described above, the first business tag can be converted from a string into an identity document (ID) value. In this way, when looking up the hash table based on the first business tag, it is only necessary to compare the ID value corresponding to the first business tag with the ID values ​​stored in each hash node in the hash table, which can greatly reduce the time occupied by string comparison.

[0051] The rate limiting metric value can be a metric value that can limit the rate of the first log data. The rate limiting metric value can be updated in real time based on the log data passed in by the business system.

[0052] For example, the rate limiting metric value may include a rate limiting flag. For instance, when the rate limiting flag is 0, it means that the first log data can be cached, and when the rate limiting flag is 1, it means that the first log data is prohibited from being cached.

[0053] For example, the rate limiting metric value can also include the rate limiting start time. For instance, if the difference between the current time and the rate limiting start time is less than a preset difference, caching of the first log data can be prohibited. That is, if a large amount of first log data is received in a short period of time, a rate limiting policy needs to be implemented on the first log data to prevent it from exhausting the log system's cache space. The current time can be the time when the rate limiting metric value is determined.

[0054] The rate limiting metric values ​​for different business tags can be associated and stored on the hash node corresponding to the business tag.

[0055] For example, after the log system obtains the first log data, it can look up the hash table to determine the rate limiting index value of the first business tag, providing a basis for the subsequent execution of the rate limiting strategy.

[0056] In S130, the rate limiting condition is used to rate limit the first log data, that is, to prohibit the storage of the first log data in the log system's cache space.

[0057] For example, when the rate limiting metric value meets the rate limiting conditions, the storage of the first log data in the log system's cache space is prohibited; when the rate limiting metric value does not meet the rate limiting conditions, the storage of the first log data in the log system's cache space is allowed.

[0058] This application embodiment can effectively filter out spam, invalid, redundant, and unrestricted junk log data by limiting the flow of log data for different business types, thus preventing these junk log data from occupying too much cache space and causing normal log data to be lost due to failure to be cached.

[0059] In some embodiments, the above-described S120 may include the following steps:

[0060] Compare the first business tag with the hash table;

[0061] If the hash table does not include the first business label, a hash node for the first business label is created and the rate limiting indicator value corresponding to the first business label is initialized. The initialized rate limiting indicator value is then determined as the log data rate limiting indicator value for the first business label.

[0062] If the hash table includes the first business tag, the log data flow limit index value of the first business tag is determined based on the hash node corresponding to the first business tag.

[0063] In some embodiments, in order to improve the efficiency of table lookup, a hash function can be used to perform a hash operation on the first business tag to obtain a first hash value, and the first hash value can be compared with each hash value in the hash table. If the first hash value is the same as a certain hash value in the hash table, it can be determined that the first business tag exists in the hash table. If the first hash value is different from each hash value in the hash table, it can be determined that the first business tag does not exist in the hash table.

[0064] In some embodiments, if the first service tag does not exist in the hash table, a hash node for the first service tag can be created, and the rate limiting metric value of the first service tag can be initialized. For example, a new hash node can be created, and a hash function can be used to perform a hash operation on the first service tag to obtain a hash value, which is then stored in the newly created hash node.

[0065] For example, assuming the hash table initially contains 5 hash nodes, a new hash node can be added at the end of the hash table.

[0066] When adding a new hash node, the rate limiting metric value for the first business tag can be initialized. For example, the statistical time and the starting value of the statistical value can be initialized to 1, and the rate limiting flag can be set to 0. In this case, there is no rate limiting start time. The statistical time is the recording time corresponding to the starting value of the statistical value.

[0067] At this point, the initial rate limiting metric value can be used as the rate limiting metric value for the first business tag. When the first log data is received subsequently, the rate limiting metric value stored on this hash node can be updated.

[0068] In some embodiments, if the first service tag exists in a hash table, the log data rate limiting metric value of the first service tag can be determined based on the hash node corresponding to the first service tag. For example, the rate limiting metric value stored on the corresponding hash node can be determined as the log data rate limiting metric value of the first service tag.

[0069] This application embodiment can compare a first service tag with a hash table to determine the rate limiting index value of the first service tag, providing a basis for subsequent execution of rate limiting strategies. Furthermore, when comparing the first service tag with the hash table, a hash operation can first be performed on the first service tag based on a hash function to obtain a first hash value. Then, the first hash value is compared with the hash values ​​of each hash node in the hash table. This significantly reduces the time spent on string comparisons and improves the efficiency of table lookups.

[0070] In some embodiments, the hash function mentioned in the above embodiments can be selected as needed. For example, several common hash algorithms can be selected, and then several hash algorithms can be tested based on several test data to select the hash algorithm with the best performance for use in the embodiments of this application.

[0071] For example, three hash algorithms can be selected, namely:

[0072] Algorithm 1: Add the ASCII codes of each character in each string, and then take the remainder after taking the remainder after 1024.

[0073] Algorithm 2: Based on Algorithm 1, add the ASCII code of the last character of the string to the remainder obtained;

[0074] Algorithm 3: Based on the BKDRHash algorithm, take the remainder of 1024.

[0075] For example, test data may include the following three types:

[0076] Data 1: 100 randomly generated strings, each with a length of 8;

[0077] Data 2: 500 randomly generated strings, each with a length of 8;

[0078] Data 3: 700 randomly generated strings, each with a length of 8;

[0079] For each algorithm, the following two methods can be used to create and retrieve tables:

[0080] Method 1: Calculate a value for data X using a hash function, denoted as tagid. When building the hash table, if there is no data at the corresponding tagid (hash node), write it directly; if there is already data at the corresponding tagid and the key does not match, add a new hash node. When searching the hash table, if the tagid does not match, traverse to the next hash node and continue searching. Here, data X can be data 1, data 2, and data 3 as described above, and the hash function can be algorithm 1, algorithm 2, or algorithm 3 as described above.

[0081] Method 2: Calculate a value for data X using a hash function, denoted as tagid. When building the hash table, if there is no data at the location corresponding to tagid (hash node), write it directly; if there is already data at the location corresponding to tagid and the key does not match, traverse the hash table from the beginning and fill the empty space; when searching the hash table, if tagid is inconsistent, traverse the hash table from the beginning and continue searching.

[0082] For each algorithm, you can test data 1, data 2 and data 3 respectively according to method one and method two above to obtain parameters such as the total time for table creation, the total time for table lookup and the average time for each string. For details, please refer to Table 1 and Table 2.

[0083] Table 1 Parameters corresponding to Method 1

[0084]

[0085] Table 2 Parameters corresponding to Method 2

[0086]

[0087]

[0088] By analyzing the above parameters, it can be seen that the number of collisions between Method 1 and Algorithm 3 is smaller and the average time consumption is shorter. Therefore, Algorithm 3 can be used as the hash function in this application embodiment to perform hash operation on the first business tag.

[0089] In a hash table, two hash values ​​are said to have the same value but different keys in their corresponding strings, indicating a collision. For example, when two hash values ​​are the same but have different keys, a new hash node can be added to the hash table.

[0090] The above algorithms 1, 2 and 3 are just examples. In actual applications, other algorithms can also be tested, and the process is similar.

[0091] The first hash value in the above embodiments is the tagid in the above example. It can be obtained by performing a calculation on the first business tag using the BKDRHash algorithm, and then taking the remainder of the result by %1024. In other words, this embodiment can convert the first business tag from a string to a numerical value. Therefore, when searching the hash table for the first business tag, string comparison can be converted to numerical comparison, which greatly improves the efficiency of table lookup.

[0092] For example, when comparing the first business label and the hash table, you can start from the first hash node of the hash table, that is, compare the first hash value with the hash value of the first hash node. If they are different, continue to traverse a hash node, that is, perform the table lookup in the manner described in Method 1 above, which can reduce the table lookup time.

[0093] In this embodiment of the application, the rate limiting condition is related to the rate limiting index value. For example, the rate limiting index value may include a rate limiting flag and a rate limiting start time.

[0094] Accordingly, the rate limiting index value meets the rate limiting conditions including: the rate limiting flag is the first flag, and the first difference between the rate limiting start time and the first time is less than the first threshold, where the first time is the time when the log data rate limiting index value of the first business tag is determined according to the hash table.

[0095] The rate limiting start time is the time during which the first log data is cached. For example, the rate limiting start time can be determined based on the rate limiting flag. For instance, a rate limiting flag of 0 indicates no rate limiting, and a rate limiting flag of 1 indicates rate limiting. Therefore, the rate limiting start time can be the time when the rate limiting flag changes from 0 to 1. For example, if the rate limiting flag changes from 0 to 1 at 9:05, then the rate limiting start time is 9:05.

[0096] The first flag is a flag that restricts the storage of the first log data; for example, the first flag can be set to 1.

[0097] The first time is the time when the log data rate limiting indicator value of the first business tag is determined according to the hash table, that is, the time when the rate limiting indicator value is determined.

[0098] For example, when the rate limiting flag of the first service tag is 1, and the first difference between the first time and the rate limiting start time is less than the first threshold, the storage of the first log data in the log system's cache space can be prohibited. The size of the first threshold can be set according to actual needs, for example, it can be set to 1 minute or 2 minutes.

[0099] This application embodiment can limit the flow of a large amount of log data entering the business system in a short period of time, thereby avoiding the problem of other normal log data being lost due to the exhaustion of cache space, and improving the caching balance of log data of various business types.

[0100] In some embodiments, the rate limiting metric value may also include a statistical value and a statistical time, wherein the statistical value represents the number of first log data obtained, and the statistical time is the recording time corresponding to the starting value of the statistical value;

[0101] Accordingly, such as Figure 2 As shown, the business processing method may include the following steps:

[0102] S210. Obtain the first log data uploaded by the business system.

[0103] S220. Determine the log data flow limit index value of the first business tag based on the hash table.

[0104] S230. If the rate limiting index value meets the rate limiting conditions, the first log data shall not be stored in the cache space of the log system.

[0105] S240. If at least one of the current limiting flag and the first difference does not meet the current limiting condition, add 1 to the statistical value to obtain the updated statistical value.

[0106] S250. If the updated statistical value is greater than the rate limiting threshold, obtain the second difference between the first time and the statistical time.

[0107] S260. If the second difference is less than the second threshold, the storage of the first log data in the log system's cache space shall be prohibited.

[0108] S270, reset statistics, rate limiting flag, and rate limiting start time.

[0109] The process of S210-S230 is the same as the process of S110-S130 above. For details, please refer to S110-S130 above. For the sake of brevity, it will not be repeated here.

[0110] The other steps described above are explained in detail below:

[0111] In S240, for example, at least one of the current limiting flag and the first difference does not meet the current limiting condition. This can be that the current limiting flag is the second flag and the first difference is less than the first threshold; or the current limiting flag is the first flag and the first difference is not less than the first threshold; or the current limiting flag is the second flag and the first difference is not less than the first threshold.

[0112] The second flag is used to indicate that no rate limiting is applied to the first log data. For example, the second flag can be set to 0.

[0113] The statistical value is used to count the number of first log data acquired by the log system. For example, if at least one of the rate limiting flag and rate limiting start time does not meet the rate limiting conditions, the statistical value can be incremented by 1 to update the number of first log data acquired by the log system, providing a basis for whether to rate limit the first log data in the future.

[0114] For example, when the rate limiting flag of the first service tag is 0, the statistical value can be incremented by 1. For example, when the first difference between the first time and the rate limiting start time is not less than the first threshold, the statistical value can be incremented by 1. For example, when the rate limiting flag is 0 and the first difference between the first time and the rate limiting start time is not less than the first threshold, the statistical value can be incremented by 1.

[0115] In S250, if the updated statistical value is greater than the rate limiting threshold, the second difference between the first time and the statistical time can be obtained. The statistical time is the recording time corresponding to the starting value of the statistical value. The starting value of the statistical value can be 0 or 1, as detailed in the description below.

[0116] In S260, if the second difference between the first time and the statistical time is less than a second threshold, storing the first log data in the log system's cache space can be prohibited; if the second difference between the first time and the statistical time is not less than the second threshold, storing the first log data in the log system's cache space can be allowed. The size of the second threshold can be set according to actual needs. For example, the second threshold can be set to 1 minute.

[0117] In S270, if the second difference between the first time and the statistical time is less than the second threshold, in addition to prohibiting the storage of the first log data in the log system's cache space, it is also necessary to update the statistical value, the rate limiting flag, and the rate limiting start time.

[0118] For example, the statistics value can be reset to 0, the rate limiting flag can be reset from 0 to 1, and the rate limiting start time can be set. In this case, the statistics time does not need to be updated.

[0119] In practical applications, S260 and S270 can be executed simultaneously, or S260 can be executed first and then S270, or S270 can be executed first and then S260. S230 and S240 can be executed separately.

[0120] For example, if the second difference between the first time and the statistical time is not less than the second threshold, in addition to allowing the first log data to be stored in the log system's cache space, it is also necessary to update the statistical value, the statistical time, and the rate limiting flag.

[0121] For example, the statistical value can be reset to 0, and the statistical time can be reset, meaning the statistical time starts counting from when the statistical value is reset to 0. For hash nodes that have already been rate-limited, the rate-limiting flag can be updated from 1 to 0, thus removing the rate limit. In other words, this embodiment only applies rate limiting to log data within a short period of time.

[0122] In this embodiment of the application, when at least one of the rate limiting flag and the rate limiting start time does not meet the rate limiting conditions, the first log data can be further rate-limited based on the statistical value and statistical time of the first log data. That is, this embodiment of the application can rate-limit a large amount of log data output by the business system in a short period of time to avoid it from excessively occupying the cache space of other normal log data.

[0123] In some embodiments, after S110, the log processing method may further include the following steps:

[0124] Record the second time when the first log data is obtained, and start timing from the second time;

[0125] If the first log data is not received within the preset timeout period, the rate limiting indicator value corresponding to the first log data in the hash table will be deleted from the first hash node corresponding to the first service tag.

[0126] For example, for each piece of first log data acquired, a second time when the first log data was acquired can be recorded, and a timer can be started from that second time. If no more first log data is received within a preset time period, the hash value of the first service tag and the corresponding rate limiting flag value can be removed from the corresponding hash node, while the hash node can be retained. For example, the preset time period can be set to 3 minutes.

[0127] Taking the first business tag corresponding to the first hash node as an example, if the log system does not receive the first log data within the preset time period, the hash value of the first business tag and the corresponding rate limiting index value stored on the first hash node can be removed.

[0128] This reduces the memory footprint of the hash table and the probability of collisions between corresponding hash values ​​in the hash table.

[0129] Since the log data of a certain business tag has not been transmitted to the log system for a period of time, some hash nodes may be empty, that is, they do not contain hash values ​​and corresponding rate limiting flag values. Therefore, if the hash value of the first business tag is different from the hash values ​​of each hash node, the hash value of the first business tag and the corresponding rate limiting flag value can be directly inserted into the empty hash nodes, that is, the table is built according to the above method one.

[0130] This application embodiment distinguishes different business types by using business tags and establishes a hash table based on a hash function to count the log output frequency and output quantity of different business types. For log data exceeding the threshold, its storage is restricted, thereby achieving the purpose of real-time monitoring and rate limiting of log data.

[0131] The following example illustrates the log processing procedure provided in this application embodiment:

[0132] like Figure 3 As shown, the log system receives log data uploaded by the business system. This log data is distinguished by business tags (strings). The business tags are hashed using a hash algorithm to obtain hash values. Then, it is determined whether the hash value exists in the hash table. If the hash value exists in the hash table, the rate limiting flag and the difference between the first time and the rate limiting start time are further checked. If the rate limiting flag is 1 and the difference is less than the first threshold, the log data corresponding to the hash value is prohibited from being stored in the log system's cache space, and the next log data is checked.

[0133] If the rate limiting flag is 0, and / or the difference is not less than the first threshold, the statistical value of the hash node corresponding to the hash value is incremented by 1, and then it is further determined whether the updated statistical value is greater than the rate limiting threshold. If the updated statistical value is not greater than the rate limiting threshold, the log data corresponding to the hash value is allowed to be stored in the cache space of the log system. If the updated statistical value is greater than the rate limiting threshold, the relationship between the difference between the first time and the statistical time and the second threshold can be further determined.

[0134] If the difference between the first time and the statistical time is less than the second threshold, the statistical value, rate limiting flag, and rate limiting start time of the corresponding hash node are reset, and the corresponding log data is prohibited from being stored in the log system's cache space. If the difference between the first time and the statistical time is not less than the second threshold, the corresponding log data is allowed to be stored in the log system's cache space, and the statistical value (set to 0) and statistical time of the corresponding hash node are reset. For hash nodes that have already been rate-limited, their rate limiting flag is updated to 0, and rate limiting is lifted.

[0135] If the hash value does not exist in the hash table, then create the corresponding hash node and set the statistical time, statistical value (set to 1), and rate limiting flag (set to 0).

[0136] The above solution significantly improves the balance of log output across different business types, preventing insufficient cache space and log loss due to excessive log traffic from a single business type. In other words, a portion of logs from each business type can be stored in the cache. Furthermore, hashing business tags using a hash function converts string comparisons into hash value comparisons, greatly enhancing the logging system's processing performance.

[0137] It should be noted that, in this embodiment of the application, when the rate limiting flag value meets the rate limiting conditions, in addition to prohibiting the storage of log data in the cache space, an alarm can also be triggered, or the rate-limited log data can be added to the blacklist, and the un-rate-limited log data can be added to the whitelist, etc.

[0138] It should be noted that the log processing method provided in this application embodiment can be executed by a log processing device or a processing module within that log processing device for executing the log processing method. This application embodiment uses the execution of the log processing method by a log processing device as an example to illustrate the log processing device provided in this application embodiment.

[0139] Figure 4 This is a schematic diagram of the structure of a log processing device provided in an embodiment of this application.

[0140] like Figure 4 As shown, the log processing device 400 may include:

[0141] The acquisition module 401 is used to acquire the first log data uploaded by the business system, wherein the first log data includes a first business tag.

[0142] The determining module 402 is used to determine the log data rate limiting index value of the first business tag according to the hash table. The hash table includes at least one hash node. The hash node is established based on different business tags and hash functions. The hash node is used to store the correspondence between business tags and rate limiting index values.

[0143] The processing module 403 is used to prohibit the storage of the first log data in the cache space of the log system when the rate limiting index value meets the rate limiting conditions.

[0144] This application embodiment obtains first log data uploaded by a business system. The first log data includes a first business tag. A rate-limiting index value for the log data of the first business tag is determined based on a hash table. The hash table includes at least one hash node, which is established based on different business tags and hash functions. The hash node stores the correspondence between business tags and rate-limiting index values. If the rate-limiting index value meets the rate-limiting conditions, the first log data is prohibited from being stored in the log system's cache space. In other words, after obtaining the first log data uploaded by the business system, this application embodiment can determine the rate-limiting flag value of the first business tag from the hash table. When the rate-limiting flag value meets the rate-limiting conditions, the first log data is prohibited from being stored in the log system's cache space. This avoids a situation where a large volume of log data output for a certain business type exhausts the log system's cache, leading to the inability to cache and loss of log data for other business types. This effectively improves the balance of log data caching across different business types.

[0145] In some possible implementations of the embodiments of this application, the determining module 401 is specifically used for:

[0146] Compare the first business tag with the hash table;

[0147] If the hash table does not include the first business label, a hash node for the first business label is created and the rate limiting indicator value corresponding to the first business label is initialized. The initialized rate limiting indicator value is then determined as the log data rate limiting indicator value for the first business label.

[0148] If the hash table includes the first business tag, the log data flow limit index value of the first business tag is determined based on the hash node corresponding to the first business tag.

[0149] In some possible implementations of the embodiments of this application, the rate limiting index value includes a rate limiting flag and a rate limiting start time;

[0150] The conditions for meeting the rate limiting criteria include: the rate limiting flag is the first flag, and the first difference between the rate limiting start time and the first time is less than the first threshold. The first time is the time when the log data rate limiting metric value of the first business tag is determined according to the hash table.

[0151] In some possible implementations of the embodiments of this application, the rate limiting indicator value also includes a statistical value and a statistical time. The statistical value represents the number of first log data obtained, and the statistical time is the recording time corresponding to the starting value of the statistical value.

[0152] The processing module 403 is also used to, after the determining module 402 determines the log data flow limiting index value of the first business tag according to the hash table, add 1 to the statistical value to obtain the updated statistical value if at least one of the flow limiting flag and the first difference does not meet the flow limiting condition.

[0153] The acquisition module 401 is also used to acquire the second difference between the first time and the statistical time when the updated statistical value is greater than the rate limiting threshold.

[0154] The processing module 403 is also used to prevent the storage of the first log data in the cache space of the log system when the second difference is less than the second threshold; and to reset the statistical value, the rate limiting flag and the rate limiting start time.

[0155] In some possible implementations of the embodiments of this application, the processing module 403 is further configured to, after the acquisition module acquires the first log data uploaded by the business system, record a second time for acquiring the first log data, and start timing from the second time; and,

[0156] If the first log data is not received within the preset timeout period, the rate limiting indicator value corresponding to the first log data in the hash table will be deleted from the first hash node corresponding to the first service tag.

[0157] This application embodiment distinguishes different business types by using business tags and establishes a hash table based on a hash function to count the log output frequency and output quantity of different business types. For log data exceeding the threshold, its storage is restricted, thereby achieving the purpose of real-time monitoring and rate limiting of log data.

[0158] The log processing device applied to an electronic device in this application embodiment can be a device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope.

[0159] The electronic device in this application embodiment can be an electronic device with an operating system. The operating system can be Android, iOS, or other possible operating systems, and this application embodiment does not specifically limit it.

[0160] The log processing device for electronic devices provided in this application embodiment can achieve... Figures 1 to 3 The various processes in the embodiments of the log processing method applied to electronic devices will not be described again here to avoid repetition.

[0161] like Figure 5 As shown, this application embodiment also provides an electronic device, including a processor 501 and a memory 502. The memory 502 stores programs or instructions that can run on the processor 501. When the program or instructions are executed by the processor 501, they implement the various steps of the above-described screen projection method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0162] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0163] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0164] The electronic device 600 includes, but is not limited to, components such as: radio frequency unit 601, network module 602, audio output unit 603, input unit 604, sensor 605, display unit 606, user input unit 607, interface unit 608, memory 609, and processor 610.

[0165] Those skilled in the art will understand that the electronic device 600 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 610 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0166] The input unit 604 is used to obtain the first log data uploaded by the business system, wherein the first log data includes a first business tag.

[0167] Processor 610 is used to determine the log data rate limiting indicator value of the first service tag according to a hash table. The hash table includes at least one hash node, which is established based on different service tags and hash functions. The hash node is used to store the correspondence between service tags and rate limiting indicator values; and...

[0168] If the rate limiting metric value meets the rate limiting conditions, the first log data must not be stored in the log system's cache space.

[0169] This application embodiment obtains first log data uploaded by a business system. The first log data includes a first business tag. A rate-limiting index value for the log data of the first business tag is determined based on a hash table. The hash table includes at least one hash node, which is established based on different business tags and hash functions. The hash node stores the correspondence between business tags and rate-limiting index values. If the rate-limiting index value meets the rate-limiting conditions, the first log data is prohibited from being stored in the log system's cache space. In other words, after obtaining the first log data uploaded by the business system, this application embodiment can determine the rate-limiting flag value of the first business tag from the hash table. When the rate-limiting flag value meets the rate-limiting conditions, the first log data is prohibited from being stored in the log system's cache space. This avoids a situation where a large volume of log data output for a certain business type exhausts the log system's cache, leading to the inability to cache and loss of log data for other business types. This effectively improves the balance of log data caching across different business types.

[0170] In some possible implementations of embodiments of this application, the processor 610 is specifically used for:

[0171] Compare the first business tag with the hash table;

[0172] If the hash table does not include the first business label, a hash node for the first business label is created and the rate limiting indicator value corresponding to the first business label is initialized. The initialized rate limiting indicator value is then determined as the log data rate limiting indicator value for the first business label.

[0173] If the hash table includes the first business tag, the log data flow limit index value of the first business tag is determined based on the hash node corresponding to the first business tag.

[0174] In some possible implementations of the embodiments of this application, the rate limiting index value includes a rate limiting flag and a rate limiting start time;

[0175] The conditions for meeting the rate limiting criteria include: the rate limiting flag is the first flag, and the first difference between the rate limiting start time and the first time is less than the first threshold. The first time is the time when the log data rate limiting metric value of the first business tag is determined according to the hash table.

[0176] In some possible implementations of the embodiments of this application, the rate limiting indicator value also includes a statistical value and a statistical time. The statistical value represents the number of first log data obtained, and the statistical time is the recording time corresponding to the starting value of the statistical value.

[0177] The processor 610 is also used to, after the determining module determines the log data flow limiting index value of the first service tag according to the hash table, add 1 to the statistical value to obtain the updated statistical value if at least one of the flow limiting flag and the first difference does not meet the flow limiting condition.

[0178] The input unit 604 is also used to obtain a second difference between the first time and the statistical time when the updated statistical value is greater than the flow limiting threshold;

[0179] The processor 610 is also configured to, if the second difference is less than the second threshold, prohibit the storage of the first log data in the log system's cache space; and, reset the statistics, the rate limiting flag, and the rate limiting start time.

[0180] In some possible implementations of the embodiments of this application, the processor 610 is further configured to, after the acquisition module acquires the first log data uploaded by the business system, record a second time for acquiring the first log data, and start timing from the second time; and,

[0181] If the first log data is not received within the preset timeout period, the rate limiting indicator value corresponding to the first log data in the hash table will be deleted from the first hash node corresponding to the first service tag.

[0182] This application embodiment distinguishes different business types by using business tags and establishes a hash table based on a hash function to count the log output frequency and output quantity of different business types. For log data exceeding the threshold, its storage is restricted, thereby achieving the purpose of real-time monitoring and rate limiting of log data.

[0183] It should be understood that, in this embodiment, the input unit 604 may include a graphics processing unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0184] The memory 609 can be used to store software programs and various data. The memory 609 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 609 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 609 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0185] Processor 610 may include one or more processing units; optionally, processor 610 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 610.

[0186] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described log processing method embodiment applied to an electronic device and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0187] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0188] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described log processing method embodiments applied to electronic devices, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0189] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0190] This application provides a computer program product stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the log processing method embodiment applied to electronic devices described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0191] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0193] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A log processing method, characterized in that, include: Obtain the first log data uploaded by the business system, the first log data including the first business tag; The log data rate limiting index value of the first business tag is determined according to a hash table. The hash table includes at least one hash node, which is established based on different business tags and hash functions. The hash node is used to store the correspondence between business tags and rate limiting index values. The rate limiting index value is used to limit the rate of the first log data. If the rate limiting index value meets the rate limiting conditions, the first log data shall not be stored in the cache space of the log system.

2. The method according to claim 1, characterized in that, The step of determining the log data rate limiting index value of the first service tag according to the hash table includes: Compare the first service tag with the hash table; If the hash table does not include the first service tag, a hash node for the first service tag is established and the rate limiting index value corresponding to the first service tag is initialized, and the initialized rate limiting index value is determined as the log data rate limiting index value of the first service tag. If the hash table includes the first service tag, the log data flow limiting index value of the first service tag is determined based on the hash node corresponding to the first service tag.

3. The method according to claim 1, characterized in that, The rate limiting index value includes the rate limiting flag and the rate limiting start time; The rate limiting indicator value meets the rate limiting conditions as follows: the rate limiting flag is a first flag, and the first difference between the rate limiting start time and the first time is less than a first threshold, wherein the first time is the time when the log data rate limiting indicator value of the first service tag is determined according to the hash table.

4. The method according to claim 3, characterized in that, The rate limiting indicator value also includes a statistical value and a statistical time. The statistical value represents the number of the first log data obtained, and the statistical time is the recording time corresponding to the starting value of the statistical value. After determining the log data rate limiting metric value of the first service tag according to the hash table, the method further includes: If at least one of the current limiting flag and the first difference does not meet the current limiting condition, the statistical value is incremented by 1 to obtain the updated statistical value; If the updated statistical value is greater than the rate limiting threshold, obtain the second difference between the first time and the statistical time; If the second difference is less than the second threshold, the first log data shall not be stored in the cache space of the log system. Reset the statistical value, the rate limiting flag, and the rate limiting start time.

5. The method according to claim 1, characterized in that, After obtaining the first log data uploaded by the business system, the method further includes: Record the second time when the first log data is obtained, and start timing from the second time; If the first log data is not received within the preset timeout period, the rate limiting indicator value corresponding to the first log data in the hash table is deleted from the first hash node corresponding to the first service tag.

6. A log processing device, characterized in that, include: The acquisition module is used to acquire the first log data uploaded by the business system, wherein the first log data includes a first business tag. The determination module is used to determine the log data rate limiting index value of the first business tag according to a hash table. The hash table includes at least one hash node, which is established based on different business tags and hash functions. The hash node is used to store the correspondence between business tags and rate limiting index values. The rate limiting index value is used to limit the rate of the first log data. The processing module is used to prohibit the storage of the first log data in the cache space of the log system when the rate limiting index value meets the rate limiting conditions.

7. The apparatus according to claim 6, characterized in that, The determining module is specifically used for: Compare the first service tag with the hash table; If the hash table does not include the first service tag, a hash node for the first service tag is established and the rate limiting index value corresponding to the first service tag is initialized, and the initialized rate limiting index value is determined as the log data rate limiting index value of the first service tag. If the hash table includes the first service tag, the log data flow limiting index value of the first service tag is determined based on the hash node corresponding to the first service tag.

8. The apparatus according to claim 6, characterized in that, The rate limiting index value includes the rate limiting flag and the rate limiting start time; The rate limiting indicator value meets the rate limiting conditions as follows: the rate limiting flag is a first flag, and the first difference between the rate limiting start time and the first time is less than a first threshold, wherein the first time is the time when the log data rate limiting indicator value of the first service tag is determined according to the hash table.

9. The apparatus according to claim 8, characterized in that, The rate limiting indicator value also includes a statistical value and a statistical time. The statistical value represents the number of the first log data obtained, and the statistical time is the recording time corresponding to the starting value of the statistical value. The processing module is further configured to, after the determining module determines the log data rate limiting index value of the first service tag according to the hash table, add 1 to the statistical value to obtain an updated statistical value if at least one of the rate limiting flag and the first difference does not meet the rate limiting condition. The acquisition module is further configured to acquire a second difference between the first time and the statistical time when the updated statistical value is greater than the rate limiting threshold. The processing module is further configured to prohibit the storage of the first log data in the cache space of the log system when the second difference is less than the second threshold; and to reset the statistical value, the rate limiting flag and the rate limiting start time.

10. The apparatus according to claim 6, characterized in that, The processing module is further configured to record a second time for acquiring the first log data after the acquisition module acquires the first log data uploaded by the business system, and start timing from the second time. as well as, If the first log data is not received within the preset timeout period, the rate limiting indicator value corresponding to the first log data in the hash table is deleted from the first hash node corresponding to the first service tag.

11. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1 to 5.

Images