Log processing method and device
By passing the filter list between the server and the client, uploading log identifiers that do not meet the preset usage conditions is prohibited, the resource waste caused by full buried points is solved and the efficiency of log processing is improved.
Patent Information
- Application Number
- CN202110955536.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-08-19
AI Technical Summary
The existing log processing method of fully buried points is easy to generate and upload a large number of useless logs, resulting in wasted resources.
Get the application's business analysis data on the server, determine the log identifier of the historical log that does not meet the preset usage conditions, and add it to the filter list, and send it to the client to prohibit uploading logs that match the log identifier from being uploaded to the server .
It reduces the upload of useless logs and waste of computing resources, and improves the efficiency of log processing.
Smart Images

Figure CN113608989B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of log processing technology, and in particular to a log processing method. The present application also relates to a log processing device, a log processing system, a computing device, and a computer-readable storage medium. Background Art
[0002] Tracking, also known as event tracking, is a method of data collection. Tracking can include: collecting and recording data about specific behaviors of users in applications (Apps, Applications), obtaining and uploading logs.
[0003] In the related technology, tracking can be achieved through full tracking. Specifically, the target code running in the application can be "intercepted", or the corresponding tracking code can be "inserted" before or after the target code to collect and record data about specific behaviors, and then obtain and upload logs. Among them, the target code is all the codes corresponding to the specific behavior of the user in the application.
[0004] However, there are often a large number of specific behaviors of users in applications. Therefore, the above-mentioned full-point embedding method is prone to generate and upload a large number of logs, and there are likely to be useless logs in the large number of logs, resulting in a waste of upload resources and computing resources. Summary of the invention
[0005] In view of this, an embodiment of the present application provides a log processing method. The present application also relates to a log processing device, a log processing system, a computing device, and a computer-readable storage medium, so as to solve the problem that a large number of useless logs are obtained by the log processing method in the form of full-embedded points in the prior art, resulting in a waste of upload resources and computing resources.
[0006] According to a first aspect of an embodiment of the present application, a log processing method is provided, which is applied to a server, and includes:
[0007] Acquire analysis data for business analysis of the application, the analysis data including usage information of historical logs during the business analysis process;
[0008] Adding log identifiers of historical logs whose usage information does not meet the preset usage conditions to the filter list;
[0009] The filtering list is sent to the client; the filtering list is used by the client to prohibit uploading the to-be-uploaded logs that match the log identifiers in the filtering list to the server.
[0010] According to a second aspect of an embodiment of the present application, a log processing method is provided, which is applied to a client and includes:
[0011] Receive a filter list sent by the server; wherein the filter list is obtained by the server adding log identifiers of historical logs whose usage information does not meet preset usage conditions; the usage information refers to the usage of historical logs in the process of business analysis of the application;
[0012] Determine the log ID of the log to be uploaded;
[0013] If the log identifier matches the filter list, uploading the log to be uploaded to the server is prohibited.
[0014] According to a third aspect of an embodiment of the present application, a log processing device is provided, which is applied to a server, including:
[0015] A data acquisition module is configured to acquire analysis data for business analysis of an application, wherein the analysis data includes usage information of historical logs during the business analysis process;
[0016] A filter list determination module, configured to add log identifiers of historical logs whose usage information does not meet preset usage conditions to the filter list;
[0017] The sending module is configured to send the filtering list to the client; the filtering list is used by the client to prohibit uploading the to-be-uploaded logs that match the log identifiers in the filtering list to the server.
[0018] According to a fourth aspect of an embodiment of the present application, a log processing device is provided, which is applied to a client and includes:
[0019] The receiving module is configured to receive a filter list sent by the server; wherein the filter list is obtained by the server adding a log identifier of a historical log whose usage information does not meet a preset usage condition; the usage information refers to the usage of the historical log in the process of business analysis of the application;
[0020] A log identification determination module, configured to determine the log identification of the log to be uploaded;
[0021] The log upload management module is configured to prohibit uploading the log to be uploaded to the server if the log identifier matches the filter list.
[0022] According to a fifth aspect of an embodiment of the present application, there is provided a log processing system, including: a server and a client;
[0023] The server is configured to obtain analysis data for business analysis of the application, the analysis data including usage information of historical logs during the business analysis; add log identifiers of historical logs whose usage information does not meet preset usage conditions to a filter list; and send the filter list to the client;
[0024] The client is configured to receive a filter list sent by a server; determine a log identifier of a log to be uploaded; and if the log identifier matches the filter list, prohibit the log to be uploaded from being uploaded to the server.
[0025] According to a sixth aspect of an embodiment of the present application, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the log processing method when executing the instructions.
[0026] According to a seventh aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer instructions, and when the instructions are executed by a processor, the steps of the log processing method are implemented.
[0027] In the solution provided by the present application, the server obtains analysis data for business analysis of the application, and the analysis data includes usage information of historical logs uploaded by the client during the business analysis process; determines whether the usage information of the historical logs meets the preset usage conditions; adds the log identifiers of the historical logs that do not meet the preset usage conditions to the filter list; the filter list is used by the client to prohibit uploading the logs to be uploaded that match the log identifiers in the filter list to the server. Among them, the filter list is obtained by adding the log identifiers of the historical logs that do not meet the preset usage conditions. Therefore, the log identifier matches the filter list, indicating that the log to be uploaded does not meet the preset usage conditions for the use of the log by the business analysis, which is equivalent to a useless log. Therefore, the client prohibits uploading the logs to be uploaded that match the filter list to the server, which can reduce the waste of upload resources and computing resources caused by uploading useless logs. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 This is a flow chart of a log processing method provided by an embodiment of the present application;
[0029] Figure 2 is a flow chart of a log processing method provided by another embodiment of the present application;
[0030] Figure 3 It is a structural diagram of a log processing system provided by an embodiment of the present application;
[0031] Figure 4 is a flow chart of a log processing method applied to a log processing system provided by an embodiment of the present application;
[0032] Figure 5 is a flow chart of a log processing method applied to a log processing system provided by another embodiment of the present application;
[0033] Figure 6 is an example diagram of an architecture of a log processing method applied to a log processing system provided by another embodiment of the present application;
[0034] Figure 7 It is a structural diagram of a log processing device provided by an embodiment of the present application;
[0035] Figure 8 is a structural schematic diagram of a log processing device provided by another embodiment of the present application;
[0036] Fig. 9 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0037] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.
[0038] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.
[0039] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0040] In the present application, a log processing method is provided. The present application also relates to a log processing device, a computing device, a log processing system, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0041] Figure 1 A flow chart of a log processing method provided according to an embodiment of the present application is shown, which is applied to a server and specifically includes the following steps:
[0042] S101, obtaining analysis data for business analysis of an application; the analysis data includes usage information of historical logs during the business analysis process.
[0043] In specific applications, operation and maintenance personnel can use historical logs to perform business analysis on the application corresponding to the historical logs. For example, the historical logs record the number of times users use each function in the application, so that the operation and maintenance personnel can analyze which function needs to be optimized. In addition, the usage information of the historical logs may include at least one of the following information: whether the historical logs are used for business analysis, and the number of times they are used for business analysis. In addition, the analysis data may be in the form of a report. In order to save storage space, the usage information of the historical logs may be the usage information of the log identifier of the historical logs.
[0044] S102, adding log identifiers of historical logs whose usage information does not meet preset usage conditions to a filtering list.
[0045] There may be multiple preset usage conditions. Exemplarily, failure to meet the preset usage conditions may include: the historical log has not been used in at least one business analysis of the application; or the historical log has not been used for business analysis of the application within a preset period of time. Any preset usage condition that can be used to determine that the historical log is a useless log can be used in this application, and this embodiment does not limit this.
[0046] Furthermore, determining whether the historical log meets the preset usage conditions may specifically include: identifying whether the usage information matches the preset usage conditions, and if not, determining that the historical log does not meet the preset usage conditions; or inputting the usage information into a pre-trained binary classification model to obtain a classification result of whether the usage information meets the preset usage conditions. The binary classification model is a neural network model trained using sample usage information and classification result labels of the sample usage information. Furthermore, the classification result labels are set according to the preset usage conditions.
[0047] Furthermore, the filter list can be in various forms. For example, the filter list can be in the form of a list, a table, an array, etc. In addition, there can be various ways to generate log identifiers. For ease of understanding and reasonable layout, the following will be described in this application. Figure 2 Specific description is given in the optional embodiments.
[0048] S103, sending a filtering list to the client; the filtering list is used by the client to prohibit uploading the to-be-uploaded logs that match the log identifiers in the filtering list to the server.
[0049] In specific applications, there are many opportunities to send the filter list. For example, the server can send the filter list when it obtains the filter list; or the server can send the filter list when it receives a request from the client to obtain the filter list; or the server can send the filter list when the filter list is updated, which are all reasonable.
[0050] In the solution provided by the present application, the server obtains analysis data for business analysis of the application, and the analysis data includes usage information of historical logs uploaded by the client during the business analysis process; determines whether the usage information of the historical logs meets the preset usage conditions; adds the log identifiers of the historical logs that do not meet the preset usage conditions to the filter list; the filter list is used by the client to prohibit uploading the logs to be uploaded that match the log identifiers in the filter list to the server. Among them, the filter list is obtained by adding the log identifiers of the historical logs that do not meet the preset usage conditions. Therefore, the log identifier matches the filter list, indicating that the log to be uploaded does not meet the preset usage conditions for the use of the log by the business analysis, which is equivalent to a useless log. Therefore, the client prohibits uploading the logs to be uploaded that match the filter list to the server, which can reduce the waste of upload resources and computing resources caused by uploading useless logs.
[0051] In an optional implementation, after adding the log identifiers of the historical logs whose usage information does not meet the preset usage conditions to the filtering list, the log processing method provided in the embodiment of the present application may further include the following steps:
[0052] Determine the type label of each user based on the historical logs of each user;
[0053] For each type of tag, determine the target log identifier for recording the same user behavior for different users under the same type of tag;
[0054] Identify whether the filter list contains the target log identifier;
[0055] If included, the target log identifier is removed from the filter list to obtain an updated filter list, and the updated filter list is sent to the client.
[0056] In a specific application, based on the historical logs of each user, the type label of each user can be determined in multiple ways. Exemplarily, the historical logs of each user can be respectively input into the first classification model obtained by pre-training to obtain the type label of the user. Among them, the first classification model is a neural network model trained using sample logs of sample users and type labels of sample users. Or, exemplarily, the historical logs of all users can be clustered to obtain multiple clusters, and the type label of the cluster can be determined based on the user behavior recorded in the historical logs of each cluster, and the type label is used as the type label of the user to whom the historical log in the cluster belongs. Among them, clustering refers to dividing similar data together. When dividing specifically, the label of this type is not concerned. The division goal is to aggregate similar data together. Clustering is an unsupervised learning method. In order to facilitate understanding and reasonable layout, the second example will be specifically described in the form of an optional embodiment.
[0057] Moreover, the target log identifier for recording the same user behavior for different users under the same type of label indicates that the same user behavior exists in different users belonging to the same type of label, and the same user behavior is recorded in the historical log corresponding to the target log identifier. Therefore, the user behaviors recorded in the historical logs of different users under the same type of label can be compared, and the log identifier corresponding to the same historical log of the recorded user can be used as the target log identifier. On this basis, the target log identifier is included in the filter list, indicating that the logs that can represent the same behavior of different users under a certain type of label will be filtered. Therefore, by removing the target log identifier from the filter list, obtaining an updated filter list, and sending the updated filter list to the client, the situation that the logs that can represent the same behavior of different users under a certain type of label will be filtered can be reduced. Therefore, this optional embodiment can ensure that the log corresponding to the target log identifier is re-uploaded by the client to obtain the same behavior of different users under a certain type of label, thereby improving the accuracy of the analysis results of the subsequent use of the log for business analysis of the application program.
[0058] Exemplarily, in the history logs of multiple users with the type label of "one-time user", there is a history log L1 that records the user behavior of "click to view the User Privacy Agreement", and the identifier corresponding to the history log L1 is the target log identifier ID1. In addition, the log identifier ID1 is added to the filter list because the history log L1 is not used for business analysis of the application. In this way, the filter list contains the target log identifier ID1. Therefore, the target log identifier ID1 is removed from the filter list, and an updated filter list is obtained, and the updated filter list is sent to the client. When the client uploads the log according to the updated filter list, it uploads the log corresponding to the target log identifier ID1 that records the user behavior of "click to view the User Privacy Agreement". In this way, the operation and maintenance personnel can analyze the business of the "User Privacy Agreement" in the application based on the log that records the user behavior of "click to view the User Privacy Agreement", and obtain the analysis result: the description of the user privacy information in the content of the "User Privacy Agreement" is unclear, resulting in some users becoming "one-time users": the loss of users with the type label of "one-time users". Therefore, the operation and maintenance personnel modify the description of user privacy information in the "User Privacy Agreement" to reduce the user churn rate. It can be seen that this optional embodiment can ensure that the log corresponding to the target log identifier is re-uploaded by the client to obtain the same behavior of different users under a certain type of tag, thereby improving the accuracy of the analysis results of subsequent business analysis of the application using the log.
[0059] In an optional implementation, the above-mentioned determination of the type label of each user based on the historical logs of each user may specifically include the following steps:
[0060] Cluster all users' historical logs to obtain multiple clusters;
[0061] According to the user behaviors recorded in the historical logs of each cluster, a type label of the cluster is determined, and the type label is used as the type label of the user to whom the historical logs in the cluster belong.
[0062] In a specific application, the clustering algorithm used to cluster the historical logs of all users may be multiple. Exemplarily, the clustering algorithm may be a k-means clustering algorithm, a k-medoids clustering algorithm, and the like. Any clustering algorithm that can cluster the historical logs of all users may be used in this application, and this embodiment does not limit this.
[0063] The contents of historical logs in the same cluster are similar. Therefore, the type label of the cluster determined based on the user behavior recorded in the historical logs of each cluster can represent the type label of the user to whom the historical logs belong in the cluster. Therefore, the effect of dividing the user types according to the user behavior recorded in the historical logs can be achieved. Exemplarily, the user behavior recorded in the historical logs may include starting the game, game duration, click behavior, number of clicks, type of page entered, page browsing route, and payment behavior, etc. Among them, the page browsing route refers to the order in which the user browses the pages, such as browsing the user page first, then browsing the homepage, and finally browsing the display page of a certain image, etc. Payment behavior may include payment time, payment amount, and purchase object, etc.
[0064] And, exemplarily, determining the type label of the cluster based on the user behavior recorded in the historical logs of each cluster may specifically include: for each cluster, from the preset correspondence between user behavior and type label, searching for the type label corresponding to the user behavior recorded in the historical logs of the cluster as the type label of the cluster. For example, user behavior: only launching the application once, corresponding type label: "one-time user"; user behavior: the cumulative payment amount is greater than the amount threshold or the number of recharges within the preset time reaches the number threshold, corresponding type label: the type label is "high krypton". Or, exemplarily, determining the type label of the cluster based on the user behavior recorded in the historical logs of each cluster may specifically include: for each cluster, inputting the user behavior recorded in the historical logs of the cluster into a pre-trained second classification model to obtain the type label of the cluster. Among them, the second classification model is a neural network model trained using the user behavior recorded in the sample logs and the type label corresponding to the sample logs.
[0065] This optional embodiment determines the user's tag type by clustering the user's historical logs, so there is no need to obtain additional user data for determining the tag type; and clustering is unsupervised learning, and no pre-training is required. Therefore, the efficiency of log processing can be improved and the development cost can be reduced.
[0066] In an optional implementation, the log processing method provided in the embodiment of the present application may further include the following steps:
[0067] Determine the type label of each user based on the historical logs of each user;
[0068] Determine the sampling weight corresponding to each type of label;
[0069] For each type of label, determine the target user of the label according to the sampling weight corresponding to the label and the number of users with the label;
[0070] Send an enable command to the client of the target user, and the enable command is used to instruct the client of the target user to upload the logs to be uploaded that do not match the filter list to the server.
[0071] This optional embodiment, based on the filter list, sends an open instruction to the client of the target user, instructing the client of the target user to upload the logs to be uploaded that do not match the filter list to the server, which is equivalent to further filtering the clients allowed to upload logs according to the user tag type. Therefore, the waste of upload resources and computing resources caused by useless logs can be further reduced.
[0072] In a specific application, the type label of each user is determined based on the historical logs of each user, which is the same as the step of determining the type label in the embodiment of updating the filter list, and will not be repeated here. For details, please refer to the description of the embodiment of updating the filter list. In addition, the sampling weight corresponding to any type label is used to determine the number of target users who are allowed to upload logs among the users with the type label. There are many ways to determine the sampling weight, which are specifically described in the form of optional embodiments below.
[0073] In an optional implementation, the above-mentioned determination of the sampling weight corresponding to each type of label may include the following steps:
[0074] From the preset correspondence between type labels and sampling weights, find the sampling weight corresponding to each type label.
[0075] The sampling weights in the correspondence between the preset type labels and the sampling weights may be set based on experience.
[0076] In another optional implementation, the above-mentioned determination of the sampling weight corresponding to each type of label may specifically include the following steps:
[0077] For each type of tag, calculate the benchmark user ratio of the type of tag based on the total number of users and the number of users with the type of tag, and use the benchmark user ratio to determine the ratio threshold;
[0078] According to the preset offset, the initial sampling weight is adjusted to obtain the adjusted sampling weight;
[0079] For each type of tag, determine the target user of the type of tag according to the adjusted sampling weight of the type of tag and the number of users of the type of tag;
[0080] For each type of label, determine the percentage of verified users corresponding to the adjusted sampling weight based on the total number of users and the number of target users of that type of label;
[0081] If the proportion of verified users does not reach the proportion threshold, the sampling weight is further adjusted according to the preset offset, and the target users of each type of label are determined according to the adjusted sampling weight of the type of label and the number of users of the type of label.
[0082] If the proportion of verified users reaches the proportion threshold, the last adjusted sampling weight is determined as the sampling weight corresponding to the type of label.
[0083] In specific applications, the proportion of benchmark users of any type of label is the proportion of the number of users of this type of label in the total number of users. For example, the total number of users, that is, the number of all users of the application, is 10 million, and the number of users with the label type "Gaokrypton" is 300,000, then the proportion of benchmark users with the label type "Gaokrypton" is 3%. In addition, there are many ways to determine the proportion threshold using the benchmark user proportion. Exemplarily, it can be the benchmark user proportion minus or plus a preset proportion of the benchmark user proportion to obtain the proportion threshold, for example, the proportion threshold = 3% ± 3% × 5%, that is, the proportion threshold is 2.85% or 3.15%. Alternatively, it can be to find the proportion threshold corresponding to the benchmark user proportion of this type of label from the correspondence between the preset benchmark user proportion and the proportion threshold. For any type of label, the proportion threshold of this type of label can ensure that the number of target users subsequently determined according to the adjusted sampling weight of this type of label can meet the upward or downward adjustment requirements relative to the benchmark user.
[0084] In addition, the initial sampling weight can be set according to experience or the need to increase or decrease relative to the benchmark users. For example, the number of users with the tag type "high krypton" is 300,000, and the benchmark users with the tag type "high krypton" account for 3%: the initial sampling weight can be 100%; or, if the upward adjustment requirement is 0.05%, the initial sampling weight can be (30+30×0.05%)÷30=105%; or if the downward adjustment requirement is 0.05%, the initial sampling weight can be (30-30×0.05%)÷30=95%.
[0085] Furthermore, adjusting the initial sampling weight according to the preset offset to obtain the adjusted sampling weight may include: calculating the difference between the initial sampling weight and the preset offset to obtain the adjusted sampling weight. The preset offset may correspond to the upward or downward adjustment requirements relative to the benchmark user, and may carry a positive or negative sign. For example, Wi * =Wi-offset, Wi is the initial sampling weight, Wi * The preset offset is the adjusted sampling weight and can be a positive or negative value.
[0086] And, for each type of label, according to the number of target users of the type of label and the total number of users, determine the proportion of verified users corresponding to the adjusted sampling weight, including: calculating the proportion of the number of target users of the type of label in the total number of users, and obtaining the proportion of verified users corresponding to the adjusted sampling weight. Exemplarily, the initial sampling weight is 1, i.e., 100%, the preset offset offset is 5%, and the proportion threshold is less than the proportion threshold of 2.85%. In this regard, the adjusted sampling weight is 95%. The proportion of verified users corresponding to the adjusted sampling weight is [(30×95%)÷1000]×100%=2.85%. The proportion of verified users of 2.85% does not reach the proportion threshold. Continue to adjust the sampling weight: 95%-5%=90%, at this time the adjusted sampling weight is 90%, and the proportion of verified users corresponding to the adjusted sampling weight is [(30×90%)÷1000]×100%=2.7%. The proportion of verified users of 2.7% is less than 2.85%, reaching the proportion threshold. Therefore, the last adjusted sampling weight of 95% is determined as the sampling weight corresponding to the type of label.
[0087] This optional embodiment determines a percentage threshold for each type of tag based on the total number of users and the number of users of the type of tag, and continuously adjusts the sampling weight, and then determines the target users in the type of tag that are allowed to upload logs according to the adjusted sampling weight that reaches the percentage threshold. In this way, when the total number of users and the number of users under the type tag change, the sampling weight changes accordingly. Therefore, the number of target users allowed to report logs can be adjusted more reasonably and dynamically, improving the rationality of reducing useless logs in log processing. In addition, the method for determining the sampling weight in the corresponding relationship between the preset type tag and the sampling weight can be similar to the method for determining the sampling weight in this optional embodiment, with the difference that: when determining the sampling weight in the corresponding relationship between the preset type tag and the sampling weight, the total number of users and the number of users of each type of tag can be determined based on sample users, or the total number of users and the number of users of each type of tag can be set using a simulation tool.
[0088] Furthermore, for each type of tag, there may be multiple ways to determine the target users of the type of tag according to the sampling weight corresponding to the type of tag and the number of users of the type of tag, which are specifically described below in the form of optional embodiments.
[0089] In an optional implementation, for each type of tag, according to the sampling weight corresponding to the type of tag and the number of users with the type of tag, determining the target user of the type of tag may specifically include the following steps:
[0090] For each type of tag, calculate the product of the sampling weight corresponding to the type of tag and the number of users with the type of tag to obtain the target number of target users allowed to enable uploading;
[0091] Among the users of this type of label, a target number of users who meet the preset user conditions are selected as the target users of this type of label.
[0092] There may be multiple preset user conditions. For example, the preset user conditions may be users whose recharge amount reaches a threshold amount, or users whose membership level reaches a level threshold, or users whose total time of using the application reaches a time threshold. Alternatively, among users with this type of label, a target number of users are selected according to the size of the user identifier as target users of this type of label.
[0093] For example, for any type of tag: the user identifier is a numerical value with the last digit randomly distributed between 00 and 99, and the sampling weight is 80%, then users whose last two digits of the user identifier are 00 to 79 are selected as target users. Alternatively, the user identifier is a numerical value, and a target number of users are selected as target users in the order of the user identifier from small to large or from large to small.
[0094] In another optional implementation, determining the target user of the type tag according to the sampling weight corresponding to the type tag and the number of users of the type tag may specifically include the following steps:
[0095] Calculate the product of the sampling weight corresponding to the type of label and the number of users with the type of label to obtain the target number of target users allowed to enable uploading;
[0096] Among the users with this type of label, a target number of users are randomly selected as target users.
[0097] This optional embodiment randomly selects a target number of users as target users, which can ensure the randomness of target user selection, reduce the problem of some users always being target users and other users never being target users caused by non-random selection, and improve the rationality of log processing.
[0098] In specific applications, there can be many ways of random selection. For example, each user can be marked with a non-repeating value between [1-N], arranged by the size of the mark, and a target number of users are selected as target users in turn. Where N is the number of users with this type of label. Or, for example, a random number between 0 and 1 is generated for each user, and the random number corresponding to the user and the sampling weight of this type of label are input into the score formula to obtain the score of the user; the users of this type of label are sorted from small to large according to the score of each user, and the first target number of users are selected as target users. Wherein, the score formula is Si=R 1 / Wi, where Si is the user's score, Wi is the sampling weight of the type label, and R is the random number corresponding to the user.
[0099] In an optional implementation, after determining the target user of the type tag according to the sampling weight corresponding to the type tag and the number of users of the type tag, the log processing method provided in the embodiment of the present application may further include the following steps:
[0100] Send a close command to the client of the non-target user, the close command is used to instruct the client of the non-target user to prohibit uploading the to-be-uploaded logs that do not match the filter list to the server, and the client of the non-target user is other clients except the client of the target user.
[0101] This optional embodiment can instruct the clients of non-target users, that is, other clients except the client of the target user, to prohibit uploading logs to be uploaded that do not match the filter list to the server by sending a close instruction to the clients of non-target users, thereby further reducing the uploading of useless logs and the computing cost.
[0102] Figure 2 A flow chart of a log processing method provided according to another embodiment of the present application is shown, which is applied to a client and specifically includes the following steps:
[0103] S201, receiving a filter list sent by a server; wherein the filter list is obtained by the server adding log identifiers of historical logs whose usage information does not meet preset usage conditions; the usage information refers to the usage of historical logs in the process of business analysis of the application.
[0104] In a specific application, the client can send a data request to the server and receive a filter list returned by the server upon receiving the data request. Alternatively, the client can directly receive a filter list actively sent by the server. In addition, the server determines the filter list in the same manner as in the present application. Figure 1 The method for determining the filter list in the embodiment is the same and will not be repeated here. For details, see this application Figure 1 Description of embodiments.
[0105] S202: Determine the log identifier of the log to be uploaded.
[0106] In a specific application, there may be multiple ways to determine the log identifier of the log to be uploaded. Exemplarily, the log to be uploaded may be numbered according to the generation time of the log to be uploaded in a sequential numbering manner, and the number may be used as the log identifier of the log to be uploaded. For example, the log identifier may be a continuous number such as 1, 2, 3, etc. Or, exemplarily, the attribute information and control information of the client page corresponding to the user behavior recorded in the log to be uploaded may be obtained, and the attribute information and control information may be integrated, a hash value may be taken for the integrated information, and the hash value may be used as the log identifier of the log to be uploaded. In order to facilitate understanding and reasonable layout, the second exemplary case will be specifically described in the form of an optional embodiment.
[0107] S203: If the log identifier matches the filter list, uploading the log to be uploaded to the server is prohibited.
[0108] Among them, prohibiting the logs to be uploaded from being uploaded to the server may specifically include: discarding the logs to be uploaded, or storing the logs to be uploaded so that after the log identifier of the log to be uploaded is removed from the filter list, the log to be uploaded is used for business analysis of the application.
[0109] In the solution provided by the present application, the server obtains analysis data for business analysis of the application, and the analysis data includes usage information of historical logs uploaded by the client during the business analysis process; determines whether the usage information of the historical logs meets the preset usage conditions; adds the log identifiers of the historical logs that do not meet the preset usage conditions to the filter list; the filter list is used by the client to prohibit uploading the logs to be uploaded that match the log identifiers in the filter list to the server. Among them, the filter list is obtained by adding the log identifiers of the historical logs that do not meet the preset usage conditions. Therefore, the log identifier matches the filter list, indicating that the log to be uploaded does not meet the preset usage conditions for the use of the log by the business analysis, which is equivalent to a useless log. Therefore, the client prohibits uploading the logs to be uploaded that match the filter list to the server, which can reduce the waste of upload resources and computing resources caused by uploading useless logs.
[0110] Corresponding to the above method embodiment, the present application also provides a log processing system embodiment, Figure 3 FIG. 1 shows a schematic diagram of the structure of a log processing system provided by an embodiment of the present application. Figure 3 As shown, the system includes: a server 301 and a client 302;
[0111] The server 301 is configured to obtain analysis data for business analysis of the application, the analysis data including usage information of historical logs during the business analysis process; add log identifiers of historical logs whose usage information does not meet the preset usage conditions to a filter list; and send the filter list to the client 302;
[0112] The client 302 is configured to receive the filter list sent by the server; determine the log identifier of the log to be uploaded; if the log identifier matches the filter list, prohibit uploading the log to be uploaded to the server.
[0113] For ease of understanding, the following is an exemplary description of the application. Figure 3 The process of the log processing method of the log processing system shown in FIG. Figure 4 As shown, another embodiment of the present application provides a log processing method applied to a log processing system, comprising the following steps:
[0114] S401, the server obtains analysis data for business analysis of the application.
[0115] The analysis data includes the usage information of historical logs during the business analysis process.
[0116] S402, the server adds the log identifiers of the historical logs whose usage information does not meet the preset usage conditions to the filtering list.
[0117] S403, the server sends the filtering list to the client.
[0118] S404, the client receives the filtering list sent by the server, and determines the log identifier of the log to be uploaded.
[0119] S405: When the log identifier matches the filter list, the client prohibits uploading the log to be uploaded to the server.
[0120] The above is a schematic scheme of a log processing system of this embodiment. It should be noted that the technical scheme of the log processing device and the technical scheme of the log processing method described above are of the same concept, and the details not described in detail in the technical scheme of the log processing device can be referred to the description of the technical scheme of the log processing method described above.
[0121] In the solution provided by the present application, the server obtains analysis data for business analysis of the application, and the analysis data includes usage information of historical logs uploaded by the client during the business analysis process; determines whether the usage information of the historical logs meets the preset usage conditions; adds the log identifiers of the historical logs that do not meet the preset usage conditions to the filter list; the filter list is used by the client to prohibit uploading the logs to be uploaded that match the log identifiers in the filter list to the server. Among them, the filter list is obtained by adding the log identifiers of the historical logs that do not meet the preset usage conditions. Therefore, the log identifier matches the filter list, indicating that the log to be uploaded does not meet the preset usage conditions for the use of the log by the business analysis, which is equivalent to a useless log. Therefore, the client prohibits uploading the logs to be uploaded that match the filter list to the server, which can reduce the waste of upload resources and computing resources caused by uploading useless logs.
[0122] In an optional implementation, after determining the log identifier of the log to be uploaded, the log processing method provided in the embodiment of the present application may further include the following steps:
[0123] If a shutdown command is received from the server, uploading the logs to be uploaded to the server is prohibited;
[0124] If the opening instruction sent by the server is received and the log identifier does not match the filter list, the log to be uploaded will be uploaded to the server.
[0125] The closing instruction and the opening instruction of this optional embodiment are respectively the same as those of the present application. Figure 1 The closing instruction and the opening instruction in the optional embodiment are the same and will not be repeated here. Figure 1 Description of optional embodiments. In addition, the present application manages whether to upload logs through closing instructions and opening instructions, thereby further reducing the waste of resources transmitted on a large number of logs on the basis of filtering useless logs by using a filter list.
[0126] In an optional implementation manner, the above-mentioned determination of the log identifier of the log to be uploaded may specifically include the following steps:
[0127] Obtaining the property information and control information of the target client page, where the target client page is the client page corresponding to the user behavior of the log record to be uploaded;
[0128] Integrate the attribute information and the control information to obtain integrated information;
[0129] Determine the hash value of the integrated information as the log identifier of the log to be uploaded.
[0130] This embodiment uses the attribute information and control information of the target client page to determine the log identifier, which can ensure that the log identifier corresponds to the client page corresponding to the user behavior of the log record to be uploaded, and the log itself can be used, reducing the additional process of obtaining the log identifiers of other logs caused by determining the identifier by number. Among them, the attribute information of the target client page can specifically include: page name and module name. Exemplarily, the page name is a unified name defined for the page during the page development process, such as: the page name of the home page is "home", the page name of the user information page is "user", and so on. The module name is a unified name defined for the module that implements a certain function during the page development process, such as: the module implements the advertising carousel function, the module name is "banner", the module implements the recommendation list function, the module name is "recommend_list", and so on.
[0131] Furthermore, in a specific application, corresponding to different log identifiers, the control information of the target client page is different, which is described below in the form of an optional embodiment.
[0132] In an optional implementation, the log identifier includes: an absolute log identifier indicating the uniqueness of the log to be uploaded, and the control information includes: a control identifier, a position index of a parent control to which the control belongs, a hierarchical structure of the control, and data processed by the control.
[0133] In this optional embodiment, different pages have different page names, different modules have different module names, different controls have different control types, and the same page, the same module, and the same control type have different position indexes of the parent controls where the controls are located and the control's own attribute content, so as to ensure the uniqueness of the absolute log identifier, that is, the absolute log identifier of each log is different.
[0134] In another optional implementation, the log identifier includes: a relative log identifier indicating the uniqueness of the log type to which the to-be-uploaded log belongs, and the control information includes: a hierarchical structure of the control.
[0135] In this optional embodiment, the log types to be uploaded are the same, which means that the hierarchical structure of the controls is the same. Therefore, in order to ensure that the relative log identifiers of the logs to be uploaded that belong to the same log type are the same, for the relative log identifier, the control information includes: the hierarchical structure of the control. The relative log identifier can be used to obtain the logs triggered by the same type of controls, and then use the logs triggered by the same type of controls to perform business analysis corresponding to the same type of controls. It can be seen that this optional embodiment can improve the convenience and scope of application of using logs for business analysis and log processing.
[0136] Moreover, a control is an object with which a user can interact in order to input or manipulate data. The above control identifier is the id set for the control during the page development process, which is unique in the entire application, such as: the control identifier of the login button is "login_btn". Moreover, during the application development process, if the layout of the page belongs to the nested mode, the control nested is the child control of another control in which the control is nested. For example, control C1 is nested with control C2, and control C2 is nested with control C3. Then, in this nested scenario, control C1 is the parent control of control C2 and control C3; control C2 is the parent control of control C3. Exemplarily, the position index of the parent control to which the control belongs can be: the parent control is a list control, and the child control is 10 option controls nested in the list control, then the position index of the parent control to which the first control belongs is 0. The hierarchical structure of the control is to trace the type of the parent control from the control itself, all the way back to the root node of the page, and the types of all the parent controls traced back are spliced with the "$" symbol to obtain the hierarchical structure of the control. Exemplarily, the data processed by a text control is the text data set by the text control; the data processed by a picture control is the image data processed by the control; for some complex controls, the data processed may include image data, text data, links and other data.
[0137] For example, a homepage recommended product list contains a total of 100 products. During the page development process, a group control (ViewGroup) is set to display the information of these 100 products. When the first product is clicked, the tracking point is triggered to generate a log. For the log generated by clicking the first product, the absolute log identifier (distinct_id) and the relative log identifier (lite_distinct_id) are determined. Similarly, when the second product is clicked, the tracking point is also triggered to generate a log. For the log generated by clicking the second product, the absolute log identifier (distinct_id) and the relative log identifier (lite_distinct_id) are determined. Among them, the position index of the parent control to which the control corresponding to the first product and the control corresponding to the second product belong is different. Therefore, the absolute log identifier of the log generated by clicking the first product is different from that of the log generated by clicking the second product. In addition, the parent control to which the control corresponding to the first product and the control corresponding to the second product belong is the same group control. Therefore, the relative log identifier of the log generated by clicking the first product and the log generated by clicking the second product is the same. When performing business analysis on an application, you can directly retrieve the click status of the home page recommendation list using relative log identifiers, or you can retrieve the difference in click volume for each position in the home page recommendation list using different absolute log identifiers.
[0138] For ease of understanding, the following exemplary description is given of the architecture diagram of the integrated solution of the above-mentioned embodiments of the present application and the processing flow.
[0139] For example, Figure 5 As shown. Take the application as a game application as an example. The architecture of the log processing method provided in the embodiment of the present application may include: a business layer, a protocol layer, a collection layer, a storage layer, a transport layer, a data layer and a service layer. Among them, the business layer includes a game software development kit, a data center and a game center, which can realize the functions of the game application and provide services about the game. The protocol layer can submit user information and obtain filtering strategies. User information can specifically include data such as logs and user information. The filtering strategy may include a filter list, an opening instruction and a closing instruction. The collection layer can collect device information, network information, user information, environmental information and behavior information. The storage layer can store data through memory cache, database cache and memory mapped file method (MMAP). The transport layer reports data and performs disaster recovery degradation. The data layer uses an open source data analysis and visualization platform (Kibana), an application (Kafka) for building real-time data pipelines and streaming, and a data warehouse tool (hive) to analyze, manage and display data. The service layer can perform data analysis, policy issuance, machine learning and weight calculation. Among them, the strategy includes a filter list, an opening instruction and a closing instruction. Machine learning is used to obtain the user's type label, and weight calculation is used to determine the sampling weight.
[0140] For example, Figure 6 Another embodiment of the present application provides a log processing method process, including the following steps:
[0141] The client starts the application, submits user information to the server, obtains the policy from the server, triggers the full embedding point, generates a log identifier, and determines whether the log is filtered. If filtered, log reporting is prohibited and the process ends. If not filtered, logs are stored or reported, and the process ends.
[0142] The computing device uses machine learning to analyze the data to be analyzed from the data warehouse, including: analyzing the use of logs, automatically synchronizing the continuous unused data to the filtering strategy, and analyzing the activity, retention rate, conversion rate, etc., automatically setting the weights, and synchronizing the weights to the server.
[0143] The server receives data synchronized by the computing device, determines a filtering strategy, and receives and stores logs uploaded by the client to obtain a data warehouse; wherein the determined filtering strategy may include a filtering list, an opening instruction, and a closing instruction.
[0144] In this embodiment, the steps performed by the client are the same as those described above. Figure 2 The steps performed by the client in the embodiment and the optional embodiment are similar, except that the description method is different for the sake of simplicity. Figure 1 The steps performed by the server in the embodiment and the optional embodiment are similar, except that the description method is different for the sake of simplicity. In addition, the computing device in this embodiment may specifically be a server. The same parts will not be repeated here, and the details are described in the above embodiments. The specific differences are:
[0145] For the client: user information is equivalent to logs; obtaining strategies from the server is equivalent to receiving filtering strategies sent by the server. The filtering strategies may include: opening instructions or closing instructions, and filtering lists.
[0146] For the server: Analyze the usage of logs, and if they are not used continuously, automatically synchronize them to the filtering strategy, which is equivalent to the step of determining the filtering list; analyze the activity, retention rate, conversion rate, etc., and automatically set the weight, which is equivalent to obtaining the user's type label, and determining the sampling weight corresponding to the type label. Among them, the analysis of activity, retention rate, and conversion rate are all reflected by the user behavior recorded in the log. Determining the filtering strategy is equivalent to sending the filtering list to the client, and sending an opening instruction or a closing instruction to the client. The data to be analyzed includes: the user's log, and the log usage data.
[0147] Corresponding to the above method embodiment, the present application also provides a log processing device embodiment, Figure 7 A structural schematic diagram of a log processing device applied to a server provided in an embodiment of the present application is shown.
[0148] like Figure 7 As shown, the device comprises:
[0149] The data acquisition module 701 is configured to acquire analysis data for business analysis of the application, wherein the analysis data includes usage information of historical logs during the business analysis process;
[0150] The filter list determination module 702 is configured to add the log identifiers of the historical logs whose usage information does not meet the preset usage conditions to the filter list;
[0151] The sending module 703 is configured to send the filtering list to the client; the filtering list is used by the client to prohibit uploading the to-be-uploaded logs that match the log identifiers in the filtering list to the server.
[0152] In the solution provided by the present application, the server obtains analysis data for business analysis of the application, and the analysis data includes usage information of historical logs uploaded by the client during the business analysis process; determines whether the usage information of the historical logs meets the preset usage conditions; adds the log identifiers of the historical logs that do not meet the preset usage conditions to the filter list; the filter list is used by the client to prohibit uploading the logs to be uploaded that match the log identifiers in the filter list to the server. Among them, the filter list is obtained by adding the log identifiers of the historical logs that do not meet the preset usage conditions. Therefore, the log identifier matches the filter list, indicating that the log to be uploaded does not meet the preset usage conditions for the use of the log by the business analysis, which is equivalent to a useless log. Therefore, the client prohibits uploading the logs to be uploaded that match the filter list to the server, which can reduce the waste of upload resources and computing resources caused by uploading useless logs.
[0153] In an optional implementation, the device further includes: a type tag determination module configured to:
[0154] Determine the type label of each user based on the historical logs of each user;
[0155] For each type of tag, determine the target log identifier for recording the same user behavior for different users under the same type of tag;
[0156] Identifying whether the target log identifier is included in the filtering list;
[0157] If included, the target log identifier is removed from the filter list to obtain an updated filter list, and the updated filter list is sent to the client.
[0158] In an optional implementation, the device further includes: a type label determination module configured to: determine a type label for each user based on the historical logs of each user;
[0159] The device further comprises: a weight determination module configured to:
[0160] Determine the sampling weight corresponding to each type of label;
[0161] For each type of label, determine the target user of the label according to the sampling weight corresponding to the label and the number of users with the label;
[0162] Send an opening instruction to the client of the target user, wherein the opening instruction is used to instruct the client of the target user to upload the logs to be uploaded that do not match the filtering list to the server.
[0163] In an optional implementation, the weight determination module is further configured as follows:
[0164] For each type of tag, calculate the benchmark user ratio of the type of tag based on the total number of users and the number of users of the type of tag, and use the benchmark user ratio to determine the ratio threshold;
[0165] According to the preset offset, the initial sampling weight is adjusted to obtain the adjusted sampling weight;
[0166] For each type of tag, determine the target user of the type of tag according to the adjusted sampling weight of the type of tag and the number of users of the type of tag;
[0167] For each type of label, determine the proportion of verified users corresponding to the adjusted sampling weight according to the total number of users and the number of target users of the type of label;
[0168] If the proportion of verified users does not reach the proportion threshold, the sampling weight is further adjusted according to the preset offset, and the process is repeated for each type of tag, and the target user of the type of tag is determined according to the adjusted sampling weight of the type of tag and the number of users of the type of tag;
[0169] If the proportion of verified users reaches the proportion threshold, the last adjusted sampling weight is determined as the sampling weight corresponding to the type of label.
[0170] In an optional implementation, the weight determination module is further configured to:
[0171] Calculate the product of the sampling weight corresponding to the type of label and the number of users with the type of label to obtain the target number of target users allowed to enable uploading;
[0172] Among the users with this type of label, the target number of users are randomly selected as target users.
[0173] In an optional implementation, the weight determination module is further configured to:
[0174] After determining the target user of the type tag according to the sampling weight corresponding to the type tag and the number of users of the type tag, a close instruction is sent to the client of the non-target user, wherein the close instruction is used to instruct the client of the non-target user to prohibit uploading the logs to be uploaded that do not match the filter list to the server, and the client of the non-target user is other clients except the client of the target user.
[0175] In an optional implementation, the type tag determination module is further configured to:
[0176] Cluster all users' historical logs to obtain multiple clusters;
[0177] According to the user behaviors recorded in the historical logs of each cluster, a type label of the cluster is determined, and the type label is used as the type label of the user to whom the historical logs in the cluster belong.
[0178] Corresponding to the above method embodiment, the present application also provides a log processing device embodiment, Figure 8 A schematic diagram of the structure of a log processing device applied to a client provided in an embodiment of the present application is shown.
[0179] like Figure 8 As shown, the device comprises:
[0180] The receiving module 801 is configured to receive a filtering list sent by the server; wherein the filtering list is obtained by the server adding a log identifier of a historical log whose usage information does not meet a preset usage condition; the usage information refers to the usage of the historical log in the process of business analysis of the application;
[0181] The log identification determination module 802 is configured to determine the log identification of the log to be uploaded;
[0182] The log upload management module 803 is configured to prohibit uploading the log to be uploaded to the server if the log identifier matches the filter list.
[0183] In the solution provided by the present application, the server obtains analysis data for business analysis of the application, and the analysis data includes usage information of historical logs uploaded by the client during the business analysis process; determines whether the usage information of the historical logs meets the preset usage conditions; adds the log identifiers of the historical logs that do not meet the preset usage conditions to the filter list; the filter list is used by the client to prohibit uploading the logs to be uploaded that match the log identifiers in the filter list to the server. Among them, the filter list is obtained by adding the log identifiers of the historical logs that do not meet the preset usage conditions. Therefore, the log identifier matches the filter list, indicating that the log to be uploaded does not meet the preset usage conditions for the use of the log by the business analysis, which is equivalent to a useless log. Therefore, the client prohibits uploading the logs to be uploaded that match the filter list to the server, which can reduce the waste of upload resources and computing resources caused by uploading useless logs.
[0184] In an optional implementation, the log upload management module 803 is further configured to:
[0185] If a closing instruction is received from the server, uploading the log to be uploaded to the server is prohibited;
[0186] If an opening instruction sent by the server is received and the log identifier does not match the filter list, the log to be uploaded is uploaded to the server.
[0187] In an optional implementation, the log identifier determination module 802 is further configured to:
[0188] Acquire attribute information and control information of a target client page, where the target client page is a client page corresponding to the user behavior of the log record to be uploaded;
[0189] Integrate the attribute information and the control information to obtain integrated information;
[0190] A hash value of the integrated information is determined as a log identifier of the log to be uploaded.
[0191] In an optional implementation, the log identifier includes: an absolute log identifier indicating the uniqueness of the log to be uploaded, and the control information includes: a control identifier, a position index of a parent control to which the control belongs, a hierarchical structure of the control, and data processed by the control.
[0192] In an optional implementation, the log identifier includes: a relative log identifier indicating the uniqueness of a log type to which the log to be uploaded belongs, and the control information includes: a hierarchical structure of controls.
[0193] The above is a schematic scheme of a log processing device of this embodiment. It should be noted that the technical scheme of the log processing device and the technical scheme of the log processing method described above are of the same concept, and the details not described in detail in the technical scheme of the log processing device can be found in the description of the technical scheme of the log processing method described above.
[0194] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present application is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and the database 950 is used to store data.
[0195] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0196] In one embodiment of the present application, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.
[0197] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 900 may also be a mobile or stationary server.
[0198] Among them, when the processor 920 executes the instructions, the steps of the log processing method are implemented.
[0199] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the log processing method described above are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the log processing method described above.
[0200] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the log processing method as described above.
[0201] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the log processing method described above are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the log processing method described above.
[0202] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0203] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0204] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0205] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0206] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.
Claims
1. A log processing method, characterized in that: Applied to the server, the method includes: Acquire analysis data for business analysis of the application, the analysis data including usage information of historical logs during the business analysis process; Adding log identifiers of historical logs whose usage information does not meet the preset usage conditions to the filter list; Send the filtering list to the client; the filtering list is used by the client to prohibit uploading the to-be-uploaded logs matching the log identifiers in the filtering list to the server; The method further comprises: Determine the target user according to the type label of each user, the sampling weight corresponding to each type label and the number of users corresponding to each type label, and the client of the target user uploads the logs to be uploaded that do not match the filter list to the server; Determining the sampling weight corresponding to each type of label includes: For each type of tag, calculate the benchmark user ratio of the type of tag based on the total number of users and the number of users of the type of tag, and use the benchmark user ratio to determine the ratio threshold; According to the preset offset, the initial sampling weight is adjusted to obtain the adjusted sampling weight; For each type of label, determine the proportion of verified users corresponding to the adjusted sampling weight according to the total number of users and the number of target users of the type of label; If the proportion of verified users does not reach the proportion threshold, the sampling weight continues to be adjusted according to the preset offset; If the proportion of verified users reaches the proportion threshold, the last adjusted sampling weight is determined as the sampling weight corresponding to the type of label.
2. The method according to claim 1, characterized in that After adding the log identifiers of the historical logs whose usage information does not meet the preset usage conditions to the filtering list, the method further includes: Determine the type label of each user based on the historical logs of each user; For each type of tag, determine the target log identifier for recording the same user behavior for different users under the same type of tag; Identifying whether the target log identifier is included in the filtering list; If included, the target log identifier is removed from the filter list to obtain an updated filter list, and the updated filter list is sent to the client.
3. The method according to claim 1, characterized in that The method further comprises: Determine the type label of each user based on the historical logs of each user; Determine the sampling weight corresponding to each type of label; For each type of label, determine the target user of the label according to the sampling weight corresponding to the label and the number of users with the label; Send an opening instruction to the client of the target user, wherein the opening instruction is used to instruct the client of the target user to upload the logs to be uploaded that do not match the filtering list to the server.
4. The method according to claim 3, characterized in that The step of determining the sampling weight corresponding to each type of label includes: For each type of tag, calculate the benchmark user ratio of the type of tag based on the total number of users and the number of users of the type of tag, and use the benchmark user ratio to determine the ratio threshold; According to the preset offset, the initial sampling weight is adjusted to obtain the adjusted sampling weight; For each type of tag, determine the target user of the type of tag according to the adjusted sampling weight of the type of tag and the number of users of the type of tag; For each type of label, determine the proportion of verified users corresponding to the adjusted sampling weight according to the total number of users and the number of target users of the type of label; If the proportion of verified users does not reach the proportion threshold, the sampling weight is further adjusted according to the preset offset, and the process is repeated for each type of tag, and the target user of the type of tag is determined according to the adjusted sampling weight of the type of tag and the number of users of the type of tag; If the proportion of verified users reaches the proportion threshold, the last adjusted sampling weight is determined as the sampling weight corresponding to the type of label.
5. The method according to claim 3, characterized in that: The determining the target user of the type tag according to the sampling weight corresponding to the type tag and the number of users of the type tag includes: Calculate the product of the sampling weight corresponding to the type of label and the number of users with the type of label to obtain the target number of target users allowed to enable uploading; Among the users with this type of label, the target number of users are randomly selected as target users.
6. The method according to claim 3, characterized in that After determining the target user of the type tag according to the sampling weight corresponding to the type tag and the number of users of the type tag, the method further includes: Send a close instruction to the client of the non-target user, wherein the close instruction is used to instruct the client of the non-target user to prohibit uploading the to-be-uploaded logs that do not match the filter list to the server, and the client of the non-target user is other clients except the client of the target user.
7. The method according to claim 2 or 3, characterized in that: Determining the type label of each user based on the historical logs of each user includes: Cluster all users' historical logs to obtain multiple clusters; According to the user behaviors recorded in the historical logs of each cluster, a type label of the cluster is determined, and the type label is used as the type label of the user to whom the historical logs in the cluster belong.
8. A log processing method, characterized in that: Applied to a client, the method comprises: Receive a filter list sent by the server; wherein the filter list is obtained by the server adding log identifiers of historical logs whose usage information does not meet preset usage conditions; the usage information refers to the usage of historical logs in the process of business analysis of the application; Determine the log ID of the log to be uploaded; If the log identifier matches the filter list, uploading the log to be uploaded to the server is prohibited; The method further comprises: Receive the target user sent by the server, the target user is determined according to the type tag of each user, the sampling weight corresponding to each type tag and the number of users corresponding to each type tag, and the client of the target user uploads the log to be uploaded that does not match the filter list to the server; Determining the sampling weight corresponding to each type of label includes: For each type of tag, calculate the benchmark user ratio of the type of tag based on the total number of users and the number of users of the type of tag, and use the benchmark user ratio to determine the ratio threshold; According to the preset offset, the initial sampling weight is adjusted to obtain the adjusted sampling weight; For each type of label, determine the proportion of verified users corresponding to the adjusted sampling weight according to the total number of users and the number of target users of the type of label; If the proportion of verified users does not reach the proportion threshold, the sampling weight continues to be adjusted according to the preset offset; If the proportion of verified users reaches the proportion threshold, the last adjusted sampling weight is determined as the sampling weight corresponding to the type of label.
9. The method according to claim 8, characterized in that After determining the log identifier of the log to be uploaded, the method further includes: If a closing instruction is received from the server, uploading the log to be uploaded to the server is prohibited; If an opening instruction sent by the server is received and the log identifier does not match the filter list, the log to be uploaded is uploaded to the server.
10. The method according to claim 8, characterized in that The step of determining the log identifier of the log to be uploaded includes: Acquire attribute information and control information of a target client page, where the target client page is a client page corresponding to the user behavior of the log record to be uploaded; Integrate the attribute information and the control information to obtain integrated information; A hash value of the integrated information is determined as a log identifier of the log to be uploaded.
11. The method according to claim 10, characterized in that The log identifier includes: an absolute log identifier indicating the uniqueness of the log to be uploaded, and the control information includes: a control identifier, a position index of a parent control to which the control belongs, a hierarchical structure of the control, and data processed by the control.
12. The method according to claim 10, characterized in that The log identifier includes: a relative log identifier indicating the uniqueness of the log type to which the to-be-uploaded log belongs, and the control information includes: a hierarchical structure of controls.
13. A log processing device, characterized in that: Applied to the server, the device comprises: A data acquisition module is configured to acquire analysis data for business analysis of an application, wherein the analysis data includes usage information of historical logs during the business analysis process; A filter list determination module, configured to add log identifiers of historical logs whose usage information does not meet preset usage conditions to the filter list; A sending module is configured to send the filtering list to the client; the filtering list is used by the client to prohibit uploading the to-be-uploaded logs matching the log identifiers in the filtering list to the server; The weight determination module determines the target user according to the type label of each user, the sampling weight corresponding to each type label and the number of users corresponding to each type label, and the client of the target user uploads the logs to be uploaded that do not match the filter list to the server; The weight determination module is also used to: For each type of tag, calculate the benchmark user ratio of the type of tag based on the total number of users and the number of users of the type of tag, and use the benchmark user ratio to determine the ratio threshold; According to the preset offset, the initial sampling weight is adjusted to obtain the adjusted sampling weight; For each type of label, determine the proportion of verified users corresponding to the adjusted sampling weight according to the total number of users and the number of target users of the type of label; If the proportion of verified users does not reach the proportion threshold, the sampling weight continues to be adjusted according to the preset offset; If the proportion of verified users reaches the proportion threshold, the last adjusted sampling weight is determined as the sampling weight corresponding to the type of label.
14. A log processing device, characterized in that: Applied to a client, the device comprises: The receiving module is configured to receive a filter list sent by the server; wherein the filter list is obtained by the server adding a log identifier of a historical log whose usage information does not meet a preset usage condition; the usage information refers to the usage of the historical log in the process of business analysis of the application; A log identification determination module, configured to determine the log identification of the log to be uploaded; A log upload management module, configured to prohibit uploading the log to be uploaded to the server if the log identifier matches the filter list; The receiving module is configured to receive the target user sent by the server, wherein the target user is determined according to the type tag of each user, the sampling weight corresponding to each type tag and the number of users corresponding to each type tag, and the client of the target user uploads the log to be uploaded that does not match the filtering list to the server; Determining the sampling weight corresponding to each type of label includes: For each type of tag, calculate the benchmark user ratio of the type of tag based on the total number of users and the number of users of the type of tag, and use the benchmark user ratio to determine the ratio threshold; According to the preset offset, the initial sampling weight is adjusted to obtain the adjusted sampling weight; For each type of label, determine the proportion of verified users corresponding to the adjusted sampling weight according to the total number of users and the number of target users of the type of label; If the proportion of verified users does not reach the proportion threshold, the sampling weight continues to be adjusted according to the preset offset; If the proportion of verified users reaches the proportion threshold, the last adjusted sampling weight is determined as the sampling weight corresponding to the type of label.
15. A log processing system, characterized in that: The system comprises: a server and a client; The server is configured to obtain analysis data for business analysis of the application, the analysis data including usage information of historical logs during the business analysis; add log identifiers of historical logs whose usage information does not meet preset usage conditions to a filter list; and send the filter list to the client; The client is configured to receive a filter list sent by the server; determine a log identifier of a log to be uploaded; if the log identifier matches the filter list, prohibit uploading the log to be uploaded to the server; The server is configured as Determine the target user according to the type label of each user, the sampling weight corresponding to each type label and the number of users corresponding to each type label, and the client of the target user uploads the logs to be uploaded that do not match the filter list to the server; Determining the sampling weight corresponding to each type of label includes: For each type of tag, calculate the benchmark user ratio of the type of tag based on the total number of users and the number of users of the type of tag, and use the benchmark user ratio to determine the ratio threshold; According to the preset offset, the initial sampling weight is adjusted to obtain the adjusted sampling weight; For each type of label, determine the proportion of verified users corresponding to the adjusted sampling weight according to the total number of users and the number of target users of the type of label; If the proportion of verified users does not reach the proportion threshold, the sampling weight continues to be adjusted according to the preset offset; If the proportion of verified users reaches the proportion threshold, the last adjusted sampling weight is determined as the sampling weight corresponding to the type of label.
16. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the instructions, the steps of the method described in any one of claims 1-7 or 8-12 are implemented.
17. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by the processor, the steps of the method described in any one of claims 1-7 or 8-12 are implemented.
18. A computer program product, comprising computer instructions, characterized in that: When the instruction is executed by the processor, the steps of the method described in any one of claims 1-7 or 8-12 are implemented.
