Data processing method and device, equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method, device and equipment and a storage medium. The method comprises: obtaining a condition related to abnormal data generated by a client in a first time period; determining a sampling condition according to the related condition; and saving the sampling condition, wherein the sampling condition is used to indicate a condition required to be met by the client for reporting abnormal data to the server. Therefore, the embodiments of the present application solve the problem that the amount of abnormal data reported by the client to the server is too large when the client is abnormal, thereby increasing the pressure on the server.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0002] Front-end applications run in complex environments, which may cause various front-end exceptions due to differences in devices, network environments, app versions, operating systems, and code errors.

[0003] In existing technologies, when an anomaly occurs on the client side, abnormal data can be reported to the server, allowing developers to analyze and resolve the aforementioned front-end anomaly issues based on this data. However, the amount of user behavior data before and after an anomaly often is very large. This requires the client to report a large amount of data to the server, thus increasing the server's storage pressure. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium to solve the problem in the prior art where the amount of abnormal data reported by the client to the server when an anomaly occurs is too large, thereby increasing the storage pressure on the server.

[0005] In a first aspect, embodiments of this application provide a data processing method applied to a server, the method comprising:

[0006] Obtain the relevant conditions for abnormal data generated by the client within the first time period;

[0007] Based on the aforementioned relevant conditions, determine the sampling conditions;

[0008] The sampling conditions are saved, wherein the sampling conditions are used to indicate the conditions that the client needs to meet to report abnormal data to the server.

[0009] Secondly, embodiments of this application provide a data processing method applied to a client, the method comprising:

[0010] When an exception occurs on the client side, a target request is sent to the server, wherein the target request is used to request the reporting of exception data;

[0011] When the client meets the sampling conditions, it receives the first indication information sent by the server and reports abnormal data to the server, wherein the first indication information is used to indicate that abnormal data needs to be reported;

[0012] If the client does not meet the sampling conditions, the client receives a second indication message sent by the server, wherein the second indication message is used to indicate that abnormal data does not need to be reported.

[0013] The sampling conditions are determined by the server based on the relevant conditions of abnormal data generated by the client within the first time period.

[0014] Thirdly, embodiments of this application provide a data processing apparatus applied to a server, the apparatus comprising:

[0015] The first acquisition module is used to acquire the relevant conditions for abnormal data generated by the client within the first time period.

[0016] The first determining module is used to determine the sampling conditions based on the relevant conditions;

[0017] The first storage module is used to store the sampling conditions, wherein the sampling conditions are used to indicate the conditions that the client needs to meet to report abnormal data to the server.

[0018] Fourthly, embodiments of this application provide a data processing apparatus applied to a client, the apparatus comprising:

[0019] The first sending module is used to send a target request to the server when an exception occurs on the client side, wherein the target request is used to request the reporting of abnormal data;

[0020] The first receiving module is configured to receive first indication information sent by the server when the client meets the sampling conditions, and report abnormal data to the server, wherein the first indication information is used to indicate that abnormal data needs to be reported;

[0021] The second receiving module is configured to receive a second indication message sent by the server when the client does not meet the sampling conditions, wherein the second indication message is used to indicate that abnormal data does not need to be reported.

[0022] The sampling conditions are determined by the server based on the relevant conditions of abnormal data generated by the client within the first time period.

[0023] Fifthly, embodiments of this application provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the above-described server-side or client-side data processing method.

[0024] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described server-side or client-side data processing method.

[0025] In the technical solution of this application embodiment, the server can obtain the relevant conditions for the client to generate abnormal data in the first time period, and then determine the sampling conditions that the client needs to meet to instruct the server to report abnormal data based on the relevant conditions, and save the sampling conditions. In this way, the server can decide which clients can report abnormal data when they have abnormalities based on the sampling conditions.

[0026] Therefore, in this embodiment of the application, when an anomaly occurs on the client side, it is not necessary for every client to report the anomaly data to the server. Instead, the sampling conditions mentioned above determine which clients can report the anomaly data. That is, in this embodiment of the application, the sampling conditions for collecting user abnormal behavior trajectories (i.e., anomaly data) are determined by analyzing the relevant conditions for the client to generate anomaly data, which reduces the storage pressure on the server side and enhances the accuracy of anomaly reporting. Attached Figure Description

[0027] Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application;

[0028] Figure 2 This is a flowchart of another data processing method provided in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram illustrating a specific implementation of the data processing method provided in this application.

[0030] Figure 4 This is a structural block diagram of a data processing apparatus provided in an embodiment of this application;

[0031] Figure 5 This is a structural block diagram of another data processing device provided in the embodiments of this application;

[0032] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0034] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0035] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0036] Firstly, referring to Figure 1 The diagram illustrates a flowchart of a data processing method according to an embodiment of this application. This method can be applied to a server and may include the following steps 101 to 103:

[0037] Step 101: Obtain the relevant conditions for the client to generate abnormal data within the first time period.

[0038] The abnormal data may include user behavior data within a certain period before and after the client-side error occurred. Here, behavior data refers to the user's data requests and interactive processes on the front-end page.

[0039] In addition, the database can store abnormal data from different clients. In step 101, the server can obtain abnormal data from different clients within the first time period from the database, thereby obtaining the relevant conditions that generated this abnormal data.

[0040] In addition, the relevant conditions for generating abnormal data are used to indicate the conditions that the client has when an abnormality occurs, such as the Uniform Resource Locator (URL) page displayed by the client, the network status, and the device type corresponding to the client.

[0041] Step 102: Determine the sampling conditions based on the relevant conditions.

[0042] The sampling conditions are used to indicate the conditions that a client needs to meet when reporting abnormal data to the server. In other words, the sampling conditions include the conditions for a client to report abnormal data when an abnormality occurs. That is, the sampling conditions are used to determine whether a client that has an abnormality needs to report abnormal data to the server.

[0043] In addition, the relevant conditions are used to indicate the conditions that the client has when an anomaly occurs. Therefore, the sampling conditions are determined according to the relevant conditions, that is: according to the relevant conditions, the conditions that the client with an anomaly generally has are statistically analyzed. In this way, the sampling conditions can indicate "the conditions that the client device with a high probability of an anomaly has". Therefore, the client that meets the sampling conditions has a higher probability of an anomaly, while the client that does not meet the sampling conditions has a lower probability of an anomaly.

[0044] Furthermore, when a client that meets the sampling conditions encounters an anomaly, it can report the anomaly data to the server; while when a client that does not meet the sampling conditions encounters an anomaly, it does not need to report the anomaly data to the server. That is, for clients that encounter an anomaly, it is not necessary for all clients to report the anomaly data to the server, but only clients that meet the above sampling conditions need to report it. In this way, the storage pressure on the server can be alleviated.

[0045] Step 103: Save the sampling conditions.

[0046] The sampling conditions can also be visualized. For example, if the server has a display function, the sampling conditions can be displayed on the server to provide a reference for developers to solve client-side exceptions.

[0047] As can be seen from steps 101 to 103 above, in this embodiment of the application, the server can obtain the relevant conditions for the client to generate abnormal data in the first time period, and then determine the sampling conditions that the client needs to meet to instruct the server to report abnormal data based on the relevant conditions, and save the sampling conditions. In this way, the server can decide which clients can report abnormal data when they have abnormalities based on the sampling conditions.

[0048] Therefore, in this embodiment of the application, when an anomaly occurs on the client side, it is not necessary for every client to report the anomaly data to the server. Instead, the sampling conditions mentioned above determine which clients can report the anomaly data. That is, in this embodiment of the application, the sampling conditions for collecting user abnormal behavior trajectories (i.e., anomaly data) are determined by analyzing the relevant conditions for the client to generate anomaly data, which reduces the pressure on the server side and enhances the accuracy of anomaly reporting.

[0049] Optionally, step 101, "obtaining the relevant conditions for the client to generate abnormal data within the first time period," includes:

[0050] From the abnormal data generated by the client during the first time period, the conditions included in each predetermined dimension are extracted to obtain the relevant conditions.

[0051] For example, if the predetermined dimensions include "page URL, network status, and device type," then it is necessary to extract the conditions included in each dimension of "page URL, network status, and device type" from the abnormal data. Here, if we understand "dimension" as a field, then "condition" is the different values ​​of the field. For example, the conditions included in the "page URL" dimension can be URL1 and URL2; the conditions included in the "network status" dimension can be Wi-Fi connection and mobile data connection; and the conditions included in the "device type" dimension can be type 1 and type 2.

[0052] Therefore, in this embodiment of the application, conditions including different dimensions can be extracted from the abnormal data. Each condition under these dimensions is the relevant condition that generated the abnormal data.

[0053] Optionally, step 102, "determining the sampling conditions based on the relevant conditions," includes the following steps A-1 to A-4:

[0054] Step A-1: ​​Based on the maximum anomaly ratio, perform grouped queries on the relevant conditions included in each dimension to obtain query results, wherein the anomaly ratio is the ratio of the size of the abnormal data volume to the size of the normal data volume;

[0055] Step A-2: Obtain the first parameter and the second parameter, wherein the first parameter represents the amount of abnormal data under the relevant conditions included in the candidate group within the second time period, and the second parameter represents the amount of normal data under the relevant conditions included in the candidate group within the second time period, and the candidate group is a group composed of the relevant conditions indicated by the query result;

[0056] Step A-3: If the absolute value of the difference between the first parameter and the second parameter is less than or equal to a predetermined threshold, the relevant conditions included in the candidate grouping are determined as the sampling conditions;

[0057] Step A-4: If the absolute value of the difference between the first parameter and the second parameter is greater than the predetermined threshold, delete one of the dimensions and return to step A-1 (i.e., the step of grouping and querying the relevant conditions included in each dimension based on the maximum anomaly ratio to obtain the query results).

[0058] The absolute value of the difference between the first parameter and the second parameter is less than or equal to a predetermined threshold, indicating whether the above candidate groups meet the pure flow condition.

[0059] It is understandable that the first parameter can be the first value of the abnormal data volume under the relevant conditions included in the candidate group within the second time period, and the second parameter can be the second value of the normal data volume under the relevant conditions included in the candidate group within the second time period. Thus, the candidate group satisfies the pure traffic condition, which means that the values ​​of normal traffic and abnormal traffic are similar under the relevant conditions included in the candidate group.

[0060] Alternatively, the first parameter can be the ratio of a first value to the total data volume, and the second parameter can be the ratio of a second value to the total data volume. In this case, a candidate group satisfying the pure traffic condition means that the percentage of relevant conditions included in the candidate group under normal traffic and the percentage under abnormal traffic are similar. Here, the total data volume represents the total amount of data (including normal data and abnormal data) under different conditions within the second time period.

[0061] Therefore, if a candidate group satisfies the pure flow condition, the probability of anomalies occurring under the relevant conditions included in the candidate group is very high. In this case, the relevant conditions included in the candidate group can be used as the sampling conditions mentioned above. If a candidate group does not satisfy the pure flow condition, the probability of anomalies occurring under the relevant conditions included in the candidate group is low. In this case, a dimension can be deleted, and then a new candidate group can be obtained.

[0062] It should be noted that after step A-4, the process of determining the sampling conditions based on relevant criteria ends either when the sampling conditions are obtained or when all dimensions have been deleted. If the sampling conditions are not obtained before all dimensions have been deleted, a message "Unable to obtain sampling conditions" will be displayed.

[0063] For example, if there are a first dimension, a second dimension, and a third dimension, where the first dimension includes relevant conditions B1 and B2, the second dimension includes relevant conditions C1 and C2, and the third dimension includes relevant conditions D1 and D2, then firstly, the relevant conditions included in the first to third dimensions are grouped and queried. If the query result is a group consisting of "B1, B1, C1", then the first and second parameters of the group consisting of "B1, B1, C1" need to be obtained.

[0064] Wherein, if the absolute value of the difference between the first parameter and the second parameter is less than or equal to a predetermined threshold, then “B1, B1, C1” is the sampling condition.

[0065] If the absolute value of the difference between the first parameter and the second parameter is greater than a predetermined threshold, then one dimension is deleted from the first, second, and third dimensions. For example, if the first dimension is deleted, then the grouped query of each relevant condition included in the second to third dimensions continues, and so on, until the sampling conditions are obtained, or until all dimensions are deleted.

[0066] Furthermore, the second time period mentioned above can be the same as or different from the first time period mentioned above.

[0067] Optionally, step A-1, "based on the maximum anomaly ratio, grouping and querying the relevant conditions included in each dimension to obtain query results," includes:

[0068] Based on the relevant conditions included in each dimension, multiple groups are obtained, wherein the number of relevant conditions included in each group is equal to the number of existing dimensions, and the relevant conditions in each group belong to different dimensions.

[0069] Calculate the anomaly ratio under the relevant conditions for each group;

[0070] Select the group with the largest anomaly ratio to obtain the query result.

[0071] For example, if there are three dimensions: a first dimension, a second dimension, and a third dimension, where the first dimension includes relevant conditions B1 and B2, the second dimension includes relevant conditions C1 and C2, and the third dimension includes relevant conditions D1 and D2, then we can select one condition from each of the first to the third dimension to form a group, resulting in the following 8 groups:

[0072] {B1, C1, D1}, {B1, C1, D2};

[0073] {B1, C2, D1}, {B1, C2, D2};

[0074] {B2, C1, D1}, {B2, C1, D2};

[0075] {B2, C2, D1}, {B2, C2, D2};

[0076] Then, the outlier ratio under the relevant conditions included in each group can be calculated. For example, the outlier ratios corresponding to the above 8 groups are: x1, x2, x3, x4, x5, x6, x7, and x8.

[0077] Next, we can select a maximum value from x1, x2, x3, x4, x5, x6, x7, and x8. The group corresponding to this maximum value is the result of the group query.

[0078] It should be noted that the anomaly ratio under the relevant conditions included in a group is: the ratio of the amount of abnormal data under the relevant conditions included in the group to the amount of normal data under the relevant conditions included in the group.

[0079] As can be seen from the above, in this embodiment of the application, the candidate group indicated by the obtained query results is the group with the largest anomaly ratio among all groups. Furthermore, after obtaining the candidate group, it is further determined whether the candidate group meets the pure traffic condition (i.e., whether the absolute value of the difference between the first parameter and the second parameter of the candidate group is less than or equal to a predetermined threshold). It can be seen that in this embodiment of the application, in the process of determining the sampling conditions according to the relevant conditions, not only is the selection based on the ratio of abnormal data volume to normal data volume under the relevant conditions included in the group, but also the difference between the abnormal data volume and normal data volume under the relevant conditions is considered, thereby making the probability of anomalies occurring under the finally determined sampling conditions higher, and thus making the reporting of abnormal data more accurate.

[0080] It should also be noted that if there are multiple groups with the highest anomaly ratio, a group can be randomly selected from these multiple groups with the highest anomaly ratio as the query result.

[0081] Optionally, step A-4 above, "deleting one of the dimensions", includes:

[0082] Obtain the target number of relevant conditions included in each dimension;

[0083] The targets are sorted in descending order of quantity to obtain the target ranking.

[0084] Delete the first dimension in the target sort.

[0085] Therefore, if the absolute value of the difference between the first parameter and the second parameter of the candidate group is greater than a predetermined threshold, the dimension with the smallest number of relevant conditions among the currently existing dimensions can be deleted, and then grouped and queried based on the relevant conditions included in the remaining dimensions.

[0086] The larger the number of relevant conditions included in a dimension, the lower the probability of anomalies occurring on the client side under that dimension. Therefore, when deleting dimensions, the dimension with the lowest probability of anomalies can be deleted, thereby further increasing the probability of anomalies occurring under the final determined sampling conditions, and thus making the reporting of abnormal data more accurate.

[0087] Optionally, the abnormal data is associated with a target project; the method further includes:

[0088] Save the association between the sampling conditions and the target project.

[0089] Therefore, in this embodiment of the application, different projects can have different sampling conditions, so that the reporting of abnormal data is more in line with the actual situation of the project.

[0090] It is understandable that if different projects correspond to different sampling conditions, then different dimensions can correspond to different projects, so that the sampling conditions determined based on the relevant conditions included in the corresponding dimensions are more in line with the actual situation of the project.

[0091] Optionally, the method further includes:

[0092] Receive a target request sent when an exception occurs on the client, wherein the target request is used to request the reporting of exception data;

[0093] If the client meets the sampling conditions, a first indication message is sent to the client, and abnormal data reported by the client is received, wherein the first indication message is used to indicate that abnormal data needs to be reported;

[0094] If the client does not meet the sampling conditions, a second indication message is sent to the client, wherein the second indication message is used to indicate that abnormal data does not need to be reported.

[0095] Therefore, after storing the above sampling conditions on the server, if a client encounters an exception, the client can send the above target request to the server so that the server can determine whether the client meets the sampling conditions. If it does, the server will instruct the client to report the exception data; if it does not, the server will instruct the client not to report the exception data.

[0096] When different projects correspond to different sampling conditions, after receiving the above target request, the server can search for the sampling conditions corresponding to the project to which the client's currently executed business belongs from the stored sampling conditions corresponding to different projects, and then determine whether the client meets the found sampling conditions.

[0097] Optionally, after receiving the abnormal data reported by the client, the method further includes:

[0098] The sampling conditions are updated based on the abnormal data reported by the client.

[0099] Therefore, based on the stored sampling conditions, the server determines that an abnormal client can report abnormal data, and upon receiving the reported abnormal data, the server can further update the sampling conditions based on newly reported abnormal data from the client. The method for updating the sampling conditions is the same as the method for determining the sampling conditions described earlier, and will not be repeated here.

[0100] Understandably, the sampling conditions can be set to be updated periodically, meaning that the server can update the sampling conditions based on the abnormal data reported by the client within the current period.

[0101] Secondly, referring to Figure 1 The diagram illustrates a flowchart of a data processing method according to an embodiment of this application. This method can be applied to a client and may include the following steps 201 to 203:

[0102] Step 201: When an exception occurs on the client side, send a target request to the server.

[0103] The target request is used to request the reporting of abnormal data. Here, the abnormal data may include user behavior data within a certain period before and after the client-side error occurred.

[0104] Step 202: If the client meets the sampling conditions, receive the first indication information sent by the server and report abnormal data to the server.

[0105] The first indication information is used to indicate that abnormal data needs to be reported;

[0106] Step 203: If the client does not meet the sampling conditions, receive the second indication information sent by the server.

[0107] The second indication information is used to indicate that abnormal data does not need to be reported; the sampling conditions are determined by the server based on the relevant conditions of abnormal data generated by the client within the first time period.

[0108] In addition, the database can store abnormal data from different clients, so the server can obtain abnormal data from different clients within the first time period from the database, thereby obtaining the relevant conditions that generated this abnormal data.

[0109] In addition, the relevant conditions for generating abnormal data are used to indicate the conditions that the client has when an abnormality occurs, such as the URL of the page displayed by the client, the network status, and the type of device corresponding to the client.

[0110] It should also be noted that the relevant conditions are used to indicate the conditions that the client has when an anomaly occurs. Therefore, the sampling conditions are determined based on the relevant conditions, that is: based on the relevant conditions, the conditions that the client with an anomaly generally has are statistically analyzed. In this way, the sampling conditions can indicate "the conditions that the client device with a high probability of having an anomaly has". Therefore, the client that meets the sampling conditions has a higher probability of having an anomaly, while the client that does not meet the sampling conditions has a lower probability of having an anomaly.

[0111] Furthermore, when a client that meets the sampling conditions encounters an anomaly, it can report the anomaly data to the server; while when a client that does not meet the sampling conditions encounters an anomaly, it does not need to report the anomaly data to the server. That is, for clients that encounter an anomaly, it is not necessary for all clients to report the anomaly data to the server, but only clients that meet the above sampling conditions need to report it. In this way, the storage pressure on the server can be alleviated.

[0112] As can be seen from steps 201 to 203 above, in this embodiment of the application, the server can obtain the relevant conditions for the client to generate abnormal data in the first time period, and then determine the sampling conditions that the client needs to meet to instruct the server to report abnormal data based on the relevant conditions, and save the sampling conditions. In this way, when the client has an abnormality, it can send a target request to the server so that the server can determine whether the client meets the sampling conditions. If the conditions are met, the server instructs the client to report abnormal data; if the conditions are not met, the server instructs the client not to report abnormal data.

[0113] Therefore, in this embodiment of the application, when an anomaly occurs on the client side, it is not necessary for every client to report the anomaly data to the server. Instead, the sampling conditions mentioned above determine which clients can report the anomaly data. That is, in this embodiment of the application, the sampling conditions for collecting user abnormal behavior trajectories (i.e., anomaly data) are determined by analyzing the relevant conditions for the client to generate anomaly data, which reduces the storage pressure on the server side and enhances the accuracy of anomaly reporting.

[0114] In summary, the specific implementation methods of the data processing device in this application are as follows:

[0115] The server side can be divided into a data receiving layer, a data warehouse, a root cause service layer, and a data visualization layer.

[0116] During the initial preparation phase, if an exception occurs after the client opens the page, it can report the exception data to the data receiving layer. The data receiving layer then receives the exception data and stores it in the time-series database (Druid) of the data warehouse. Therefore, Druid stores exception data reported by different clients when exceptions occur.

[0117] After the preliminary preparation stage, such as Figure 3 As shown, a scheduled task is configured in the root cause service layer, which then retrieves abnormal client data for the current project within that time period from Druid at regular intervals. This abnormal data is then used for offline dimensional analysis, specifically by obtaining sampling conditions through the following process (steps H-1 to H-7):

[0118] Step H-1: Extract conditions from the abnormal data that are determined in advance for multiple dimensions (i.e., extract data fields that are positively correlated with the generation of the abnormality, such as: page URL, network status, device type, system type, app version, etc.);

[0119] Step H-2: Calculate the number of conditions in each dimension;

[0120] Step H-3: Sort the dimensions in descending order based on the number of conditions;

[0121] Step H-4: Based on the anomaly ratio, perform grouped queries on the conditions included in each dimension to obtain the query results. The specific process of grouped queries can be found in the previous text and will not be repeated here.

[0122] Step H-5: Determine whether the candidate groups indicated by the query results meet the pure traffic conditions. Here, the specific process of determining whether the candidate groups meet the pure traffic conditions can be found in the previous text and will not be repeated here.

[0123] Step H-6: If a candidate group satisfies the pure flow condition, then the probability of an anomaly in the relevant conditions included in the candidate group is very high, and the relevant conditions included in the candidate group are the sampling conditions; if a candidate group satisfies the pure flow condition, then delete the first dimension in the descending order, and then repeat steps H-4 and H-5 based on the remaining dimensions.

[0124] Step H-7: Save the one-to-one mapping relationship between the sampling conditions and the current project.

[0125] Here, the obtained sampling conditions can be stored in the remote dictionary server (Redis) in the data warehouse; and the sampling conditions can also be displayed in the data visualization layer, so that developers can analyze the high-frequency conditions for anomalies and adjust the dimensions used to determine the sampling conditions in a timely manner.

[0126] Furthermore, if an exception occurs again during the client's operation, the client can query the data receiving layer to report it. This allows the data receiving layer to retrieve the sampling conditions corresponding to the project to which the client's current business belongs from Redis, determine whether the client meets the sampling conditions, and if so, instruct the client to report the exception data. In this way, the client can report the exception data to the data receiving layer.

[0127] It should be noted that if the client meets the sampling conditions, Redis can delete the sampling conditions after the client reports abnormal data, and trigger the root cause service layer to update the sampling conditions.

[0128] Therefore, in this embodiment of the application, by analyzing the relevant conditions for the occurrence of anomalies to determine the sampling conditions for collecting user abnormal behavior trajectories, the pressure on the server is reduced, and the accuracy and real-time performance of anomaly reporting are enhanced. It also provides analytical directions for resolving anomalies.

[0129] Thirdly, embodiments of this application also provide a data processing apparatus that can be applied to a server, such as... Figure 4 As shown, the data processing device may include the following modules:

[0130] The first acquisition module 401 is used to acquire the relevant conditions for abnormal data generated by the client within the first time period.

[0131] The first determining module 402 is used to determine the sampling conditions based on the relevant conditions;

[0132] The first storage module 403 is used to store the sampling conditions, wherein the sampling conditions are used to indicate the conditions that the client needs to meet to report abnormal data to the server.

[0133] Optionally, the first acquisition module 401 is specifically used for:

[0134] From the abnormal data generated by the client during the first time period, the conditions included in each predetermined dimension are extracted to obtain the relevant conditions.

[0135] Optionally, the first determining module 402 includes:

[0136] The group query submodule is used to perform group queries on the relevant conditions included in each dimension based on the maximum anomaly ratio to obtain query results, wherein the anomaly ratio is the ratio of the size of the abnormal data volume to the size of the normal data volume;

[0137] The parameter acquisition submodule is used to acquire a first parameter and a second parameter, wherein the first parameter represents the size of the abnormal data volume under the relevant conditions included in the candidate group within the second time period, and the second parameter represents the size of the normal data volume under the relevant conditions included in the candidate group within the second time period, wherein the candidate group is a group composed of the relevant conditions indicated by the query result;

[0138] The first processing submodule is used to determine the relevant conditions included in the candidate group as the sampling conditions when the absolute value of the difference between the first parameter and the second parameter is less than or equal to a predetermined threshold.

[0139] The second processing submodule is used to delete one of the dimensions when the absolute value of the difference between the first parameter and the second parameter is greater than the predetermined threshold, and to trigger the group query submodule to return a query result by grouping the relevant conditions included in each dimension based on the maximum anomaly ratio.

[0140] Optionally, the group query submodule is specifically used for:

[0141] Based on the relevant conditions included in each dimension, multiple groups are obtained, wherein the number of relevant conditions included in each group is equal to the number of existing dimensions, and the relevant conditions in each group belong to different dimensions.

[0142] Calculate the anomaly ratio under the relevant conditions for each group;

[0143] Select the group with the largest anomaly ratio to obtain the query result.

[0144] Optionally, the second processing submodule deletes one of the dimensions, specifically for:

[0145] Obtain the target number of relevant conditions included in each dimension;

[0146] The targets are sorted in descending order of quantity to obtain the target ranking.

[0147] Delete the first dimension in the target sort.

[0148] Optionally, the abnormal data is associated with a target project; the device further includes:

[0149] The second storage module is used to store the association between the sampling conditions and the target project.

[0150] Optionally, the device further includes:

[0151] The third receiving module is used to receive a target request sent by the client when an exception occurs, wherein the target request is used to request the reporting of exception data;

[0152] The second sending module is used to send first indication information to the client when the client meets the sampling conditions, and to receive abnormal data reported by the client, wherein the first indication information is used to indicate that abnormal data needs to be reported.

[0153] The third sending module is used to send a second indication message to the client when the client does not meet the sampling conditions, wherein the second indication message is used to indicate that abnormal data does not need to be reported.

[0154] Optionally, the device further includes:

[0155] The update module is used to update the sampling conditions based on the abnormal data reported by the client.

[0156] Fourthly, embodiments of this application also provide a data processing apparatus that can be applied to a server, such as... Figure 5 As shown, the data processing device may include the following modules:

[0157] The first sending module 501 is used to send a target request to the server when an exception occurs on the client side, wherein the target request is used to request the reporting of abnormal data;

[0158] The first receiving module 502 is configured to receive first indication information sent by the server when the client meets the sampling conditions, and report abnormal data to the server, wherein the first indication information is used to indicate that abnormal data needs to be reported.

[0159] The second receiving module 503 is used to receive second indication information sent by the server when the client does not meet the sampling conditions, wherein the second indication information is used to indicate that abnormal data does not need to be reported;

[0160] The sampling conditions are determined by the server based on the relevant conditions of abnormal data generated by the client within the first time period.

[0161] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0162] On the other hand, embodiments of this application also provide an electronic device, including a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the above-described server-side or client-side data processing methods.

[0163] For example, Figure 6 A schematic diagram of the physical structure of an electronic device is shown.

[0164] like Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions stored in the memory 630, and the processor 610 is used to execute the steps in the aforementioned server-side or client-side data processing methods.

[0165] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] In another aspect, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the data processing methods provided in the above embodiments.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that, Applied to the server side, the method includes: Obtain the relevant conditions for abnormal data generated by the client within the first time period; Based on the aforementioned relevant conditions, determine the sampling conditions; The sampling conditions are saved, wherein the sampling conditions are used to indicate the conditions that the client needs to meet to report abnormal data to the server; The step of determining the sampling conditions based on the relevant conditions includes: Based on the relevant conditions included in each dimension, multiple groups are obtained, wherein the number of relevant conditions included in each group is equal to the number of existing dimensions, and the relevant conditions in each group belong to different dimensions; the anomaly ratio of the relevant conditions included in each group is calculated; the group with the largest anomaly ratio is selected to obtain the query result, wherein the anomaly ratio is the ratio of the size of the abnormal data volume to the size of the normal data volume; Obtain a first parameter and a second parameter, wherein the first parameter represents the amount of abnormal data under the relevant conditions included in the candidate group within the second time period, and the second parameter represents the amount of normal data under the relevant conditions included in the candidate group within the second time period, and the candidate group is a group composed of the relevant conditions indicated by the query result; If the absolute value of the difference between the first parameter and the second parameter is less than or equal to a predetermined threshold, the relevant conditions included in the candidate grouping are determined as the sampling conditions. If the absolute value of the difference between the first parameter and the second parameter is greater than the predetermined threshold, delete one of the dimensions and return the query results by grouping the relevant conditions included in each dimension based on the maximum anomaly ratio.

2. The method according to claim 1, characterized in that, The conditions for obtaining abnormal data generated by the client within the first time period include: From the abnormal data generated by the client during the first time period, the conditions included in each predetermined dimension are extracted to obtain the relevant conditions.

3. The method according to claim 1, characterized in that, Deleting one of the dimensions includes: Obtain the target number of relevant conditions included in each dimension; The targets are sorted in descending order of quantity to obtain the target ranking. Delete the first dimension in the target sort.

4. The method according to claim 1, characterized in that, The abnormal data is associated with the target project; the method further includes: Save the association between the sampling conditions and the target project.

5. The method according to claim 1, characterized in that, The method further includes: Receive a target request sent when an exception occurs on the client, wherein the target request is used to request the reporting of exception data; If the client meets the sampling conditions, a first indication message is sent to the client, and abnormal data reported by the client is received, wherein the first indication message is used to indicate that abnormal data needs to be reported; if the client does not meet the sampling conditions, a second indication message is sent to the client, wherein the second indication message is used to indicate that abnormal data does not need to be reported.

6. The method according to claim 5, characterized in that, After receiving the abnormal data reported by the client, the method further includes: The sampling conditions are updated based on the abnormal data reported by the client.

7. A data processing method, characterized in that, Applied to a client, the method includes: When an exception occurs on the client side, a target request is sent to the server, wherein the target request is used to request the reporting of exception data; When the client meets the sampling conditions, it receives the first indication information sent by the server and reports abnormal data to the server, wherein the first indication information is used to indicate that abnormal data needs to be reported; If the client does not meet the sampling conditions, the client receives a second indication message sent by the server, wherein the second indication message is used to indicate that abnormal data does not need to be reported. The sampling conditions are determined by the server based on the relevant conditions of abnormal data generated by the client within a first time period. The sampling condition is as follows: the server obtains multiple groups based on the relevant conditions included in each dimension, wherein the number of relevant conditions in each group is equal to the number of existing dimensions, and the relevant conditions in each group belong to different dimensions; the anomaly ratio of the relevant conditions included in each group is calculated; the group with the largest anomaly ratio is selected to obtain the query result, wherein the anomaly ratio is the ratio of the size of abnormal data to the size of normal data; a first parameter and a second parameter are obtained, wherein the first parameter represents the size of abnormal data under the relevant conditions included in the candidate group in the second time period, and the second parameter represents the size of normal data under the relevant conditions included in the candidate group in the second time period, wherein the candidate group is a group composed of the relevant conditions indicated by the query result; if the absolute value of the difference between the first parameter and the second parameter is less than or equal to a predetermined threshold, the relevant conditions included in the candidate group are determined as the sampling condition; if the absolute value of the difference between the first parameter and the second parameter is greater than the predetermined threshold, one of the dimensions is deleted, and the query result is returned based on the grouped query of the relevant conditions included in each dimension according to the largest anomaly ratio.

8. A data processing apparatus, characterized in that, Applied to the server side, the device includes: The first acquisition module is used to acquire the relevant conditions for abnormal data generated by the client within the first time period. The first determining module is used to determine the sampling conditions based on the relevant conditions; The first storage module is used to store the sampling conditions, wherein the sampling conditions are used to indicate the conditions that the client needs to meet to report abnormal data to the server; The first determining module includes: The grouping query submodule is specifically used to obtain multiple groups based on the relevant conditions included in each dimension, wherein the number of relevant conditions included in each group is equal to the number of existing dimensions, and the relevant conditions in each group belong to different dimensions; calculate the anomaly ratio under the relevant conditions included in each group; select the group with the largest anomaly ratio to obtain the query result, wherein the anomaly ratio is the ratio of the size of the abnormal data volume to the size of the normal data volume; The parameter acquisition submodule is used to acquire a first parameter and a second parameter, wherein the first parameter represents the size of the abnormal data volume under the relevant conditions included in the candidate group within the second time period, and the second parameter represents the size of the normal data volume under the relevant conditions included in the candidate group within the second time period, wherein the candidate group is a group composed of the relevant conditions indicated by the query result; The first processing submodule is used to determine the relevant conditions included in the candidate group as the sampling conditions when the absolute value of the difference between the first parameter and the second parameter is less than or equal to a predetermined threshold. The second processing submodule is used to delete one of the dimensions when the absolute value of the difference between the first parameter and the second parameter is greater than the predetermined threshold, and to trigger the group query submodule to return a query result by grouping the relevant conditions included in each dimension based on the maximum anomaly ratio.

9. A data processing apparatus, characterized in that, Applied to a client, the device includes: The first sending module is used to send a target request to the server when an exception occurs on the client side, wherein the target request is used to request the reporting of abnormal data; The first receiving module is configured to receive first indication information sent by the server when the client meets the sampling conditions, and report abnormal data to the server, wherein the first indication information is used to indicate that abnormal data needs to be reported. The second receiving module is configured to receive a second indication message sent by the server when the client does not meet the sampling conditions, wherein the second indication message is used to indicate that abnormal data does not need to be reported; The sampling conditions are determined by the server based on the relevant conditions of abnormal data generated by the client within the first time period. The sampling condition is as follows: the server obtains multiple groups based on the relevant conditions included in each dimension, wherein the number of relevant conditions in each group is equal to the number of existing dimensions, and the relevant conditions in each group belong to different dimensions; the anomaly ratio of the relevant conditions included in each group is calculated; the group with the largest anomaly ratio is selected to obtain the query result, wherein the anomaly ratio is the ratio of the size of abnormal data to the size of normal data; a first parameter and a second parameter are obtained, wherein the first parameter represents the size of abnormal data under the relevant conditions included in the candidate group in the second time period, and the second parameter represents the size of normal data under the relevant conditions included in the candidate group in the second time period, wherein the candidate group is a group composed of the relevant conditions indicated by the query result; if the absolute value of the difference between the first parameter and the second parameter is less than or equal to a predetermined threshold, the relevant conditions included in the candidate group are determined as the sampling condition; if the absolute value of the difference between the first parameter and the second parameter is greater than the predetermined threshold, one of the dimensions is deleted, and the query result is returned based on the grouped query of the relevant conditions included in each dimension according to the largest anomaly ratio.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the data processing method as described in any one of claims 1 to 6, or the steps of the data processing method as described in claim 7.

11. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the data processing method as described in any one of claims 1 to 6, or the steps of the data processing method as described in claim 7.

Images