Data processing method, device, server and computer-readable storage medium

Abstract

An embodiment of the present invention provides a data processing method, device, server and computer-readable storage medium, wherein the method includes: in response to a test operation on a target application, obtaining a covariate data set of a first user, the covariate data set including multiple covariate data; performing security verification on each covariate data in the multiple covariate data to obtain a security verification result; based on the security verification result, determining first target covariate data from the multiple covariate data, and reducing the variance of user operation data of the first user based on the first target covariate data, wherein the user operation data of the first user is detected during the process of performing the test operation on the target application, and the variance of the user operation data can be reduced, thereby reducing the acquisition of sample data required for testing the target application while ensuring the accuracy of the test results.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device, server and computer-readable storage medium. Background Art

[0002] With the continuous development of Internet technology, various applications have emerged. Before various applications are put online, it is usually necessary to test the performance of the application, and then put the application online after the test results meet the expected results. At present, the testing method for applications usually adopts the method of comparative testing of experimental groups and control groups. According to the comparison of experimental group data and control group data, the relevant performance of the application can be determined. Therefore, during the test process, it is necessary to obtain more experimental group data and control group data respectively. The acquisition of a large amount of data makes the current testing efficiency of applications low. It can be seen that how to ensure the test results for the target application while obtaining less data has become a current research hotspot. Summary of the invention

[0003] Embodiments of the present invention provide a data processing method, device, server and computer-readable storage medium, which can reduce the variance of user operation data, thereby reducing the acquisition of sample data required for the test target application while ensuring the accuracy of the test results.

[0004] On the one hand, an embodiment of the present invention provides a data processing method, including:

[0005] In response to a test operation on a target application, obtaining a covariate data set of a first user, the covariate data set including a plurality of covariate data;

[0006] Performing security verification on each covariate data among the multiple covariate data to obtain a security verification result;

[0007] Based on the security verification result, first target covariate data is determined from the multiple covariate data, and the variance of the user operation data of the first user is reduced based on the first target covariate data. The user operation data of the first user is detected during the process of performing the test operation on the target application.

[0008] In another aspect, an embodiment of the present invention provides a data processing device, comprising:

[0009] an acquisition unit, configured to acquire a covariate data set of a first user in response to a test operation on a target application, wherein the covariate data set includes a plurality of covariate data;

[0010] A verification unit, configured to perform security verification on each of the plurality of covariate data to obtain a security verification result;

[0011] a determining unit, configured to determine first target covariate data from the plurality of covariate data according to the safety verification result;

[0012] A reduction unit is used to reduce the variance of the user operation data of the first user according to the first target covariate data, where the user operation data of the first user is detected during the process of performing the test operation on the target application.

[0013] In another aspect, an embodiment of the present invention provides a server, including a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are connected to each other, wherein the memory is used to store a computer program that supports a terminal to execute the above method, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the following steps:

[0014] In response to a test operation on a target application, obtaining a covariate data set of a first user, the covariate data set including a plurality of covariate data;

[0015] Performing security verification on each covariate data among the multiple covariate data to obtain a security verification result;

[0016] Based on the security verification result, first target covariate data is determined from the multiple covariate data, and the variance of the user operation data of the first user is reduced based on the first target covariate data. The user operation data of the first user is detected during the process of performing the test operation on the target application.

[0017] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the first aspect.

[0018] In an embodiment of the present invention, the server responds to a test operation on a target application and obtains a covariate data set of a first user, the covariate data set including multiple covariate data, and performs security verification on each of the multiple covariate data to obtain a security verification result, and then determines first target covariate data from the multiple covariate data based on the security verification result, and reduces the variance of the user operation data of the first user based on the first target covariate data, thereby improving the reduction effect and improving the efficiency and accuracy of experimental testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a structural schematic diagram of a data processing system provided by an embodiment of the present invention;

[0021] Figure 2 It is a flowchart of a data processing method provided by an embodiment of the present invention;

[0022] Figure 3 is a flow chart of another data processing method provided by an embodiment of the present invention;

[0023] Figure 4a It is a flowchart of a specific data processing method provided by an embodiment of the present invention;

[0024] Figure 4b is a structural schematic diagram of a test interface provided by an embodiment of the present invention;

[0025] Figure 5 is a structural schematic diagram of a data processing device provided by an embodiment of the present invention;

[0026] Figure 6 It is a structural diagram of a server provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The embodiment of the present invention proposes a data processing method, which can reduce the variance of user operation data collected during the test phase (or experimental phase) of the target application. Based on the reduction of the variance of the user operation data, the number of user operation data collected during the test phase can be reduced while ensuring the accuracy of the test results, thereby improving the efficiency of testing the target application. In a specific implementation, a server for testing the target application can first obtain multiple covariate data of a first user when running the target application, and use the obtained multiple covariate data as a data source, wherein the first user is a user who runs the target application when testing the target application, and sends the user operation data generated by running the target application to the server to test the target application. The target application can be an independent application in the client, such as various social program clients, news clients, etc.; or, the target application can also be a small program of any independent application in the client, and the small program is a program that depends on the independent application to run and has an association relationship with the independent application, and the association relationship is, for example, login information association.

[0029] In one embodiment, after acquiring multiple covariate data, the server may perform security verification on the multiple covariate data and determine the first target covariate data from the multiple covariate data, so as to reduce the variance of the user operation data of the first user based on the first target covariate data, such as Figure 1 As shown, it is a schematic diagram of the architecture of a data processing system provided by an embodiment of the present invention, and the data processing system includes a client 101 and a server 102. In one embodiment, the client 101 is used to carry the operation of the target application to test the target application. The target application can be QQ Kandian, WeChat, etc. In addition, the client 101 can also store the user operation data performed by the user (such as the first user mentioned above) in the target application; the server 102 is used to receive multiple covariate data of the first user sent by the client 101 to test the application based on the multiple covariate data. In a specific implementation, the server 102 can directly determine the first covariate data of the first user from the target application, determine the second covariate data of the first user from the associated application of the target application, and determine the third covariate data of the first user from the external application, so that the server 102 obtains multiple covariate data of the first user.

[0030] In one embodiment, after acquiring multiple covariate data, the server 102 may further perform security verification on the multiple covariate data. In a specific implementation, the server 102 may calculate the expected value of each covariate data in the multiple covariate data, and the expected value of the corresponding control covariate data, so that the security of each covariate data may be verified by determining whether the expected value of each covariate data is equal to the expected value of the corresponding control covariate data. In one embodiment, the process of the server 102 performing security verification on the first covariate data is described in detail. The server 102 may first obtain the expected value of the first covariate data (assuming it is E1), and the expected value of the control covariate data corresponding to the first covariate data (assuming it is E2). Further, the server 102 may compare E1 and E2. If E1=E2, it is determined that the first covariate data passes the security verification. Otherwise, it is determined that the first covariate data fails the security verification.

[0031] In one embodiment, based on the result of the security verification of each covariate data, the server 102 can determine the first target covariate data from the multiple covariate data, so as to reduce the variance of the user operation data of the first user based on the first target covariate data, wherein the server 102 uses the covariate data whose expected value of each covariate data is equal to the expected value of the corresponding control covariate data as the reference covariate data, wherein the number of the reference covariate data can be one or more, and when the number of the reference covariate data is one, the reference covariate data is used as the first target covariate data, or if the number of the reference covariate data is more than one, the server 102 needs to further screen the multiple reference covariate data to determine the first target covariate data from the multiple reference covariate data. In one embodiment, when the server 102 screens the multiple reference covariate data, it can determine the reference covariate data with the greatest correlation with the user operation data of the first user from the multiple reference covariate data, and use the reference covariate data with the greatest correlation as the first target covariate data.

[0032] In one embodiment, after determining the first target covariate data, the server 102 may further perform variance reduction on the user operation data of the first user based on the first target covariate data. Assuming that the user operation data of the first user is Y_new, and the determined first target covariate data is X_new_i (where i=1, 2, 3, ...), further, the server 102 may calculate the covariance cov(Y_new, X_new_i) between Y_new and X_new_i, and calculate the variance of the first target covariate data, i.e., var(X_new_i). After obtaining the covariance cov(Y_new, X_new_i) and the variance var(X_new_i), the server 102 may use the ratio between the covariance cov(Y_new, X_new_i) and the variance var(X_new_i) as a reduction coefficient, so that the variance of the user operation data of the first user may be reduced based on the reduction coefficient. The reduction coefficient may be represented by θ, and its calculation formula is shown in Formula 1.1:

[0033]

[0034] In one embodiment, after determining the reduction coefficient θ, the server 102 may reduce the variance of the user operation data of the first user based on the reduction coefficient θ. In a specific implementation, the server 102 may first determine the mean mean of the first target covariate data and the expected value E of the first target covariate data, wherein if the first target covariate data is the above-mentioned first covariate data X_new_i, then the mean mean of the first target covariate data is mean(X_new_i), and the expected value E of the first target covariate data is E(X_new_i). Further, the server 102 may determine a first product of the mean mean and the reduction coefficient θ, and a second product of the expected value E and the reduction coefficient θ, thereby determining the unbiased estimated data Y_new_cv of the user operation data of the first user according to the user operation data Y_new of the first user, the first product, and the second product, wherein the calculation formula of the unbiased estimated data Y_new_cv may be as shown in Formula 1.2:

[0035] Y_new_cv=Y_new-θmean(X_new_i)+θE(X_new_i) Equation 1.2

[0036] After determining the unbiased estimate data Y_new_cv of the user operation data of the first user, the server 102 can determine the minimum value of the variance of the unbiased estimate data Y_new_cv, that is, the minimum value of var(Y_new_cv), and can further use the minimum value of var(Y_new_cv) as the variance of the operation data of the first user, thereby achieving a reduction in the variance of the user operation data of the first user.

[0037] In one embodiment, the first user is a new user of the target application, and a new user refers to a user who has not registered in the target application before testing the target application. Correspondingly, the second user is an old user of the target application, and an old user refers to a user who has completed registration in the target application before testing the target application. After reducing the variance of the user operation data of the first user, the server 102 can further reduce the variance of the user operation data of the second user, wherein the server 102 can also use the method of reducing the variance of the user operation data of the first user (i.e., the new user) to reduce the variance of the user operation data of the second user (i.e., the old user). Based on the reduction of the variance of the user operation data of the first user and the reduction of the variance of the user operation data of the second user, the server can further reduce the variance of the operation data of all users. When the server 102 reduces the variance of the operation data of all users, the variance of the reduced user operation data of the first user and the variance of the reduced user operation data of the second user can be integrated according to the proportion of the first user and the second user to the total number of users, thereby reducing the variance of the user operation data of all users.

[0038] See also Figure 2 , Figure 2 A flow chart of a data processing method provided by an embodiment of the present invention is shown in FIG. 1 . The data processing method described in this embodiment can be applied to the above-mentioned server, such as Figure 2 As shown, the method may include:

[0039] S201 : In response to a test operation on a target application, obtain a covariate dataset of a first user.

[0040] In one embodiment, the target application may be an independent application, such as various social applications. More specifically, the independent application may be Tencent QQ, etc. Alternatively, the target application may also be a small program that depends on an independent application to run, such as a small program that depends on Tencent QQ to run. In the embodiment of the present invention, the target application is a small program that depends on an independent application to run as an example for detailed description. When the target application is other applications, please refer to the embodiment of the present invention. Among them, the test operation for the target application refers to the detection operation of some or all performance indicators in the target application after the target application is updated and before the updated target application is released. Alternatively, the test operation for the target application may also be the detection operation of some or all performance indicators in the target application (or product) before the new target application (or product) is released, that is, before the new target application (or product) is put online, some or all performance indicators of the new target application are detected to determine the running performance of the new target application. It can be understood that the test operation is used to detect the quality of some or all of the performance indicators updated in the target application. If it is determined based on the test operation that some or all of the updated performance indicators are better than the corresponding performance indicators before the update, the updated target application is released. Alternatively, if it is determined based on the test operation that some or all of the performance indicators of the new target application meet the preset performance indicators, the new target application is released and made available online for all users to download the application, wherein the preset performance indicator refers to the indicator corresponding to each performance in the application when the product release conditions are met.

[0041] In one embodiment, the covariate data set obtained by the server includes multiple covariate data, where the covariate data refers to independent variables that are not dependent on the manipulation of the experimental group during the testing of the target application. The first user may refer to a user who only appears when the test operation is performed on the target application. It can be understood that the first user is a new user of the target application, wherein the new user refers to a user who has not registered in the target application before the target application is tested.

[0042] In one embodiment, the server may obtain the covariate data set of the first user from a target application, an associated application of the target application, and an external application, wherein the server may use the covariate data obtained from the target application as the first covariate data, or the server may use the covariate data obtained from the associated application as the second covariate data. In addition, the server may also use the covariate data obtained from the external application as the third covariate data, wherein the external application may be an application other than the target application and the associated application. It is understandable that the external application may also be an independently running application. For example, if the target application is Tencent QQ, the external application may correspondingly be WeChat, etc. Among them, taking the target application as the Kandian applet as an example, the server can obtain the first covariate data from the Kandian applet, where the first covariate data can be represented by X_new_1, or the server can obtain the second covariate data from the application associated with the Kandian applet (such as Tencent QQ), where the second covariate data can be represented by X_new_2, or the server can also obtain the third covariate data from an external application unrelated to the Kandian applet (such as WeChat), where the third covariate data can be represented by X_new_3.

[0043] In one embodiment, after the server obtains multiple different covariate data, which may be, for example, the first covariate data X_new_1, the second covariate data X_new_2, and the third covariate data X_new_3 mentioned above, the covariate data set of the first user can be determined, wherein the determined covariate data set includes: one or more of the first covariate data, the second covariate data, and the third covariate data. In one embodiment, the corresponding quantities of the above-mentioned first covariate data, the second covariate data, and the third covariate data can all be multiple.

[0044] In one embodiment, the server may determine the quality of the covariate data acquisition source in the covariate data set based on one or more of the following three aspects: ① Whether the full amount of user data can be obtained; ② The difficulty of obtaining covariate data (or engineering complexity); ③ The security of the data acquisition source. The quality of the target application, the associated application, and the external application as the covariate data acquisition source can be shown in Table 1:

[0045] Table 1

[0046]

[0047] In one embodiment, taking the target application as the Kandian applet as an example, the process of determining the covariate data set of the first user of the Kandian applet is described in detail, wherein the Kandian applet is a applet embedded in (i.e., dependent on) Tencent QQ, that is, the first user of the Kandian applet is also a user in Tencent QQ. On the other hand, the essence of Tencent QQ is a social application, and the Kandian applet is an information flow application. In terms of application functions, Tencent QQ and the Kandian applet have a certain degree of independence. Therefore, when testing the Kandian applet, it is possible to consider obtaining the covariate data of the first user (new user) from the Tencent QQ. Furthermore, the server can further expand the source of obtaining the covariate data, that is, consider other applications (Application, APP) other than the Kandian applet, such as WeChat, etc. Among them, the server can ignore the impact of the test on the Kandian applet on the covariate data obtained on WeChat. That is to say, the security of the covariate data obtained from WeChat is relatively high, but when obtaining covariate data from WeChat, there are still problems of difficulty in data acquisition and difficulty in covering all new users. In addition, the server can also obtain covariate data from the Kandian applet itself. When obtaining covariate data from the applet itself, there is a problem of low security, but it can achieve coverage of all new users, and there is an advantage of easily obtaining covariate data from the Kandian applet.

[0048] In one embodiment, the security of the server when acquiring covariate data is determined based on the degree of influence of the test operation for the target application on the server when acquiring the covariate data. It can be understood that the greater the degree of influence of the test operation for the target application on the server when acquiring the covariate data, the lower the security of the acquired covariate data; and if the smaller the degree of influence of the test operation for the target application on the server when acquiring the covariate data, the higher the security of the covariate data acquired by the server. For example, when the target application is the Kandian applet, the server can consider acquiring covariate data from Tencent QQ, the Kandian applet, and WeChat respectively. Since the server's current test operation is for the Kandian applet, and the Kandian applet is embedded in Tencent QQ, when the server tests the Kandian applet, it still needs to pay attention to the changes caused by the test operation in Tencent QQ. Since the Kandian applet and WeChat are less related, it can be considered that the test operation on the Kandian applet will not cause data changes in WeChat. Therefore, when the server acquires covariate data from Tencent QQ or the Kandian applet itself, the application degree of the test operation is greater, while when acquiring covariate data from WeChat, the application degree of the test operation is smaller. Therefore, when testing the Kandian applet, the security of the covariate data acquired by the server from WeChat is higher than the security of the covariate data acquired from Tencent QQ, and higher than the security of the covariate data acquired from the Kandian applet itself.

[0049] S202, performing security verification on each covariate data among the multiple covariate data to obtain a security verification result.

[0050] In one embodiment, security verification is performed by determining whether the expectation of each covariate data among a plurality of covariate data is equal to the expectation of the corresponding control covariate data, wherein the covariate data is obtained when testing the target application, so it can be understood that the covariate data is the experimental group data, and the control covariate data refers to the covariate data of the control group corresponding to the experimental group. In a specific implementation, the server can use AA-test (a control test method) for security verification, that is, after obtaining the covariate data of the experimental group, the server can further obtain the control covariate data corresponding to the covariate data of the experimental group, and further, the server can respectively calculate the expected value of each control covariate data and the expected value of each covariate data, thereby realizing the verification of the security of the covariate data.

[0051] In one embodiment, since the first user is an experimental group user for testing the target application, assuming that the first user is user t, the covariate data set of the first user is X_new_i_t (i=1, 2, 3, ...), and correspondingly, if the control group user is user c, the control covariate data can be recorded as X_new_i_c (i=1, 2, 3, ...), and further, the server can determine whether the expected E(X_new_i)_t of the covariate data X_new_i_t and the expected E(X_new_i)_c of the corresponding control covariate data X_new_i_c are equal to determine the security of each covariate data in the covariate data set. Specifically, if E(X_new_1)_t=E(X_new_1)_c, the security verification of the covariate data X_new_1 is passed, and if E(X_new_2)_t and E(X_new_2)_c are not equal, the covariate data X_new_2 fails the security verification.

[0052] S203, according to the security verification result, determine the first target covariate data from the multiple covariate data, and reduce the variance of the user operation data of the first user according to the first target covariate data, where the user operation data of the first user is detected during the process of performing a test operation on the target application.

[0053] In one embodiment, the server may determine one or more security-verified covariate data from multiple covariate data based on the security verification result. If the server only determines one security-verified covariate data based on the security verification result, the security-verified covariate data is used as the first target covariate data, such as the above-mentioned covariate data X_new_1. Alternatively, if the server determines multiple security-verified covariate data based on the security verification result, it is necessary to further screen the multiple security-verified covariate data to finally screen out a first target covariate data. In a specific implementation, the server can calculate the correlation coefficient between each security-verified covariate data and the user operation data of the first user, so that the covariate data with the highest correlation coefficient can be used as the first target covariate data. Among them, if the covariate data is X_new_i (i=1, 2, 3, ...), the user operation data of the first user is Y_new, assuming that the covariate data that have passed the security verification include X_new_1 and X_new_2, then the server calculates the correlation coefficients between X_new_1 and X_new_2 and the user operation data Y_new of the first user respectively. If the correlation coefficient between X_new_2 and the user operation data Y_new of the first user is the largest among the correlation coefficients calculated by the server, then X_new_2 is used as the first target covariate data.

[0054] In one embodiment, after the server determines the first target covariate data from the multiple covariate data, the variance of the user operation data of the first user can be reduced according to the first target covariate data. The user operation data of the first user refers to the number of pages browsed, the number of times the page is clicked, etc. by the first user during the target application test. In a specific implementation, after the server determines the first target covariate data from the multiple covariate data, the user operation data of the first user can be obtained, and the covariance of the first target covariate data and the user operation data of the first user, as well as the variance of the first target covariate data, can be determined according to the covariance and variance, and the reduction coefficient can be determined. The specific method of calculating the reduction coefficient can refer to the above formula 1.1. After the server determines the reduction coefficient, the server can determine the mean of the first target covariate data and the expected value of the first target covariate data. Further, the server can determine the first product of the mean and the reduction coefficient, and the second product of the expected value and the reduction coefficient, so that the unbiased estimation data of the user operation data of the first user can be determined according to the user operation data of the first user, the first product, and the second product, wherein the calculation method of the unbiased estimation data can refer to the above formula 1.2. Further, the server may determine the minimum value of the variance of the unbiased estimation data through the reduction coefficient, so as to reduce the variance of the user operation data of the first user according to the minimum value.

[0055] In an embodiment of the present invention, the server responds to a test operation on a target application and obtains a covariate data set of a first user, the covariate data set including multiple covariate data, and performs security verification on each of the multiple covariate data to obtain a security verification result, and then determines first target covariate data from the multiple covariate data based on the security verification result, and reduces the variance of the user operation data of the first user based on the first target covariate data, thereby improving the reduction effect and improving the efficiency and accuracy of experimental testing.

[0056] See also Figure 3 , Figure 3 1 is a flow chart of another data processing method provided by an embodiment of the present invention. The data processing method described in this embodiment is applied to the above-mentioned server, such as Figure 3 As shown, the method may include:

[0057] S301 : In response to a test operation on a target application, obtain a covariate dataset of a first user.

[0058] S302, performing security verification on each covariate data among the multiple covariate data to obtain a security verification result.

[0059] In step S301 and step S302, when the server acquires the covariate data set of the first user, it may acquire the covariate data from the target application, the associated application of the target application, and the external application, respectively, so as to form the covariate data set of the first user. In a specific implementation, the server may determine the first covariate data of the first user from the target application, determine the second covariate data of the first user from the associated application of the target application, and determine the third covariate data of the first user from the external application, where the external application refers to an application other than the target application and the associated application. Furthermore, after acquiring the first covariate data, the second covariate data, and the third covariate data, the server may acquire the covariate data set, which includes: one or more of the first covariate data, the second covariate data, and the third covariate data.

[0060] In one embodiment, after the server obtains the covariate data set of the first user, it can perform security verification on the multiple covariate data included in the covariate data set, wherein the security verification is performed by determining whether the expectation of each covariate data in the multiple covariate data is equal to the expectation of the corresponding control covariate data. In a specific implementation, when performing security verification, the server can first determine the control user of the first user, so as to obtain the control covariate data set of the control user, the control covariate data set includes multiple control covariate data, any control covariate data corresponds to one covariate data in the multiple covariates, and according to the expected value of each control covariate data and the expected value of each covariate data, each covariate data is security verified, thereby obtaining a security verification result. In one embodiment, after the server obtains the covariate data set of the first user, and performs security verification on each covariate data included in the covariate data set, and determines the security verification result of each covariate data, it can further reduce the variance of the user operation data of the first user according to the security verification result, that is, turn to execute step S303.

[0061] S303, according to the security verification result, determine the first target covariate data from the multiple covariate data, and reduce the variance of the user operation data of the first user according to the first target covariate data, where the user operation data of the first user is detected during the test operation performed on the target application.

[0062] In one implementation, since each covariate data in the covariate data set has corresponding control covariate data, the server can compare the expectation of each covariate data in the multiple covariate data with the expectation of the corresponding control covariate data according to the corresponding relationship, and determine the reference covariate data equal to the expectation of the corresponding control covariate data from the multiple covariate data according to the comparison result, that is, the reference covariate data refers to the covariate data whose expectation is equal to the expectation of the corresponding control covariate data. After the server determines the reference covariate data, if the server determines that the number of the reference covariate data is one, the reference covariate data is used as the first target covariate data; or, if the server determines that the number of the reference covariate data is at least two, the correlation coefficient between each reference covariate data and the user operation data of the first user is determined, and the reference covariate data corresponding to the maximum correlation coefficient is used as the first target covariate data. Wherein, the corresponding relationship refers to the relationship that each covariate data in the multiple covariate data corresponds to one control covariate data.

[0063] In one embodiment, it can be assumed that the first user is user t, and the multiple covariate data (covariate data sets) corresponding to user t obtained by the server are X_new_i_t (i=1, 2, 3,...), and the control user of user t is user c. Then the control covariate data corresponding to the multiple covariate data respectively obtained by the server can be expressed as X_new_i_c (where i=1, 2, 3,...). It can be understood that X_new_1_t corresponds to X_new_1_c, X_new_2_t corresponds to X_new_2_c, X_new_3_t corresponds to X_new_3_c, and so on. Further, the server may compare the expected value of each covariate data with the expected value of the corresponding control covariate data, such as comparing E(X_new_1_t) and E(X_new_1_c). If the server determines that E(X_new_1_t)=E(X_new_1_c), that is, the number of reference covariate data X_new_1_t determined by the server is one, then the server may determine the determined reference covariate data X_new_1_t as the target covariate data. In one embodiment, when the number of reference covariate data determined by the server from the first covariate data set is multiple, when the server determines the target covariate data from the multiple reference covariate data, the server may first determine the correlation coefficient between each reference covariate data and the user operation data of the first user. Since the greater the correlation between the reference covariate data and the user operation data of the first user, the better the effect of variance reduction, the server may use the reference covariate data corresponding to the maximum correlation coefficient as the first target covariate data, thereby improving the variance reduction effect.

[0064] In one embodiment, if the server determines based on the comparison between the expectations of the covariate data of user t and the expectations of the control covariate data of user c, the reference covariate data determined from the covariate data set of user t include X_new_1_t and X_new_3_t. Further, in order to determine the target covariate data from the two reference covariate data, the correlation coefficient between the reference covariate data X_new_1_t and the user operation data of user t (i.e., the first user) (the user operation data can be expressed as Y_new) and the correlation coefficient between the reference covariate data X_new_3_t and the user operation data Y_new of user t can be determined respectively. Further, the server can use the reference covariate data corresponding to the maximum correlation coefficient as the first target covariate data. Among them, the server can use the correlation coefficient calculation method to calculate the correlation coefficient between the reference covariate data X_new_1_t and the user operation data Y_new, as well as the correlation coefficient between the reference covariate data X_new_3_t and the user operation data Y_new. If the server determines that the correlation coefficient between X_new_3_t and the user operation data Y_new is greater than the correlation coefficient between X_new_1_t and the user operation data Y_new, then X_new_3_t can be determined as the first target covariate data.

[0065] In one embodiment, after determining the first target covariate data of the first user, the server may reduce the variance of the user operation data of the first user based on the determined first target covariate data, wherein the server may use an enhanced controlled experiment CUPED-Plus (Controlled-experiment Using Pre-ExperimentData-Plus, CUPED-Plus) algorithm to reduce the variance of the user operation data of the first user. In addition, the server may also reduce the variance of the user operation data of the second user based on the second target covariate data of the second user, thereby reducing the variance of the user operation data of all users, thereby improving the efficiency of the server in testing the target application. When the server reduces the variance of the user operation data of the second user, step S304 may be executed first.

[0066] S304: Acquire second target covariate data of the second user, where the second target covariate data is acquired before performing the test operation on the target application.

[0067] In one embodiment, the second user is an old user of the target application, and correspondingly, the first user is a new user of the target application. It can be understood that the old user refers to a user who has completed registration in the target application before testing the target application, and the new user refers to a user who has not registered and browsed the target application before testing the target application. In one embodiment, the second user can be, for example, a user who appeared in the historical time period before the target application performed the test operation process, for example, a user who completed registration and browsed in the target application within 7 days before testing the target application.

[0068] In one embodiment, if the running period before the target application performs the test operation is a dry run period, the process of performing the test operation on the target application is called an experimental period, wherein the server may use the data of the second user in the dry run period as the covariate data of the second user, and further, the server may determine the second target covariate data from the covariate data of the second user.

[0069] 305 : Reduce the variance of user operation data of a second user according to the second target covariate data, where the user operation data of the second user is detected during a process of performing a test operation on the target application.

[0070] In one embodiment, when the server reduces the variance of the user operation data of the second user based on the second target covariate data, the server may use a controlled experiment CUPED (Controlled-experiment Using Pre-Experiment Data, CUPED) algorithm to reduce the variance of the user operation data of the second user. In a specific implementation, the server may first determine the unbiased estimate data of the user operation data of the second user based on the second target covariate data, and determine the minimum variance of the unbiased estimate data of the user operation data of the second user, and reduce the variance of the user operation data of the second user based on the minimum variance of the unbiased estimate data. In a specific implementation, the second target covariate data is X, the server may obtain the user operation data Y of the first user, and determine the covariance cov(Y, X) of the second target covariate data and the user operation data of the second user, and the variance var(X) of the first target covariate data. The reduction coefficient θ may be determined based on the covariance and variance. The specific method for calculating the reduction coefficient may refer to the above formula 2.1:

[0071]

[0072] After determining the reduction coefficient θ, the server can determine the mean value mean(X) of the second target covariate data and the expected value E(X) of the first target covariate data. Further, the server can determine the first product θ*mean(X) of the mean value and the reduction coefficient and the second product θ*E(X) of the expected value and the reduction coefficient, thereby determining the unbiased estimated data Y_cv of the user operation data of the first user according to the user operation data of the first user, the first product, and the second product. The calculation method of the unbiased estimated data can refer to Formula 2.2 as shown in:

[0073] Y_cv=Y-θmean(X)+θE(X) Equation 2.2

[0074] After determining the unbiased estimate data Y_cv of the user operation data of the second user, the server can determine the minimum value of the variance of the unbiased estimate data Y_cv, that is, the minimum value of var(Y_cv). It can be understood that the minimum value of var(Y_cv) is smaller than the minimum value of var(Y). Therefore, after determining the minimum value of var(Y_cv), the server can use the minimum value of var(Y_cv) as the variance of the user operation data of the first user, thereby reducing the variance of the user operation data of the second user.

[0075] 306 , reducing the variance of the user operation data of all users according to the reduced variance of the user operation data of the first user and the reduced variance of the user operation data of the second user.

[0076] In a feasible embodiment, all users refer to all users in the target application test process, including the first user and the second user. The server integrates the variance of the reduced user operation data of the first user and the variance of the reduced user operation data of the second user according to the proportion of the first user and the second user to the total number of users, thereby reducing the variance of the user operation data of the total number of users. Since the minimum sample size is proportional to the variance, reducing the variance of the total number of users can not only improve the reduction effect, but also reduce the sample size required for the experiment, thereby improving the efficiency of the experiment.

[0077] In one embodiment, Figure 4aIn the example, it is assumed that the first user is a new user and the second user is an old user. The embodiment of the present invention provides two variance reduction methods for calculating new and old users. First, it is determined whether the user is an old user based on the user who appears in the idle period and the experimental period of the target application. If the user appears in the idle period, it is determined that the user is an old user. The CUPED algorithm can be used to use the idle period data X of the old user as the covariate data of the experimental period data Y of the old user, and the unbiased estimated data Y of the old user is calculated by formula 2.1 and formula 2.2. cv , thus determining var(Y cv ) minimum value, according to var(Y cv ) minimum value to reduce the variance of the experimental data of old users.

[0078] If a user does not appear during the idle period, it is determined that the user is a new user (i.e., the first user mentioned above), and the server can use the CUPED-Plus algorithm to reduce the variance of the user operation data of the new user. Specifically, the server can first obtain the experimental period related data of the new user in the target application as the covariate data X_new_1, the data in the associated application associated with the target application as the covariate data X_new_2, and the data in the external application as the covariate data X_new_3, and verify through AA-test whether E(X_new_i)_t=E(X_new_i)_c is satisfied, where i=1, 2, 3, ..., if E(X_new_i)_t=E(X_new_i)_c is satisfied, the covariate data X_new_i can be preliminarily obtained, if there are multiple covariate data X_new_i, the correlation coefficient between X_new_i and the experimental period data of the new user is calculated, and the X_new_i with the highest correlation is selected as the final covariate data. Furthermore, the unbiased estimated data Y_cv_new of the new user is determined by equations 1.1 and 1.2, thereby determining the minimum value of var(Y_cv_new), and thus reducing the variance of the experimental data of the new user according to the minimum value of var(Y_cv_new).

[0079] based on Figure 4aThe data processing method flow obtains the comparative effect shown in Table 2, where the variance of the data and the minimum sample size required for the experiment are reduced to 0 based on the calculation without using any algorithm, and the experimental time is not reduced. After using the above-mentioned CUPED algorithm, the variance of the experimental data of old users is reduced by 50%-60%. Since the minimum sample size required for the experiment is proportional to the variance, the minimum sample size required for the experiment is also reduced by 50%-60%, that is, 100,000 people were originally required to observe the realization of significant effects, but after using CUPED, only 40,000-50,000 users are required. In addition, assuming that it takes 5 days for an experiment to reach a cumulative number of 40,000 users, since users need to be deduplicated in the experiment, the original experiment requires 100,000 people, which takes about 15 days or even 20 days, that is, the experimental time is reduced to 1 / 3-1 / 4 of the time without the algorithm. Using the CUPED-Plus algorithm, in addition to reducing the variance of old users, the variance of new users' experimental data is also reduced to a relatively good level. In summary, the overall variance can be reduced by 70%-80%, that is, the minimum sample size required for the experiment is also reduced by 70%-80%, and the experimental time is reduced to 1 / 4 or even less.

[0080] Table 2

[0081]

[0082]

[0083] In a feasible embodiment, the server outputs a test interface, which includes a test indicator corresponding to the test operation and an indicator corresponding to the test indicator, the indicator being used to indicate whether the target application can be applied to all users based on the reduction of the variance of the operation data of the first user and the reduction of the operation data of the second user. If the indicator is a first indicator, it is determined that the target application can be applied to all users according to the first indicator. If the indicator is a second indicator, it is determined that the target application cannot be applied to all users according to the second indicator. If the indicator is a third indicator, it is impossible to determine whether the target application can be applied to all users according to the third indicator. The test indicator can be page views, page clicks, etc. Page views can be expressed in PV per person, and page clicks can be expressed in VV per person. In a specific implementation, before performing a test operation on a target application, the server may pre-set a test indicator and set an indication mark corresponding to the test indicator. In response to performing a test operation on the target application, the server may output a test interface, which includes a test indicator corresponding to the test operation and an indication mark corresponding to the test indicator. According to the indication mark corresponding to the test indicator, it can be determined whether the target application can be applied to all users based on the reduction of the variance of the operation data of the first user and the reduction of the operation data of the second user.

[0084] Among them, the test interface is as follows Figure 4b As shown in the example, the test interface is a test interface outputted for testing the target application program, the Kandian applet, wherein the pre-set test indicators are page views, page clicks, the average APP time per person (seconds), and the average video time per person (seconds), and the indicator corresponding to the test indicator is a battery grid, which is used to indicate whether the Kandian applet can be applied to all users based on the reduction of the variance of the operation data of the first user and the reduction of the operation data of the second user. If the battery grid is full (i.e., the first indicator mentioned above), the Kandian applet can be applied to all users based on the reduction of the variance of the operation data of the first user and the reduction of the operation data of the second user. If the battery grid is empty (i.e., the second indicator mentioned above), the Kandian applet cannot be applied to all users based on the reduction of the variance of the operation data of the first user and the reduction of the operation data of the second user. If the battery grid is cracked (i.e., the third indicator mentioned above), it is impossible to determine whether the Kandian applet can be applied to all users based on the reduction of the variance of the operation data of the first user and the reduction of the operation data of the second user.

[0085] In an embodiment of the present invention, the server responds to a test operation on a target application and obtains a covariate data set of a first user, the covariate data set including multiple covariate data, and performs security verification on each of the multiple covariate data to obtain a security verification result, and then determines first target covariate data from the multiple covariate data based on the security verification result, and reduces the variance of the user operation data of the first user based on the first target covariate data, obtains second target covariate data of the second user, and reduces the variance of the user operation data of the first user based on the first target covariate data, and then reduces the variance of the user operation data of all users based on the reduced variance of the user operation data of the first user and the reduced variance of the user operation data of the second user, which can reduce the quantity requirement for collecting user operation data in the test phase while ensuring the accuracy of the test results, thereby improving the efficiency of testing the target application.

[0086] Based on the description of the above data processing method embodiment, the present invention also proposes a data processing device, which may be a computer program (including program code) running on the above server. The data processing device may be used to execute the following steps: Figure 2 as well as Figure 3 For the data processing method, please refer to Figure 5The data processing device includes: an acquisition unit 501, a verification unit 502, a determination unit 503 and a reduction unit 504.

[0087] An acquiring unit 501 is configured to acquire a covariate data set of a first user in response to a test operation on a target application, wherein the covariate data set includes a plurality of covariate data;

[0088] A verification unit 502, configured to perform security verification on each of the plurality of covariate data to obtain a security verification result;

[0089] A determining unit 503, configured to determine first target covariate data from the plurality of covariate data according to the security verification result;

[0090] The reduction unit 504 is configured to reduce the variance of the user operation data of the first user according to the first target covariate data, where the user operation data of the first user is detected during the process of performing the test operation on the target application.

[0091] In one embodiment, the acquisition unit 501 is specifically configured to:

[0092] determining first covariate data for the first user from the target application;

[0093] determining second covariate data of the first user from an application associated with the target application;

[0094] determining third covariate data of the first user from an external application, wherein the external application refers to an application other than the target application and the associated application;

[0095] A covariate data set is obtained, where the covariate data set includes one or more of the first covariate data, the second covariate data, and the third covariate data.

[0096] In one embodiment, the verification unit 502 is specifically used to:

[0097] Determine a control user of the first user, and obtain a control covariate data set of the control user, wherein the control covariate data set includes a plurality of control covariate data, and any control covariate data corresponds to one covariate data of the plurality of covariates;

[0098] According to the expected value of each control covariate data and the expected value of each covariate data, safety verification is performed on each covariate data.

[0099] In one embodiment, the verification unit 502 is specifically used to:

[0100] According to the corresponding relationship, comparing the expectation of each covariate data in the plurality of covariate data with the expectation of the corresponding control covariate data;

[0101] According to the comparison result, determining reference covariate data that is equal to the expectation of the corresponding control covariate data from the plurality of covariate data;

[0102] If the number of the reference covariate data is one, the reference covariate data is used as the first target covariate data;

[0103] If the number of the reference covariate data is at least two, a correlation coefficient between each reference covariate data and the user operation data of the first user is determined, and the reference covariate data corresponding to the maximum correlation coefficient is used as the first target covariate data.

[0104] In one embodiment, the acquisition unit 501 is further used to acquire second target covariate data of a second user, where the second target covariate data is acquired before performing the test operation on the target application;

[0105] The reduction unit 504 is further configured to reduce the variance of the user operation data of the second user according to the second target covariate data, where the user operation data of the second user is detected during the process of performing the test operation on the target application.

[0106] In one embodiment, the reduction unit 504 is further used to reduce the variance of the user operation data of all users based on the reduced variance of the user operation data of the first user and the reduced variance of the user operation data of the second user, where the all users include the first user and the second user.

[0107] In one embodiment, the user operation data of the first user is used as the target operation data, or the user operation data of the second user is used as the target operation data, and the reduction unit 504 is specifically used to:

[0108] Determine the mean and expected value of the covariate data corresponding to the target operation data;

[0109] determining a reduction factor, and determining a first product of the mean value and the reduction factor, and a second product of the expected value and the reduction factor;

[0110] Determining unbiased estimation data of the target operation data based on the target operation data, the first product, and the second product;

[0111] According to the reduction coefficient, a minimum value of the variance of the unbiased estimation data is determined, and the variance of the target operation data is reduced according to the minimum value.

[0112] In one embodiment, the reduction unit 504 is specifically configured to:

[0113] Determining the target operation data and the covariance between the covariate data corresponding to the target operation data;

[0114] Determining the variance of the covariate data corresponding to the target operation data;

[0115] The ratio between the covariance and the variance is used as a reduction coefficient.

[0116] In one embodiment, the device further includes: an output unit 505 .

[0117] an output unit 505, configured to output a test interface, the test interface including a test indicator corresponding to the test operation and an indicator corresponding to the test indicator, the indicator being configured to indicate whether the target application can be applied to all users based on the reduction of the variance of the operation data of the first user and the reduction of the operation data of the second user;

[0118] If the indicator is the first indicator, determining, according to the first indicator, that the target application can be applied to all users;

[0119] If the indicator is the second indicator, determining, according to the second indicator, that the target application cannot be applied to all users;

[0120] If the indicator is the third indicator, it is impossible to determine whether the target application can be applied to all users based on the third indicator.

[0121] In an embodiment of the present invention, the acquisition unit 501 responds to a test operation on a target application and acquires a covariate data set of a first user, the covariate data set including multiple covariate data, the verification unit 502 performs security verification on each of the multiple covariate data to obtain a security verification result, and then the determination unit 503 can determine the first target covariate data from the multiple covariate data based on the security verification result, so that the reduction unit 504 can reduce the variance of the user operation data of the first user based on the first target covariate data, thereby reducing the variance of the user operation data, thereby improving the test efficiency of the target application while ensuring the test accuracy of the target application.

[0122] See also Figure 6 , is a schematic block diagram of the structure of a server provided in an embodiment of the present invention. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. The terminal may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited thereto. Figure 6 The server in the embodiment shown may include: one or more processors 601; one or more input devices 602, one or more output devices 603 and a memory 604. The processor 601, the input device 602, the output device 603 and the memory 604 are connected via a bus 605. The memory 604 is used to store a computer program, the computer program includes program instructions, and the processor 601 is used to execute the program instructions stored in the memory 604.

[0123] The memory 604 may include a volatile memory, such as a random-access memory (RAM); the memory 604 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the memory 604 may also include a combination of the above types of memory.

[0124] The processor 601 may be a central processing unit (CPU). The processor 601 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. The PLD may be a field-programmable gate array (FPGA), a generic array logic (GAL), etc. The processor 601 may also be a combination of the above structures.

[0125] In the embodiment of the present invention, the memory 604 is used to store a computer program, the computer program includes program instructions, and the processor 601 is used to execute the program instructions stored in the memory 604 to implement the above-mentioned Figure 2 and Figure 3 The steps of the corresponding method in .

[0126] In one embodiment, the processor 601 is configured to call the program instructions to execute:

[0127] In response to a test operation on a target application, obtaining a covariate data set of a first user, the covariate data set including a plurality of covariate data;

[0128] Performing security verification on each covariate data among the multiple covariate data to obtain a security verification result;

[0129] Based on the security verification result, first target covariate data is determined from the multiple covariate data, and the variance of the user operation data of the first user is reduced based on the first target covariate data. The user operation data of the first user is detected during the process of performing the test operation on the target application.

[0130] In one embodiment, the processor 601 is configured to call the program instructions to execute:

[0131] determining first covariate data for the first user from the target application;

[0132] determining second covariate data of the first user from an application associated with the target application;

[0133] determining third covariate data of the first user from an external application, wherein the external application refers to an application other than the target application and the associated application;

[0134] A covariate data set is obtained, where the covariate data set includes one or more of the first covariate data, the second covariate data, and the third covariate data.

[0135] In one embodiment, the processor 601 is configured to call the program instructions to execute:

[0136] Determine a control user of the first user, and obtain a control covariate data set of the control user, wherein the control covariate data set includes a plurality of control covariate data, and any control covariate data corresponds to one covariate data of the plurality of covariates;

[0137] According to the expected value of each control covariate data and the expected value of each covariate data, safety verification is performed on each covariate data.

[0138] In one embodiment, the processor 601 is configured to call the program instructions to execute:

[0139] According to the corresponding relationship, comparing the expectation of each covariate data in the plurality of covariate data with the expectation of the corresponding control covariate data;

[0140] According to the comparison result, determining reference covariate data that is equal to the expectation of the corresponding control covariate data from the plurality of covariate data;

[0141] If the number of the reference covariate data is one, the reference covariate data is used as the first target covariate data;

[0142] If the number of the reference covariate data is at least two, a correlation coefficient between each reference covariate data and the user operation data of the first user is determined, and the reference covariate data corresponding to the maximum correlation coefficient is used as the first target covariate data.

[0143] In one embodiment, the processor 601 is configured to call the program instructions and further to execute:

[0144] Acquire second target covariate data of a second user, where the second target covariate data is acquired before performing the test operation on the target application;

[0145] The variance of the user operation data of the second user is reduced according to the second target covariate data, where the user operation data of the second user is detected during the process of performing the test operation on the target application.

[0146] In one embodiment, the processor 601 is configured to call the program instructions and is further configured to execute:

[0147] According to the reduced variance of the user operation data of the first user and the reduced variance of the user operation data of the second user, the variance of the user operation data of all users is reduced, and the all users include the first user and the second user.

[0148] In one embodiment, the user operation data of the first user is used as the target operation data, or the user operation data of the second user is used as the target operation data, and the processor 601 is configured to call the program instruction to execute:

[0149] Determine the mean and expected value of the covariate data corresponding to the target operation data;

[0150] determining a reduction factor, and determining a first product of the mean value and the reduction factor, and a second product of the expected value and the reduction factor;

[0151] Determining unbiased estimation data of the target operation data based on the target operation data, the first product, and the second product;

[0152] According to the reduction coefficient, a minimum value of the variance of the unbiased estimation data is determined, and the variance of the target operation data is reduced according to the minimum value.

[0153] In one embodiment, the processor 601 is configured to call the program instructions to execute:

[0154] Determining the target operation data and the covariance between the covariate data corresponding to the target operation data;

[0155] Determining the variance of the covariate data corresponding to the target operation data;

[0156] The ratio between the covariance and the variance is used as a reduction coefficient.

[0157] In one embodiment, the processor 601 is configured to call the program instructions and is further configured to execute:

[0158] Outputting a test interface, the test interface including a test indicator corresponding to the test operation and an indicator corresponding to the test indicator, the indicator being used to indicate whether the target application can be applied to all users based on the reduction of the variance of the operation data of the first user and the reduction of the operation data of the second user;

[0159] If the indicator is the first indicator, determining, according to the first indicator, that the target application can be applied to all users;

[0160] If the indicator is the second indicator, determining, according to the second indicator, that the target application cannot be applied to all users;

[0161] If the indicator is the third indicator, it is impossible to determine whether the target application can be applied to all users based on the third indicator.

[0162] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method described above.

[0163] The embodiment of the present invention further provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned Figure 2 and Figure 3 The method embodiment shown in the figure. Wherein, the computer-readable storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0164] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A data processing method, characterized in that: include: In response to a test operation on a target application, obtaining a covariate data set of a first user, the covariate data set including a plurality of covariate data; Performing security verification on each of the multiple covariate data to obtain a security verification result; wherein, performing security verification on each of the multiple covariate data includes: determining a control user of the first user, and obtaining a control covariate data set of the control user, the control covariate data set including multiple control covariate data, any control covariate data corresponding to one of the multiple covariates; performing security verification on each of the covariate data according to an expected value of each control covariate data and an expected value of each covariate data; Based on the security verification result, first target covariate data is determined from the multiple covariate data, and the variance of the user operation data of the first user is reduced based on the first target covariate data. The user operation data of the first user is detected during the process of performing the test operation on the target application.

2. The method according to claim 1, characterized in that: The obtaining of the covariate dataset of the first user includes: determining first covariate data for the first user from the target application; determining second covariate data of the first user from an application associated with the target application; determining third covariate data of the first user from an external application, wherein the external application refers to an application other than the target application and the associated application; A covariate data set is obtained, where the covariate data set includes one or more of the first covariate data, the second covariate data, and the third covariate data.

3. The method according to claim 1, characterized in that The step of determining first target covariate data from the plurality of covariate data according to the safety verification result includes: According to the corresponding relationship, comparing the expectation of each covariate data in the plurality of covariate data with the expectation of the corresponding control covariate data; According to the comparison result, determining reference covariate data that is equal to the expectation of the corresponding control covariate data from the plurality of covariate data; If the number of the reference covariate data is one, the reference covariate data is used as the first target covariate data; If the number of the reference covariate data is at least two, a correlation coefficient between each reference covariate data and the user operation data of the first user is determined, and the reference covariate data corresponding to the maximum correlation coefficient is used as the first target covariate data.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire second target covariate data of a second user, where the second target covariate data is acquired before performing the test operation on the target application; The variance of the user operation data of the second user is reduced according to the second target covariate data, where the user operation data of the second user is detected during the process of performing the test operation on the target application.

5. The method according to claim 4, characterized in that The method further comprises: According to the reduced variance of the user operation data of the first user and the reduced variance of the user operation data of the second user, the variance of the user operation data of all users is reduced, and the all users include the first user and the second user.

6. The method according to claim 1 or 4, characterized in that: Using the user operation data of the first user as target operation data, or using the user operation data of the second user as target operation data, and reducing the variance of the target operation data, includes: Determine the mean and expected value of the covariate data corresponding to the target operation data; determining a reduction factor, and determining a first product of the mean value and the reduction factor, and a second product of the expected value and the reduction factor; Determining unbiased estimation data of the target operation data based on the target operation data, the first product, and the second product; According to the reduction coefficient, a minimum value of the variance of the unbiased estimation data is determined, and the variance of the target operation data is reduced according to the minimum value.

7. The method according to claim 6, characterized in that Determining the reduction factor includes: Determining the target operation data and the covariance between the covariate data corresponding to the target operation data; Determining the variance of the covariate data corresponding to the target operation data; The ratio between the covariance and the variance is used as a reduction coefficient.

8. The method according to claim 5, characterized in that the method further comprises: Outputting a test interface, the test interface including a test indicator corresponding to the test operation and an indicator corresponding to the test indicator, the indicator being used to indicate whether the target application can be applied to all users based on the reduction of the variance of the operation data of the first user and the reduction of the operation data of the second user; If the indicator is the first indicator, determining, according to the first indicator, that the target application can be applied to all users; If the indicator is the second indicator, determining, according to the second indicator, that the target application cannot be applied to all users; If the indicator is the third indicator, it is impossible to determine whether the target application can be applied to all users based on the third indicator.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 8.

10. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program comprises program instructions. When the program instructions are called by a processor, the processor is caused to execute the method according to any one of claims 1 to 8.

Images