Test method and device of experimental parameters and electronic equipment

Abstract

The invention provides an experimental parameter testing method and device and electronic equipment, and the method comprises the steps: determining a business index of a target object, and obtaining a plurality of business index values of the business index from a plurality of databases; determining a quantile index value corresponding to the service index according to the plurality of service index values; obtaining a plurality of first real index values corresponding to the experimental group of the business indexes of the plurality of databases and a plurality of second real index values corresponding to the control group; and according to the plurality of first real index values, the plurality of second real index values and the quantile index value, determining a test result of the experimental parameter, the test result being that the test is passed or not passed. And the test accuracy of experimental parameters is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a method, device and electronic equipment for testing experimental parameters. Background Art

[0002] Before adding new features to an application, developers need to test the impact of the feature on certain business indicators (such as startup time).

[0003] At present, since business indicators can be stored in different databases, electronic devices can evaluate business indicators based on the mean of business indicators in different databases. However, for business data of business indicators stored in quantiles (such as 50 quantiles), the mean cannot accurately reflect the distribution of business indicators in each database, and the accuracy of the data is low, which in turn leads to low accuracy of experimental evaluation. Summary of the invention

[0004] The present disclosure provides a method, device and electronic equipment for testing experimental parameters, which are used to solve one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides a method for testing an experimental parameter, the method comprising:

[0006] Determine a business indicator of a target object, and obtain multiple business indicator values ​​of the business indicator from multiple databases;

[0007] Determine, according to the multiple business indicator values, a quantile indicator value corresponding to the business indicator;

[0008] Acquire a plurality of first real indicator values ​​corresponding to an experimental group and a plurality of second real indicator values ​​corresponding to a control group of business indicators of the plurality of databases;

[0009] A test result of the experimental parameter is determined according to the multiple first true index values, the multiple second true index values ​​and the quantile index value, and the test result is test pass or test fail.

[0010] In a second aspect, the present disclosure provides a testing device for experimental parameters, the testing device for experimental parameters comprising a first determination module, a first acquisition module, a second determination module, a second acquisition module and a third determination module, wherein:

[0011] The first determination module is used to determine the business indicator of the target object;

[0012] The first acquisition module is used to acquire multiple business indicator values ​​of the business indicator from multiple databases;

[0013] The second determination module is used to determine the quantile indicator value corresponding to the business indicator according to the multiple business indicator values;

[0014] The second acquisition module is used to acquire a plurality of first real indicator values ​​corresponding to the experimental group and a plurality of second real indicator values ​​corresponding to the control group of the business indicators of the plurality of databases;

[0015] The third determination module is used to determine a test result of the experimental parameter according to the multiple first true indicator values, the multiple second true indicator values ​​and the quantile indicator value, and the test result is a test pass or a test fail.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device including: a processor and a memory;

[0017] The memory stores computer-executable instructions;

[0018] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the testing method of the first aspect and various experimental parameters that may be involved in the first aspect as described above.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the testing method of the first aspect and various experimental parameters that may be involved in the first aspect are implemented.

[0020] The present disclosure provides a test method, device and electronic device for experimental parameters, the electronic device can determine the business indicators of the target object, and obtain multiple business indicator values ​​of the business indicators from multiple databases, determine the quantile indicator value corresponding to the business indicator according to the multiple business indicator values, obtain multiple first real indicator values ​​corresponding to the experimental group of the business indicators of multiple databases and multiple second real indicator values ​​corresponding to the control group, and determine the test result of the experimental parameter according to the multiple first real indicator values, multiple second real indicator values ​​and quantile indicator values, wherein the test result is the test passed or the test failed. In the above method, since the electronic device can accurately determine the superiority or deterioration result of the experimental parameter on the business indicator in combination with the quantile indicator value, based on the real indicator value of the control group and the real indicator value of the experimental group, the accuracy of the experimental evaluation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of a flow chart of a method for testing experimental parameters provided in an embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of obtaining a business indicator value provided by an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of a first real index value provided by an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of determining a test result provided by an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of a method for determining a quantile index value provided by an embodiment of the present disclosure;

[0028] Figure 7 A process schematic diagram of a method for testing experimental parameters provided by an embodiment of the present disclosure;

[0029] Figure 8 A schematic diagram of the structure of a testing device for experimental parameters provided in an embodiment of the present disclosure;

[0030] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0033] Terminal device: a device with wireless transceiver function. The terminal device can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted. The terminal device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a vehicle-mounted terminal device, a wireless terminal in self-driving, a wireless terminal device in remote medical, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, a wireless terminal device in smart home, a wearable terminal device, etc. The terminal device involved in the embodiments of the present disclosure can also be called a terminal, a user equipment (UE), an access terminal device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote terminal device, a mobile device, a UE terminal device, a wireless communication device, a UE agent or a UE device, etc. Terminal devices can also be fixed or mobile.

[0034] Next, combine Figure 1 , the application scenarios of the embodiments of the present disclosure are described.

[0035] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 , including an experimental group and a control group. The experimental group and the control group both include database 1, database 2, and database 3. The 50th percentile of the startup time in database 1 of the experimental group is 2500 milliseconds, the 50th percentile of the startup time in database 2 is 2000 milliseconds, and the 50th percentile of the startup time in database 3 is 1500 milliseconds. The 50th percentile of the startup time in database 1 of the control group is 1500 milliseconds, the 50th percentile of the startup time in database 2 is 1500 milliseconds, and the 50th percentile of the startup time in database 3 is 1500 milliseconds.

[0036] See also Figure 1, the electronic device can average the 50th percentile startup time in database 1, database 2, and database 3, and obtain the average 50th percentile of the startup time of the experimental group is 2000 milliseconds, and the average 50th percentile of the startup time of the control group is 1500 milliseconds. Since 2000 milliseconds is greater than 1500 milliseconds, the electronic device can determine that the experiment does not optimize the startup time, and the experimental test fails.

[0037] It should be noted that Figure 1 The application scenarios of the embodiments of the present disclosure are merely illustrative and are not intended to limit the application scenarios of the embodiments of the present disclosure.

[0038] In the related art, before adding a new function to an application, the developer needs to test the impact of the function on some business indicators. For example, the developer needs to optimize the startup time of the application, so the developer can develop multiple functions, and needs to test whether each function optimizes the startup time. At present, since business indicators need to be stored in different data blocks, the electronic device needs to obtain the average value corresponding to the business indicator in different data blocks, and then evaluate the experiment based on the average value. For example, if the average startup time before the experiment is 1000 milliseconds and the average startup time after the experiment is 800 milliseconds, the electronic device can determine that the experiment optimizes the startup time. However, the values ​​of some business indicators in multiple databases are stored based on quantiles, and the electronic device calculates the average value cannot accurately reflect the distribution information indicated by the quantile. The value of the quantile is statistically calculated based on the average method, which makes the accuracy of the value of the business indicator low, and then leads to low accuracy of the experimental evaluation.

[0039] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides a test method for experimental parameters, wherein the electronic device can determine the business indicator of the target object, and obtain multiple business indicator values ​​of the business indicator from multiple databases, determine the quantile indicator value corresponding to the business indicator according to the multiple business indicator values, obtain multiple first real indicator values ​​corresponding to the experimental group of the business indicators of the multiple databases and multiple second real indicator values ​​corresponding to the control group, calculate the first permeability indicator associated with the experimental group, calculate the second permeability indicator associated with the control group, if the first permeability indicator is greater than or equal to the second permeability indicator, then determine that the test result is that the test fails, if the first permeability indicator is less than the second permeability indicator, then determine that the test result is that the test passes. In this way, since the permeability indicator can indicate the proportion of the real indicator value that meets the preset size relationship with the quantile indicator value, the influence of the sudden change of the real indicator value (such as a sudden increase or a sudden decrease) on the experimental result can be avoided, and the electronic device can accurately determine the change of the experimental group associated with the business indicators of the multiple databases compared with the control group based on the first permeability indicator value and the second permeability indicator value, thereby improving the accuracy of the experimental test.

[0040] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0041] Figure 2 A flow chart of a method for testing experimental parameters provided by an embodiment of the present disclosure. Figure 2 , the method may include:

[0042] S201. Determine the business indicators of the target object.

[0043] The execution subject of the embodiments of the present disclosure may be an electronic device, or a device for determining experimental parameters set in the electronic device. Among them, the device for determining experimental parameters may be implemented based on software, or the device for determining experimental parameters may be implemented based on a combination of software and hardware, which is not limited in the embodiments of the present disclosure. Optionally, the electronic device may be any device with on-end computing capabilities, such as a computer or an electronic device, which is not limited in the embodiments of the present disclosure.

[0044] Optionally, the target object may be an application to be optimized. For example, if the electronic device optimizes application 1, the target object may be application 1; if the electronic device optimizes application 2, the target object may be application 2.

[0045] Optionally, the target object may also be a function to be optimized. For example, if the electronic device optimizes function 1 in the application, the target object may be function 1; if the electronic device optimizes function 2 in the application, the target object may be function 2.

[0046] It should be noted that the electronic device can determine the target object based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0047] Optionally, the business indicator may be an indicator of the experiment impact in the target object. For example, if the experiment affects the startup time of the application, the business indicator may be the startup time of the application; if the experiment affects the playback frame rate of the application, the business indicator may be the playback frame rate of the application.

[0048] It should be noted that the business indicator can be an indicator of any business in the application, which is not limited in the embodiments of the present disclosure. In addition, the electronic device can determine the business indicator of the target object based on any feasible implementation method, which is not limited in the embodiments of the present disclosure.

[0049] S202: Acquire multiple business indicator values ​​of business indicators from multiple databases.

[0050] The service indicator value may be a value corresponding to the service indicator. For example, if the service indicator is the startup duration, the service indicator value may be 1000 milliseconds, 2000 milliseconds, 3000 milliseconds, etc., and if the service indicator is the playback frame rate, the service indicator value may be 30 frames, 60 frames, 120 frames, etc.

[0051] Among them, multiple business indicator values ​​of the business indicator can be stored in different databases. For example, the business indicator value corresponding to the startup time of region 1 can be stored in database A, the business indicator value corresponding to the startup time of region 2 can be stored in database B, and the business indicator value corresponding to the startup time of region 3 can be stored in database C.

[0052] Optionally, after the electronic device determines the business indicator, it can obtain the business indicator value corresponding to the business indicator in multiple databases. For example, if the business indicator is the startup duration, the electronic device can determine multiple databases storing data of the startup duration, and obtain the business indicator value corresponding to the startup duration in the multiple databases; if the business indicator is the playback frame rate, the electronic device can determine multiple databases storing data of the playback frame rate, and obtain the business indicator value corresponding to the playback frame rate in the multiple databases.

[0053] It should be noted that the electronic device can determine the database for storing the data of the business indicator based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0054] Optionally, for any database, the electronic device may input the identifier of the business indicator into the database, and then obtain multiple business indicator values ​​corresponding to the business indicator. For example, the database may store multiple business indicator values ​​corresponding to business indicator 1 and multiple business indicator values ​​corresponding to business indicator 2. If the electronic device inputs the identifier of business indicator 1 into the database, the electronic device may obtain multiple business indicator values ​​corresponding to business indicator 1 in the database.

[0055] It should be noted that the electronic device may also obtain multiple business indicator values ​​of the business indicators from multiple databases based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0056] Next, combine Figure 3 , a process of an electronic device obtaining multiple business indicator values ​​of business indicators from multiple databases is described.

[0057] Figure 3 A schematic diagram of obtaining a business indicator value provided by an embodiment of the present disclosure. Figure 3 , including database A and database B, wherein database A includes startup duration, playback frame rate, and multiple values ​​corresponding to the startup duration and multiple values ​​corresponding to the playback frame rate. Database B also includes startup duration, playback frame rate, and multiple values ​​corresponding to the startup duration and multiple values ​​corresponding to the playback frame rate. If the electronic device ( Figure 3 (not shown) if the service indicator is determined to be the startup duration, the electronic device can obtain 1000 milliseconds, 1200 milliseconds, 1300 milliseconds and 1400 milliseconds.

[0058] S203: Determine a quantile indicator value corresponding to the business indicator according to the multiple business indicator values.

[0059] Optionally, the quantile index value may be a 10th quantile index value, a 50th quantile index value, an 80th quantile index value, etc., which is not limited in the present disclosure. For example, if the quantile index value is a 50th quantile index value, the quantile index value may be a median.

[0060] For example, if the business indicator is the startup duration, and the quantile indicator value is the 50 quantile indicator value, the quantile indicator value may be the median of multiple business indicator values ​​of the startup duration obtained from multiple databases.

[0061] Optionally, the electronic device may determine the quantile indicator value corresponding to the business indicator based on a dichotomy method. For example, the electronic device may determine the maximum business indicator value and the minimum business indicator value among multiple business indicator values, and then determine the average value of the maximum business indicator value and the minimum business indicator value. The electronic device uses the average value as the initial value and determines the quantile indicator value among multiple business indicator values ​​by a dichotomy method.

[0062] S204: Acquire a plurality of first real indicator values ​​corresponding to the experimental group and a plurality of second real indicator values ​​corresponding to the control group of business indicators of the plurality of databases.

[0063] Optionally, the first real indicator value may be an indicator value corresponding to a business indicator in an experimental group. For example, the experimental group may be a group for conducting experiments on a target object, wherein the electronic device may conduct experiments on the target object based on experimental parameters.

[0064] Optionally, the second real indicator value may be an indicator value corresponding to a business indicator in a control group. For example, the control group may be a group that does not conduct an experiment on the target object, wherein the electronic device may determine the degree of influence of the experimental parameters on the business indicator of the target object based on the data of the control group.

[0065] It should be noted that the electronic device can determine the experimental parameters of the target object based on any feasible implementation method (such as receiving experimental parameters sent by other devices, etc.), and the embodiments of the present disclosure are not limited to this.

[0066] It should be noted that the electronic device can obtain multiple first real indicator values ​​corresponding to the experimental group and multiple second real indicator values ​​corresponding to the control group based on any feasible implementation method (for example, the electronic device can obtain the first real indicator value and the second real indicator value in the database), and the embodiments of the present disclosure are not limited to this.

[0067] Next, combine Figure 4 , the first true indicator value is described in detail.

[0068] Figure 4 A schematic diagram of a first real indicator value provided by an embodiment of the present disclosure. Figure 4 , including a database. The service indicators included in the database may be a startup duration and a playback frame rate, wherein the service indicator value of the startup duration is 1200 milliseconds, 1300 milliseconds, and 1500 milliseconds, and the service indicator value of the playback frame rate is 30 frames, 35 frames, and 38 frames. Electronic device ( Figure 4(Not shown) After conducting experiments based on the experimental parameters, the business indicator values ​​of the startup duration in the database are 1100 milliseconds, 1200 milliseconds and 1400 milliseconds (multiple first real indicator values ​​corresponding to the startup duration), and the business indicator values ​​of the playback frame rate are 30 frames, 35 frames and 38 frames (multiple first real indicator values ​​corresponding to the playback frame rate).

[0069] S205. Determine test results of the experimental parameters according to the multiple first true index values, the multiple second true index values, and the quantile index value.

[0070] The test result is a test pass or a test fail. For example, if the test result is a test pass, it means that the experimental parameters can optimize the business indicators, and if the test result is a test fail, it means that the experimental parameters do not optimize the business indicators.

[0071] The electronic device can determine the test result of the experimental parameter based on the following feasible implementation: calculate the first permeability index associated with the experimental group, calculate the second permeability index associated with the control group, and determine the test result of the experimental parameter according to the first permeability index and the second permeability index. In this way, the electronic device can determine the result of the experimental test based on the relationship between the first permeability index and the second permeability index, thereby improving the accuracy of determining the experimental result.

[0072] The first permeability index is used to indicate the proportion of the first real index value that satisfies the preset size relationship with the quantile index value. For example, the preset size relationship may include: the first real index value is greater than or equal to the quantile index value, and the first real index value is less than the quantile index value. For example, if the first real index value is 1000, of which 500 first real index values ​​satisfy the preset size relationship with the quantile index value, the first permeability index may be 50%.

[0073] The second permeability index is used to indicate the proportion of the second real index value that satisfies the preset size relationship with the quantile index value. For example, the preset size relationship may include: the second real index value is greater than or equal to the quantile index value, and the second real index value is less than the quantile index value. For example, if the second real index value is 1000, of which 400 second real index values ​​satisfy the preset size relationship with the quantile index value, the second permeability index may be 40%.

[0074] Optionally, the electronic device may calculate the first permeability index associated with the experimental group based on the following feasible implementation method: obtaining the number of first real index values ​​in each database, and adding the numbers of first real index values ​​in multiple databases to obtain a first number, determining the number of first target index values ​​in each database, and adding the numbers of first target index values ​​in multiple databases to obtain a second number, and determining the ratio of the second number to the first number as the first permeability index

[0075] The first target index value and the quantile index value satisfy a preset size relationship. For example, the preset size relationship is that the first real index value is greater than or equal to the quantile index value, if the first real index value 1 is greater than the quantile index value, the electronic device can determine the first real index value 1 as the first target index value, and if the first real index value 2 is greater than the quantile index value, the electronic device can determine the first real index value 2 as the first target index value.

[0076] Optionally, the first number may be the sum of the first real indicator values ​​in multiple databases. For example, if database A includes 100 first real indicator values, database B includes 300 first real indicator values, and database C includes 600 first real indicator values, the electronic device may determine that the first number is 1000.

[0077] Optionally, the second number may be the sum of the numbers of the first target indicator values ​​in multiple databases. For example, if database A includes 40 first real indicator values ​​(first target indicator values ​​in database A) that satisfy a preset size relationship with the quantile indicator value, and database B includes 60 first real indicator values ​​(first target indicator values ​​in database B) that satisfy a preset size relationship with the quantile indicator value, the electronic device may determine that the second number is 100.

[0078] It should be noted that the electronic device can determine the number of first real indicator values ​​and the number of first target indicator values ​​in each database based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0079] Optionally, after the terminal device determines the first quantity and the second quantity, the ratio of the second quantity to the first quantity can be determined as the first permeability index. For example, if the sum of the first real index values ​​in multiple databases is 1000 and the sum of the first target index values ​​in multiple databases is 500, then the first permeability index is 50%.

[0080] Optionally, the electronic device may determine the first permeability index based on the following formula:

[0081]

[0082] Among them, B can be the first permeability index; COUNT(X) is the sum of the number of first real index values ​​in multiple databases (second number); A is the quantile index value; X is the first real index value. This formula can express the proportion of the first real index value greater than or equal to the quantile index value in the multiple first real index values. In this way, based on the method of the first permeability index, the changes in the business indicators can be accurately reflected, thereby improving the accuracy of the experimental test.

[0083] Optionally, the electronic device may calculate the second permeability index associated with the control group based on the following feasible implementation method: obtaining the number of second real index values ​​in each database, and adding the numbers of second real index values ​​in multiple databases to obtain a third number, determining the number of second target index values ​​in each database, and adding the numbers of second target index values ​​in multiple databases to obtain a fourth number, and determining the ratio of the fourth number to the third number as the second permeability index

[0084] The second target indicator value and the quantile indicator value satisfy a preset size relationship. For example, the preset size relationship is that the second real indicator value is greater than or equal to the quantile indicator value, if the second real indicator value 1 is greater than the quantile indicator value, the electronic device can determine the second real indicator value 1 as the second target indicator value, and if the second real indicator value 2 is greater than the quantile indicator value, the electronic device can determine the second real indicator value 2 as the second target indicator value.

[0085] Optionally, the third number may be the sum of the second real indicator values ​​in multiple databases. For example, if database A includes 100 second real indicator values, database B includes 200 second real indicator values, and database C includes 200 second real indicator values, the electronic device may determine that the third number is 500.

[0086] Optionally, the fourth number may be the sum of the number of second target indicator values ​​in multiple databases. For example, if database A includes 10 second real indicator values ​​(second target indicator values ​​in database A) that satisfy a preset size relationship with the quantile indicator value, and database B includes 40 second real indicator values ​​(second target indicator values ​​in database B) that satisfy a preset size relationship with the quantile indicator value, the electronic device may determine that the fourth number is 50.

[0087] It should be noted that the electronic device can determine the number of second real indicator values ​​and the number of second target indicator values ​​in each database based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0088] Optionally, after the terminal device determines the third number and the fourth number, the ratio of the fourth number to the third number can be determined as the second permeability index. For example, if the sum of the second real index values ​​in multiple databases is 1000 and the sum of the second target index values ​​in multiple databases is 400, then the first permeability index is 40%.

[0089] Optionally, the electronic device may determine the first permeability index based on the following formula:

[0090]

[0091] Wherein, C can be the second permeability index; COUNT(Y) is the sum of the number of second real index values ​​in multiple databases (fourth number); A is the quantile index value; Y is the second real index value. This formula can represent the proportion of the second real index value greater than or equal to the quantile index value in the multiple second real index values. In this way, the electronic device can combine the first permeability index and the second permeability index to determine the impact of the experimental group on the business index, and then accurately evaluate the experimental results.

[0092] Optionally, the electronic device determines the test result of the experimental parameter based on the first permeability indicator and the second permeability indicator. Specifically, it can be: the first permeability indicator and the second permeability indicator are processed based on a hypothesis test to obtain a P value associated with the hypothesis test, and when the P value indicates that there is a significant difference between the first permeability indicator and the second permeability indicator, the test result of the experimental parameter is determined to be a passed test.

[0093] The P value is used to indicate whether there is a significant difference between the first permeability index and the second permeability index. For example, if the P value is greater than or equal to 0.05, the terminal device can determine that there is no significant difference between the first permeability index and the second permeability index; if the P value is less than 0.05, the terminal device can determine that there is a significant difference between the first permeability index and the second permeability index.

[0094] For example, if there is no significant difference between the first permeability index and the second permeability index, it means that the electronic device has not optimized the business index in the experiment on the target object, and therefore, the electronic device can determine that the test result is a test failure.

[0095] For example, if there is a significant difference between the first permeability index and the second permeability index, it means that the electronic device optimizes the business index of the experiment on the target object, and therefore, the electronic device can determine that the test result is a pass.

[0096] Next, combine Figure 5 , describing the process of determining the test results.

[0097] Figure 5 A schematic diagram of determining a test result provided by an embodiment of the present disclosure. Figure 5 In the embodiment shown, the service indicator is the startup time, see Figure 5 , the 1000 first real index values ​​corresponding to the startup duration are 1000 milliseconds, 1200 milliseconds, ..., 1700 milliseconds. Among them, the quantile index value is 1500 milliseconds, the second penetration index is 50%, the electronic device determines that the number of startup durations greater than 1500 milliseconds is 400, and the first penetration index is 40%. Therefore, the electronic device can determine that the experiment optimizes the startup duration (with significant differences), and then determine that the test result is a pass.

[0098] Optionally, after the electronic device determines the test result of the experimental parameter, if the test result is that the test fails, a prompt message is generated and sent to a preset device, or the prompt message is displayed.

[0099] Optionally, the prompt information is used to indicate that there is a risk of the experimental parameters going online. For example, in the process of actual application, after the electronic device determines that the test result of the experimental parameter is that the test failed, the electronic device can generate a prompt information indicating that the experiment degrades the business indicator, thereby prompting the user that there is a risk of degrading the business indicator after the experiment goes online. For example, after the electronic device generates the prompt information, it can send the prompt information to a preset device, and the preset device can be any pre-set device, which is not limited to this in the embodiments of the present disclosure. For example, after the electronic device generates the prompt information, it can also display the prompt information, thereby reminding the user that there is a risk of degrading the business indicator of the experiment, thereby improving the reliability of the experiment.

[0100] Optionally, after the electronic device determines that the test result is a test failure, the electronic device can calculate the quantile of the business indicator value of the business indicator in each database and generate associated prompt information, prompting that the business indicator will be degraded by the experimental parameters. For example, if the electronic device determines that the experimental parameters will increase the value of the startup time, the electronic device can calculate the 50th quantile of the startup time in database A and generate prompt information, and the electronic device can calculate the 50th quantile of the startup time in database B and generate prompt information.

[0101] The disclosed embodiment provides a method for testing experimental parameters, wherein an electronic device can determine a business indicator of a target object, and obtain multiple business indicator values ​​of the business indicator from multiple databases, determine a quantile indicator value corresponding to the business indicator based on the multiple business indicator values, obtain multiple first real indicator values ​​corresponding to an experimental group of business indicators from multiple databases and multiple second real indicator values ​​corresponding to a control group, calculate a first permeability indicator associated with the experimental group, calculate a second permeability indicator associated with the control group, and if the first permeability indicator is greater than or equal to the second permeability indicator, determine that the test result is that the test has not passed, and if the first permeability indicator is less than the second permeability indicator, determine that the test result is that the test has passed. In this way, since the permeability indicator can indicate the proportion of real indicator values ​​that meet a preset size relationship with the quantile indicator value, the electronic device can accurately determine the change of the business indicators of the experimental group in multiple databases based on the size relationship between the first permeability indicator value and the second permeability indicator value, thereby improving the accuracy of the experimental test.

[0102] exist Figure 2 Based on the embodiment shown below, combined with Figure 6 , in the testing method of the above experimental parameters, a method for determining the quantile indicator value corresponding to the business indicator according to multiple business indicator values ​​is described.

[0103] Figure 6 A schematic diagram of a method for determining a quantile index value provided by an embodiment of the present disclosure. Figure 6 The method flow includes:

[0104] S601. Determine an initial indicator value.

[0105] Optionally, the electronic device may determine the initial indicator value based on the maximum business indicator value and the minimum business indicator value. For example, the electronic device may determine the average of the maximum business indicator value and the minimum business indicator value of the business indicator as the initial indicator value. For example, if the business indicator values ​​of the startup duration in database A are 1000 milliseconds and 1500 milliseconds, and the business indicator values ​​of the startup duration in database B are 2000 milliseconds and 1300 milliseconds, then the electronic device may determine that the maximum business indicator value of the startup duration is 2000 milliseconds and the minimum business indicator value is 1000 milliseconds, and the electronic device may determine that the initial indicator value is 1500 milliseconds (the average of 2000 milliseconds and 1000 milliseconds).

[0106] Optionally, the electronic device may determine the initial indicator value based on any other feasible implementation manner and based on multiple business indicator values, which is not limited in the embodiments of the present disclosure.

[0107] S602: Determine a plurality of target business indicator values ​​from a plurality of business indicator values ​​according to the initial indicator value.

[0108] The target service indicator value may satisfy a preset size relationship with the initial indicator value. For example, the preset size relationship may be that the target service indicator value is greater than or equal to the initial indicator value, or the preset size relationship may be that the target service indicator value is less than the initial indicator value, which is not limited in the embodiments of the present disclosure.

[0109] For example, the preset size relationship is that the target business indicator value is greater than or equal to the initial indicator value. If business indicator value A, business indicator value B, and business indicator value C are greater than the initial indicator value, the electronic device can determine business indicator value A, business indicator value B, and business indicator value C as the target business indicator value. For example, the preset size relationship is that the target business indicator value is less than the initial indicator value, the business indicator is the startup duration, and the initial indicator value is 1000 milliseconds. If business indicator value A is 1200 milliseconds, business indicator value B is 1300 milliseconds, and business indicator value C is 600 milliseconds, the electronic device can determine business indicator value C as the target business indicator value.

[0110] It should be noted that the electronic device may also determine the target service indicator value based on any other feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0111] S603: Determine a fifth number of the plurality of target business indicator values ​​and a sixth number of the plurality of business indicator values.

[0112] The fifth number may be the number of target business indicator values. For example, if the number of business indicator values ​​of the business indicator is 1000, if 500 business indicator values ​​and the initial indicator value satisfy a preset size relationship, the electronic device may determine that the fifth number of the target business indicator values ​​is 500, and if 800 business indicator values ​​and the initial indicator value satisfy a preset size relationship, the electronic device may determine that the fifth number of the target business indicator values ​​is 800.

[0113] It should be noted that the electronic device can determine the fifth number of the target business indicator value based on any feasible implementation method, and the embodiment of the present disclosure is not limited to this.

[0114] Optionally, the sixth number may be the number of multiple business indicator values. For example, in actual application, if the number of business indicator values ​​of business indicators determined by the electronic device in multiple databases is 1000, the electronic device may determine that the sixth number is 1000. For example, if there are 100 business indicator values ​​for the startup duration in database A, 900 business indicator values ​​for the startup duration in database B, and 500 business indicator values ​​for the startup duration in database C, the electronic device may determine that the sixth number of business indicator values ​​for the startup duration is 1500.

[0115] It should be noted that the electronic device can determine the sixth number of the service indicator value based on any feasible implementation method, and the embodiment of the present disclosure is not limited to this.

[0116] S604: Determine a quantile index value according to the fifth quantity, the sixth quantity, the preset quantile and the initial index value.

[0117] Optionally, the preset quantile may be a value set based on the quantile index value. For example, if the quantile index value is the 10th quantile index value, the electronic device may determine that the preset quantile is 10%, and if the quantile index value is the 50th quantile index value, the electronic device may determine that the preset quantile is 50%.

[0118] Among them, the electronic device can determine the fractional index value based on the following feasible implementation method: determine the ratio between the fifth number and the sixth number, if the ratio is equal to the preset quantile, determine the initial index value as the quantile index value, if the ratio is not equal to the preset quantile, update the initial index value, until the ratio determined according to the updated initial index value is equal to the preset quantile, and determine the updated initial index value as the quantile index value.

[0119] The ratio between the fifth number and the sixth number may be a ratio between the number of target business indicator values ​​and the number of business indicator values. For example, the ratio may be a proportion of the target business indicator value in the business indicator value. For example, if the fifth number of the target business indicator value is 500 and the sixth number of the business indicator value is 1000, the electronic device may determine that the proportion of the target business indicator value in the business indicator value is 50%.

[0120] Optionally, if the ratio between the fifth number and the sixth number is equal to a preset quantile, the electronic device can determine the initial indicator value as the quantile indicator value. For example, if the quantile indicator value is the 50th quantile indicator value, the preset quantile value can be 50%, and if the ratio is equal to 50%, the electronic device can determine the initial indicator value as the quantile indicator value. For example, the quantile indicator value is the 50th quantile indicator value, the business indicator is the startup duration, and the initial indicator value determined by the electronic device is 1500 milliseconds. If the number of target business indicator values ​​greater than or equal to 1500 milliseconds is 500, and the number of business indicator values ​​is 1000, the electronic device can determine the second ratio to be 50% (equal to the preset quantile), and then determine the quantile indicator value to be 1500.

[0121] It should be noted that in the embodiments of the present disclosure, if the ratio is within a preset range of a preset quantile, the electronic device may also determine the initial indicator value as the quantile indicator value. For example, if the preset quantile is 50%, and the ratio between the fifth quantity and the sixth quantity determined by the electronic device based on the initial indicator value is 49.5%, the electronic device may also determine the initial indicator value as the quantile indicator value.

[0122] Optionally, if the ratio between the fifth quantity and the sixth quantity is not equal to the preset quantile, the electronic device may update the initial indicator value until the ratio determined according to the updated initial indicator value is equal to the preset quantile, and the updated initial indicator value is determined as the quantile indicator value. For example, if the ratio between the fifth quantity and the sixth quantity is not equal to the preset quantile, the electronic device needs to update the initial indicator value, and after the update, the ratio between the fifth quantity and the sixth quantity may be recalculated based on the updated initial indicator value, and if the ratio is equal to the preset quantile, the electronic device may determine the updated initial indicator value as the quantile indicator value. For example, the quantile index value is the 50th quantile index value, the business index is the startup duration, and the initial index value determined by the electronic device is 1500 milliseconds. If the number of target business index values ​​greater than or equal to 1500 milliseconds is 300, and the number of business index values ​​is 1000, then the electronic device can determine that the ratio between the fifth number and the sixth number is 30% (not equal to the preset quantile), and the electronic device can update 1500 milliseconds to 1000 milliseconds. If the number of target business index values ​​greater than or equal to 1000 milliseconds is 500, then the electronic device can determine that the ratio between the fifth number and the sixth number is 50% (equal to the preset quantile), and then determine 1000 milliseconds as the quantile index value. In this way, the accuracy of the determined quantile index value can be improved, and then the test accuracy of the experimental parameters can be improved.

[0123] Optionally, the electronic device may update the initial indicator value based on the following feasible implementation method: if the ratio between the fifth number and the sixth number is less than a preset percentile, the electronic device may lower the initial indicator value; if the ratio between the fifth number and the sixth number is greater than a preset percentile, the electronic device may increase the initial indicator value.

[0124] Optionally, if the ratio between the fifth number and the sixth number is less than the preset quantile, it means that the initial indicator value is high, and the number of target business indicator values ​​greater than or equal to the initial indicator value is small. Therefore, the electronic device can reduce the initial indicator value, so that more business indicator values ​​can meet the preset size relationship and improve the accuracy of the quantile indicator value. For example, the business indicator is the startup duration, and the preset quantile is 50%. If the initial indicator value is 1500 milliseconds, the number of startup durations greater than or equal to 1500 milliseconds is 300, and the total number of startup durations is 1000, then the electronic device can determine that the ratio between the fifth number and the sixth number is 30%, and then determine that the initial indicator value is too large. The electronic device can update the initial indicator value to 1000 milliseconds. If the number of startup durations greater than or equal to 1000 milliseconds is 500, then the electronic device can determine that the ratio is equal to the preset quantile, and then determine 1000 milliseconds as the quantile indicator value.

[0125] Optionally, if the ratio between the fifth number and the sixth number is greater than a preset percentile, it means that the initial indicator value is low, and the number of target business indicator values ​​greater than or equal to the initial indicator value is large. Therefore, the electronic device can increase the initial indicator value, so that fewer business indicator values ​​can meet the preset size relationship, thereby improving the accuracy of the percentile indicator value. For example, the business indicator is the startup duration, and the preset percentile is 50%. If the initial indicator value is 1500 milliseconds, the number of startup durations greater than or equal to 1500 milliseconds is 700, and the total number of startup durations is 1000, then the electronic device can determine that the ratio is 70%, and then determine that the initial indicator value is too small. The electronic device can update the initial indicator value to 2000 milliseconds. If the number of startup durations greater than or equal to 2000 milliseconds is 500, then the electronic device can determine that the ratio is equal to the preset percentile, and then determine 2000 milliseconds as the percentile indicator value.

[0126] It should be noted that the method for an electronic device to increase or decrease the initial indicator value is related to the preset size relationship, and is also related to the optimization of the business indicator and the change in the business indicator value. For example, if the preset size relationship is that the target business indicator value is less than the initial business indicator value, then the ratio between the fifth number and the sixth number is less than the preset quantile, then the initial indicator value is increased, and the ratio between the fifth number and the sixth number is greater than the preset quantile, then the initial indicator value is reduced. For example, if the preset size relationship is that the target business indicator value is greater than or equal to the initial indicator value, and the business indicator is the playback frame rate (the higher the playback frame rate, the better the optimization), then the ratio between the fifth number and the sixth number is less than the preset quantile, then the initial indicator value is increased, and the ratio between the fifth number and the sixth number is greater than the preset quantile, then the initial indicator value is reduced.

[0127] It should be noted that the electronic device can determine the update size of each update of the initial indicator value based on any feasible implementation method (such as based on dichotomy, etc.), and the embodiments of the present disclosure are not limited to this.

[0128] The disclosed embodiment provides a method for determining a quantile index value, determining an initial index value, determining multiple target business index values ​​from multiple business index values ​​according to the initial index value, determining a fifth number of multiple target business index values, and a sixth number of multiple business index values, determining a ratio between the fifth number and the sixth number, and if the ratio is equal to a preset quantile, determining the initial index value as the quantile index value, and if the ratio is not equal to the preset quantile, updating the initial index value until the ratio determined according to the updated initial index value is equal to the preset quantile, and determining the updated initial index value as the quantile index value. In this way, the electronic device can determine the quantile index values ​​associated with multiple business index values ​​based on the dichotomy method, improve the accuracy of determining the quantile index value, and thereby improve the accuracy of the experimental test.

[0129] Based on any of the above embodiments, Figure 7 , the process of the testing method of the above experimental parameters is explained.

[0130] Figure 7 A process diagram of a method for testing experimental parameters provided by an embodiment of the present disclosure. Figure 7 , including a control group and an experimental group without the experiment. The database in the control group includes the startup time, and the business indicator values ​​of the startup time include 1000 milliseconds, 1100 milliseconds, ..., 1500 milliseconds (1000), and the database in the experimental group includes the startup time, and the business indicator values ​​of the startup time include 1000 milliseconds, 1100 milliseconds, ..., 1500 milliseconds (before the experiment in the experimental group, the business indicator value of the startup time is the same as that of the control group).

[0131] See also Figure 7 The electronic device can determine that the quantile index value of the startup time in the experimental group and the control group is 1200 milliseconds, and the second penetration index of the control group is 50%. The electronic device can conduct experiments on the experimental group based on the experimental parameters, and then update the business index value of the startup time in the experimental group.

[0132] See also Figure 7, the business indicator values ​​of the startup time in the updated experimental group include 800 milliseconds, 900 milliseconds, ..., 1300 milliseconds. The electronic device can determine that the number of startup times greater than or equal to 1200 milliseconds is 400, and the first permeability indicator of the electronic device experimental group is 40%. Since the first permeability indicator of the experimental group is less than the second permeability indicator of the control group, it means that after the experiment, the business indicator value of the startup time has decreased as a whole. Therefore, the electronic device can determine that the test result of the experiment is a pass (that is, the experiment optimizes the startup time). In this way, since the permeability indicator range is 0-1 (this value is a ratio), it is less affected by mutation data and also includes distribution information. Therefore, the electronic device can accurately determine the test results of the experiment and improve the test accuracy.

[0133] Figure 8 A schematic diagram of a test device for experimental parameters provided in an embodiment of the present disclosure. Figure 8 The experimental parameter testing device 800 includes a first determination module 801, a first acquisition module 802, a second determination module 803, a second acquisition module 804 and a third determination module 805, wherein:

[0134] The first determination module 801 is used to determine the business indicator of the target object;

[0135] The first acquisition module 802 is used to acquire multiple business indicator values ​​of the business indicator from multiple databases;

[0136] The second determination module 803 is used to determine the quantile indicator value corresponding to the business indicator according to the multiple business indicator values;

[0137] The second acquisition module 804 is used to acquire a plurality of first real indicator values ​​corresponding to the experimental group and a plurality of second real indicator values ​​corresponding to the control group of the business indicators of the plurality of databases;

[0138] The third determination module 805 is used to determine the test result of the experimental parameter according to the multiple first true index values, the multiple second true index values ​​and the quantile index value, and the test result is test pass or test fail.

[0139] According to one or more embodiments of the present disclosure, the third determination module 805 is specifically configured to:

[0140] Calculating a first permeability index associated with the experimental group, the first permeability index being used to indicate a ratio of a first real index value that satisfies a preset magnitude relationship with the quantile index value;

[0141] Calculating a second permeability index associated with the control group; the second permeability index is used to indicate the proportion of a second real index value that satisfies a preset magnitude relationship with the quantile index value;

[0142] The test results of the experimental parameters are determined according to the first permeability index and the second permeability index.

[0143] According to one or more embodiments of the present disclosure, the third determination module 805 is specifically configured to:

[0144] Obtaining the number of first true indicator values ​​in each database, and adding the numbers of first true indicator values ​​in multiple databases to obtain a first number;

[0145] In each database, determine the number of first target indicator values, and add the numbers of first target indicator values ​​in multiple databases to obtain a second number, wherein the first target indicator value and the quantile indicator value satisfy a preset size relationship;

[0146] The ratio of the second quantity to the first quantity is determined as the first permeability index.

[0147] According to one or more embodiments of the present disclosure, the third determination module 805 is specifically configured to:

[0148] Obtaining the number of second true indicator values ​​in each database, and adding the numbers of second true indicator values ​​in multiple databases to obtain a third number;

[0149] In each database, determining the number of second target indicator values, and adding the numbers of second target indicator values ​​in multiple databases to obtain a fourth number, wherein the second target indicator value and the quantile indicator value satisfy a preset size relationship;

[0150] The ratio of the fourth quantity to the third quantity is determined as the second permeability index.

[0151] According to one or more embodiments of the present disclosure, the third determination module 805 is specifically configured to:

[0152] Processing the first permeability indicator and the second permeability indicator based on a hypothesis test to obtain a P value associated with the hypothesis test, wherein the P value is used to indicate whether there is a significant difference between the first permeability indicator and the second permeability indicator;

[0153] When the P value indicates that there is a significant difference between the first permeability index and the second permeability index, the test result of the experimental parameter is determined to be a test pass.

[0154] According to one or more embodiments of the present disclosure, the second determining module 803 is specifically configured to:

[0155] Determine the initial indicator value;

[0156] According to the initial indicator value, determining a plurality of target business indicator values ​​from the plurality of business indicator values, wherein the target business indicator values ​​and the initial indicator values ​​satisfy a preset size relationship;

[0157] determining a fifth number of the plurality of target business indicator values ​​and a sixth number of the plurality of business indicator values;

[0158] The quantile index value is determined according to the fifth quantity, the sixth quantity, a preset quantile and the initial index value.

[0159] According to one or more embodiments of the present disclosure, the second determining module 803 is specifically configured to:

[0160] determining a ratio between the fifth amount and the sixth amount;

[0161] If the ratio is equal to the preset quantile, determining the initial index value as the quantile index value;

[0162] If the ratio is not equal to the preset quantile, the initial index value is updated until the ratio determined according to the updated initial index value is equal to the preset quantile, and the updated initial index value is determined as the quantile index value.

[0163] According to one or more embodiments of the present disclosure, the third determining module 805 is further configured to:

[0164] If the test result is that the test fails, a prompt message is generated, where the prompt message is used to indicate that there is a risk of the experimental parameter being put online;

[0165] The prompt information is sent to a preset device, or the prompt information is displayed.

[0166] The testing device for experimental parameters provided in the embodiment of the present disclosure can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0167] Fig. 9 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig. 9, which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure. The electronic device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0168] like Fig. 9 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0169] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0170] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0171] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0172] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0173] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0174] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, various methods that may be involved in the above embodiments are implemented.

[0175] An embodiment of the present disclosure provides a computer program product, including a computer program, which implements various methods that may be involved in the above embodiments when executed by a processor.

[0176] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0177] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0179] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0180] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0181] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0182] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0183] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0184] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server or storage medium that performs the operation of the technical solution of the present disclosure based on the prompt message. As an optional but non-limiting implementation method, in response to receiving an active request from a user, the method of sending a prompt message to the user can be, for example, a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0185] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0186] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the relevant laws and regulations. The data may include information, parameters and messages, such as flow switching indication information.

[0187] In a first aspect, according to one or more embodiments of the present disclosure, the present disclosure includes a method for testing an experimental parameter, comprising:

[0188] Determine a business indicator of a target object, and obtain multiple business indicator values ​​of the business indicator from multiple databases;

[0189] Determine, according to the multiple business indicator values, a quantile indicator value corresponding to the business indicator;

[0190] Acquire a plurality of first real indicator values ​​corresponding to an experimental group and a plurality of second real indicator values ​​corresponding to a control group of business indicators of the plurality of databases;

[0191] A test result of the experimental parameter is determined according to the multiple first true index values, the multiple second true index values ​​and the quantile index value, and the test result is test pass or test fail.

[0192] According to one or more embodiments of the present disclosure, determining the test result of the experimental parameter according to the multiple first true index values, the multiple second true index values ​​and the quantile index value includes:

[0193] Calculating a first permeability index associated with the experimental group, the first permeability index being used to indicate a ratio of a first real index value that satisfies a preset magnitude relationship with the quantile index value;

[0194] Calculating a second permeability index associated with the control group; the second permeability index is used to indicate the proportion of a second real index value that satisfies a preset magnitude relationship with the quantile index value;

[0195] The test results of the experimental parameters are determined according to the first permeability index and the second permeability index.

[0196] According to one or more embodiments of the present disclosure, the calculating the first permeability index associated with the experimental group includes:

[0197] Obtaining the number of first true indicator values ​​in each database, and adding the numbers of first true indicator values ​​in multiple databases to obtain a first number;

[0198] In each database, determine the number of first target indicator values, and add the numbers of first target indicator values ​​in multiple databases to obtain a second number, wherein the first target indicator value and the quantile indicator value satisfy a preset size relationship;

[0199] The ratio of the second quantity to the first quantity is determined as the first permeability index.

[0200] According to one or more embodiments of the present disclosure, the calculating the second permeability index associated with the control group includes:

[0201] Obtaining the number of second true indicator values ​​in each database, and adding the numbers of second true indicator values ​​in multiple databases to obtain a third number;

[0202] In each database, determining the number of second target indicator values, and adding the numbers of second target indicator values ​​in multiple databases to obtain a fourth number, wherein the second target indicator value and the quantile indicator value satisfy a preset size relationship;

[0203] The ratio of the fourth quantity to the third quantity is determined as the second permeability index.

[0204] According to one or more embodiments of the present disclosure, determining the test result of the experimental parameter according to the first permeability index and the second permeability index includes:

[0205] Processing the first permeability indicator and the second permeability indicator based on a hypothesis test to obtain a P value associated with the hypothesis test, wherein the P value is used to indicate whether there is a significant difference between the first permeability indicator and the second permeability indicator;

[0206] When the P value indicates that there is a significant difference between the first permeability index and the second permeability index, the test result of the experimental parameter is determined to be a test pass.

[0207] According to one or more embodiments of the present disclosure, determining, according to the multiple business indicator values, a quantile indicator value corresponding to the business indicator includes:

[0208] Determine the initial indicator value;

[0209] According to the initial indicator value, determining a plurality of target business indicator values ​​from the plurality of business indicator values, wherein the target business indicator values ​​and the initial indicator values ​​satisfy a preset size relationship;

[0210] determining a fifth number of the plurality of target business indicator values ​​and a sixth number of the plurality of business indicator values;

[0211] The quantile index value is determined according to the fifth quantity, the sixth quantity, a preset quantile and the initial index value.

[0212] According to one or more embodiments of the present disclosure, determining the quantile index value according to the fifth number, the sixth number, the preset quantile and the initial index value includes:

[0213] determining a ratio between the fifth amount and the sixth amount;

[0214] If the ratio is equal to the preset quantile, determining the initial index value as the quantile index value;

[0215] If the ratio is not equal to the preset quantile, the initial index value is updated until the ratio determined according to the updated initial index value is equal to the preset quantile, and the updated initial index value is determined as the quantile index value.

[0216] According to one or more embodiments of the present disclosure, after determining the test results of the experimental parameters, the method further includes:

[0217] If the test result is that the test fails, a prompt message is generated, where the prompt message is used to indicate that there is a risk of the experimental parameter being put online;

[0218] The prompt information is sent to a preset device, or the prompt information is displayed.

[0219] In a second aspect, according to one or more embodiments of the present disclosure, the present disclosure includes a testing device for experimental parameters, the testing device for experimental parameters includes a first determination module, a first acquisition module, a second determination module, a second acquisition module and a third determination module, wherein:

[0220] The first determination module is used to determine the business indicator of the target object;

[0221] The first acquisition module is used to acquire multiple business indicator values ​​of the business indicator from multiple databases;

[0222] The second determination module is used to determine the quantile indicator value corresponding to the business indicator according to the multiple business indicator values;

[0223] The second acquisition module is used to acquire a plurality of first real indicator values ​​corresponding to the experimental group and a plurality of second real indicator values ​​corresponding to the control group of the business indicators of the plurality of databases;

[0224] The third determination module is used to determine a test result of the experimental parameter according to the multiple first true indicator values, the multiple second true indicator values ​​and the quantile indicator value, and the test result is a test pass or a test fail.

[0225] According to one or more embodiments of the present disclosure, the third determination module is specifically configured to:

[0226] Calculating a first permeability index associated with the experimental group, the first permeability index being used to indicate a ratio of a first real index value that satisfies a preset magnitude relationship with the quantile index value;

[0227] Calculating a second permeability index associated with the control group; the second permeability index is used to indicate the proportion of a second real index value that satisfies a preset magnitude relationship with the quantile index value;

[0228] The test results of the experimental parameters are determined according to the first permeability index and the second permeability index.

[0229] According to one or more embodiments of the present disclosure, the third determination module is specifically configured to:

[0230] Obtaining the number of first true indicator values ​​in each database, and adding the numbers of first true indicator values ​​in multiple databases to obtain a first number;

[0231] In each database, determine the number of first target indicator values, and add the numbers of first target indicator values ​​in multiple databases to obtain a second number, wherein the first target indicator value and the quantile indicator value satisfy a preset size relationship;

[0232] The ratio of the second quantity to the first quantity is determined as the first permeability index.

[0233] According to one or more embodiments of the present disclosure, the third determination module is specifically configured to:

[0234] Obtaining the number of second true indicator values ​​in each database, and adding the numbers of second true indicator values ​​in multiple databases to obtain a third number;

[0235] In each database, determining the number of second target indicator values, and adding the numbers of second target indicator values ​​in multiple databases to obtain a fourth number, wherein the second target indicator value and the quantile indicator value satisfy a preset size relationship;

[0236] The ratio of the fourth quantity to the third quantity is determined as the second permeability index.

[0237] According to one or more embodiments of the present disclosure, the third determination module is specifically configured to:

[0238] Processing the first permeability indicator and the second permeability indicator based on a hypothesis test to obtain a P value associated with the hypothesis test, wherein the P value is used to indicate whether there is a significant difference between the first permeability indicator and the second permeability indicator;

[0239] When the P value indicates that there is a significant difference between the first permeability index and the second permeability index, the test result of the experimental parameter is determined to be a test pass.

[0240] According to one or more embodiments of the present disclosure, the second determining module is specifically configured to:

[0241] Determine the initial indicator value;

[0242] According to the initial indicator value, determining a plurality of target business indicator values ​​from the plurality of business indicator values, wherein the target business indicator values ​​and the initial indicator values ​​satisfy a preset size relationship;

[0243] determining a fifth number of the plurality of target business indicator values ​​and a sixth number of the plurality of business indicator values;

[0244] The quantile index value is determined according to the fifth quantity, the sixth quantity, a preset quantile and the initial index value.

[0245] According to one or more embodiments of the present disclosure, the second determining module is specifically configured to:

[0246] determining a ratio between the fifth amount and the sixth amount;

[0247] If the ratio is equal to the preset quantile, determining the initial index value as the quantile index value;

[0248] If the ratio is not equal to the preset quantile, the initial index value is updated until the ratio determined according to the updated initial index value is equal to the preset quantile, and the updated initial index value is determined as the quantile index value.

[0249] According to one or more embodiments of the present disclosure, the third determination module is further configured to:

[0250] If the test result is that the test fails, a prompt message is generated, where the prompt message is used to indicate that there is a risk of the experimental parameter being put online;

[0251] The prompt information is sent to a preset device, or the prompt information is displayed.

[0252] In a third aspect, an embodiment of the present disclosure provides an electronic device including: a processor and a memory;

[0253] The memory stores computer-executable instructions;

[0254] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the first aspect as described above and various methods that may be involved in the first aspect.

[0255] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the first aspect and various possible methods involved in the first aspect are implemented.

[0256] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0257] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0258] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for testing experimental parameters, characterized in that: include: Determine a business indicator of a target object, and obtain multiple business indicator values ​​of the business indicator from multiple databases; Determine, according to the multiple business indicator values, a quantile indicator value corresponding to the business indicator; Acquire a plurality of first real indicator values ​​corresponding to an experimental group and a plurality of second real indicator values ​​corresponding to a control group of business indicators of the plurality of databases; A test result of the experimental parameter is determined according to the multiple first true index values, the multiple second true index values ​​and the quantile index value, and the test result is test pass or test fail.

2. The method according to claim 1, characterized in that The step of determining the test result of the experimental parameter according to the plurality of first true index values, the plurality of second true index values ​​and the quantile index value comprises: Calculating a first permeability index associated with the experimental group, the first permeability index being used to indicate a ratio of a first real index value that satisfies a preset magnitude relationship with the quantile index value; Calculating a second permeability index associated with the control group; the second permeability index is used to indicate the proportion of a second real index value that satisfies a preset magnitude relationship with the quantile index value; The test results of the experimental parameters are determined according to the first permeability index and the second permeability index.

3. The method according to claim 2, characterized in that The calculating the first permeability index associated with the experimental group includes: Obtaining the number of first true indicator values ​​in each database, and adding the numbers of first true indicator values ​​in multiple databases to obtain a first number; In each database, determine the number of first target indicator values, and add the numbers of first target indicator values ​​in multiple databases to obtain a second number, wherein the first target indicator value and the quantile indicator value satisfy a preset size relationship; The ratio of the second quantity to the first quantity is determined as the first permeability index.

4. The method according to claim 2, characterized in that: The calculating the second permeability index associated with the control group includes: Obtaining the number of second true indicator values ​​in each database, and adding the numbers of second true indicator values ​​in multiple databases to obtain a third number; In each database, determining the number of second target indicator values, and adding the numbers of second target indicator values ​​in multiple databases to obtain a fourth number, wherein the second target indicator value and the quantile indicator value satisfy a preset size relationship; The ratio of the fourth quantity to the third quantity is determined as the second permeability index.

5. The method according to any one of claims 2 to 4, characterized in that: Determining the test result of the experimental parameter according to the first permeability index and the second permeability index includes: Processing the first permeability indicator and the second permeability indicator based on a hypothesis test to obtain a P value associated with the hypothesis test, wherein the P value is used to indicate whether there is a significant difference between the first permeability indicator and the second permeability indicator; When the P value indicates that there is a significant difference between the first permeability index and the second permeability index, the test result of the experimental parameter is determined to be a test pass.

6. The method according to any one of claims 1 to 4, characterized in that: The determining, according to the multiple business indicator values, a quantile indicator value corresponding to the business indicator comprises: Determine the initial indicator value; According to the initial indicator value, determining a plurality of target business indicator values ​​from the plurality of business indicator values, wherein the target business indicator values ​​and the initial indicator values ​​satisfy a preset size relationship; determining a fifth number of the plurality of target business indicator values ​​and a sixth number of the plurality of business indicator values; The quantile index value is determined according to the fifth quantity, the sixth quantity, a preset quantile and the initial index value.

7. The method according to claim 6, characterized in that The determining the quantile index value according to the fifth quantity, the sixth quantity, the preset quantile and the initial index value includes: determining a ratio between the fifth amount and the sixth amount; If the ratio is equal to the preset quantile, determining the initial index value as the quantile index value; If the ratio is not equal to the preset quantile, the initial index value is updated until the ratio determined according to the updated initial index value is equal to the preset quantile, and the updated initial index value is determined as the quantile index value.

8. The method according to any one of claims 1 to 4, characterized in that: After determining the test results of the experimental parameters, the method further includes: If the test result is that the test fails, a prompt message is generated, where the prompt message is used to indicate that there is a risk of the experimental parameter being put online; The prompt information is sent to a preset device, or the prompt information is displayed.

9. A device for testing experimental parameters, characterized in that: It includes a first determination module, a first acquisition module, a second determination module, a second acquisition module and a third determination module, wherein: The first determination module is used to determine the business indicator of the target object; The first acquisition module is used to acquire multiple business indicator values ​​of the business indicator from multiple databases; The second determination module is used to determine the quantile indicator value corresponding to the business indicator according to the multiple business indicator values; The second acquisition module is used to acquire a plurality of first real indicator values ​​corresponding to the experimental group and a plurality of second real indicator values ​​corresponding to the control group of the business indicators of the plurality of databases; The third determination module is used to determine a test result of the experimental parameter according to the multiple first true indicator values, the multiple second true indicator values ​​and the quantile indicator value, and the test result is a test pass or a test fail.

10. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the test method for the experimental parameters according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the method for testing the experimental parameters according to any one of claims 1 to 8 is implemented.

Images