Page test method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a page testing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps of displaying a page corresponding to a first page version based on a page code and a test code of a to-be-tested page, and determining operation data executed in the page; the operation data is sent to an experiment platform, a second page version returned by the experiment platform is received, and the second page version is determined from the multiple first page versions based on the operation data; and deleting the test code, and solidifying the page code based on the second page version. Through the method and the device, the page code corresponding to the test result can be automatically solidified after the test is finished, the development period of the page is shortened, and the development cost is reduced.

Description

Technical Field

[0001] The present application relates to data analysis technology, and more particularly to a page testing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] As the business volume increases, every time the business party innovates a new page, it generally needs to connect to the AB experiment configuration platform for testing. First, the test diversion is carried out, the diversion results are reported to the tracking point, and the user behavior is recorded. Then, the test results are manually counted or calculated on the tracking point platform to find the page version with better effect. The tester makes corresponding code modifications according to the determined page version, and submits the modified code to continuous integration and continuous deployment (CICD) before the new page version can be launched. Summary of the invention

[0003] The embodiments of the present application provide a page testing method, device, electronic device, computer-readable storage medium, and computer program product, which can automatically solidify the page code corresponding to the test result after the test is completed, shorten the page development cycle and reduce the development cost.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a page testing method, the method comprising:

[0006] Based on the page code and the test code of the page to be tested, display a page corresponding to the first page version, and determine the operation data executed in the page;

[0007] Sending the operation data to an experiment platform, and receiving a second page version returned by the experiment platform, wherein the second page version is determined from a plurality of the first page versions based on the operation data;

[0008] The test code is deleted, and the page code is solidified based on the second page version.

[0009] The present application provides a page testing device, including:

[0010] A display module, configured to display a page corresponding to the first page version based on the page code and the test code of the page to be tested, and determine operation data executed in the page;

[0011] A first sending module, used for sending the operation data to the experimental platform;

[0012] A first receiving module, configured to receive a second page version returned by the experiment platform, where the second page version is determined from a plurality of first page versions based on the operation data;

[0013] A deletion module is used to delete the test code and solidify the page code based on the second page version.

[0014] The present application also provides a page testing method, the method comprising:

[0015] Receiving operation data sent by a client and executed in a page corresponding to a first page version, wherein the page corresponding to the first page version is obtained based on a page code of a page to be tested and a test code;

[0016] Based on the operation data, a second page version is determined from multiple first page versions, and a first instruction carrying the second page version is returned to the client, wherein the first instruction is used to instruct the client to delete the test code and solidify the page code based on the second page version.

[0017] The present application also provides a page testing device, including:

[0018] A second receiving module is used to receive operation data sent by a client and executed in a page corresponding to a first page version, wherein the page corresponding to the first page version is obtained based on a page code and a test code of the page to be tested;

[0019] a determination module, configured to determine a second page version from a plurality of the first page versions based on the operation data;

[0020] The second sending module is used to return a first instruction carrying the second page version to the client, wherein the first instruction is used to instruct the client to delete the test code and solidify the page code based on the second page version.

[0021] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0022] A memory for storing computer executable instructions or computer programs;

[0023] The processor is used to implement the page testing method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0024] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing a page testing method provided in an embodiment of the present application when executed by a processor.

[0025] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the page testing method provided by the embodiment of the present application is implemented.

[0026] The embodiments of the present application have the following beneficial effects:

[0027] The embodiment of the present application can display the page corresponding to the first page version in the client based on the page code and test code of the page to be tested, and then the client can send the operation data performed by the user in the page to the experimental platform. The experimental platform determines the second page version from multiple first page versions based on the operation data sent by the client and returns it to the client. Finally, the client can delete the test code according to the second page version determined by the experimental platform, and solidify the page code based on the second page version. In this way, there is no need to modify the page code and republish it. The page code corresponding to the test result can be automatically solidified after the test is completed, shortening the page development cycle and reducing development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the architecture of the page testing system provided in the embodiment of the present application;

[0029] Figure 2A is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application;

[0030] Figure 2B is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application;

[0031] Figure 3 It is a first flow chart of the page testing method provided in an embodiment of the present application;

[0032] Figure 4 is a second flow chart of the page testing method provided in an embodiment of the present application;

[0033] Figure 5 is a third flow chart of the page testing method provided in the embodiment of the present application;

[0034] Figure 6 is a fourth flow chart of the page testing method provided in an embodiment of the present application;

[0035] Figure 7 It is a schematic diagram of the principle of the experiment-to-online function provided in an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of the principle of the curing function provided in the embodiment of the present application.

[0037] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0039] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0041] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0042] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0043] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0044] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0045] 1) AB Experiment (AB Test): It is a method of statistically evaluating and comparing the performance differences between two or more versions of a page or product. By randomly assigning users to different versions of pages or products, and comparing their performance on a certain target indicator, to determine which version is better. This kind of test can help optimize websites, advertisements, email templates, etc., increase user engagement, sales, etc. When conducting AB testing, it is necessary to set clear test goals and diversion strategies, and consider factors such as sample size, statistical significance, and test duration. If the test version shows statistically significant superior performance, it may be adopted as the standard version.

[0046] 2) Feature: refers to the winning strategy determined in ABTest. For example, the embodiment of the present application can determine the second page version (ie, the optimal solution) from multiple first page versions (ie, candidate solutions) included in the page to be tested based on the AB experiment.

[0047] 3) Sample distribution verification algorithm: Verify whether the traffic in ABTest is diverted according to the pre-configured traffic ratio, which can effectively verify whether the traffic of ABTest is configured correctly.

[0048] 4) Statistical efficacy algorithm: Check whether the current traffic in ABTest is sufficient to support the grouping results of ABTest.

[0049] 5) Confidence interval algorithm: When the positive and negative signs of the confidence interval data are the same, it can indicate that the experimental results of ABTest are credible.

[0050] 6) P value algorithm: P value represents the statistical significance of the results. When the P value is less than 0.05, it means that there is a significant difference between the two groups of indicators; otherwise, it is considered that there is no significant difference between the experimental group and the control group.

[0051] 7) Funnel algorithm: focus on the survival indicators of users in different time windows.

[0052] 8) Retention algorithm: Focus on the number of people who stayed on a page some time ago and the number of people who are currently on the page.

[0053] 9) Tracking: refers to inserting code or tools into the application to record the behavior and attributes of certain events, such as recording the user's clicks, browsing, purchases, registrations, and other operations in the application. This data can be used to analyze user behavior, optimize product functions, improve user experience, etc. For example, in ABTest, through tracking, user operation data can be collected and sent to the experimental platform for analysis.

[0054] In the related art, before a new page needs to be launched, different versions of the new page need to be tested through an AB experiment platform. After determining the optimal version of the page, testers usually extract the code corresponding to the optimal version of the page and submit the code to CICD before launching it online. This requires testers to perform secondary development before launching the new page, which lengthens the cycle of developing new pages and increases costs.

[0055] In view of this, the embodiments of the present application provide a page testing method, device, electronic device, computer-readable storage medium and computer program product, which can automatically solidify the page code corresponding to the test result after the test is completed, shorten the page development cycle and reduce development costs.

[0056] See also Figure 1 , Figure 1 Schematic diagram of the architecture of the page test system provided in the embodiment of the present application, in order to realize an application supporting an automatic solidification of the page code corresponding to the test result after the test is completed, such as Figure 1 As shown, in system 10, terminal 100 (on which a client may be running) is connected to a business backend 200 (for example, it may be a backend server of a client running on terminal 100) via network 300. Network 300 may be a wide area network or a local area network, or a combination of the two.

[0057] In some embodiments, when the business backend 200 wants to publish a new page (i.e., the page to be tested), it can access the experimental platform 400 (e.g., the AB experimental platform), and the client running on the terminal 100 can display the page corresponding to the first page version (e.g., one of multiple candidate advertisements can be randomly displayed) based on the page code and test code of the page to be tested, and determine the operation data performed on the page (e.g., the length of stay on the page, etc.), and send the operation data to the experimental platform 400. After receiving the operation data sent by the client participating in the experiment, the experimental platform 400 can determine the second page version (i.e., the optimal page) from multiple first page versions based on the operation data, and notify the business backend 200 to send a first instruction (e.g., a solidification instruction) to the terminal 100, and the first instruction carries the second page version. After receiving the first instruction sent by the business backend 200, the client running on the terminal 100 can delete the test code and solidify the page code based on the second page version.

[0058] In some embodiments, the above-mentioned business backend 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal and the business backend can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of the present application.

[0059] The structure of the electronic device provided in the embodiments of the present application is described below.

[0060] See also Figure 2A , Figure 2A is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, taking the electronic device as a terminal as an example, Figure 2A The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2A Various buses are labeled as bus system 540 .

[0061] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.

[0064] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0066] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0067] A network communication module 552, used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0068] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0069] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0070] In some embodiments, the page testing device provided in the embodiments of the present application can be implemented in a software manner. Figure 2A The page test device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a display module 5551, a first sending module 5552, a first receiving module 5553 and a deletion module 5554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0071] See also Figure 2B , Figure 2B 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application, taking the electronic device 600 as a server as an example, Figure 2B The electronic device 600 shown includes: at least one processor 610, a memory 640 and at least one network interface 620. The various components in the electronic device 600 are coupled together via a bus system 630. The memory 640 includes: an operating system 641 and a network communication module 642. It should be noted that, Figure 2B The function of the structure in Figure 2A The functions of the structures in are similar.

[0072] In some embodiments, the page testing device provided in the embodiments of the present application can be implemented in a software manner. Figure 2B The page test device 643 stored in the memory 640 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a second receiving module 6431, a determination module 6432, and a second sending module 6433. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0073] In other embodiments, the page testing device provided in the embodiments of the present application can be implemented in hardware. As an example, the page testing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the page testing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0074] The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals. The following describes the page testing method provided by the embodiment of the present application in combination with the exemplary application and implementation of the terminal provided by the embodiment of the present application.

[0075] See also Figure 3 , Figure 3 This is a first flow chart of the page testing method provided in the embodiment of the present application, which will be combined with Figure 3 Steps 101 to 103 are shown for explanation.

[0076] In step 101, based on the page code and the test code of the page to be tested, a page corresponding to the first page version is displayed, and operation data executed in the page is determined.

[0077] In some embodiments, the page code of the page to be tested may include a page subcode corresponding to the first page version, see Figure 4 , Figure 4 is a second flow chart of the page testing method provided in an embodiment of the present application, Figure 3 The step 101 of "displaying a page corresponding to the first page version based on the page code of the page to be tested and the test code" can be implemented by following the steps 1011 to 1012, which are described in detail below.

[0078] In some embodiments, the page to be tested includes multiple first page versions, and the page code of the page to be tested includes multiple page sub-codes corresponding to the multiple first page versions one by one.

[0079] In step 1011 , a first page version is selected from a plurality of first page versions based on the test code.

[0080] In some examples, the client can select a first page version from multiple first page versions based on the test code of the page to be tested. For example, the test code carried in the page to be tested can be a random function, and in the process of running the page to be tested, a first page version can be randomly selected from multiple first page versions based on the random function; the test code carried in the page to be tested can also be a judgment function, and a first page version can be selected from multiple first page versions based on the judgment function.

[0081] As an example, during page testing, in order to ensure that the test traffic is evenly distributed to each first page version, the client may randomly select a first page version from multiple first page versions based on the test code carried by the page to be tested.

[0082] As an example, in the process of page testing, in order to ensure that the test traffic is evenly distributed to each first page version, some measures can also be taken to ensure the randomness and fairness of the traffic. For example, any of the following schemes can be adopted: 1. Random number generation: A random number can be generated for each client, and the client should be assigned to which first page version according to the random number. For example, taking two first page versions, first page version A and first page version B as an example, if the random number is less than 0.5, the client is assigned to the first page version A; if the random number is greater than or equal to 0.5, the client is assigned to the first page version B. 2. Timestamp allocation: The timestamp when the client sends a request to the experimental platform plus some random offsets is used for allocation. For example, taking two first page versions, first page version A and first page version B as an example, if the low digit of the timestamp is less than a certain threshold, the client is assigned to the first page version A; otherwise, it is assigned to the first page version B. 3. Hash algorithm: Use a hash algorithm (such as MD5) to convert the client's unique identifier (such as login ID) into a hash value, and then take the modulus of the hash value to determine the client allocation. For example, taking three first page versions, first page version A, first page version B and first page version C as examples, the client's login ID is hashed using the MD5 hash algorithm to obtain a hash value, and the hash value is modulo processed (that is, the hash value is divided by the number of first page versions 3 (3 in this example)), to obtain a remainder between 0 and 2. After that, it can be determined according to the remainder to which first page version it is assigned: a remainder of 0 is assigned to the first page version A, a remainder of 1 is assigned to the first page version B, and a remainder of 2 is assigned to the first page version C. 4. Round-robin allocation: If the client passes the preliminary random or hash allocation, but encounters an allocation failure (for example, a group has reached the traffic limit), a round-robin method can be adopted, that is, the client is assigned to each group in turn according to a predetermined order. It should be noted that the embodiment of the present application does not limit this, and the allocation method can be selected according to the specific situation. The above allocation method can be written in the code to form a corresponding test code, so that the client selects a first page version from multiple first page versions according to the test code.

[0083] In step 1012, based on the page sub-code corresponding to the selected first page version, the page corresponding to the selected first page version is displayed.

[0084] In some examples, the page code of the page to be tested includes multiple page sub-codes corresponding to multiple first page versions. After the client selects a first page version, it displays the page corresponding to the first page version according to the page sub-code corresponding to the first page version.

[0085] As an example, for example, the page to be tested is a display page of a product. After the client selects a first page version, if the user initiates a request in the client and hopes to view the display page of the product, the client will display the page of the selected first page version to the user. First, the built-in rendering engine of the client begins to parse the page subcode corresponding to the first page version (for example, Hyper Text Markup Language (HTML) code) and construct a Document Object Model (DOM) tree. The client will load Cascading Style Sheets (CSS) and JavaScript files, as well as other resources (such as images, videos, etc.) according to the links in the HTML. The rendering engine of the client applies CSS styles to DOM elements to determine the layout and style of the elements. The built-in JavaScript engine executes JavaScript code, operates on the DOM, and adds interactive functions or dynamic content. The rendering engine of the client will perform layout calculations, determine the position and size of the DOM elements, and then draw them on the screen to present the product page.

[0086] Through steps 1011 to 1012, the first page version is selected using the test code, and the page is displayed according to its corresponding page sub-code, so that each first page version can be randomly selected with the same probability of being selected, thereby ensuring the fairness of the test and improving the accuracy of the test results.

[0087] In some examples, the client may determine operation data performed on the page in response to the user's operation on the page.

[0088] As an example, after the client displays the page of the selected first page version, the user can browse the product information, such as viewing product details, pictures, and reviews, and perform operations such as purchase and adding to shopping cart. The client determines corresponding operation data (e.g., browsing time, number of clicks, whether to purchase, purchase quantity, etc.) based on these operations.

[0089] Continue to see Figure 3 , and the description will continue with step 101 above.

[0090] In step 102, the operation data is sent to the experiment platform, and the second page version returned by the experiment platform is received.

[0091] Here, the second page version is determined by the experimental platform from multiple first page versions based on operation data, which is explained in detail below.

[0092] In some examples, clients participating in page testing can send operation data performed on the page corresponding to the selected first page version to the experimental platform. The experimental platform will determine the second page version from multiple first page versions based on the operation data sent by multiple clients respectively, and return the second page version to the client.

[0093] As an example, for example, an online bookstore wants to test the impact of the layout of its website homepage on user purchasing behavior. The business backend can set two different layouts for the website homepage. The client will execute the layout assigned by the experimental platform. The real user will generate some operation data on the client (i.e., the operation data for the first page version executed by the client). The operation data can be click-through rate, page dwell time, purchase conversion rate, etc. The client will record and collect these data through tracking points and send them to the experimental platform. For example, tracking points for user click operations can be set in advance on the client. When the user's operation triggers the preset tracking point event, the tracking point code will collect relevant operation data and send the collected operation data to the experimental platform. After receiving the operation data, the experimental platform will summarize the operation data sent by all participating clients and compare the operation data of the two first page versions to determine which first page version corresponds to the homepage layout that is more attractive for users to stay and buy. Assuming that the homepage layout corresponding to the first page version B is more attractive for users to stay and buy, it can also be evaluated whether the obtained results are significant. This can be evaluated by finding the confidence interval, p-value, and statistical power. The following is a detailed explanation.

[0094] It should be noted that when calculating whether the results are significant, the first page version needs to be divided into an experimental group and a control group. If there are only two first page versions, one first page version is the control group and the other first page version is the experimental group; if there are three or more first page versions, one first page version is randomly selected as the control group and the remaining first page versions are used as the experimental group.

[0095] The following uses two first page versions, first page version A and first page version B, as examples to illustrate that the confidence interval can be calculated through the standard deviation of the control group, the standard deviation of the experimental group, the sample size of the two groups, and the mean of the two groups through mathematical formulas. Under normal experimental conditions, the interval values ​​are all positive or negative, indicating that the direction of change of the effect of the experimental group relative to the effect of the control group is an increase or decrease. The expression is as follows:

[0096]

[0097] in, is the mean of the samples of the first page version A; is the mean of the samples of the first page version B; is the critical value of the Z distribution for a given significance level (usually found from a normal distribution table), for a 95% confidence level, It is equal to 1.96, and the Z value is used to convert the sample standard error to the confidence interval of the standard normal distribution; represents the square of the standard deviation of the sample of the first page version A; is the square of the standard deviation of the sample of the first page version B; N A is the sample size of the first page version A, N B is the sample size of the sample of version B of the first page; It is the square root of the weighted average of the two sample standard deviations. It is used to estimate the standard error of the difference between the two sample means and can reflect the impact of the standard deviation of the two samples on the mean difference.

[0098] The P value can be calculated using a mathematical formula using the standard deviation of the control group, the standard deviation of the experimental group, the sample size of the two groups, and the mean of the two groups. A p value < 0.05 indicates that the experiment is significant, and a p value > 0.05 indicates that the experimental result is not significant. The expression is as follows:

[0099]

[0100] in, It is the cumulative distribution function of the standard normal distribution and is used to calculate the probability of the z value in the normal distribution. The z value is a standardized statistic used to compare sample data with a population parameter.

[0101] The statistical power can be calculated using a mathematical formula through the standard deviation of the control group, the standard deviation of the experimental group, the sample size of the two groups, and the means of the two groups. It is usually assumed that if the statistical power reaches 80%, the statistical power is normal. The expression is as follows:

[0102]

[0103] Among them, 1-β refers to the confidence level, that is, the probability of correctly rejecting the null hypothesis in the test, and β is the probability of incorrectly rejecting the null hypothesis, that is, the false positive rate; is a z-score, indicating that in a two-tailed test, the cumulative distribution function reaches In a two-tailed test, α is the overall significance level, usually 0.05, which means that if the test result shows that the statistic falls within If the value of is within the two intervals, the null hypothesis can be rejected at a 95% confidence level; z is a statistic calculated from sample data, which is used to determine whether the difference between the sample data and a population parameter is statistically significant. It is calculated by dividing the difference between the sample data and the population parameter by the sample standard error; is the mean of the samples of the first page version A; is the mean of the samples of the first page version B; is the variance of the sample of version A of the first page; is the variance of the sample of the first page version B; N A is the sample size of the first page version A, N B is the sample size of the sample of version B of the first page.

[0104] Assume that the experimental platform data shows that the homepage layout of the first page version B increases the average user stay time by 10% and the purchase rate by 5% compared to the first page version A, and statistical analysis through the above algorithm shows that the confidence interval values ​​are all positive, the p-value is <0.0001, and the statistical power is greater than 80%, then a positive and significant conclusion can be drawn. Therefore, the homepage layout corresponding to the first page version B is effective and can increase user engagement and purchasing intention.

[0105] In some embodiments, Figure 3 The "receiving the second page version returned by the experimental platform" in step 102 shown can be implemented through the following technical solution: receiving a first instruction sent by the experimental platform, the first instruction is used to instruct to solidify the page code; parsing the first instruction to obtain the second page version carried in the first instruction.

[0106] In some examples, the client receives a first instruction sent by the experimental platform, and the first instruction is used to instruct the client to solidify the page code. After that, the client parses the first instruction and obtains the identifier of the second page version carried in the first instruction. The first instruction is sent by the experimental platform after determining the second page version from multiple first page versions based on multiple operation data.

[0107] As an example, when the test is completed, that is, the experimental platform has determined the second page version from multiple first page versions, at this time, the experimental platform can notify the business background to send a first instruction (such as a solidified instruction) to the client, and the first instruction can carry the identifier of the second page version. After receiving the first instruction, the client parses the first instruction to obtain the identifier of the second page version carried in the first instruction, and then can solidify the page sub-code of the second page version into the code that needs to be executed when displaying the page based on the identifier of the second page version carried by the first instruction.

[0108] In step 103, the test code is deleted, and the page code is solidified based on the second page version.

[0109] In some embodiments, the above step 103 can be implemented by the following technical solution: in response to the first instruction, the test code is deleted, and the page code is solidified based on the second page version.

[0110] In some embodiments, the above-mentioned “solidifying the page code based on the second page version” can be implemented through the following technical solution: solidifying the page sub-code corresponding to the second page version into the code that needs to be executed when displaying the page.

[0111] In some examples, the client deletes the test code carried by the page to be tested according to the first instruction sent by the business backend, and solidifies the page sub-code corresponding to the second page version into the code that needs to be executed when the page is displayed.

[0112] As an example, before conducting a page test, a software development kit (SDK) can be introduced, the location where the page needs to be developed is marked, and the corresponding page code and test code for the page test are developed, and then the experiment is started. After the second page version is determined from multiple first page versions, the client receives the first instruction, and the client begins to delete the test code (such as "if", "else" and other judgment codes), and solidifies the page subcode of the second page version into the code that needs to be executed when the page is displayed according to the identifier of the second page version, and displays the corresponding page according to the page subcode of the second page version. At this time, after the solidification is completed, the page to be tested is no longer displayed as the page corresponding to other first page versions, but is fixedly displayed as the corresponding page of the second page version.

[0113] The following example illustrates that, for example, multiple first page versions can be multiple first pages associated with a certain control, and the second page version can be a second page determined by the experimental platform from multiple first pages based on operation data. When the client receives a trigger operation for the control in the process of running the page to be tested, according to the identifier of the second page carried by the first instruction, it can be determined that the page associated with the control at this time is the second page, and the second page is displayed on the client. For example, during testing, multiple first pages associated with the "Home" control of a shopping platform are set. After the experimental platform determines the second page from multiple first pages based on operation data, when the client receives a trigger operation for the "Home" control in the process of running the page to be tested, according to the identifier of the second page carried by the first instruction, it can be determined that the page associated with the "Home" control at this time is the second page, and the second page is displayed on the client.

[0114] For example, the multiple first page versions may be multiple first recommendation information set for specific items (for example, multiple candidate advertisements, assumed to be Advertisement 1, Advertisement 2, and Advertisement 3), and the second page version may be the second recommendation information (i.e., the optimal advertisement, such as Advertisement 2) determined by the experimental platform from the multiple first recommendation information based on operation data. When the client is running the page to be tested, according to the identifier of the second recommendation information carried by the first instruction, it can be determined that the recommendation information set for the specific item at this time is the second recommendation information, and the second recommendation information is displayed on the client.

[0115] In some embodiments, after the page code is solidified based on the second page version, when there is a problem with the second page version, the following processing can also be performed: in response to the second instruction sent by the experimental platform, the solidified page code is rolled back, and the test code carried by the second instruction is added to the page code.

[0116] In some examples, the client receives a second instruction sent by the experimental platform, rolls back the solidified page code, and adds the test code carried by the second instruction to the page code. When a problem is found in the second page version, the user can click the rollback control of the experimental platform to withdraw. When the experimental platform receives the user's click operation, it will notify the business background to send a second instruction (such as a rollback instruction) to the client, wherein the second instruction carries the test code. After the client receives the second instruction, it obtains the test code carried by the second instruction, adds the test code to the page code, and continues to display the page corresponding to the first page version according to the page code and the test code, and determines the operation data executed in the page. For the implementation method of continuing to display the page corresponding to the first page version according to the page code and the test code, and determining the operation data executed in the page, please refer to step 101, which will not be repeated here.

[0117] Through the above technical solution, a rollback function can be provided through the experimental platform, so that when a failure occurs when running the second page version, the page can be restored and retested, and the client can quickly return to normal use to avoid negative impact on user experience.

[0118] In some embodiments, Figure 3 After step 101 shown, the following processing can also be performed: sending a request to the experimental platform through the diversion interface, wherein the request carries the client's identification; receiving the page code and test code of the page to be tested sent by the experimental platform to notify the business background when the verification is passed.

[0119] In some examples, the client identifier may be used by the experimental platform to verify the client, that is, to determine whether the current client is a client that meets the experimental conditions.

[0120] The client may receive the page code and the test code of the page to be tested sent by the business background, wherein the page to be tested includes multiple first page versions, and the first page versions may be different candidate pages set by the business background for the same page to be tested.

[0121] For example, to test the function of a certain control in an application, the business backend can set up multiple candidate pages for the control based on specific business needs; an online bookstore wants to test the impact of the layout of its website's homepage on user purchasing behavior, and the business backend can set up multiple different layouts for the website's homepage; a social media platform wants to optimize the message push function of its mobile app to increase user activity and reduce user churn, and the business platform can set up multiple different types of push content for the social media platform.

[0122] After starting the page test, the client sends a request to the experimental platform through the diversion interface. The request carries the client's identifier, which can be a login identity identifier (ID) or a device ID. The experimental platform can verify the client based on the client's identifier. The experimental platform can be an ABTest platform.

[0123] During the page test, the client accesses ABSDK, starts accessing the diversion interface, and sends a request to the experimental platform. Taking the client's identification as the login ID as an example, the experimental platform verifies the request based on the login ID after receiving it. The experimental platform will first set the experimental traffic, for example, the traffic is set to 1% (it can be understood that when the total traffic is 10,000 visits, 1% of the test traffic is 100 visits). You can choose a hash algorithm, such as Message Digest Algorithm 5 (MD5) to generate a hash value, and use the MD5 hash algorithm to hash the login ID to obtain a hash value, and then divide the hash value by 10,000 (ie, the total traffic) to obtain a random number between 0 and 1. If this random number falls between 0 and 1% (ie, the test traffic), then the client corresponding to this login ID passes the verification.

[0124] After the client verification is passed, the experimental platform will notify the business background to send the page code and test code of the page to be tested to the client, and the client receives the page code and test code of the page to be tested sent by the business background.

[0125] Through the above technical solution, the page code and test code of the page to be tested can be sent only to the verified client. If there is a problem with the page to be tested, the potential impact on the overall user experience can be reduced, and the testing cost can also be reduced.

[0126] Through the embodiments of the present application, the page corresponding to the first page version can be displayed based on the page code and test code of the page to be tested, and the operation data performed in the page can be sent to the experimental platform. The experimental platform determines the second page version from multiple first page versions based on the operation data, and finally deletes the test code based on the determined second page version, and solidifies the page code based on the second page version. In this way, there is no need to modify the page code and republish it. The page code corresponding to the test result can be automatically solidified after the test is completed, shortening the page development cycle and reducing development costs.

[0127] In some embodiments, see Figure 5 , Figure 5 This is a third flow chart of the page testing method provided in the embodiment of the present application, which is described in detail below in conjunction with step 201 to step 202.

[0128] In step 201, operation data executed in a page corresponding to a first page version and sent by a client is received.

[0129] The page corresponding to the first page version is obtained based on the page code and the test code of the page to be tested.

[0130] In some examples, multiple operation data are received respectively from multiple clients participating in the test, and each operation data corresponds to a first page version. If there are more than one client participating in the test, the experimental platform can receive multiple operation data respectively sent by multiple clients participating in the test, and each operation data corresponds to a first page version, that is, each operation data comes from a first page version.

[0131] As an example, suppose a test is set up on message push optimization of a social media platform. The social media platform hopes to optimize the message push function of its mobile application to increase user activity and reduce user churn. Currently, the social media platform pushes the same content to all users on the recommendation page. The company wants to test the impact of different types of push content (such as news, promotions, and user interactive content) on user behavior. The experimental platform first determines the goal of the test: to increase the number of times users open the application to view messages and the active time of users in the application. According to the preset traffic ratio, the client participating in the page test can be randomly selected. The page code and test code of the page to be tested sent by the business background to the client include different types of messages: news, promotion, user interaction, and control group (no push is sent). The client can execute any of the four first page versions according to the page code and test code of the page to be tested. The user's operation data on the client will be sent to the experimental platform by the client. The operation data may include: whether the user opens the message, the application usage time, whether the user retains, and other operation data. Each client participating in the test needs to send the operation data to the experimental platform.

[0132] In step 202, based on the operation data, a second page version is determined from a plurality of first page versions, and a first instruction carrying the second page version is returned to the client.

[0133] The first instruction is used to instruct the client to delete the test code and solidify the page code based on the second page version.

[0134] In some embodiments, see Figure 6 , Figure 6 This is a fourth flow chart of the page testing method provided in an embodiment of the present application. Figure 5 The step 202 of "determining the second page version from multiple first page versions based on the operation data" can be implemented by following the steps 2021 to 2022, which are described in detail below.

[0135] In step 2021, for each first page version, statistics are collected on the operation data corresponding to the first page version to obtain statistical results.

[0136] In some embodiments, the experimental platform can count the operation data corresponding to the first page version for each first page version to obtain statistical results. In the test, there are more than one clients executing the first page version, and the operation data corresponding to the first page version needs to be counted before the corresponding data can be analyzed.

[0137] As an example, still taking setting up a test on message push optimization on a social media platform as an example, each client will randomly push a type of message, and the client will also record operational data such as whether the user opens the message, application usage time, and whether the user retains the application under the current first page version. After the experimental platform receives the client's operational data, it groups and summarizes it according to the source of the operational data for statistics. For example, for pushed news messages, the total number of clients that push news messages is first counted, and then the number of clients that open the messages is counted. The number of clients that open the messages is divided by the total number of clients that push news messages to obtain the user message opening rate, which can be used as a statistical result. It should be noted that the user message opening rate is used as a statistical result only for example description, and the embodiments of the present application are not limited to this and can be set according to actual conditions.

[0138] In step 2022, based on a plurality of statistical results respectively corresponding to the plurality of first page versions, a second page version is determined from the plurality of first page versions.

[0139] In some embodiments, the above-mentioned determination of the second page version from multiple first page versions based on multiple statistical results corresponding to multiple first page versions can be achieved through the following technical solutions: determining the difference between multiple statistical results; when the difference is greater than a difference threshold, using the first page version corresponding to the maximum statistical result as the second page version; when the difference is less than the difference threshold, notifying the business background to send the page code and test code of the page to be tested to the second client.

[0140] As an example, a statistical algorithm can be used to evaluate the difference between multiple statistical results. For example, still taking a test of message push optimization on a social media platform as an example, the page to be tested includes different types of messages: news, promotion, user interaction, and control group (i.e., four first page versions). The user message opening rate is set as the statistical result of each first page version. Referring to the implementation method of step 102, the statistical results of the news, promotion, and user interaction categories are compared with the statistical results of the control group using a confidence interval algorithm, a p-value algorithm, and a statistical power algorithm. When two or three groups of comparison results indicate that the statistical results are significant, the message push type corresponding to the maximum statistical result can be used as the winning page version (i.e., the second page version). When only one group of comparison results indicates that the statistical results are significant, the message push type corresponding to the significant result can be used as the winning page version (i.e., the second page version); when the comparison result indicates that the statistical results are not significant, it indicates that the test has failed, and the experimental background can notify the business background to send the page code and the test code of the page to be tested to the second client to continue the test. It should be noted that the second client refers to other clients that are different from the client. The number of second clients that continue to participate in the test can be calculated by the remaining sample size algorithm. The two first page versions, first page version A and first page version B, are still used as an example for explanation. The expression is as follows:

[0141] R B =n B +1-N B (5)

[0142] N A =kN B (6)

[0143]

[0144] Among them, R B Indicates the required remaining sample size; n B N represents the total sample size of the first page version B when the statistical power reaches 80%; B N represents the sample size of the current first page version B; A is the sample size of the current first page version A; σ is S in formula (8) p ; is a constant with a value of 1.96; z 1-β is a constant with a value of 0.8416; μ A is the sample mean of the samples of the current first page version A; μ B is the sample mean of the samples of the current first page version B; is the variance of the sample of the current first page version A; is the variance of the sample of the current first page version B.

[0145] Through the above technical solution, the second page version can be determined through the experimental platform, and when the second page version cannot be determined, the page code and test code of the page to be tested can be sent to the second client to continue testing. The test process can be dynamically adjusted to improve the test success rate.

[0146] Through step 2021 to step 2022, the experimental platform can determine the second page version by performing statistical analysis on the client's operation data, thereby improving the objectivity and accuracy of the test.

[0147] In some embodiments, after returning the first instruction carrying the second page version to the client, the experimental platform can also be used to perform the following processing: in response to a trigger operation on a rollback control, sending a second instruction to the client, wherein the second instruction is used to instruct the client to roll back the solidified page code and add the test code carried by the second instruction to the page code.

[0148] As an example, a rollback control can be provided on the experimental platform. When a problem is found in the second page version, the user can click the rollback control of the experimental platform to withdraw. When the experimental platform receives the user's click operation, it will notify the business background to send a second instruction (such as a rollback instruction) to the client, wherein the second instruction carries a test code. After receiving the second instruction, the client obtains the test code carried by the second instruction, adds the test code to the page code, and continues to display the page corresponding to the first page version according to the page code and the test code, and determines the operation data executed in the page. For the implementation method of continuing to display the page corresponding to the first page version according to the page code and the test code, and determining the operation data executed in the page, refer to step 101, which will not be repeated here.

[0149] Through the above technical solution, a rollback function can be provided through the experimental platform, so that when a failure occurs when running the second page version, the page can be restored and retested, and the client can quickly return to normal use to avoid negative impact on user experience.

[0150] In some embodiments, the experimental platform may also be used to perform at least one of the following processes:

[0151] At least one of the conversion rate, the publishing success rate, and the withdrawal rate of the page to be tested is counted, and the statistical result is displayed.

[0152] A real-time report for the plurality of first page versions is generated based on the operational data received in real time.

[0153] Based on the operational data received within a set period of time, a timing report for the plurality of first page versions is generated.

[0154] As an example, the conversion rate of the page to be tested is equal to the number of solidified second page versions divided by the number of pages to be tested. This indicator can intuitively show which pages to be tested on the experimental platform have been successfully verified and used online. The release success rate is equal to the number of clients that successfully solidified the second page version divided by the number of all clients participating in the test. This indicator can intuitively show what proportion of users have used the page corresponding to the second page version. The withdrawal rate is equal to the number of withdrawn clients divided by the number of all clients. This indicator can intuitively show what proportion of users have withdrawn the page corresponding to the second page version.

[0155] As an example, the experimental platform can generate real-time reports for multiple first page versions based on the operational data received in real time. The report can be updated each time the operational data returned by the client is received. The report can be displayed in the order of core indicators, confidence intervals, statistical power, p-values, required sample sizes, and significant results. Testers can adjust the experiment in real time based on the data presented in the report. For example, when they see that the significant results are not significant, they can adjust the sample size of the test according to the required sample size.

[0156] As an example, a scheduled report for a plurality of first schemes may be generated based on the operation data received within a set time period. For example, the time period may be preset, including daily, hourly, etc., and may be set according to actual conditions.

[0157] Through the embodiments of the present application, the second page version can be determined from multiple first page versions based on the operation data sent by the client and performed in the page corresponding to the first page version, and the first instruction carrying the second page version can be returned to the client, so that the client can solidify the page code according to the first instruction. In this way, there is no need to modify the page code and republish it. The page code corresponding to the test result can be automatically solidified after the test is completed, shortening the page development cycle and reducing development costs.

[0158] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0159] See also Figure 7 , Figure 7 It is a schematic diagram of the principle of the experiment-to-online function provided in the embodiment of the present application. Figure 7The content shown is the core process of the experimental platform from the beginning of ABTest to the online function. The starting point is to launch a new function or business (i.e. the page to be tested), but there are multiple solutions (i.e. multiple first page versions) to be selected. The purpose of ABTest is to screen out a better solution or strategy (i.e. the second page version) from the actual diversion situation. There are four major steps in total: 1. Configure experimental parameters, traffic distribution, and grouping; 2. Run the experiment. During the use process, each user requests the diversion interface, obtains the configured parameters in a group, and displays the page or function corresponding to the parameters. After diversion, report the buried point (user behavior record) to indicate the group; 3. After a certain experimental cycle, observe the configured indicator report in the platform, such as checking the conversion rate value, to see which group has a more significant experimental effect and which group wins; 4. After the experimental results are obtained, the winning function is solidified as Feature based on this experiment. The following is a detailed explanation.

[0160] First, determine the business to be launched, and prepare to create an experiment on the experimental platform, including: creating a new traffic layer, creating new indicators that need to be paid attention to, editing the experimental cycle, editing the control group and experimental group, and selecting the indicators that need to be associated. Then, determine the type of experiment (client-side experiment, server-side experiment), divert the traffic through the server, and conduct experiments. After a certain experimental time, different group results are obtained and reported. Then, use the sample distribution test algorithm, statistical power algorithm, confidence interval algorithm, p-value algorithm, retention algorithm, and funnel algorithm to perform significance analysis on different group results, and only recommend analysis reports. Finally, after solidifying the Feature, dynamically publish the Feature and end the experiment.

[0161] When the client receives the command from the server to solidify the Feature, it recognizes that the experiment has been converted to a Feature, and stops requesting the diversion interface, and constantizes the solidified function parameters. The client code automatically converts to the logic of taking constant values ​​to avoid further judgment. At this point, a new function has been released, and this function will not change due to disconnection from the server. In addition, the command to release the solidified feature can also be released in proportion (sending instructions to some clients), thereby realizing the grayscale release and gradual full release.

[0162] See also Figure 8 , Figure 8 Schematic diagram of the curing function provided in the embodiment of the present application. Figure 8As shown, the server first diverts traffic according to the device or login identity, thereby starting the experiment. During the experimental stage, the client will call the server to obtain a strategy periodically or in real time (that is, it is equivalent to displaying the page corresponding to the first page version based on the page code and test code of the page to be tested). After the server has generated a winning group, the management end (that is, the business background) selects the winning strategy (that is, the second page version) and determines the solidification function. The server issues a solidification function command. After receiving the command, the client generates a constant, stops calling the server to divert traffic, and stops judging (that is, deleting the test code), solidifies the function (that is, solidifies the page code based on the second page version), and ends the release.

[0163] The following is an example of an actual application scenario to illustrate the embodiment of the present application. For example, there is a scenario where a company wants to place an advertisement for a certain product, and the design department has designed three advertisement plans for the product (temporarily named: Advertisement 1, Advertisement 2, Advertisement 3). The design department wants to select an advertisement that can increase transaction volume through experiments.

[0164] First, you need to create an experiment on the experimental platform (which can be an ABTest platform), set the experiment name (for example, advertising test of product xx), experiment type (client experiment, server experiment, this application embodiment selects client experiment), experiment period (up to 365 days, this application embodiment selects 30 days), audience identification (that is, choose to divert by login ID or device ID, because this application embodiment focuses on the transaction results of people, so the login ID is selected as the unique diversion identifier), experimental traffic (referring to using part of the traffic or all of the traffic, this application embodiment initially sets 1% of the traffic, a sample size of about 1 million people), set rules (those whose diversion request parameters meet the set rules can participate in the experiment, this application embodiment has not set specific rules yet), configure the experimental group (this application embodiment sets advertisement 1 as the control group, advertisement 2 and advertisement 3 as the experimental group, and then Then set 3 groups of traffic to be evenly distributed), set core indicators (the results displayed by the core indicators are used to measure the final experimental effect. The embodiment of the present application uses the core indicators to dynamically prompt whether the experimental statistical samples are sufficient. If not, it will also prompt how much sample size needs to be increased, whether the final experimental results are significant, and whether there are obvious differences. In the embodiment of the present application, the successful transaction rate of the product is set as the core indicator, that is: the number of successful payments in the end / the number of advertisements clicked), set other indicators (other indicators can be used as additional reference indicators during the experiment. The embodiment of the present application is configured with a funnel indicator (which can record the processes from the start action to the final action in the client, as well as the number of times or the number of people, for example, the number of advertisements clicked -> the number of times the product details page is entered -> the number of orders submitted -> the number of payments -> the number of completed orders), and now an experiment has been created.

[0165] After the experiment is launched, it starts running, and the client accesses ABSDK to start accessing the diversion interface. Diversion logic: When diverting, the embodiment of the present application will request the diversion interface according to the login ID, and filter through layers of traffic layers (because different experiments have different traffic layers, and a traffic layer can have one or more experiments. In the same traffic layer, a login ID can only hit one of the experiments). The client falls to the traffic layer where the experiment is set. After the diversion service obtains the login ID, it hashes the login ID to obtain the hash value, and then divides the hash value by 1000 to obtain the calculation result. If the result falls into the top 1% (1% of the traffic is set when configuring the experiment), the experiment is hit, and then the experimental value of one of the three experimental groups is hit. After the client obtains the experimental value, it displays the corresponding advertisement, and then automatically reports the tracking point to record that the advertisement has been opened once. Then the user will have a series of other operations that will be recorded through the tracking point.

[0166] Secondly, the experimental platform supports scheduled daily reports, and will notify the experimenter to pay attention to the experiment or adjust the experiment according to different results; the experimental platform also supports real-time statistics and reporting of query indicators by hour or minute. In the embodiment of the present application, after the first day, the three advertisements were divided into groups of about 10,000 people, and the core indicators (conversion rate) of the three groups were almost all about 1%, with one positive and one negative confidence interval, indicating that the difference in indicators was not significant, and the statistical power was 35%. The system defaults to a statistical power of 80%, which is considered normal. Therefore, it also shows how much sample size is needed to achieve the corresponding statistical power (which can be calculated using the remaining sample size algorithm). After seeing that the difference is not significant and the system prompts that the sample size is still needed, the experimental traffic can be adjusted to 3%, and then the report results for the next few days can be observed. A few days after the adjustment, the sample size was sufficient and began to show differences in the core indicators of the three groups. After one month of experimentation, the experimental platform gave significant experimental results. Among them, the conversion rate of Ad 2 increased by 10.34% compared with Ad 1, and the conversion rate of Ad 3 decreased by 3.12% compared with Ad 1. Therefore, Ad 2 won (system prompt information: core indicator conversion rate 34%, confidence interval: [2.23%, 4.72%], p-value: <0.0001, positive and significant).

[0167] In addition, the configured funnel indicators can also show during the experiment how many users clicked into the product details page after opening the ad, how many clicked on the order operation, how many completed the payment, how many refunded the money, and how many finally completed the transaction. The funnel report can more fully support the results of the core indicators.

[0168] The following is an introduction to the algorithms involved in the experimental platform of the embodiment of the present application:

[0169] Confidence interval algorithm: The confidence interval is calculated by mathematical formula using the standard deviation of the control group, the standard deviation of the experimental group, the sample size of the two groups, and the mean of the two groups. Under normal experimental conditions, the interval values ​​are positive or negative, indicating that the direction of change of the effect of the experimental group relative to the effect of the control group is increasing or decreasing. The formula here can be formula (1) shown above.

[0170] p-value calculation method: The p-value is calculated using a mathematical formula using the standard deviation of the control group, the standard deviation of the experimental group, the sample size of the two groups, and the mean of the two groups. A p-value < 0.05 indicates that the experiment is significant, and a p-value > 0.05 indicates that the experimental result is not significant. The formula here can be the formula (2) shown above.

[0171] Calculation method of statistical power: In ABTest, the significance of the experimental sample is tested through the confidence interval. If the confidence interval shows that the difference is not significant, but the actual situation may be significant, the real effect may be missed (the experimental result is not significant, but it does not mean that there is no difference). At this time, if the calculated statistical power is low, then when the difference really exists, it may be wrongly judged that there is no difference between the two groups. If the statistical power is defined at 80%, the result is considered to be relatively reliable. The statistical power is calculated by the standard deviation of the control group, the standard deviation of the experimental group, the sample size of the two groups, and the mean of the two groups using a mathematical formula. The formula here can be formula (3) and formula (4) shown above.

[0172] Residual sample size algorithm: When the calculated statistical power is less than 80%, the residual sample size needs to be calculated. The formula here can be formula (5) to formula (8) shown above.

[0173] Finally, the Feature is released online. The embodiment of the present application has screened out Advertisement 2 with significant effects after one month, so Advertisement 2 can be released, and Feature can be released based on this experimental platform (called solidified Feature on the platform). In the created experiment, click the "Solidify Feature" control, select Advertisement 2, and then select full release. After receiving the server instruction, the client SDK will discard the code logic of the previous experiment, change the original dynamic variables into immutable constants, delete the judgment logic (that is, automatically detach from the experimental code), and then the client will follow the fixed logic to achieve the online, thereby fixing the winning Advertisement 2. The code here will not actively interact with the server, and will not cause abnormalities due to disconnection with the server. Feature also has grayscale release (that is, some clients are online), and has a full release function. If there is a problem, there is also a rollback function, so the Feature function can be used to control the final release. The so-called rollback, the platform has a "rollback" control, after clicking it, the client instruction will be issued, and the client will receive the instruction, and it will reset the constant value to the previous value and restore the previous function.

[0174] In addition, the experimental platform also has the functions of feature conversion rate statistics, release success rate, and rollback rate statistics:

[0175] 1) Feature conversion rate: The feature conversion rate is equal to the number of solidified features divided by the number of configured experiments. This indicator can intuitively show which experiments have been successfully verified and used online.

[0176] 2) Release success rate: The release success rate is equal to the number of successfully solidified features divided by the number of all clients. This indicator can intuitively show what proportion of users have used the feature.

[0177] 3) Rollback rate: The rollback success rate is equal to the number of rollbacks divided by the number of all clients. This indicator can intuitively show what proportion of users have rolled back the function.

[0178] At this point, a business has been completed from the beginning of the experiment to the final launch. The embodiment of the present application covers the main functions of the experimental platform. The remaining functions are some supporting management functions, such as: traffic layer management, indicator management, approval management, system commonly used variables management, authority management, etc.

[0179] It is understandable that in the embodiments of the present application, related data such as user operation data are involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards.

[0180] The following continues to describe the exemplary structure of the page testing device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2A As shown, the software modules stored in the page testing device 555 of the memory 550 may include:

[0181] Display module 5551 is used to display the page corresponding to the first page version based on the page code and test code of the page to be tested, and determine the operation data performed in the page; the first sending module 5552 is used to send the operation data to the experimental platform; the first receiving module 5553 is used to receive the second page version returned by the experimental platform, and the second page version is determined from multiple first page versions based on the operation data; the deletion module 5554 is used to delete the test code and solidify the page code based on the second page version.

[0182] In some embodiments, the first receiving module 5553 is also used to receive a first instruction sent by the experimental platform, the first instruction is used to instruct to solidify the page code; parse the first instruction to obtain the second page version carried in the first instruction; the deletion module 5554 is also used to respond to the first instruction, delete the test code, and solidify the page code based on the second page version.

[0183] In some embodiments, the page code includes a page sub-code corresponding to the first page version; the deletion module 5554 is also used to solidify the page sub-code corresponding to the second page version into a code that needs to be executed when displaying the page.

[0184] In some embodiments, the display module 5551 is also used to select a first page version from multiple first page versions based on the test code; and display the page corresponding to the selected first page version based on the page sub-code corresponding to the selected first page version.

[0185] In some embodiments, the apparatus further comprises:

[0186] The first recovery module is used to roll back the solidified page code in response to the second instruction sent by the experimental platform, and add the test code carried by the second instruction to the page code.

[0187] The following continues to describe the exemplary structure of the page testing device 643 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2B As shown, the software modules stored in the page test device 643 of the memory 640 may include:

[0188] In some embodiments, the second receiving module 6431 is used to receive operation data sent by the client and performed in a page corresponding to the first page version, wherein the page corresponding to the first page version is obtained based on the page code and test code of the page to be tested; the determination module 6432 is used to determine the second page version from multiple first page versions based on the operation data; the second sending module 6433 is used to return a first instruction carrying the second page version to the client, wherein the first instruction is used to instruct the client to delete the test code and solidify the page code based on the second page version.

[0189] In some embodiments, the determination module 6432 is also used to count the operation data corresponding to the first page version for each first page version to obtain statistical results; and determine the second page version from the multiple first page versions based on multiple statistical results corresponding to the multiple first page versions.

[0190] In some embodiments, the apparatus further comprises:

[0191] The second recovery module is used to send a second instruction to the client in response to a trigger operation on the rollback control, wherein the second instruction is used to instruct the client to roll back the solidified page code and add the test code carried by the second instruction to the page code.

[0192] In some embodiments, statistics are collected for at least one of the conversion rate, publication success rate, and withdrawal rate of the page to be tested, and the statistical results are displayed; based on the operation data received in real time, a real-time report is generated for multiple first page versions; based on the operation data received within a set time period, a scheduled report is generated for multiple first page versions.

[0193] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the page test method described in the embodiment of the present application.

[0194] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the page testing method provided by the present application embodiment, for example, Figures 3 to 6 The page testing method is shown.

[0195] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0196] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0197] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0198] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0199] In summary, the embodiment of the present application can display the page corresponding to the first page version based on the page code and test code of the page to be tested, and send the operation data executed in the page to the experimental platform. The experimental platform determines the second page version from multiple first page versions based on the operation data, and finally deletes the test code based on the determined second page version, and solidifies the page code based on the second page version. In this way, there is no need to modify the page code and republish it. The page code corresponding to the test result can be automatically solidified after the test is completed, shortening the page development cycle and reducing development costs; using the test code to select the first page version, and displaying the page according to its corresponding page sub-code, each first page version can be randomly selected with the same probability of being selected, ensuring the fairness of the test and improving the accuracy of the test results; the experimental platform can provide a rollback function, so that when a failure occurs when running the second page version, the page can be restored and retested, and can be quickly restored to normal use status to avoid negative impact on user experience; the page code and test code of the page to be tested can be sent only to the verified client. If there is a problem with the page to be tested, the page code and test code can be restored to the client only. In the case of problems, the potential impact on the overall user experience can be reduced, and the testing cost can also be reduced; the second page version can be determined through the experimental platform, and when the second page version cannot be determined, the page code and test code of the page to be tested can be sent to the second client to continue testing, and the test process can be dynamically adjusted to improve the test success rate; the experimental platform can determine the second page version by statistically analyzing the client's operation data, which improves the objectivity and accuracy of the test; using a variety of algorithms, combined with background timing tasks, it can timely correct whether the configuration is correct; promptly inform the user whether the experiment has been significant, how significant it is, whether it is necessary to adjust the traffic size, and whether it is sufficient to publish significant results, thereby improving the efficiency of testing and obtaining test results; visual perception and operation on an experimental platform reduces the use of servers, and the process from starting testing to obtaining test results to publishing online is very simple and easy to operate; after obtaining the test results, the client avoids secondary development and then publishing, and the client that receives the server instruction can automatically convert it into a fixed logic function, thereby completing the release and online, and completing the required business benefits, thereby reducing costs and increasing efficiency.

[0200] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A page testing method, characterized in that: The method comprises: Based on the page code and the test code of the page to be tested, display a page corresponding to the first page version, and determine the operation data executed in the page; Sending the operation data to the experiment platform, and receiving a second page version returned by the experiment platform, where the second page version is determined from a plurality of the first page versions based on the operation data; The test code is deleted, and the page code is solidified based on the second page version.

2. The method according to claim 1, characterized in that The receiving the second page version returned by the experimental platform includes: Receiving a first instruction sent by the experimental platform, wherein the first instruction is used to instruct to solidify the page code; Parsing the first instruction to obtain the second page version carried in the first instruction; The deleting the test code and solidifying the page code based on the second page version includes: In response to the first instruction, the test code is deleted, and the page code is solidified based on the second page version.

3. The method according to claim 1, characterized in that The page code includes a page subcode corresponding to the first page version; The solidifying the page code based on the second page version includes: The page subcode corresponding to the second page version is solidified into the code that needs to be executed when the page is displayed.

4. The method according to claim 3, characterized in that The step of displaying a page corresponding to the first page version based on the page code and the test code of the page to be tested includes: Based on the test code, selecting a first page version from a plurality of first page versions; Based on the page sub-code corresponding to the selected first page version, a page corresponding to the selected first page version is displayed.

5. The method according to any one of claims 1 to 4, characterized in that: After solidifying the page code based on the second page version, the method further includes: In response to the second instruction sent by the experimental platform, the solidified page code is rolled back, and the test code carried by the second instruction is added to the page code.

6. A page testing method, characterized in that: The method comprises: Receiving operation data sent by a client and executed in a page corresponding to a first page version, wherein the page corresponding to the first page version is obtained based on a page code of a page to be tested and a test code; Based on the operation data, a second page version is determined from multiple first page versions, and a first instruction carrying the second page version is returned to the client, wherein the first instruction is used to instruct the client to delete the test code and solidify the page code based on the second page version.

7. The method according to claim 6, characterized in that The determining, based on the operation data, a second page version from a plurality of the first page versions comprises: For each of the first page versions, statistics are collected on the operation data corresponding to the first page version to obtain statistical results; Based on the multiple statistical results respectively corresponding to the multiple first page versions, a second page version is determined from the multiple first page versions.

8. The method according to claim 6, characterized in that After returning the first instruction carrying the second page version to the client, the method further includes: In response to the trigger operation for the rollback control, a second instruction is sent to the client, wherein the second instruction is used to instruct the client to roll back the solidified page code and add the test code carried by the second instruction to the page code.

9. The method according to any one of claims 6 to 8, characterized in that: The method further comprises: Collecting statistics on at least one of the conversion rate, publishing success rate, and withdrawal rate of the page to be tested, and displaying the statistical results; generating a real-time report for the plurality of first page versions based on the operational data received in real time; Based on the operation data received within a set period of time, a timed report for the plurality of first page versions is generated.

10. A page testing device, characterized in that: The device comprises: A display module, configured to display a page corresponding to the first page version based on the page code and the test code of the page to be tested, and determine operation data executed in the page; A first sending module, used for sending the operation data to the experimental platform; A first receiving module, configured to receive a second page version returned by the experiment platform, where the second page version is determined from a plurality of first page versions based on the operation data; A deletion module is used to delete the test code and solidify the page code based on the second page version.

11. A page testing device, characterized in that: The device comprises: A second receiving module is used to receive operation data sent by a client and executed in a page corresponding to a first page version, wherein the page corresponding to the first page version is obtained based on a page code and a test code of the page to be tested; a determination module, configured to determine a second page version from a plurality of the first page versions based on the operation data; The second sending module is used to return a first instruction carrying the second page version to the client, wherein the first instruction is used to instruct the client to delete the test code and solidify the page code based on the second page version.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; The processor is used to implement the page testing method described in any one of claims 1 to 5 or any one of claims 6 to 9 when executing the computer executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the page testing method described in any one of claims 1 to 5 or any one of claims 6 to 9 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the page testing method described in any one of claims 1 to 5 or any one of claims 6 to 9 is implemented.

Images