Methods, devices, electronic equipment, and storage media for evaluating the effectiveness of version updates

Abstract

This specification provides one or more embodiments of a method for evaluating the effectiveness of a version update. The method includes: during the testing phase of the current iteration cycle, importing business traffic into a test version application and an original version application according to a first preset ratio; evaluating the test effectiveness gain of the test version application compared to the original version application; if the effectiveness gain is greater than a preset threshold, moving from the testing phase to the formal launch phase, and importing business traffic into the test version application and the original version application according to a second preset ratio; during the formal launch phase, evaluating the formal launch effectiveness gain of the test version application compared to the original version application according to a preset evaluation period, and calculating the long-term effectiveness gain of the test version application within the current iteration cycle based on all obtained formal launch effectiveness gains.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and more particularly to a method, apparatus, electronic device, and storage medium for evaluating the effectiveness of version updates. Background Technology

[0002] With the rapid development of the internet industry, many internet applications iterate weekly or even daily. The typical iteration method involves directly releasing the iterated application (i.e., the test version) to all users. If the iterated application doesn't meet expectations, emergency version optimization or rollback is necessary. To avoid this problem, a solution based on A / B testing has been proposed. Before officially releasing the test version, a control group (the original application) and an experimental group (the test version) are set up, and a subset of users are imported into each group. This allows the test version to demonstrate its performance gain compared to the original version. Only when the test version's performance gain meets expectations is it released to all users.

[0003] However, once the test version of the application is released to all users, or in other words, once the test version of the application is officially launched, the benefits of the test version of the application compared to the official launch of the original application cannot be obtained, nor can the long-term benefits of the test version of the application after its official launch be obtained. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a method for evaluating the effectiveness of version updates.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method for evaluating the effectiveness of a version update is proposed, the method comprising:

[0007] During the testing phase of the current iteration cycle, business traffic is diverted to the test version application and the original version application respectively according to the first preset ratio;

[0008] Evaluate the performance gain of the test version of the application compared to the original version of the application;

[0009] If the effect gain is greater than a preset threshold, the process moves from the testing phase to the formal launch phase, whereby business traffic is imported into the test version application and the original version application respectively according to a second preset ratio.

[0010] During the formal launch phase, the effect gain of the test version application compared to the original version application is evaluated according to a preset evaluation cycle, and the long-term effect gain of the test version application within the current iteration cycle is calculated based on all the obtained formal launch effect gains.

[0011] Optionally, the test version application accounts for a smaller proportion of traffic during the testing phase than the original version application; and the test version application accounts for a larger proportion of traffic during the launch phase than the original version application.

[0012] Optionally, the calculation of the long-term effect gain of the test version applied within the current iteration cycle based on all obtained official launch effect gains includes:

[0013] Calculate the arithmetic mean of all the effects gained from the official launch over the current iteration period;

[0014] The arithmetic mean is determined as the long-term effect gain of the test version applied within the current iteration cycle.

[0015] Optionally, determining the long-term gain of the test version application within the current iteration cycle based on multiple calculation results includes:

[0016] Calculate the weighted average of all the effects gained from the official launch within the current iteration period;

[0017] The weighted average value is determined as the long-term effect gain of the test version applied in the current iteration cycle.

[0018] Optional, also includes:

[0019] If a favorable event occurs for the test version application during any evaluation period, the weight of the gain in effectiveness between the test version application and the original version application upon official launch will be reduced; and / or,

[0020] If an adverse event occurs to the test version application during any evaluation period, the weight corresponding to the gain in performance of the test version application compared to the official launch of the original version application will be increased.

[0021] Optionally, the current iteration cycle is the Nth iteration cycle for the application; wherein:

[0022] When N is an integer greater than 1, the test version application is the application after the Nth version update, and the original version application is the test version application used in the (N-1)th iteration cycle.

[0023] When N equals 1, the test version application is the application after the first version update, and the original version application is the application of the first version.

[0024] Optionally, the updates to the test version application compared to the original version application include: algorithm updates and / or user interface updates.

[0025] According to a second aspect of one or more embodiments of this specification, a version update effect evaluation apparatus is provided, the apparatus comprising:

[0026] The test import unit is used to import business traffic into the test version application and the original version application respectively according to a first preset ratio during the test phase of the current iteration cycle.

[0027] The test performance evaluation unit is used to evaluate the test performance gain of the test version application compared to the original version application;

[0028] The formal import unit is used to move from the testing phase to the formal launch phase when the effect gain is greater than a preset threshold, so as to import business traffic into the test version application and the original version application respectively according to a second preset ratio.

[0029] The long-term effect evaluation unit is used to evaluate the gain of the test version application compared with the original version application in terms of the official launch effect during the official launch phase according to a preset evaluation period, and to calculate the long-term effect gain of the test version application in the current iteration period based on all the obtained gains of the official launch effect.

[0030] Optionally, the test version application accounts for a smaller proportion of traffic during the testing phase than the original version application; and the test version application accounts for a larger proportion of traffic during the launch phase than the original version application.

[0031] Optionally, the long-term effect evaluation unit is specifically used for:

[0032] Calculate the arithmetic mean of all the effects gained from the official launch over the current iteration period;

[0033] The arithmetic mean is determined as the long-term effect gain of the test version applied within the current iteration cycle.

[0034] Optionally, the long-term effect evaluation unit is specifically used for:

[0035] Calculate the weighted average of all the effects gained from the official launch within the current iteration period;

[0036] The weighted average value is determined as the long-term effect gain of the test version applied in the current iteration cycle.

[0037] Optionally, the device further includes:

[0038] A weight adjustment unit is used to reduce the weight corresponding to the gain in performance between the test version application and the original version application during any evaluation period if a favorable event occurs for the test version application; and / or,

[0039] If an adverse event occurs to the test version application during any evaluation period, the weight corresponding to the gain in performance of the test version application compared to the official launch of the original version application will be increased.

[0040] Optionally, the current iteration cycle is the Nth iteration cycle for the application; where:

[0041] When N is an integer greater than 1, the test version application is the application after the Nth version update, and the original version application is the test version application used in the (N-1)th iteration cycle.

[0042] When N equals 1, the test version application is the application after the first version update, and the original version application is the application of the first version.

[0043] Optionally, the updates to the test version application compared to the original version application include: algorithm updates and / or user interface updates.

[0044] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0045] processor;

[0046] Memory used to store processor-executable instructions;

[0047] The processor executes the executable instructions to implement the steps of the method as described in the first aspect above.

[0048] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect above.

[0049] This specification provides a method for evaluating the effectiveness of version updates. During the testing phase of an iteration cycle, the test effect gain can be obtained. If the test effect gain exceeds a preset threshold, the test version is officially launched. Even after the official launch of the test version, some traffic is still reserved for the original version application. This allows the original version application to serve as a control group, and the effect gain of the official launch can be evaluated multiple times according to a preset evaluation period. Furthermore, based on the effect gains of all official launches, the long-term effect gain of the test version application within the current iteration cycle can be obtained. As can be seen from the technical solution of this specification, after the test version application is officially launched, some traffic is still allocated to the original version application. This allows the original version application to serve as a long-term control group, obtaining the effect gain of the official launch. Furthermore, based on the effect gains of all official launches, the long-term effect gain of the test version application within the current iteration cycle can be calculated, thus obtaining a more realistic long-term effect gain. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the architecture of a version update effect evaluation system provided in an exemplary embodiment.

[0051] Figure 2 This is a flowchart of an exemplary embodiment of a method for evaluating the effectiveness of a version update.

[0052] Figure 3 This is a schematic diagram of an effect gain time decay provided in an exemplary embodiment.

[0053] Figure 4 This is a schematic diagram of effect gain time decay at different stages provided in an exemplary embodiment.

[0054] Figure 5 This is a schematic diagram of a multi-round iteration provided in an exemplary embodiment.

[0055] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment.

[0056] Figure 7 This is a block diagram of an exemplary embodiment of a version update effect evaluation device. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification.

[0058] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0059] With the rapid development of the internet industry, many internet applications iterate weekly or even daily. This means that new versions can be released daily or weekly. The typical iteration method involves directly releasing the iterated application (i.e., the test version) to all users. If the iterated application doesn't meet expectations, emergency version optimization or rollback is necessary. To avoid this problem, a solution based on A / B testing has been proposed. Before officially releasing the test version, a control group (the original version) and an experimental group (the test version) are set up, and a subset of users are imported into each group. This allows for the measurement of the test version's performance gain compared to the original version. Once the test version's performance gain meets expectations, it is then released to all users. However, once the test version is released to all users, or even officially launched, all traffic is allocated to it. Without the control group, the performance gains from the official launch cannot be obtained, let alone long-term performance gains. Therefore, in related technologies, the test effect gain is sometimes directly used as the long-term effect gain of the test version within the current iteration cycle. For example, during the testing phase, based on the results of A / B testing, the test effect gain is 5%, which could be a 5% increase in user click-through rate. Based on this, the test version is directly released to all users, and this 5% is used as the long-term effect gain of the test version applied within the current iteration cycle. However, because effect gains have decay characteristics, for example, over time, users may experience visual fatigue with the UI of the test version, and its effect gain will decay over time; similarly, in the case of LBS (Location Based Services), the algorithm's gain also exhibits the decay characteristics described above. In reality, both algorithms (with or without LBS attributes) and user interfaces exhibit the aforementioned time decay characteristic, differing only in the rate of decay. Therefore, considering the decay characteristic of effect gain, directly using the test effect gain as the long-term effect gain for the test version applied in the current iteration cycle will result in significant errors. Moreover, in order to obtain the cumulative long-term effect gain after multiple iterations, related technologies directly add up the test effect gains of each round to obtain the cumulative long-term effect gain, which can even yield a cumulative long-term effect gain exceeding 100%.

[0060] In view of this, the version update effect evaluation method provided in this specification improves upon the version update effect evaluation methods in related technologies to solve the aforementioned technical problems existing in those technologies. The following section combines... Figures 1-6 This document provides a detailed explanation of the method for evaluating the effectiveness of version updates.

[0061] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a version update effect evaluation system provided in an exemplary embodiment. For example... Figure 1 As shown, the system may include a server 11, a network 12, and several electronic devices (such as mobile phones 13-15).

[0062] The server 11 can be deployed on a physical server containing an independent host, or it can be deployed on a virtual server (such as a cloud server) hosted in a host cluster; this specification does not impose any restrictions on this. The server 11 can be configured with the version update effect evaluation method described in this specification, which directs business traffic to different version applications (test version application and original version application) through a first preset and a second preset ratio, thereby evaluating the effect gain of all officially launched applications and calculating the long-term effect gain of the test version application within the current iteration cycle.

[0063] Mobile phones 13-15 represent a type of electronic device that a user can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. The electronic device (such as mobile phones 13-15) can run different versions of an application (test version or original version). These different versions of the application can refer to a client based on a C / S architecture (Client / Server), in which case the client can be a native application client configured on the electronic device, or a mini-program implemented based on H5 technology; these different versions of the application can also refer to a webpage based on a B / S architecture (Browser / Server), in which case different versions of the webpage can be displayed to different users when a user accesses the webpage, i.e., business traffic can be directed to different versions of the webpage at a preset ratio, thereby calculating the gain of the official launch effect and the long-term gain within the current iteration cycle. This specification does not limit this. It should be noted that the importation of business traffic into the test version application and the original version application described in this specification can refer to different users' electronic devices running different versions of the application according to the allocation of the server 11. That is, mobile phone 13 can run the test version application, mobile phone 14 can run the original version application, and mobile phone 15 can run the test version application, allowing users to interact with different versions of the application through their electronic devices. In other words, the application is divided into different versions based on the electronic device. However, in some embodiments, the application can be divided into different versions based on the user account. For example, if user A has logged into their own user account on mobile phone 13 and mobile phone 14, and user B has logged into their own user account on mobile phone 15, then mobile phone 13 and mobile phone 14 can run the test version application, and mobile phone 15 can run the original version application. This specification does not impose any restrictions on this.

[0064] For ease of description, the method for evaluating the effectiveness of version updates described in this specification will be described in detail below with reference to the accompanying drawings.

[0065] Please see Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of a method for evaluating the effectiveness of a version update. For example... Figure 2 As shown, the following steps may be included:

[0066] Step 202: During the testing phase of the current iteration cycle, business traffic is imported into the test version application and the original version application respectively according to the first preset ratio.

[0067] The updates to the test version application compared to the original version application described in this specification may include: algorithm updates and / or user interface updates. Algorithm updates may include recommendation algorithms, for example, the recommendation algorithm in the original version application has a 3-second delay, while the delay in the test version application is reduced to 10 milliseconds, or the recommendation algorithm in the test version application is more in line with user habits, etc. User interface updates may include changes to icon size, color, etc., for example, the original version application uses blue icons, while the test version application replaces the blue icons with red icons.

[0068] The first preset ratio can be determined based on testing requirements. However, since it's during the testing phase of the iteration cycle, it's impossible to determine whether the test version application will generate a positive effect gain, or whether the test effect gain of the test version application will reach the preset threshold. Therefore, allocating business traffic according to the first preset ratio ensures that the business traffic of the test version application is less than that of the original version application, thus avoiding the assumption that the negative effect gain of this test version application will affect the user experience of most users. For example, the first preset ratio can be 1:19, where 1:19 means importing 5% of the business traffic into the test version application and 95% into the original version application. Although in some embodiments, the sample sizes of the control group and the experimental group need to be consistent, since the effect gain described in this specification is generally a percentage value such as user click-through rate or order rate, the sample sizes of the test version application and the original version application in this specification can be different, that is, the business traffic can be allocated according to the first preset ratio as mentioned above. Of course, in other embodiments, 5% of the business traffic can also be evenly imported into the test version application and the original version application, and this specification does not limit this. By directing business traffic to different versions of the application at a predetermined ratio, corresponding test performance gains can be obtained. Whether this gain reaches the expected threshold determines whether the test version should be officially launched. This ensures that the launched application achieves the expected results, avoiding version rollbacks or emergency version optimizations. Furthermore, by only directing a small portion of business traffic to the test version application at the predetermined ratio, it effectively avoids impacting the normal experience of most users.

[0069] Step 204: Evaluate the performance gain of the test version of the application compared to the original version of the application.

[0070] The test effect gains mentioned in this manual can be user click-through rate, user order rate, etc. In fact, the test effect gains mentioned in this manual are inseparable from the updated content. In other words, different updated content can correspond to different effect gains. Taking the interactive interface update as an example, in order to determine whether the interactive interface update is successful, it is often reflected by the user click-through rate. For example, the test version of the application uses a red icon, while the original version of the application uses a blue icon. Since it is impossible to determine whether users prefer blue or red, it can be indirectly reflected by the user click-through rate. Assuming that the click-through rate of the test version of the application is higher than that of the original version, it can reflect that users prefer the red icon, and then the test version of the application can be officially launched.

[0071] Step 206: If the effect gain is greater than a preset threshold, the process moves from the testing phase to the formal launch phase, in which business traffic is imported into the test version application and the original version application respectively according to the second preset ratio.

[0072] The size of the preset threshold can be determined according to actual needs. For example, if the test effect gain is greater than 3%, the test phase can be moved to the formal launch phase. Even the preset threshold can be zero. In this case, as long as there is a positive test effect gain, the test phase can be moved to the formal launch phase.

[0073] After the official launch phase, the traffic share of the test version application is greater than that of the original version application. Since this is the official launch phase, it's necessary for most users to use the test version. Business traffic can be directed to both the test version and the original version application at a second preset ratio. For example, this second preset ratio could be 19:1, meaning 95% of the business traffic is directed to the test version application and 5% to the original version application, thereby promoting the test version and achieving the official launch. Although in this embodiment, the first and second preset ratios are reciprocals of each other (i.e., 19 and 1 / 19), in some embodiments, the first and second preset ratios can be any values. For example, during the testing phase, 5% of the business traffic is directed to the test version application and 95% to the original version application; during the official launch phase, 98% of the business traffic is directed to the test version application and 2% to the original version application. This specification does not impose any limitations on this. However, regardless of whether the first and second preset ratios are reciprocals, the traffic share of the test version application during the testing phase is less than that of the original version application; and the traffic share of the test version application during the launch phase is greater than that of the original version application.

[0074] Step 208: During the formal launch phase, the formal launch effect gain of the test version application compared to the original version application is evaluated according to a preset evaluation period, and the long-term effect gain of the test version application within the current iteration period is calculated based on all the formal launch effect gains obtained.

[0075] As mentioned earlier, because the effect gain has a decay characteristic, such as Figure 3 As shown, Figure 3 This is a schematic diagram of effect gain time decay provided in an exemplary embodiment, by Figure 3 It is known that the effect gain will decay over time, and this decay will occur during the testing phase and the official launch phase (e.g., Figure 4 As shown in the image, after the test version is officially launched, the effect gains in the official launch will also decay over time. The decay rate (i.e., the slope of the curve) is different for different effect gains, but regardless of whether it is an algorithm or an interface update, it will have the same characteristics. Figure 4 The attenuation characteristics shown differ only in their attenuation rates. The evaluation method described in this specification retains a certain proportion of business traffic for the original application version. Therefore, even after the test version application is officially launched, the performance gain of the test version application compared to the official launch of the original application can still be obtained. Furthermore, the calculation of the long-term performance gain within the current iteration cycle takes into account the aforementioned attenuation characteristics, resulting in a long-term performance gain that more closely reflects the actual long-term performance gain.

[0076] The preset evaluation period can be determined according to actual needs. For example, if the preset evaluation period is 1 day, then the performance gain of the test version of the application compared to the official launch version can be calculated every day. As another example, if the preset evaluation period is 4 hours, then the performance gain of the test version of the application compared to the official launch version can be calculated every 4 hours.

[0077] In one embodiment, the arithmetic mean of all official launch effect gains within the current iteration period can be calculated; this arithmetic mean is determined as the long-term effect gain of the test version applied within the current iteration period. Assuming the preset evaluation period is 1 day and the current iteration period is 5 days, all official launch effect gains can be obtained as shown in Table 1:

[0078] Assessment time Day 1 the next day Day 3 Day 4 Day 5 Effect gain 8％ 7.5％ 6.5％ 6％ 6％

[0079] Table 1

[0080] Based on the information shown in Table 1, the arithmetic mean within the current iteration cycle can be obtained, specifically 6.8%. This 6.8% (i.e., the arithmetic mean) can be determined as the long-term effect gain of the test version application within the current iteration cycle. The version update effect evaluation method described in this specification does not directly use the test effect gain as the long-term effect gain within the current iteration cycle. Instead, it fully considers the decay characteristics of effect gain. By retaining the original version application during the official launch phase and allocating a small amount of business traffic to it, multiple official launch effect gains can be obtained. After taking the arithmetic mean, this arithmetic mean is used as the long-term effect gain, making the final calculated long-term effect gain more realistic.

[0081] In one embodiment, a weighted average of all official release effect gains within the current iteration period can be calculated; this weighted average is determined as the long-term effect gain of the test version applied within the current iteration period, and the weights of each official release effect gain can be determined according to actual needs (such as the specific value range of the weights), as shown in Table 2 below:

[0082] Assessment time Day 1 the next day Day 3 Day 4 Day 5 Effect gain 8％ 7.5％ 6.5％ 6％ 6％ Weight 5 6 7 7 9

[0083] Table 2

[0084] Based on the information shown in Table 2, the weighted average value within the current iteration cycle is 6.6%. Since the effect gain of the test version application after its official launch is typically higher in the initial stage than in the later stage, weights can be assigned to all official launch effect gains, showing an overall upward trend as shown in Table 2, to obtain a weighted average value. This weighted average value can then be determined as the long-term effect gain, resulting in a more accurate representation of the actual long-term gain. Of course, the weight values ​​shown in Table 2 are merely illustrative; different test effect gains can be configured with different weights, and this specification does not impose any restrictions on this.

[0085] In one embodiment, if a favorable event occurs for the test version application during any evaluation period, the weight corresponding to the gain in performance of the test version application compared to the official launch of the original version application is reduced; and / or, if an unfavorable event occurs for the test version application during any evaluation period, the weight corresponding to the gain in performance of the test version application compared to the official launch of the original version application is increased. For example, suppose a promotional activity is set up on a certain day during the evaluation period, resulting in a significant increase in user click-through rate. However, this gain in performance is not entirely due to updates (such as algorithm updates or interface updates), but rather to interference from an occasional event. To eliminate interference caused by favorable or unfavorable events like those described above, the corresponding weights can be reduced and / or increased, thereby making the calculated long-term performance gain more reflective of the true effect of the actual long-term performance gain.

[0086] As mentioned earlier, internet applications require multiple iterations. The current iteration cycle described in this specification is the Nth iteration cycle for the application. Specifically: when N is an integer greater than 1, the test version application is the application after the Nth version update, and the original version application is the test version application used in the (N-1)th iteration cycle; when N equals 1, the test version application is the application after the 1st version update, and the original version application is the first version of the application. To better understand the above iteration cycle, you can participate in... Figure 5 , Figure 5 This is a schematic diagram illustrating a multi-round iteration as provided in an exemplary embodiment. When multiple iterations are involved, regardless of the iteration number, the previous version of that iteration is used as a control group, so the corresponding official launch effect gain can still be calculated. At this point, the effect gains from multiple iterations can be summed to obtain the corresponding arithmetic mean or weighted average of the multiple iterations, and this average is used as the cumulative long-term effect gain over the multiple iteration cycles.

[0087] As can be seen from the above technical solution, the version update effect evaluation method described in this specification can obtain the test effect gain of the test during the testing phase of the iteration cycle. When the test effect gain is greater than a preset threshold, the test version is officially launched, avoiding the risk of version rollback or emergency optimization implied by directly launching the test version. Moreover, after the test version is officially launched, some traffic is still reserved for the original version application, so that the original version application can be used as a control group. The effect gain of the official launch can be evaluated multiple times according to the preset evaluation cycle. Then, based on the effect gain of all official launches, the long-term effect gain of the test version application in the current iteration cycle can be obtained, and the obtained long-term gain effect is more in line with the actual long-term gain effect.

[0088] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 6 At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, memory 608, and non-volatile memory 610, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into memory 608 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0089] Please refer to Figure 7 , Figure 7This is a block diagram of a version update effect evaluation apparatus provided in an exemplary embodiment of this specification. This apparatus can be applied to, for example... Figure 6 The device shown, in order to implement the technical solution of this specification, includes:

[0090] Test import unit 702 is used to import business traffic into the test version application and the original version application respectively according to a first preset ratio during the test phase of the current iteration cycle.

[0091] The test effect evaluation unit 704 is used to evaluate the test effect gain of the test version application compared to the original version application.

[0092] The formal import unit 706 is used to enter the formal launch phase from the testing phase when the effect gain is greater than a preset threshold, so as to import the business traffic into the test version application and the original version application respectively according to the second preset ratio.

[0093] The long-term effect evaluation unit 708 is used to evaluate the gain of the test version application compared with the original version application in terms of the official launch effect during the official launch phase according to a preset evaluation period, and to calculate the long-term effect gain of the test version application in the current iteration period based on all the obtained gains of the official launch effect.

[0094] Optionally, the test version application accounts for a smaller proportion of traffic during the testing phase than the original version application; and the test version application accounts for a larger proportion of traffic during the launch phase than the original version application.

[0095] Optionally, the long-term effect evaluation unit 708 is specifically used for:

[0096] Calculate the arithmetic mean of all the effects gained from the official launch over the current iteration period;

[0097] The arithmetic mean is determined as the long-term effect gain of the test version applied within the current iteration cycle.

[0098] Optionally, the long-term effect evaluation unit 708 is specifically used for:

[0099] Calculate the weighted average of all the effects gained from the official launch within the current iteration period;

[0100] The weighted average value is determined as the long-term effect gain of the test version applied in the current iteration cycle.

[0101] Optionally, the device further includes:

[0102] The weight adjustment unit 710 is configured to, if a favorable event occurs for the test version application within any evaluation period, reduce the weight corresponding to the gain in performance between the test version application and the original version application upon official launch; and / or,

[0103] If an adverse event occurs to the test version application during any evaluation period, the weight corresponding to the gain in performance of the test version application compared to the official launch of the original version application will be increased.

[0104] Optionally, the current iteration cycle is the Nth iteration cycle for the application; where:

[0105] When N is an integer greater than 1, the test version application is the application after the Nth version update, and the original version application is the test version application used in the (N-1)th iteration cycle.

[0106] When N equals 1, the test version application is the application after the first version update, and the original version application is the application of the first version.

[0107] Optionally, the updates to the test version application compared to the original version application include: algorithm updates and / or user interface updates.

[0108] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0109] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0110] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0111] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0112] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0114] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0115] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0116] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A method for evaluating the effectiveness of a version update, characterized in that, The method includes: During the testing phase of the current iteration cycle, business traffic is diverted to the test version application and the original version application respectively according to the first preset ratio; Evaluate the performance gain of the test version of the application compared to the original version of the application; If the test effect gain is greater than a preset threshold, the test phase will be transitioned to the formal launch phase, and business traffic will be imported into the test version application and the original version application respectively according to the second preset ratio. During the formal launch phase, multiple evaluation periods are set within the current iteration cycle. For each evaluation period, the formal launch effect gain of the test version application compared to the original version application is determined, and the formal launch effect gain of the evaluation period is obtained. Calculate the weighted average of the formal launch effect gain for each of the multiple evaluation periods, and determine the weighted average as the long-term effect gain of the test version application within the current iteration period; wherein, if a favorable event for the test version application occurs in any evaluation period, the weight corresponding to the formal launch effect gain of the test version application compared to the original version application is reduced; and / or, if an unfavorable event for the test version application occurs in any evaluation period, the weight corresponding to the formal launch effect gain of the test version application compared to the original version application is increased. The effect gain is characterized by a specific quantitative indicator, which is set based on the updated content of the test version application compared to the original version application, and the effect gain has a time decay characteristic.

2. The method according to claim 1, characterized in that, The test version application has a lower traffic share during the testing phase than the original version application; and the test version application has a higher traffic share during the launch phase than the original version application.

3. The method according to claim 1, characterized in that, The current iteration cycle is the Nth iteration cycle for the application; wherein: When N is an integer greater than 1, the test version application is the application after the Nth version update, and the original version application is the test version application used in the (N-1)th iteration cycle. When N equals 1, the test version application is the application after the first version update, and the original version application is the application of the first version.

4. The method according to claim 1, characterized in that, The updates to the test version of the application compared to the original version include: algorithm updates and / or user interface updates.

5. A device for evaluating the effectiveness of a version update, characterized in that, The device includes: The test import unit is used to import business traffic into the test version application and the original version application respectively according to a first preset ratio during the test phase of the current iteration cycle. The test performance evaluation unit is used to evaluate the test performance gain of the test version application compared to the original version application; The formal import unit is used to move from the testing phase to the formal launch phase when the test effect gain is greater than a preset threshold, so as to import business traffic into the test version application and the original version application respectively according to a second preset ratio. The long-term effect evaluation unit is used to set multiple evaluation periods within the current iteration cycle during the formal launch phase, and for each evaluation period, determine the formal launch effect gain of the test version application compared to the original version application within that evaluation period, thereby obtaining multiple formal launch effect gains for that evaluation period. Calculate the weighted average of the formal launch effect gain of each of the multiple evaluation periods within the current iteration period; The weighted average value is determined as the long-term effect gain of the test version application within the current iteration cycle; wherein, if a favorable event occurs for the test version application in any evaluation cycle, the weight corresponding to the effect gain of the test version application compared to the official launch of the original version application is reduced; and / or, if an unfavorable event occurs for the test version application in any evaluation cycle, the weight corresponding to the effect gain of the test version application compared to the official launch of the original version application is increased. The effect gain is characterized by a specific quantitative indicator, which is set based on the updated content of the test version application compared to the original version application, and the effect gain has a time decay characteristic.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 4.

Images