Online test method and device, equipment and storage medium

Abstract

The invention provides an online testing method and device, equipment and a storage medium, and relates to the technical field of data processing, in particular to the technical fields of artificial intelligence, intelligent recommendation and the like. According to the specific implementation scheme, under the condition that a target function service completes a first online process in online operation, execution of other online processes in the online operation is stopped; determining time consumption characteristic information of the target function service; and determining whether to continue to execute other online processes in the online operation or not based on the time consumption feature information of the target function service.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to technical fields such as artificial intelligence and intelligent recommendation. Background Art

[0002] In software testing, the response time of backend services (also called time consumption) is an important indicator of software performance testing. Especially in recommendation systems, the time consumption of backend services will directly determine the waiting time of users, and high time consumption will seriously affect the user experience. Therefore, how to accurately and efficiently evaluate the time consumption of recommendation services is of great significance. Summary of the invention

[0003] The present disclosure provides an online testing method, device, equipment and storage medium.

[0004] According to one aspect of the present disclosure, there is provided an online testing method, comprising:

[0005] When the target functional service completes the first on-line process in the on-line operation, stop executing other on-line processes in the on-line operation;

[0006] Determine time-consuming characteristic information of the target function service;

[0007] Based on the time-consuming characteristic information of the target function service, it is determined whether to continue to execute other online processes in the online operation.

[0008] According to another aspect of the present disclosure, there is provided an online testing device, comprising:

[0009] An online testing unit, configured to stop executing other online processes in the online operation when the target functional service completes the first online process in the online operation;

[0010] An information processing unit is used to determine the time-consuming characteristic information of the target function service; and determine whether to continue to execute other online processes in the online operation based on the time-consuming characteristic information of the target function service.

[0011] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0012] at least one processor; and

[0013] a memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0017] In this way, the disclosed solution can obtain the online time-consuming characteristic information of the target function service in a timely manner after the first online process in the online operation is completed, and then determine whether the target function service should continue to go online based on the online time-consuming characteristic information. In this way, a small-scale online performance test is implemented, which provides strong support for timely discovery of problems and making adjustments. At the same time, it also lays the foundation for ensuring that the user experience is not affected and ensuring the stable operation of the system.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 This is a schematic flow chart of an online testing method according to an embodiment of the present application. Figure 1 ;

[0021] Figure 2 This is a schematic diagram of the process of an online testing method in an example according to an embodiment of the present application. Figure 1 ;

[0022] Figure 3 is a schematic diagram of a scenario for collecting performance data according to an embodiment of the present application;

[0023] Figure 4 This is a schematic flow chart of an online testing method according to an embodiment of the present application. Figure 2 ;

[0024] Figure 5 This is a schematic diagram of a scenario of online operation according to an embodiment of the present application;

[0025] Figure 6 This is a schematic diagram of the process of an online testing method in an example according to an embodiment of the present application. Figure 2 ;

[0026] Figure 7This is a schematic diagram of a scenario of a link structure of a target egress service according to an embodiment of the present application;

[0027] Figure 8 is a structural schematic diagram of an online testing device according to an embodiment of the present application;

[0028] Fig. 9 It is a block diagram of an electronic device used to implement the online testing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0031] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0032] The disclosed solution provides an online testing method that can collect and evaluate online data of new services during the online process, and then quickly obtain the time consumption of the new services during the online process. This effectively improves the testing efficiency and provides strong support for reducing labor costs.

[0033] Specifically, Figure 1 This is a schematic flow chart of an online testing method according to an embodiment of the present application. Figure 1The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0034] Further, the method includes at least part of the following contents. Figure 1 As shown, including:

[0035] Step S101: when the target functional service completes the first online process in the online operation, stop executing other online processes in the online operation.

[0036] Here, the target functional service may specifically be a newly added or updated functional service. For example, in one example, the target functional service may specifically be an updated service (such as an updated background algorithm, etc.) based on the recommendation service in the recommendation system.

[0037] It should be noted that the online operation may specifically include multiple online processes in stages, such as the first online process, the second online process, and the third online process. Furthermore, the online traffic of online processes at different stages may be different. In other words, the target functional service faces different traffic in online processes at different stages. For example, in one example, for the first online process, the target functional service may face 10% of the online traffic; for the second online process, the target functional service may face 50% of the online traffic; for the third online process, the target functional service may face 100% of the online traffic. In other words, after completing the third online process, the entire online operation for the target functional service is completed.

[0038] Step S102: Determine time-consuming characteristic information of the target functional service.

[0039] It can be understood that the time-consuming characteristic information of the target function service can be specifically understood as the time-consuming characteristic information of the target function service after completing the first online process in the online operation, for example, the total time consumed to execute the target function service determined after completing the first online process.

[0040] Step S103: Determine whether to continue executing other online processes in the online operation based on the time-consuming characteristic information of the target functional service.

[0041] That is to say, in this example, when the first online process in the online operation of the target function service is completed, the time-consuming characteristic information of the target function service in the online process is obtained, and then according to the time-consuming characteristic information in the online process, it is determined whether the target function service continues to execute other online processes. In this way, during the online process, the target function service is subjected to performance testing, and the process is transferred from offline testing to online testing, which effectively improves the test efficiency of performance testing and the accuracy of test results.

[0042] In this way, the disclosed solution can obtain the online time-consuming characteristic information of the target function service in a timely manner after the first online process in the online operation is completed, and then determine whether the target function service should continue to go online based on the online time-consuming characteristic information. In this way, a small-scale online performance test is implemented, which provides strong support for timely discovery of problems and making adjustments. At the same time, it also lays the foundation for ensuring that the user experience is not affected and ensuring the stable operation of the system.

[0043] Moreover, the disclosed solution can also provide a decision-making basis for the subsequent full-flow launch of target functional services based on the time-consuming characteristic information, thereby effectively reducing the risks brought about by the full-flow launch of target functional services.

[0044] Furthermore, since the disclosed solution introduces an online evaluation method, that is, through a small-scale online performance test, the online time-consuming characteristic information of the target functional service is obtained, the accuracy of the time-consuming evaluation results of the target functional service is greatly improved, thereby effectively solving the problem of inaccurate time-consuming evaluation results in offline performance tests. At the same time, on the basis of ensuring the stable operation of the system, the maintenance cost of the performance test is reduced.

[0045] Further, in a specific example, the time-consuming characteristic information may be obtained in the following manner; specifically, the above-mentioned determination of the time-consuming characteristic information of the target function service (for example, step S102) specifically includes:

[0046] Step S102-1: Obtain online feature information of the target functional service.

[0047] Step S102-2: Based on the online characteristic information of the target function service, obtain the time-consuming characteristic information of the target function service.

[0048] In a specific example, in order to smoothly execute the online operation for the target function service, before executing the online operation, it is also necessary to determine all program codes corresponding to the target function service; here, the program code corresponding to the target function service can be called the experiment to be tested; further, in actual applications, the target function service may include multiple experiments to be tested. For example, in an actual scenario, multiple developers develop the target function service at the same time. At this time, the target function service may include multiple experiments to be tested developed by multiple developers.

[0049] Furthermore, for the above-mentioned development scenario, in order to determine all the experiments to be tested corresponding to the target functional service, an incremental code scan may be performed on the target application corresponding to the target functional service to determine the newly added code or the modified code (here, the newly added code or the modified code may be collectively referred to as incremental code), thereby obtaining all the experiments to be tested corresponding to the target functional service.

[0050] It should be noted that in the actual development process, in order to facilitate the identification of incremental codes, identification information (also called experiment number) can be set for each experiment to be tested. Furthermore, after scanning the incremental code, the experiment numbers of all experiments to be tested corresponding to the target functional service can be obtained. In this way, it is convenient to call the corresponding experiment to be tested by the experiment number in the future, and then quickly conduct online performance testing.

[0051] Furthermore, in one example, when the target function service includes multiple experiments to be tested, the above step S102-1 may specifically include: obtaining the online feature information corresponding to each experiment to be tested of the target function service in the online process, and obtaining the online feature information of the target function service based on the online feature information corresponding to each experiment to be tested, thereby providing strong support for the subsequent rapid and accurate calculation of the time-consuming feature information of the target function service.

[0052] Furthermore, in a specific example, the online characteristic information of the experiment to be tested includes but is not limited to at least one of the following: target computer room time consumption (for example, the time consumed by the experiment to be tested in the target computer room), the traffic proportion of the experiment to be tested (for example, the ratio of the traffic occupied by the experiment to be tested to the total traffic occupied by the target application), and baseline time consumption (for example, the time consumption of the comparative experiment corresponding to the experiment to be tested in the target control group).

[0053] For example, in one example, the incremental code file of the target functional service may include multiple experiments to be tested, such as the experiment number of each experiment to be tested, and each experiment number corresponds to a configuration file related to the experiment to be tested corresponding to the experiment number.

[0054] It should be noted that in this example, the experiment to be tested can be located on the trunk of the target application or on the branch of the target application, and the disclosed solution does not impose specific restrictions on this. For example, the following methods can be used to identify the experiment to be tested on the trunk: (1) Use a preset tool to identify the branch merged into the trunk to scan and obtain the experiment to be tested located on the branch; (2) The difference between the code merged into the trunk and the determined experiment to be tested is further scanned to obtain the experiment to be tested that needs to be launched.

[0055] Furthermore, for the experiment to be tested located on the branch, before going online, the experiment to be tested may be merged into the trunk of the target application, or may not be merged, and the disclosed solution does not impose any specific limitation on this.

[0056] Furthermore, if Figure 2 As shown, when there is a target functional service that needs to be launched, an incremental code scan is performed on the target application corresponding to the target functional service (for example, a scan is performed to add / modify the experimental number in the code of the target application) to obtain the experimental number of each experiment to be tested contained in the incremental code file of the target functional service. Here, in order to further ensure the accuracy of the experiments to be tested that need to be launched, the experimental numbers of each experiment to be tested obtained by scanning can also be displayed to facilitate manual confirmation of the experimental number, thereby ensuring the correctness of the data collected in the subsequent online process.

[0057] Further, after confirming all the experiments to be tested that need to be put online, the first online process in the online operation is executed, and after the execution is completed, the following is adopted: Figure 3 The process shown is to obtain the online characteristic information of each experiment to be tested, for example, to obtain the online characteristic information of each experiment to be tested that is actively reported by the online service, and to perform data collection and calculation to obtain the online characteristic information corresponding to each experiment to be tested (for example, each experiment number) of the target functional service, and then according to the online characteristic information corresponding to each experiment number, calculate the time-consuming characteristic information of the target functional service, and store the data, so as to facilitate subsequent data query and provide a decision-making basis for the subsequent full-flow online launch of the target functional service.

[0058] Furthermore, the obtained time-consuming characteristic information is evaluated to determine whether to dynamically intercept the online target functional service. This not only reduces the manpower cost required for test optimization, but also improves the accuracy of test results. Moreover, problems can be discovered and adjusted in a timely manner to ensure the quality and efficiency of functional service delivery, while also ensuring the stable operation of the online system.

[0059] In this way, the disclosed solution can use the online feature information obtained after the target function service is launched to obtain the time-consuming feature information of the target function service. In this way, the time-consuming result of the target function service is obtained by using the evaluation result of the online evaluation, which greatly improves the accuracy of the online performance test results of the target function service and provides a decision-making basis for the subsequent full-flow launch of the target function service. Moreover, the process is efficient and does not require manual intervention. Therefore, on the basis of effectively reducing labor costs, the test efficiency is further improved.

[0060] Figure 4 This is a schematic flow chart of an online testing method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figures 1 to 3 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0061] Further, the method includes at least part of the following contents. Figure 4 As shown, including:

[0062] Step S401: performing a comparative test on the target control group and the target experimental group (ie, performing an online comparative test).

[0063] Here, the target control group is used to run the target application online; the target experimental group is used to launch the target application with the target function service added. That is to say, when running the target application online and launching the target application with the target function service added, during the above comparative test, the target application is used as the target control group, and the target application with the target function service added is used as the target experimental group, so that the time-consuming characteristic information of the target function service can be obtained by comparing the two.

[0064] It should be noted that after a new service (for example, a service with the target function described above) is developed, it is necessary to evaluate the pros and cons of the service before full traffic is launched. At this time, A (which can be used as a control group, such as the target application before the version change) / B (which can be used as an experimental group, such as the target application after the version change, such as a target application with a newly added target function service) experiment can be used for comparative evaluation (also called ABdiff experiment), that is, by executing the new strategy (corresponding to the above target application with the newly added target function service) and the old strategy (corresponding to the above target application), comparative analysis and evaluation are carried out, so as to evaluate the online performance of the new service.

[0065] Step S402: when the target functional service completes the first online process in the online operation, stop executing other online processes in the online operation.

[0066] It should be noted that the relevant description of the target function service can be found in the above examples and will not be repeated here.

[0067] Further, in a specific example, the online operation at least includes: the target computer room is online; the whole computer room is online. Here, the whole computer room is online at least after the target computer room is online.

[0068] Furthermore, the online launch of the entire computer room may specifically refer to the online launch of the entire traffic, for example, the online launch is carried out in all computer rooms (for example, N computer rooms). In other words, the online launch of the entire computer room faces a large amount of traffic. Compared with the online launch of the entire computer room, the online launch of the target computer room may refer to the online launch of part of the traffic, for example, the online launch is carried out in one or more of the N computer rooms. In other words, the online launch of the target computer room faces a small amount of traffic.

[0069] Further, in an example, the first online process in the online operation may include going online in a target computer room, and further, may specifically refer to going online in a single computer room.

[0070] For example, if Figure 5 As shown, the online operation may include the first online of a single computer room, the online of a single computer room, the first online of all computer rooms, and the online of all computer rooms; further, the first online process mentioned above may be specifically the online of a single computer room. In other words, in this example, when the single computer room online in the online operation of the target function service is completed, the online processes such as the first computer room and the whole computer room may be stopped, and an online evaluation may be performed on the target function service completed by the single computer room online, and then based on the evaluation results, it may be determined whether to continue to execute other online processes.

[0071] It should be pointed out that the "first launch in a single computer room" in this example can specifically mean that the target functional service is launched on a server in a computer room, "single computer room launch" can specifically mean that the target functional service is launched in a computer room, "first launch in all computer rooms" can specifically mean that the target functional service is launched on a server in all computer rooms, and "all computer room launch" can specifically mean that the target functional service is launched in all computer rooms.

[0072] In this way, the disclosed solution can utilize the target computer room online operation during the online operation to enable the target functional service to perform online performance testing in a "small range", thereby providing a decision-making basis for the subsequent "large range" (also known as full traffic) online launch of the target functional service. Moreover, since the test data is collected during the online process, the test results are more accurate. Compared with traditional tests (such as simulation tests), the disclosed solution omits the environmental simulation process, effectively avoiding the problem of inaccurate test results caused by insufficient simulation. On the basis of ensuring the stable operation of the system, it reduces the maintenance cost of performance testing and, at the same time, reduces the cost of human analysis.

[0073] Step S403: Determine time-consuming characteristic information of the target functional service.

[0074] Step S404: Determine whether to continue executing other online processes in the online operation based on the time-consuming characteristic information of the target functional service.

[0075] In this way, the disclosed scheme can use the target control group and the target experimental group for comparative testing, and obtain the online time-consuming characteristic information of the target functional service from the target experimental group, and then determine whether to continue to execute other online processes based on the online time-consuming characteristic information. In this way, the performance of the target functional service in the online environment can be accurately evaluated, providing a more scientific and reliable basis for the subsequent decision on whether to go online with full traffic, thereby effectively reducing the risks brought about by the full traffic launch of the target functional service.

[0076] For example, if Figure 6 As shown, after the development of the target function service is completed, a comparative test is performed; at this time, the newly added target function service is put online, and after completing the single computer room online process, other online processes are stopped. Further, the target function service is evaluated online. For example, during the online evaluation process, the online feature information corresponding to each experimental number of the target function service is collected, and the time consumption analysis is performed on the online feature information corresponding to each experimental number to obtain the time consumption feature information of the target function service, and then according to the threshold strategy, it is determined whether the time consumption feature information exceeds the threshold. If not, continue to execute other online processes in the online operation, such as the first online of the entire computer room, the online of the entire computer room, etc.; otherwise, online interception is performed.

[0077] Here, if the target functional service needs to be intercepted, the experiment can be stopped and the code can be rolled back, or approval can be reported; further, after approval, other online processes in the online operation can be continued. In this way, the risks brought by the full flow online of the target functional service are effectively reduced, the stable operation of the online system is ensured, and the quality and efficiency of the functional service delivery are effectively guaranteed.

[0078] In a specific example of the disclosed solution, the following threshold strategy can be used to determine whether the target functional service can be launched with full traffic, thereby enhancing the sensitivity and response capability of the online system to abnormal functional services; specifically, it includes:

[0079] Method 1: The above-mentioned determination of whether to continue to execute other online processes in the online operation (for example, step S404) based on the time-consuming characteristic information of the target function service specifically includes:

[0080] Step S404-1-1: Based on the time consumption characteristic information of the target functional service, determine the total time consumption of the long tail link corresponding to the target export service in the target experimental group.

[0081] It should be pointed out that, in actual applications, the overall system time consumption of the target experimental group depends on the time consumption of the target export service, and the time consumption of the target export service depends on the total time consumption of the long-tail link corresponding to the target export service; here, the long-tail link corresponding to the target export service is determined from the multiple links corresponding to the target export service based on historical time consumption data.

[0082] For example, if Figure 7 As shown, the target export service (for example, it can be recorded as A) corresponds to multiple links. According to the historical time consumption data, the link with the largest historical total time consumption can be determined among the multiple links corresponding to the target export service A. For example, the historical total time consumption of link A->D->H is 1100 milliseconds (hereinafter referred to as ms), which is greater than the historical total time consumption of other links. At this time, link A->D->H is the long-tail link corresponding to the target export service A, while link A->D->G, link A->C->F, link A->B->E, and link A->B->D are all non-long-tail links.

[0083] Step S404-1-2: Determine whether the total time consumption of the long tail link corresponding to the target egress service in the target experimental group exceeds the total time consumption of the long tail link corresponding to the target egress service in the target control group. If not, proceed to step S404-1-3; otherwise, proceed to step S404-1-4.

[0084] Step S404-1-3: Continue to execute other online processes in the online operation, such as the first online of the entire computer room, the online of the entire computer room, and other processes.

[0085] Step S404-1-4: Stop executing other online processes in the online operation and perform code rollback; or, report for approval and continue with other online processes after approval.

[0086] That is to say, in this example, the total time consumption of the long-tail link corresponding to the target export service in the target experimental group is determined based on the time consumption characteristic information of the target functional service; then, it is determined whether the total time consumption of the long-tail link corresponding to the target export service in the target experimental group exceeds the total time consumption of the long-tail link corresponding to the target export service in the target control group. If it exceeds, it indicates that the newly added target functional service increases the overall time consumption of the system. At this time, it is necessary to stop other online processes in the online operation for the target functional service; otherwise, it indicates that the newly added target functional service has no effect on the overall time consumption of the system. At this time, other online processes in the online operation can continue to be executed. In other words, in this example, interception is only performed when the time consumption of the system's target export service increases, and the increase in time consumption of non-long-tail services that have no effect on system delays is exempted.

[0087] In this way, the disclosed solution can utilize the total time consumption of the long-tail link corresponding to the target export service to determine whether the target functional service can continue to execute other online processes in the online operation. In this way, the online system's judgment and response capabilities for new functional services are enhanced, and the decision-making efficiency of the full-flow online launch of the target functional service is effectively improved. At the same time, the effectiveness of interception of online functional services is improved, thereby effectively reducing the risks brought by the full-flow online launch of the target functional service, thereby ensuring the stable operation of the online system.

[0088] Method 2: The above-mentioned determination of whether to continue to execute other online processes in the online operation (for example, step S404) based on the time-consuming characteristic information of the target function service specifically includes:

[0089] Step S404-2-1: Determine a first ratio of the traffic occupied by the target functional service to the total traffic occupied by the target experimental group.

[0090] Step S404-2-2: Based on the first ratio, determine the time consumption increase threshold corresponding to the target function service.

[0091] Step S404-2-3: Determine whether the time-consuming characteristic information of the target function service exceeds the time-consuming increase threshold corresponding to the target function service. If not, proceed to step S404-2-4; otherwise, proceed to step S404-2-5.

[0092] Step S404-2-4: Continue to execute other online processes in the online operation, such as the first online of the entire computer room, the online of the entire computer room, and other online processes.

[0093] Step S404-2-5: Stop executing other online processes in the online operation and perform code rollback; or, report for approval and continue with other online processes after approval.

[0094] That is to say, in this example, the time-consuming characteristic information of the target functional service can be used directly to determine whether to continue to execute other online processes in the online operation; specifically, before using the time-consuming characteristic information to make a judgment, it is necessary to determine the first ratio of the traffic occupied by the target functional service to the total traffic occupied by the target experimental group, and then determine the time-consuming increase threshold corresponding to the target functional service based on the first ratio; then compare the time-consuming characteristic information with the time-consuming increase threshold.

[0095] Here, in one example, the correspondence between the first ratio and the time-consuming increase threshold value may be preset, as shown in Table 1, and then the time-consuming increase threshold value corresponding to the first ratio may be determined according to the correspondence shown in Table 1 below:

[0096] Table 1

[0097] First ratio Time consumption increase threshold / ms Less than 0.01 100 [0.01,0.25] 50 (0.25,0.5] 10 (0.5,2] 5

[0098] In this way, the disclosed solution can use the traffic proportion of the target functional service in the target experimental group to determine the time-consuming increase threshold corresponding to the target functional service, and then judge whether the time-consuming characteristic information of the target functional service exceeds the time-consuming increase threshold. In this way, an interception method with a dynamic threshold is introduced, thereby improving the effectiveness of intercepting functional services with problems, facilitating timely processing after problems are discovered, effectively reducing the risks brought about by the full traffic launch of the target functional service, and ensuring the stable operation of the online system.

[0099] Mode 3: namely, determining whether to continue to execute other online processes (for example, step S404) in the online operation based on the time-consuming characteristic information of the target function service as described above, specifically includes:

[0100] Step S404-3-1: Based on the time consumption characteristic information of the target functional service, determine the total time consumption of the long tail link corresponding to the target export service in the target experimental group.

[0101] It should be noted that the relevant content about the long tail link can refer to the above example, which will not be repeated here.

[0102] Step S404-3-2: Determine whether the total time consumption of the long tail link corresponding to the target export service in the target experimental group exceeds the total time consumption of the long tail link corresponding to the target export service in the target control group. If so, proceed to step S404-3-6; otherwise, proceed to step S404-3-3.

[0103] Step S404-3-3: Determine a first ratio of the traffic occupied by the target functional service to the total traffic occupied by the target experimental group. Go to step S404-3-4.

[0104] Step S404-3-4: Based on the first ratio, determine the time consumption increase threshold corresponding to the target functional service. Go to step S404-3-5.

[0105] It should be noted that the relevant content about the time-consuming increase threshold can be referred to the above example and will not be repeated here.

[0106] Step S404-3-5: Determine whether the time-consuming characteristic information of the target function service exceeds the time-consuming increase threshold corresponding to the target function service. If so, proceed to step S404-3-6; otherwise, proceed to step S404-3-7.

[0107] Step S404-3-6: Stop executing other online processes in the online operation and perform code rollback; or, report for approval and continue with other online processes after approval.

[0108] Step S404-3-7: Continue to execute other online processes in the online operation, such as the first online of the entire computer room, the online of the entire computer room, and other online processes.

[0109] That is to say, in this example, we can first determine whether the total time consumption of the long tail link corresponding to the target export service in the target experimental group exceeds the total time consumption of the long tail link corresponding to the target export service in the target control group. If it exceeds, it means that the overall system time consumption of the target experimental group is abnormal due to the newly added target function service. At this time, it is necessary to intercept the target function service online; if it does not exceed, it means that the overall system time consumption of the target experimental group is not affected. At this time, it is determined whether the time consumption of the target function service is abnormal, that is, whether the time consumption characteristic information of the target function service exceeds the time consumption increase threshold corresponding to the target function service. If it exceeds, it is necessary to intercept the target function service online, otherwise continue to execute other online processes in the online operation.

[0110] It should be noted that the target function service may act on a long-tail link or a non-long-tail link; if the target function service acts on a non-long-tail link corresponding to the target export service in the target experimental group, then the target function service will not increase the total time consumption of the long-tail link. In other words, the overall system time consumption of the target experimental group may not be affected. However, in this scenario, since the target function service acts on a non-long-tail link, it may cause the time consumption of the non-long-tail link to increase. At this time, the above-mentioned dynamic threshold strategy can be further used to determine whether the time consumption of the target function service is abnormal. In other words, if only the time consumption of the long-tail link is determined, it may not be possible to determine whether the time consumption of the target function service acting on the non-long-tail link is abnormal. Therefore, the above-mentioned dynamic threshold strategy can be further used to determine whether the time consumption of the target function service is abnormal.

[0111] In this way, the performance of the target functional service is evaluated as a whole by combining the long-tail link time consumption strategy and the dynamic threshold strategy, which improves the effectiveness of intercepting functional services with problems, facilitates timely processing after problems are discovered, further reduces the risks brought by the full-flow launch of the target functional service, and ensures the stable operation of the online system.

[0112] Mode 4: namely, determining whether to continue to execute other online processes (for example, step S404) in the online operation based on the time-consuming characteristic information of the target function service as described above, specifically includes:

[0113] Step S404-4-1: Determine a first ratio of the traffic occupied by the target functional service to the total traffic occupied by the target experimental group.

[0114] Step S404-4-2: Based on the first ratio, determine the time consumption increase threshold corresponding to the target function service.

[0115] It should be noted that the relevant content about the time-consuming increase threshold can be referred to the above example and will not be repeated here.

[0116] Step S404-4-3: Determine whether the time-consuming characteristic information of the target function service exceeds the time-consuming increase threshold corresponding to the target function service. If not, proceed to step S404-4-4; otherwise, proceed to step S404-4-6.

[0117] Step S404-4-4: Based on the time consumption characteristic information of the target functional service, determine the total time consumption of the long tail link corresponding to the target export service in the target experimental group, and proceed to step S404-4-5.

[0118] It should be noted that the relevant content about the long tail link can refer to the above example, which will not be repeated here.

[0119] Step S404-4-5: Determine whether the total time consumption of the long tail link corresponding to the target export service in the target experimental group exceeds the total time consumption of the long tail link corresponding to the target export service in the target control group. If so, proceed to step S404-4-6; otherwise, proceed to step S404-4-7.

[0120] Step S404-4-6: Stop executing other online processes in the online operation and perform code rollback; or, report for approval and continue with other online processes after approval.

[0121] Step S404-4-7: Continue to execute other online processes in the online operation, such as the first online of the entire computer room, the online of the entire computer room, and other online processes.

[0122] That is to say, in this example, it is possible to first determine whether the time consumption of the target function service is abnormal, that is, to determine whether the time consumption characteristic information of the target function service exceeds the time consumption increase threshold corresponding to the target function service. If it exceeds, it indicates that the time consumption of the newly added target function service does not meet the standard. At this time, it is necessary to intercept the target function service online; if it does not exceed, it indicates that the time consumption of the newly added target function service is within the preset range. At this time, it is further determined whether the newly added target function service has an impact on the overall time consumption of the system, that is, to determine whether the total time consumption of the long tail link corresponding to the target export service in the target experimental group exceeds the total time consumption of the long tail link corresponding to the target export service in the target control group. If it exceeds, it indicates that the overall system time consumption of the target experimental group is abnormal due to the newly added target function service. At this time, it is necessary to intercept the target function service online; if it does not exceed, it indicates that the newly added target function service has no impact on the overall system time consumption of the target experimental group. At this time, other online processes in the online operation can continue to be executed.

[0123] In this way, the performance of the target functional service is evaluated as a whole by combining the long-tail link time consumption strategy and the dynamic threshold strategy, which improves the effectiveness of intercepting functional services with problems, facilitates timely processing after problems are discovered, effectively reduces the risks brought by the full-flow launch of the target functional service, and ensures the stable operation of the online system.

[0124] In summary, compared with traditional tests (such as simulation tests), the disclosed solution has the following advantages:

[0125] First, the test is more accurate. Compared with traditional testing, the disclosed solution moves the performance evaluation of the changed target functional service from the offline testing stage to the online testing stage (such as the single machine room inspection stage), which improves the accuracy of the obtained performance data. For example, the accuracy of the time-consuming evaluation is increased from the original 56% to 100%, effectively avoiding the problem of inaccurate test results caused by insufficient simulation, making the test more accurate. Moreover, the disclosed solution innovatively combines offline testing with online testing, making the testing of newly added functional services more scientific and reliable.

[0126] Second, low cost. Compared with the simulation test in the traditional test, the performance test of the target functional service in the disclosed solution does not need to build facilities for simulating the online environment, and saves the environment simulation process, reducing the maintenance cost and human analysis cost required for the performance test.

[0127] Third, new interception strategies. After obtaining the performance data, the disclosed solution makes full use of the non-long-tail time-consuming exemption strategy and the dynamic threshold strategy to analyze the obtained performance data, thereby enhancing the online system's ability to judge and respond to new functional services, and at the same time improving the interception capability of changed functional services, thereby reducing the risk brought by the full flow of the target functional service online.

[0128] The disclosed solution also provides an online testing device, such as Figure 8 As shown, including:

[0129] The online testing unit 801 is used to stop executing other online processes in the online operation when the target functional service completes the first online process in the online operation;

[0130] The information processing unit 802 is used to determine the time-consuming characteristic information of the target function service; and determine whether to continue to execute other online processes in the online operation based on the time-consuming characteristic information of the target function service.

[0131] In a specific example of the disclosed solution, the information processing unit is specifically used to:

[0132] Obtaining online feature information of the target functional service;

[0133] Based on the online characteristic information of the target function service, the time-consuming characteristic information of the target function service is obtained.

[0134] In a specific example of the disclosed solution, the online testing unit is further used for:

[0135] A target control group and a target experimental group are subjected to a comparative test, wherein the target control group is used to run the target application online; and the target experimental group is used to perform online operation on the target application newly provided with the target functional service.

[0136] In a specific example of the disclosed solution, the online operation at least includes: the target computer room is online; the entire computer room is online; wherein the entire computer room is online at least after the target computer room is online.

[0137] In a specific example of the disclosed solution, the first online process includes going online in a target computer room.

[0138] In a specific example of the disclosed solution, the information processing unit is specifically used to:

[0139] Based on the time-consuming characteristic information of the target functional service, determining the total time consumption of the long-tail link corresponding to the target export service in the target experimental group;

[0140] When it is determined that the total time consumption of the long tail links corresponding to the target egress service in the target experimental group does not exceed the total time consumption of the long tail links corresponding to the target egress service in the target control group, other online processes in the online operation are continued to be executed.

[0141] In a specific example of the disclosed solution, the information processing unit is specifically used to:

[0142] Determine a first ratio of the traffic occupied by the target functional service to the total traffic occupied by the target experimental group;

[0143] Based on the first ratio, determining a time consumption increase threshold corresponding to the target function service;

[0144] When it is determined that the time-consuming characteristic information of the target function service does not exceed the time-consuming increase threshold corresponding to the target function service, other online processes in the online operation continue to be executed.

[0145] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0146] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0147] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0148] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0149] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0150] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as online testing methods. For example, in some embodiments, the online testing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the online testing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the online testing method in any other appropriate manner (e.g., by means of firmware).

[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0157] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An online testing method, comprising: When the target functional service completes the first on-line process in the on-line operation, stop executing other on-line processes in the on-line operation; Determine time-consuming characteristic information of the target function service; Based on the time-consuming characteristic information of the target function service, it is determined whether to continue to execute other online processes in the online operation.

2. The method according to claim 1, wherein: The determining of the time-consuming characteristic information of the target function service includes: Obtaining online feature information of the target functional service; Based on the online characteristic information of the target function service, the time-consuming characteristic information of the target function service is obtained.

3. The method according to claim 1 or 2, further comprising: A target control group and a target experimental group are subjected to a comparative test, wherein the target control group is used to run the target application online; and the target experimental group is used to perform online operation on the target application newly provided with the target functional service.

4. The method according to claim 3, wherein: The online operation at least includes: the target computer room is online; the whole computer room is online; wherein the whole computer room is online at least performed after the target computer room is online.

5. The method according to claim 4, wherein: The first online process includes going online in a target computer room.

6. The method according to any one of claims 3 to 5, wherein: The determining whether to continue to execute other online processes in the online operation based on the time-consuming characteristic information of the target functional service includes: Based on the time-consuming characteristic information of the target functional service, determining the total time consumption of the long-tail link corresponding to the target export service in the target experimental group; When it is determined that the total time consumption of the long tail links corresponding to the target egress service in the target experimental group does not exceed the total time consumption of the long tail links corresponding to the target egress service in the target control group, other online processes in the online operation are continued to be executed.

7. The method according to any one of claims 3 to 5, wherein: The determining whether to continue to execute other online processes in the online operation based on the time-consuming characteristic information of the target function service includes: Determine a first ratio of the traffic occupied by the target functional service to the total traffic occupied by the target experimental group; Based on the first ratio, determining a time consumption increase threshold corresponding to the target function service; When it is determined that the time-consuming characteristic information of the target function service does not exceed the time-consuming increase threshold corresponding to the target function service, other online processes in the online operation continue to be executed.

8. An online testing device, comprising: An online testing unit, configured to stop executing other online processes in the online operation when the target functional service completes the first online process in the online operation; An information processing unit, used to determine time-consuming characteristic information of the target functional service; Based on the time-consuming characteristic information of the target function service, it is determined whether to continue to execute other online processes in the online operation.

9. The device according to claim 8, wherein: The information processing unit is specifically used for: Obtaining online feature information of the target functional service; Based on the online characteristic information of the target function service, the time-consuming characteristic information of the target function service is obtained.

10. The device according to claim 8 or 9, wherein: The online testing unit is also used for: A target control group and a target experimental group are subjected to a comparative test, wherein the target control group is used to run the target application online; and the target experimental group is used to perform online operation on the target application newly provided with the target functional service.

11. The device according to claim 10, wherein: The online operation at least includes: the target computer room is online; the whole computer room is online; wherein the whole computer room is online at least performed after the target computer room is online.

12. The device according to claim 11, wherein The first online process includes going online in a target computer room.

13. The device according to any one of claims 10 to 12, wherein: The information processing unit is specifically used for: Based on the time-consuming characteristic information of the target functional service, determining the total time consumption of the long-tail link corresponding to the target export service in the target experimental group; When it is determined that the total time consumption of the long tail links corresponding to the target egress service in the target experimental group does not exceed the total time consumption of the long tail links corresponding to the target egress service in the target control group, other online processes in the online operation are continued to be executed.

14. The device according to any one of claims 10 to 12, wherein: The information processing unit is specifically used for: Determine a first ratio of the traffic occupied by the target functional service to the total traffic occupied by the target experimental group; Based on the first ratio, determining a time consumption increase threshold corresponding to the target function service; When it is determined that the time-consuming characteristic information of the target function service does not exceed the time-consuming increase threshold corresponding to the target function service, other online processes in the online operation continue to be executed.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images