Test methods, apparatus, electronic devices and storage media

Abstract

This disclosure provides a testing method, apparatus, electronic device, and storage medium, relating to technical fields such as speech recognition, speech synthesis, cloud storage, and cloud computing. The solution involves: obtaining task parameters associated with a task identifier of the task to be tested; loading target test cases based on test case identifiers in the task parameters; loading a target test set based on test set identifiers in the task parameters; sending test data from the target test set to a first container of services deployed in at least one server based on the target test cases; and obtaining test results obtained by performing target tests matching the task to be tested on services in at least one first container using the test data. Thus, by obtaining a target test set and target test cases matching the task to be tested, and performing target tests on services deployed in the first container based on these target test cases and test sets, the targeting and reliability of the testing can be achieved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to technologies such as speech recognition, speech synthesis, cloud storage, and cloud computing, and particularly to a testing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Automated testing of the server can determine its performance, stability, resilience, and whether any anomalies occur. For example, by setting up a test environment and pulling the latest version of testing tools, test scripts can be executed to simulate server performance and obtain test results. These results can then be used to determine the server's performance, resilience, and other capabilities. Summary of the Invention

[0003] This disclosure provides a test method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a testing method is provided, comprising:

[0005] Obtain the task to be tested, and based on the task identifier of the task to be tested, obtain the task parameters associated with the task identifier, wherein the task parameters include test case identifier and test set identifier;

[0006] Based on the test case identifier, load the target test case that matches the test case identifier;

[0007] Based on the test set identifier, load the target test set that matches the test set identifier;

[0008] Based on the target test cases, send the test data in the target test set to the first container of the service in at least one deployment server.

[0009] Obtain the test results obtained by performing target tests on the services in the at least one first container using the test data, which are matched with the task to be tested.

[0010] According to another aspect of this disclosure, a testing apparatus is provided, comprising:

[0011] The first acquisition module is used to acquire the task to be tested and, based on the task identifier of the task to be tested, acquire the task parameters associated with the task identifier, wherein the task parameters include test case identifier and test set identifier;

[0012] The loading module is used to load target test cases that match the test case identifier according to the test case identifier, and to load target test sets that match the test set identifier according to the test set identifier.

[0013] The sending module is used to send test data from the target test set to a first container of a service in at least one deployment server, based on the target test case.

[0014] The second acquisition module is used to acquire the test results obtained by using the test data to perform a target test on the services in the at least one first container that matches the task to be tested.

[0015] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the test method proposed in one aspect of this disclosure.

[0019] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided for computer instructions used to cause the computer to perform the test methods proposed in the foregoing aspect of this disclosure.

[0020] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the test method proposed in the above aspect of this disclosure.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a schematic diagram of the server-side testing framework;

[0024] Figure 2 A diagram illustrating the various stages of server-side testing;

[0025] Figure 3 This is a schematic diagram of an L2-level automated testing architecture;

[0026] Figure 4 This is a flowchart illustrating the testing method provided in Embodiment 1 of this disclosure;

[0027] Figure 5This is a schematic flowchart of the testing method provided in Embodiment 2 of this disclosure;

[0028] Figure 6 This is a flowchart illustrating the testing method provided in Embodiment 3 of this disclosure;

[0029] Figure 7 This is a flowchart illustrating the testing method provided in Embodiment 4 of this disclosure;

[0030] Figure 8 This is a flowchart illustrating the testing method provided in Embodiment 5 of this disclosure;

[0031] Figure 9 This is a flowchart illustrating the testing method provided in Embodiment Six of this disclosure;

[0032] Figure 10 This is a schematic flowchart of the testing method provided in Embodiment 7 of this disclosure;

[0033] Figure 11 This is a schematic diagram of the overall process of server-side automated testing provided in the embodiments of this disclosure;

[0034] Figure 12 This is a schematic diagram of the execution flow of the scheduling module provided in an embodiment of this disclosure;

[0035] Figure 13 This is a schematic diagram of the execution flow of the testing tool provided in the embodiments of this disclosure;

[0036] Figure 14 This is a schematic diagram of the overall architecture of the automated testing platform provided in the embodiments of this disclosure;

[0037] Figure 15 This is a schematic diagram of the server-side automated testing framework provided in the embodiments of this disclosure;

[0038] Figure 16 This is a schematic diagram of the test apparatus provided in Embodiment 8 of this disclosure;

[0039] Figure 17 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0040] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0041] Server-side testing frameworks can be like Figure 1 As shown, the first step is to set up a server-side test environment and construct test data. Then, a test script or agent (intelligent agent, a software or hardware entity that can act autonomously) simulates a user to read the test data and initiates test requests to the service in the test environment. After that, by parsing the results returned by the service, specific metrics are calculated and a test report is generated.

[0042] The test scenarios typically include the following three types:

[0043] The first type is functional testing: triggering a function specified by the server by constructing specific parameters and data, and verifying the returned results.

[0044] The second method is performance testing: using test sets to test the server's performance under different concurrency levels.

[0045] The third method is stress testing: testing the server's resilience by continuously sending test data from the test set with high concurrency.

[0046] based on Figure 1 The test framework shown includes server-side test automation. Figure 2 The five stages shown:

[0047] Phase 1: L0 (Manual Testing). No pipeline; the test environment is set up manually by using scripts to copy online machine resources for deployment, and test scripts are executed locally.

[0048] Phase 2: L1 (Pipeline Assistance). Single-module pipeline, agent self-maintenance, scheduling and distribution self-maintenance.

[0049] Phase 3: L2 (partially automated). Multi-module integrated pipeline, agents use unified image files, scheduling and distribution are self-maintained.

[0050] Phase 4: L3 (Limited Automation). A service-oriented testing pipeline with agents deployed as services. These agents are uniformly scheduled through an independent scheduling module and manually triggered to complete the automated testing process, from test environment setup and tool scheduling to test execution and metric calculation, achieving limited automation.

[0051] Phase 5: L4 (Fully Automated). An engineered pipeline that automatically triggers and executes tests without human intervention.

[0052] Currently, server-side testing falls under Level 2 automated testing. Level 2 automated testing includes... Figure 3 As shown, a multi-module integrated pipeline is used for testing. The overall testing process includes:

[0053] 1) Configure the server-side architecture topology and request container resources for each module (manually);

[0054] 2) Configure the branch, version, and other information of the test module (manual);

[0055] 3) Configure the necessary information for each stage of the pipeline according to testing needs (including test case selection, machine path of test set, etc.) (manual);

[0056] 4) Start the pipeline to perform tests (automatic).

[0057] The execution process at each stage within the pipeline includes:

[0058] 1. Environment Setup: Request the test environment platform to deploy resources on the pre-approved test environment by using custom parameters.

[0059] 2. Functional Testing: Pull the latest branch version of the testing tool code repository, read the user configuration parameters, select the test cases and test set list (including index (e.g., ID) and storage path) pre-stored in the testing tool, pull the test data (including audio, text, etc.) from the test set list, execute the test data locally on the resource container provided by the pipeline through the script, verify the test results, and generate a report.

[0060] 3. Performance Testing: Pull the latest branch version of the testing tool code repository, read the user configuration parameters, select the test cases and test sets list pre-stored in the testing tool, pull the test data (including audio, text, etc.) in the test sets, execute the script locally on the resource container provided by the pipeline, calculate performance indicators, and generate reports.

[0061] 4. Stress Testing: Pull the latest branch version of the testing tool code repository, read the user configuration parameters, select the test cases and test sets list pre-stored in the testing tool, pull the test data (including audio, text, etc.) in the test sets, and execute the script locally on the resource container provided by the pipeline. According to the user configuration parameters, specify the number of rounds to send the test sets or send them continuously.

[0062] However, L2-level automated testing pipeline configurations are highly specific, with 10 pipelines corresponding to 10 business functions. Scheduling and distribution are handled by operations and maintenance personnel, and the scheduling strategies for each direction vary greatly, making it difficult for users to operate. Furthermore, the testing tools are not service-oriented, and the modules are highly coupled. That is, the testing tools integrate test sets, test cases, test execution, and metric calculation into one, resulting in poor scalability. Modifying the configuration requires resubmitting the testing tool codebase, making it unable to cope with increasingly complex and changing test cases, data, and scenarios.

[0063] In addition, the following problems exist: 1) The work in each stage of the pipeline is highly repetitive, and the build execution takes a long time; 2) There is no project process control, the data analysis between test versions is difficult, the testing threshold is high, and the pipeline test execution failure rate is high; 3) The collection and calculation of indicators are cumbersome, the test results are not persisted, and the test data analysis is difficult; 4) The scalability is poor, and it is difficult to integrate with other businesses; 5) There are problems such as inconsistent versions of test tools, multiple copies of test data, and many parameter configurations.

[0064] Therefore, in view of at least one of the above-mentioned problems, this disclosure provides a test method, apparatus, electronic device and storage medium.

[0065] The test methods, apparatus, electronic devices, and storage media of embodiments of this disclosure are described below with reference to the accompanying drawings.

[0066] Figure 4 This is a flowchart illustrating the testing method provided in Embodiment 1 of this disclosure.

[0067] The testing method disclosed in this embodiment can be applied to the server side.

[0068] like Figure 4 As shown, the testing method may include the following steps:

[0069] Step 401: Obtain the task to be tested, and obtain the task parameters associated with the task identifier based on the task identifier of the task to be tested. The task parameters include the test case identifier and the test set identifier.

[0070] In this embodiment of the disclosure, the task to be tested can be created by relevant personnel, and the task identifier of the task to be tested is used to uniquely identify the task to be tested. For example, the task identifier can be a task ID.

[0071] In this embodiment, task parameters associated with the task identifier can be obtained or loaded based on the task identifier of the task to be tested. For example, when creating a task to be tested, relevant personnel can set the task parameters based on their experience. Alternatively, task parameters corresponding to different test tasks can be pre-configured using a configuration template and stored on a cloud platform. Thus, in this disclosure, task parameters corresponding to the task to be tested can be loaded from the cloud platform. The task parameters may include test case identifiers (e.g., test case ID) and test set identifiers (e.g., test set ID).

[0072] It should be noted that task parameters can include not only test case identifiers and test set identifiers, but also other parameters related to the task under test. For example, when the test type of the task under test is stress testing, the task parameters can also include concurrency, i.e., the number of test data to be sent subsequently, the upper limit of the concurrency setting, and other parameters.

[0073] Step 402: Load the target test case that matches the test case identifier based on the test case identifier.

[0074] In this embodiment of the disclosure, a target test case matching the test case identifier can be loaded based on the test case identifier.

[0075] As an example, the correspondence between test case identifiers and test cases can be stored on the cloud platform. Thus, in this disclosure, the above correspondence can be queried based on the test case identifier to obtain the test case corresponding to the test case identifier, which is referred to as the target test case in this disclosure.

[0076] Step 403: Load the target test set that matches the test set identifier.

[0077] In this embodiment of the disclosure, a target test set matching the test set identifier can be loaded according to the test set identifier.

[0078] As an example, the correspondence between test set identifiers and test sets can be stored on the cloud platform. Thus, in this disclosure, the above correspondence can be queried based on the test set identifier in the task parameters to obtain the test set corresponding to the test set identifier, which is denoted as the target test set in this disclosure.

[0079] The target test set may include multiple test data sets, which may include, but are not limited to, audio, video, text, and images.

[0080] Step 404: Based on the target test cases, send the test data in the target test set to the first container of the service in at least one deployment server.

[0081] In the embodiments of this disclosure, resources and services in the server can be deployed in at least one container (referred to as the first container in this disclosure). The resources may include CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), memory, etc., and the services may include transmission services, speech recognition services (or speech recognition engine), speech synthesis services, etc.

[0082] In this embodiment of the disclosure, test data from the target test set can be sent to a first container of a service in at least one deployment server, based on the target test case.

[0083] Step 405: Obtain the test results obtained by performing target tests on at least one service in the first container using test data, which are matched with the task to be tested.

[0084] The target test is determined based on the test type of the task to be tested. The test type may include, but is not limited to, performance testing, stress testing, functional testing, exception testing, and stability testing.

[0085] For example, the test type is performance testing and the target test is performance testing; another example is stress testing and the target test is stress testing; yet another example is functional testing and the target test is stress testing.

[0086] In this embodiment of the disclosure, test data can be used to perform target tests on services in at least one first container that match the task to be tested, in order to obtain test results. Thus, by performing target tests on services in at least one first container, simulated testing of the server can be achieved, thereby determining the server's performance, stress resistance, etc., based on the test results.

[0087] The testing method of this disclosure involves: obtaining task parameters associated with the task identifier based on the task identifier of the task to be tested; loading target test cases based on test case identifiers in the task parameters; loading target test sets based on test set identifiers in the task parameters; sending test data from the target test set to at least one first container of a service deployed on a server based on the target test cases; and obtaining test results obtained by performing target tests matching the task to be tested on the services in at least one first container using the test data. In this disclosure, since the test sets and test cases corresponding to different test tasks can be different, when performing target tests matching the task to be tested on the server, obtaining the target test set and target test cases matching the task to be tested, and performing target tests on the services deployed on the server in the first container based on the target test cases and test sets, can achieve targeted and reliable testing. Therefore, based on the test results, the performance, stress resistance, etc., of the server can be accurately and reliably determined.

[0088] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0089] To clearly illustrate how test data from the target test set is sent to the first container of a service in at least one deployment server based on the target test cases in the above embodiments, this disclosure also proposes a testing method.

[0090] Figure 5This is a flowchart illustrating the testing method provided in Embodiment 2 of this disclosure.

[0091] like Figure 5 As shown, the testing method may include the following steps:

[0092] Step 501: Obtain the task to be tested, and obtain the task parameters associated with the task identifier based on the task identifier of the task to be tested. The task parameters include the test case identifier and the test set identifier.

[0093] Step 502: Load the target test case that matches the test case identifier based on the test case identifier.

[0094] Step 503: Load the target test set that matches the test set identifier.

[0095] The explanation of steps 501 to 503 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0096] Step 504: Determine the target testing tool that matches the task type based on the task type to be tested.

[0097] In this disclosed embodiment, the task type and the target testing tool matching the task type can be different depending on the task to be tested. For example, when the task to be tested is a speech processing test task (such as speech recognition (Automatic Speech Recognition, or ASR) or speech synthesis test task), the target testing tool can be a speech processing test tool (such as a speech recognition test tool or a speech synthesis test tool). As another example, when the task to be tested is a Redis (Remote Dictionary Server) test task, the target testing tool can be a Redis test tool. And as yet another example, when the task to be tested is a link test (such as lc (long...)...)... When performing connection (long connection) link testing tasks, the target testing tool can be an lc link testing tool. Similarly, when the task to be tested is an interface testing task, the target testing tool can be an interface testing tool. Furthermore, when the task to be tested is an audio / video link testing task (for example, when client A initiates a voice or video call to client B, the audio and video data of client A and client B can be forwarded by the server; in the voice or video call scenario, the test data can be audio and video data, which can be used to perform audio / video link testing on the server), the target testing tool can be an audio / video link testing tool, and so on. These are just a few examples; they are not all listed here.

[0098] In this embodiment of the disclosure, the correspondence between different task types and testing tools can be pre-configured. Thus, in this disclosure, after determining the task type of the task to be tested, the above-mentioned correspondence can be queried according to the task type of the task to be tested in order to determine the target testing tool that matches the task type of the task to be tested.

[0099] Continuing with the example above, assume the voice processing test task has a task type of type 1, which corresponds to the voice processing test tool. Similarly, assume the Redis test task has a task type of type 2, which corresponds to the Redis test tool. If the task to be tested has a task type of type 1, then the target test tool can be determined to be the voice processing test tool.

[0100] Step 505: Determine the second container from the candidate containers based on the running status of the candidate containers for deploying the target testing tool.

[0101] The running status can include idle status and occupied status.

[0102] In this embodiment of the disclosure, the number of candidate containers for deploying the target testing tool can be at least one. A second container can be determined from each candidate container based on the running status of each candidate container for deploying the target testing tool, wherein the running status of the second container is an idle state.

[0103] In one possible implementation of this disclosure, at least one third container can be determined from each candidate container based on the running status of each candidate container of the deployment target testing tool, wherein the running status of the third container is idle. Then, a second container can be determined from each third container based on the test type of the task to be tested.

[0104] The test types can include functional testing, stress testing, and performance testing.

[0105] For example, in a stress testing scenario, test data needs to be sent continuously with a large number of concurrent requests (e.g., the target testing tool needs to simulate 100 or 200 clients). In this case, the second container needs to have enough space to store the log data corresponding to the target testing tool. Therefore, a third container with larger resources can be selected as the second container.

[0106] For performance testing and functional testing, a container can be randomly selected from the third container as the second container, or a third container with larger resources can be selected as the second container.

[0107] In another possible implementation of the present disclosure, the task to be tested can be associated with at least one task that has been tested. A third container can be determined from each candidate container based on the running status of each candidate container of the target testing tool, wherein the running status of the third container is idle. Then, the container called by the aforementioned task that has been tested can be determined from each third container and used as the second container.

[0108] For example, test tasks created by the same person can be associated with the same project. Test tasks under the same project are related and can be scheduled to the same container. Thus, after a test task in the project completes its test, the container called by the completed test task can be invoked for the uncompleted test task.

[0109] This allows for the determination of the containers for deploying the required testing tools based on different methods, thereby improving the flexibility and applicability of the approach.

[0110] Step 506: Invoke the target testing tool in the second container to construct a test scenario that matches the test type of the task to be tested based on the target test cases.

[0111] In this embodiment of the disclosure, the target testing tool in the second container can be invoked to construct a test scenario that matches the test type of the task to be tested, based on the target test cases.

[0112] As an example, a target test case can include two parts: one part is configuration parameters used to build the test scenario, and the other part is code logic used to compare the differences between the test results and the annotation information on the test data.

[0113] For example, when the test task is a speech recognition test task, the test result may include at least one service in the first container performing speech recognition on the test data, and the annotation information may be the text information annotated on the test data. The code logic is used to compare the differences between the speech recognition result and the text information.

[0114] In this disclosure, configuration parameters can be obtained from the target test case, and the target test tool in the second container can be invoked to construct a test scenario that matches the test type of the task to be tested based on the configuration parameters.

[0115] The configuration parameters differ for different test types. For example, the configuration parameters in the target test case corresponding to the test type of the task to be tested, which is stress testing, can be different from the configuration parameters in the target test case corresponding to the test type of the task to be tested, which is performance testing.

[0116] For example, for stress testing, which requires continuous and uninterrupted transmission of test data, a test scenario for continuously transmitting test data can be constructed based on the configuration parameters in the target test case. For functional testing and performance testing, a test scenario for short-term transmission of test data can be constructed based on the configuration parameters in the target test case.

[0117] For example, using test data as audio, in a speech recognition scenario, if continuous speech recognition is required after a single wake-up, a scenario for continuous audio transmission can be constructed based on the configuration parameters in the target test case.

[0118] In summary, it is possible to effectively construct test scenarios based on the configuration parameters in the target test cases, thereby enabling effective functional, performance, or stress testing of the server.

[0119] Step 507: In the test scenario, the target test tool in the second container is invoked to send test data from the target test set to at least one first container.

[0120] In this embodiment of the disclosure, a target testing tool in a second container can be invoked during a test scenario to send test data from a target test set to at least one first container. In other words, this disclosure allows the target testing tool to simulate client operations and send test data from the target test set to the first container deploying the service on the server, thereby enabling simulated testing of the server.

[0121] The client refers to software programs that run on electronic devices and provide services to users.

[0122] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0123] Step 508: Obtain the test results obtained by performing target tests on at least one service in the first container using test data, which are matched with the task to be tested.

[0124] The explanation of step 508 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0125] The testing method of this disclosure can decouple test cases and test sets from testing tools. When dealing with increasingly complex and changing test cases, data and scenarios, only test cases and test sets need to be updated independently, without modifying the configuration of testing tools. This can reduce the difficulty of iteration and improve the applicability and scalability of the method.

[0126] To clearly illustrate how test results are obtained in any embodiment of this disclosure, a test method is also proposed.

[0127] Figure 6 This is a flowchart illustrating the testing method provided in Embodiment 3 of this disclosure.

[0128] like Figure 6 As shown, the testing method may include the following steps:

[0129] Step 601: Obtain the task to be tested, and obtain the task parameters associated with the task identifier based on the task identifier of the task to be tested. The task parameters include the test case identifier and the test set identifier.

[0130] Step 602: Load the target test case that matches the test case identifier based on the test case identifier.

[0131] Step 603: Load the target test set that matches the test set identifier.

[0132] Step 604: Based on the target test cases, send the test data in the target test set to the first container of the service in at least one deployment server.

[0133] The explanation of steps 601 to 604 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0134] Step 605: Receive a response result sent by at least one first container, wherein the response result is generated by a service in at least one first container in response to the test data.

[0135] In this embodiment of the disclosure, at least one service in the first container can generate a response result in response to the test data.

[0136] As an example, when the task to be recognized is a speech recognition test task, the test data can be audio, and at least one service in the first container can perform speech recognition on the test data to obtain a response result (i.e., a speech recognition result).

[0137] As another example, when the task to be recognized is a speech synthesis task, the test data can be text, and at least one service in the first container can perform speech synthesis on the test data to obtain a response result (i.e., the speech synthesis result).

[0138] In one possible implementation of this disclosure, a second container can receive response results sent by at least one first container. That is, after a service in at least one first container generates a response result, it can send the response result to the second container, so that the server can receive the response result sent by at least one first container through the second container.

[0139] Step 606: During the process of testing service response data in at least one first container, the operation of at least one first container is monitored to obtain monitoring results.

[0140] In this embodiment of the disclosure, during the process of testing service response data in at least one first container, the server can monitor the operation of at least one first container to obtain monitoring results.

[0141] The monitoring results may include monitoring information such as the resources (e.g., CPU, GPU, memory, etc.) used by the service response test data in at least one first container, and the functions (e.g., phone call function, voice call function, search function, etc.) called by the service response test data in at least one first container.

[0142] Step 607: Use the response results and monitoring results as the test results.

[0143] In this embodiment of the disclosure, the response results and monitoring results can be used as test results, thereby verifying the server's performance, stress resistance, etc. based on the test results.

[0144] The testing method of this disclosure includes not only the response results generated by the service in the first container in response to the test data, but also the monitoring results obtained by monitoring the operation of the first container. This can improve the richness of the test results, thereby determining the performance, stress resistance, etc. of the server based on the test results, and improving the accuracy and reliability of the determination results.

[0145] In one possible implementation of this disclosure, when the test type of the task to be tested is a performance test, the test results can be parsed to determine the performance of the server. The above process will be described in detail below with reference to Embodiment 4.

[0146] Figure 7 This is a flowchart illustrating the testing method provided in Embodiment 4 of this disclosure.

[0147] like Figure 7 As shown, the testing method may include the following steps:

[0148] Step 701: Obtain the task to be tested, and obtain the task parameters associated with the task identifier based on the task identifier of the task to be tested. The task parameters include the test case identifier and the test set identifier.

[0149] Step 702: Load the target test case that matches the test case identifier according to the test case identifier, and load the target test set that matches the test set identifier according to the test set identifier.

[0150] Step 703: Determine the target testing tool that matches the task type based on the task type to be tested.

[0151] Step 704: Determine the second container from the candidate containers based on the running status of the candidate containers for deploying the target testing tool.

[0152] Step 705: Invoke the target testing tool in the second container to construct a test scenario that matches the test type of the task to be tested based on the target test cases.

[0153] The test type can be the first phase of performance testing.

[0154] Step 706: In the test scenario, the target test tool in the second container is invoked to send test data from the target test set to at least one first container.

[0155] The explanation of steps 701 to 706 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0156] Step 707: Receive the response result of the first stage sent by at least one first container through the second container.

[0157] The response result of the first phase is generated by at least one service in the first container performing the first phase processing on the test data.

[0158] In this embodiment of the disclosure, at least one service in a first container can perform a first-stage processing on the test data to generate a first-stage response result.

[0159] For example, if the test data is audio, the first stage could be the speech recognition stage; if the test data is text, the first stage could be the speech synthesis stage.

[0160] Step 708: During the first phase of processing of test data by the service in at least one first container, the operation of at least one first container is monitored to obtain the monitoring results of the first phase.

[0161] In this embodiment of the disclosure, during the first-stage processing of test data by a service in at least one first container, the server can monitor the operation of at least one first container to obtain the monitoring results of the first stage.

[0162] The monitoring results of the first phase may include monitoring information such as the resources used by at least one service in the first container when processing the test data in the first phase, and the functions called by at least one service in the first container when processing the test data in the first phase (such as phone call function, voice call function, search function, etc.).

[0163] Step 709: Generate test results based on the response results and monitoring results of the first phase.

[0164] In this embodiment of the disclosure, test results can be generated based on the response results and monitoring results of the first stage.

[0165] As an example, when at least one service in the first container is only used for the first phase of processing test data, the response results and monitoring results of the first phase can be used as the test results.

[0166] When a service in at least one first container is used not only for the first stage of test data processing but also for other stages of test data processing, the response results of other stages sent by at least one first container can be obtained through a second container. These response results are generated by the services in at least one first container performing other stages of test data processing. Furthermore, during the process of services in at least one first container performing other stages of test data processing, the operation of at least one first container can be monitored to obtain the monitoring results of other stages. Therefore, the response results and monitoring results corresponding to the first stage, as well as the response results and monitoring results corresponding to other stages, can be combined as the test results.

[0167] Step 710: Invoke the target test tool in the second container to execute the target test case to determine the first difference between the response result of the first stage in the test results and the expected result of the first stage marked on the test data.

[0168] In this embodiment of the disclosure, the target testing tool in the second container can be invoked to execute the code logic in the target test case to determine the first difference between the response result of the first stage and the expected result of the first stage marked on the test data.

[0169] For example, taking the test task as a speech recognition test task, the response result of the first stage may include at least one service in the first container performing speech recognition on the test data, and the expected result of the first stage may be the text information annotated on the test data (i.e., audio).

[0170] Step 711: Based on the first difference, determine the first test index corresponding to the performance test in the first stage.

[0171] In this embodiment of the disclosure, the first test index is negatively correlated with the first difference, that is, the smaller the first difference, the larger the first test index, and vice versa.

[0172] Step 712: Analyze the monitoring results of the first phase in the test results to determine the resources occupied by the service response test data in at least one first container.

[0173] In this embodiment of the disclosure, the monitoring results of the first phase can be parsed to obtain the resources occupied by the service response test data in at least one first container.

[0174] Step 713: Determine the second test metric corresponding to the performance test of the first phase based on the resources occupied by the services in at least one first container.

[0175] In this embodiment of the disclosure, a second test metric corresponding to the performance test in the first phase can be determined based on the resources occupied by the services in at least one first container. The second test metric is negatively correlated with the amount of resources occupied; that is, the smaller the amount of resources occupied, the larger the second test metric, and vice versa.

[0176] Step 714: Determine the server's performance in the first phase based on the first test metric and the second test metric.

[0177] In this embodiment of the disclosure, the performance of the server in the first stage can be determined based on the first test index and the second test index.

[0178] For example, the larger the first test metric, the better the server's performance in the first stage; conversely, the smaller the first test metric, the worse the server's performance in the first stage. Similarly, the larger the second test metric, the better the server's performance in the first stage; conversely, the smaller the second test metric, the worse the server's performance in the first stage.

[0179] The testing method of this disclosure decouples test execution and metric calculation. When dealing with increasingly complex and changing test cases, data and scenarios, only test cases and test sets need to be updated independently without modifying the test tools. This improves the applicability and scalability of the method.

[0180] To clearly illustrate how the performance of the server in the first stage is determined in any embodiment of this disclosure, a testing method is also proposed.

[0181] Figure 8 This is a flowchart illustrating the testing method provided in Embodiment 5 of this disclosure.

[0182] like Figure 8 As shown, the testing method may include the following steps:

[0183] Step 801: Obtain the task to be tested, and obtain the task parameters associated with the task identifier based on the task identifier of the task to be tested. The task parameters include the test case identifier and the test set identifier.

[0184] Step 802: Load the target test case that matches the test case identifier according to the test case identifier, and load the target test set that matches the test set identifier according to the test set identifier.

[0185] Step 803: Determine the target testing tool that matches the task type based on the task type to be tested.

[0186] Step 804: Determine the second container from the candidate containers based on the running status of the candidate containers for deploying the target testing tool.

[0187] Step 805: Invoke the target testing tool in the second container to construct a test scenario that matches the test type of the task to be tested based on the target test cases.

[0188] Step 806: In the test scenario, the target test tool in the second container is invoked to send test data from the target test set to at least one first container.

[0189] Step 807: Receive the response result of the first stage sent by at least one first container through the second container.

[0190] The response result of the first phase is generated by at least one service in the first container performing the first phase processing on the test data.

[0191] Step 808: During the first phase of processing of test data by the service in at least one first container, the operation of at least one first container is monitored to obtain the monitoring results of the first phase.

[0192] Step 809: Generate test results based on the response results and monitoring results of the first phase.

[0193] Step 810: Invoke the target test tool in the second container to execute the target test case to determine the first difference between the response result of the first phase and the expected result of the first phase marked on the test data.

[0194] Step 811: Based on the first difference, determine the first test index corresponding to the performance test in the first stage.

[0195] Step 812: Analyze the monitoring results of the first phase to determine the resources consumed by the service response test data in at least one first container.

[0196] Step 813: Determine the second test metric corresponding to the performance test of the first phase based on the resources occupied by the services in at least one first container.

[0197] The explanation of steps 801 to 813 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0198] Step 814: Determine the first packet response time, total execution time, and hard latency of the test data.

[0199] In this embodiment, the first packet response time is the duration between a first moment and a second moment. The first moment is the time when the target testing tool in the second container sends the first data packet from the test data, and the second moment is the time when the second container receives the first data packet from the response result of the first stage. The size of each data packet can be fixed and may include one frame of data or multiple frames of data; this disclosure does not impose any limitations on this.

[0200] In other words, the first packet response time is the time interval between the second container sending the first data packet and receiving the first data packet.

[0201] In this embodiment of the disclosure, the total execution time is the time between the first moment and the third moment, wherein the third moment is the moment when the second container receives the last data packet in the response result of the first stage.

[0202] In other words, the total execution time is the time interval between the second container sending the first data packet and receiving the last data packet in the response result of the first stage.

[0203] In this embodiment of the disclosure, the hard delay is the duration between the fourth time point and the third time point, wherein the fourth time point is the time when the target test tool in the second container sends the last data packet in the test data.

[0204] In other words, hard latency is the time interval between the second container sending the last data packet and receiving the last data packet in the response result of the first phase.

[0205] Step 815: Determine the third test metric corresponding to the performance test in the first phase based on at least one of the following: first frame response time, total execution time, and hard latency of the test data.

[0206] In this embodiment of the disclosure, a third test metric corresponding to the performance test can be determined based on at least one of the first frame response time, total execution time, and hard latency of the test data.

[0207] Among them, the third test index is negatively correlated with the first frame response time. That is, the shorter the first frame response time, the larger the third test index, and vice versa.

[0208] Among them, the third test metric is negatively correlated with the total execution time. That is, the shorter the total execution time, the larger the third test metric, and vice versa.

[0209] Among them, the third test metric is negatively correlated with hard latency; that is, the shorter the hard latency, the larger the third test metric, and vice versa.

[0210] Step 816: Determine the server's performance in the first phase based on the first test metric, the second test metric, and the third test metric.

[0211] In this embodiment of the disclosure, the performance of the server in the first stage can be determined based on the first test index, the second test index, and the third test index.

[0212] For example, the larger the first test metric, the better the server's performance in the first stage; conversely, the smaller the first test metric, the worse the server's performance in the first stage. Similarly, the larger the second test metric, the better the server's performance in the first stage; and the smaller the second test metric, the worse the server's performance in the first stage.

[0213] The testing method of this disclosure improves the accuracy and reliability of the determination results by determining the performance of the server based on multiple test indicators.

[0214] In one possible implementation of this disclosure, when the test type of the task to be tested is a performance test, the server performance can be determined based on the response results and monitoring results. The process will be described in detail below with reference to Embodiment Six.

[0215] Figure 9 This is a flowchart illustrating the testing method provided in Embodiment Six of this disclosure.

[0216] like Figure 9 As shown, the testing method may include the following steps:

[0217] Step 901: Obtain the task to be tested, and obtain the task parameters associated with the task identifier based on the task identifier of the task to be tested. The task parameters include the test case identifier and the test set identifier.

[0218] Step 902: Based on the test case identifier, load the target test case that matches the test case identifier, and based on the test set identifier, load the target test set that matches the test set identifier.

[0219] Step 903: Determine the target testing tool that matches the task type based on the task type to be tested.

[0220] Step 904: Determine the second container from the candidate containers based on the running status of the candidate containers for deploying the target testing tool.

[0221] Step 905: Invoke the target testing tool in the second container to construct a test scenario that matches the test type of the task to be tested based on the target test cases.

[0222] The test type can be a second-stage performance test, which can also be called a functional test.

[0223] Step 906: In the test scenario, the target test tool in the second container is invoked to send test data from the target test set to at least one first container.

[0224] The explanation of steps 901 to 906 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0225] Step 907: Receive the second-stage response result sent by at least one of the first containers through the second container.

[0226] The response result in the second phase is generated by at least one service in the first container performing second-phase processing on the test data.

[0227] In this embodiment of the disclosure, at least one service in the first container can perform a second-stage processing on the test data to generate a second-stage response result.

[0228] For example, taking test data as audio as an example, the first stage in the above embodiment can be a speech recognition stage, and the second stage can be a stage for further processing based on the speech recognition results, such as a phone call stage, a search stage, etc.

[0229] For example, assuming the test data (audio) is "call Zhang San", in the first stage, the audio can be recognized by the service in the first container based on speech recognition technology to obtain the speech recognition result. For example, the response result in the first stage can be the speech recognition result. In the second stage, the call function can be invoked based on the above speech recognition result to make a call to Zhang San. For example, the response result in the second stage can be "call to Zhang San" or "call to Zhang San".

[0230] For example, assuming the test data (audio) is "searching for popular TV series", in the first stage, the audio can be recognized by the service in the first container based on speech recognition technology to obtain the speech recognition result. For example, the response result in the first stage can be the speech recognition result. In the second stage, the search function can be invoked based on the above speech recognition result to search for popular TV series. For example, the response result in the second stage can be the search result or "popular TV series searched".

[0231] Step 908: During the second phase of the test data processing by the service in at least one first container, the operation of at least one first container is monitored to obtain the monitoring results of the second phase.

[0232] In this embodiment of the disclosure, during the second-stage processing of test data by a service in at least one first container, the server can monitor the operation of at least one first container to obtain the monitoring results of the second stage.

[0233] Step 909: Invoke the target test tool in the second container to execute the target test case to determine the second difference between the second phase response result and the expected result of the second phase marked on the test data.

[0234] In this embodiment of the disclosure, the target testing tool in the second container can be invoked to execute the code logic in the target test case to determine a second difference between the response result of the second phase and the expected result of the second phase marked on the test data.

[0235] For example, the expected result of the second stage is "Zhang San has been called", while the response result of the second stage is "Zhang San has been called". Obviously, the second stage is quite different because the calling objects are different.

[0236] Step 910: Analyze the monitoring results of the second phase to determine the first function invoked by the service response test data in at least one first container.

[0237] In this embodiment of the disclosure, the monitoring results of the second phase can be parsed to obtain the first function called by the service response test data in at least one first container.

[0238] Step 911: Determine the third difference between the first function and the second function of the second phase marked on the test data.

[0239] In this embodiment of the disclosure, the test data may be marked with the second function to be called in the second stage. For example, assuming the test data (audio) is "call Zhang San", the second function marked on the test data in the second stage is "call function".

[0240] In this embodiment of the disclosure, a third difference can be determined between the first function and the second function of the second stage marked on the test data.

[0241] Step 912: Determine the server's performance in the second phase based on the second and / or third differences.

[0242] In this embodiment of the disclosure, the server's performance in the second phase can be determined based on the second difference and / or the third difference.

[0243] For example, the smaller the second difference, the better the server's performance in the second stage; conversely, the larger the second difference, the worse the server's performance in the second stage. Similarly, the smaller the third difference, the better the server's performance in the second stage; conversely, the larger the third difference, the worse the server's performance in the second stage.

[0244] As an example, the second phase of performance testing can be called functional testing. If the second difference is less than the set first difference threshold and / or the third difference is less than the set second difference threshold, the server is determined to have passed the functional test; if the second difference is greater than or equal to the first difference threshold and / or the third difference is greater than or equal to the second difference threshold, the server is determined to have failed the functional test.

[0245] The testing method of this disclosure improves the accuracy and reliability of the determination results by determining the performance of the server based on multiple data points.

[0246] In one possible implementation of this disclosure, when the test type is stress testing, the server can be determined to have passed the stress test based on the monitoring results. The process will be described in detail below with reference to Embodiment Seven.

[0247] Figure 10 This is a flowchart illustrating the testing method provided in Embodiment 7 of this disclosure.

[0248] Step 1001: Obtain the task to be tested, and obtain the task parameters associated with the task identifier based on the task identifier of the task to be tested. The task parameters include the test case identifier and the test set identifier.

[0249] Step 1002: Load the target test case that matches the test case identifier based on the test case identifier.

[0250] Step 1003: Load the target test set that matches the test set identifier.

[0251] Step 1004: Based on the target test cases, send the test data in the target test set to the first container of the service in at least one deployment server.

[0252] Step 1005: Obtain the test results obtained by performing target tests on at least one service in the first container using test data, which are matched with the task to be tested.

[0253] The test results include response results and monitoring results. The response results are generated by the service in at least one first container responding to the test data. The monitoring results are obtained by monitoring the operation of at least one first container during the process of the service in at least one first container responding to the test data.

[0254] The explanation of steps 1001 to 1005 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0255] Step 1006: Analyze the monitoring results to determine whether any service in at least one first container experienced a runtime anomaly while responding to test data.

[0256] In this embodiment of the disclosure, the monitoring results can be parsed to determine whether any service in at least one first container has experienced operational anomalies during the response to test data, such as determining whether any service in at least one first container has experienced anomalies such as freezing or crashing.

[0257] Among them, a service freeze can be determined by the resources it uses. For example, if the resources used by a service are 0, it can be determined that the service is frozen. A crash can be determined by the exception crash code.

[0258] Step 1007: In response to a service malfunction occurring in at least one of the first containers, determine that the server failed the stress test.

[0259] In this embodiment of the disclosure, when a service in at least one first container experiences an operational anomaly, it can be determined that the server has failed the stress test, that is, the server's stress resistance is low and does not meet expectations.

[0260] Step 1008: In response to the fact that no service in at least one first container has experienced any operational anomalies, determine that the server has passed the stress test.

[0261] In this embodiment of the disclosure, when no service in at least one first container experiences any operational anomalies, it can be determined that the server has passed the stress test, that is, the server's stress resistance is high and meets expectations.

[0262] In any embodiment of this disclosure, the test results can also be stored in correspondence with task identifiers. Therefore, by persistently storing the test results, subsequent data analysis can be facilitated, allowing for optimization of the server based on the analysis results, thereby improving server performance, resilience, and other aspects.

[0263] The testing method disclosed in this embodiment can not only perform performance and functional tests on the server, but also perform stress tests on the server, thereby improving the flexibility and applicability of the method.

[0264] In any embodiment of this disclosure, the automated testing of the server provided by this disclosure may include: test project management, automated test environment setup, unified scheduling of test tools, cloud storage and loading of test sets, test case hosting, test task execution, test environment indicator monitoring, persistent storage of test results, and test indicator calculation.

[0265] The overall execution process can be as follows: Figure 11 As shown, the process consists of two parts. The first part is test environment setup. Container resources refer to machine resources. Since server-side testing is interconnected, different functional services are deployed on different machines, and these services or machines are interconnected. Therefore, before conducting server-side simulation testing, when setting up the test environment, machines can be virtualized into at least one container, and different service resources can be deployed within the container.

[0266] Test version acquisition refers to obtaining the latest version of a service after applying for container resources, since services in the container have different versions, and then deploying the latest version of the service in the container.

[0267] Resource packaging refers to the process of packaging the latest version of the service, which is code provided by the developers, into a resource package and deploying it in a container.

[0268] Test environment deployment refers to the ability to control the unpacking of resource packages in containers and start services within those containers.

[0269] The second part is: testing the server.

[0270] Log information refers to the logs of the testing tool, which record information such as the tool's operational behavior and the data packets it receives. In other words, when performing simulated testing on the server, if an anomaly occurs, it could be due to a problem on the server side or the testing tool itself. In this case, the log information of the testing tool can be used to determine whether the anomaly is on the testing tool. If the testing tool is not experiencing an anomaly, then the anomaly is on the server side. If the testing tool is experiencing an anomaly, further investigation is needed to determine if the anomaly is on the server side.

[0271] Test environment container monitoring refers to monitoring all containers on the server side and obtaining the monitoring results.

[0272] It should be noted that, Figure 11The examples provided only illustrate the test types of functional testing, performance testing, and stress testing, but this disclosure is not limited to these. In practical applications, test types can also include exception testing, stability testing, and so on.

[0273] The execution flow of the scheduling module (share-agent) can be as follows: Figure 12 As shown. Specifically, when the test task is a speech recognition test task, a speech recognition testing tool can be used to test the server; when the test task is a Redis test task, a Redis testing tool can be used to test the server; when the test task is a link test (such as lc link test), an lc link test tool can be used to test the server; and when the test task is an audio / video link test task, an audio / video link test tool (such as a general (comm) test tool) can be used to test the server.

[0274] like Figure 12 As shown, the scheduling module mainly includes the following functions:

[0275] Parameter validation: Validation of task type, test type, task identifier, and query interface;

[0276] Container filtering (or instance filtering): Based on the task type, determine the test tool that matches the task type, obtain the idle container for deploying the test tool, verify the container status (idle status, occupied status), and distribute container requests.

[0277] Based on testing tools Figure 12 The speech recognition testing tool shown is used as an example. The execution flow of the testing tool can be as follows: Figure 13 As shown. Here, clientimpl: the tool interface implementation class, used to define the process and methods for each test type; AsrBase: the base class for speech recognition; AsrFunction: a subclass for speech recognition function testing, inheriting from AsrBase; AsrPerform: a subclass for speech recognition performance testing, inheriting from AsrBase; AsrStress: a subclass for speech recognition performance testing, inheriting from AsrBase; AsrCore: the process control class for streaming speech recognition; ApiCore: the process control class for interface-based speech recognition.

[0278] in, Figure 13 The AsrBase in the system is used to load task parameters (including test set identifier, test case identifier, concurrency (the number of clients that the test tool needs to simulate), and the configuration limit of the concurrency corresponding to the container) based on the task identifier (obtained from the platform), load test sets based on the test set identifier (obtained from the Redis cache), and load test cases based on the test case identifier (obtained from the test case hosting platform).

[0279] AsrCore is used to construct data sending queues, data receiving queues, and time-stamping for data interaction with the server. In other words, in a speech recognition scenario, the client sends streaming data to the server while simultaneously receiving the speech recognition results from the server. Therefore, in a test scenario, data sending and receiving queues can be constructed to send and receive streaming data.

[0280] As an example, the overall architecture of an automated testing platform can be as follows: Figure 14 As shown, the resource overhead metric refers to the resources consumed by the services in each container, obtained by monitoring each container. For example, the resource overhead metric can be a second test metric.

[0281] Role recognition accuracy refers to the accuracy of identifying different roles in test data (such as audio). For example, if the audio contains a dialogue between a doctor and a patient, identifying the roles in the test data can determine which parts of the audio belong to the doctor's dialogue and which parts belong to the patient's dialogue. Furthermore, the accuracy of role recognition can be determined by comparing the recognition results with the annotation information on the test data.

[0282] Confidence level refers to the confidence assessment of test data (such as audio) to determine whether the test data is a human-computer interaction dialogue or noise data (noise data does not require a response). Based on the assessment result and the annotation information on the test data, the confidence level of the test data is determined. For example, if the assessment result is that the test data is a human-computer interaction dialogue, but the annotation information indicates that the test data is noise data, the confidence level is low; if the assessment result is that the test data is a human-computer interaction dialogue, and the annotation information indicates that the test data is a human-computer interaction dialogue, the confidence level is high.

[0283] A spoken language index (SLI) is a measure of the standard of pronunciation in test data (such as audio). A first calculated value is obtained to indicate the standard of pronunciation. The SLI is then determined based on this first calculated value and a second calculated value (also indicated on the test data) that further indicates the standard of pronunciation. For example, the SLI can be determined based on the difference between the first and second calculated values, where the SLI is negatively correlated with this difference.

[0284] Other characteristic indicators refer to indicators other than those mentioned above, such as time consumption indicators, which are referred to as the third test indicator in this disclosure.

[0285] based on Figure 14 The architecture shown uses test data as an example to illustrate how an automated testing framework can be implemented. Figure 15As shown. Among them, business metric collection refers to the metrics returned by the server, such as role metrics; container resource monitoring and processing metrics refer to the metrics actively acquired, such as resource consumption metrics determined based on monitoring results; characteristic metrics can include metrics related to the link, such as QPS (queries per second).

[0286] It should be noted that, Figure 15 The examples provided only illustrate the test types of functional testing, performance testing, and stress testing, but this disclosure is not limited to these. In practical applications, test types can also include exception testing, stability testing, and so on.

[0287] Figure 15 In this context, "ws" refers to WebSocket (a protocol for full-duplex communication over a single TCP (Transmission Control Protocol) connection), and "qnet" refers to a weak network.

[0288] In summary, the testing method provided in this disclosure has the following advantages:

[0289] 1. The level of test automation can be upgraded from L2 (partial automation) to L3 (limited automation), and provide underlying services for the implementation of L4 (fully automated); for example, the level of automation can be upgraded for server-side testing of services such as speech recognition, speech synthesis, natural language processing and security.

[0290] 2. Multi-service support: Platform-based design enables rapid access for different services.

[0291] 3. Lower the testing threshold: Unified agent scheduling and version management, along with a platform-based presentation of agent testing functions, enable low-threshold service-oriented testing. This standardizes the testing process, significantly lowers the testing threshold, reduces operational costs, and improves usability.

[0292] 4. Fast data loading: Data is stored in the cloud and retrieved quickly through a two-level caching mechanism of Redis + cloud.

[0293] 5. Persistent Test Results: Test results are stored in a database for easy data analysis. The testing cycle is clearly defined, enabling control over the testing process and project. Persistent test results facilitate data analysis.

[0294] 6. Modular Design: In the original testing solution, the testing tool integrates test cases, test sets, and metric calculation into one unit, resulting in high coupling between modules and difficulty in iteration. In the solution provided in this disclosure, the testing tool is only responsible for executing tests, while the test sets and test cases are hosted by the platform, and the metric calculation is handled by the platform's metric calculation module, which significantly reduces coupling.

[0295] 7. The testing process is modularized, making it easy for businesses to integrate, the testing tools are lightweight, and the testing project is deployed to the cloud.

[0296] 8. Dynamic scheduling of test resources saves resource costs. For example, when the concurrency is low, a single second container can be used to test the server, while multiple second containers can be used collaboratively to test the server when the concurrency is high. Furthermore, when the service in the first container completes testing, some resources can be dynamically released. For instance, if the service does not receive test data for a period of time, resources can be released; or if resources in the second container are not used for a period of time, they can be released.

[0297] With the above Figures 4 to 10 Corresponding to the testing methods provided in the embodiments, this disclosure also provides a testing apparatus. Since the testing apparatus provided in the embodiments of this disclosure is similar to the one described above... Figures 4 to 10 The test methods provided in the embodiments correspond to the test apparatus provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0298] Figure 16 This is a schematic diagram of the test apparatus provided in Embodiment 8 of this disclosure.

[0299] like Figure 16 As shown, the testing device 1600 may include: a first acquisition module 1601, a loading module 1602, a sending module 1603, and a second acquisition module 1604.

[0300] The first acquisition module 1601 is used to acquire the task to be tested and, based on the task identifier of the task to be tested, acquire the task parameters associated with the task identifier, wherein the task parameters include the test case identifier and the test set identifier.

[0301] The loading module 1602 is used to load target test cases that match the test case identifier based on the test case identifier, and to load target test sets that match the test set identifier based on the test set identifier.

[0302] The sending module 1603 is used to send test data from the target test set to the first container of a service in at least one deployment server, based on the target test cases.

[0303] The second acquisition module 1604 is used to acquire test results obtained by performing target tests on at least one service in the first container using test data, which are matched with the task to be tested.

[0304] In one possible implementation of this disclosure, the sending module 1603 may include:

[0305] The first determining unit is used to determine the target testing tool that matches the task type based on the task type to be tested.

[0306] The second determining unit is used to determine the second container from the candidate containers based on the running status of the candidate containers for deploying the target testing tool.

[0307] The building unit is used to call the target testing tool in the second container to build a test scenario that matches the test type of the task to be tested, based on the target test cases.

[0308] The sending unit is used to, in a test scenario, invoke the target testing tool in the second container to send test data from the target test set to at least one first container.

[0309] In one possible implementation of this disclosure, the construction unit is specifically used for: obtaining configuration parameters from the target test case; and calling the target testing tool in the second container to construct a test scenario that matches the test type of the task to be tested based on the configuration parameters.

[0310] In one possible implementation of this disclosure, the second determining unit is specifically configured to: determine a third container from each candidate container based on the running status of the candidate containers for deploying the target testing tool, wherein the running status of the third container is an idle state; and determine a second container from each third container based on the test type.

[0311] In one possible implementation of this disclosure, the task to be tested is associated with at least one task that has been tested. The second determining unit is specifically configured to: determine a third container from each candidate container based on the running status of the candidate containers for deploying the target testing tool, wherein the running status of the third container is idle; and determine a second container called by the task that has been tested from each third container.

[0312] In one possible implementation of this disclosure, the second acquisition module 1604 is specifically configured to: receive a response result sent by at least one first container through a second container, wherein the response result is generated by a service in at least one first container responding to test data; monitor the operation of at least one first container during the process of the service in at least one first container responding to test data to obtain monitoring results; and use the response result and monitoring results as test results.

[0313] In one possible implementation of this disclosure, at least one service in a first container is used to perform a first-stage processing of test data, and the test type is a first-stage performance test. The test apparatus 1600 may further include:

[0314] The first execution module is used to call the target testing tool in the second container to execute the target test cases in order to determine the first difference between the response result of the first phase and the expected result of the first phase marked on the test data.

[0315] The first determining module is used to determine the first test index corresponding to the performance test in the first stage based on the first difference.

[0316] The first parsing module is used to parse the monitoring results of the first phase to determine the resources occupied by the service response test data in at least one first container.

[0317] The second determining module is used to determine the second test metric corresponding to the performance test in the first phase based on the resources occupied by the services in at least one first container.

[0318] The third determination module is used to determine the server's performance in the first phase based on the first test index and the second test index.

[0319] In one possible implementation of this disclosure, the third determining module is specifically used to: determine the first packet response time, total execution time, and hard latency of the test data; wherein, the first packet response time is the duration between the first moment and the second moment, the total execution time is the duration between the first moment and the third moment, and the hard latency is the duration between the fourth moment and the third moment; the first moment is the moment when the target testing tool in the second container sends the first data packet of the test data, the second moment is the moment when the second container receives the first data packet of the response result of the first stage, the third moment is the moment when the second container receives the last data packet of the response result of the first stage, and the fourth moment is the moment when the target testing tool in the second container sends the last data packet of the test data; determine the third test index corresponding to the performance test of the first stage based on at least one of the first frame response time, total execution time, and hard latency of the test data; and determine the performance of the server in the first stage based on the first test index, the second test index, and the third test index.

[0320] In one possible implementation of this disclosure, at least one service in a first container is used to perform a second-stage processing of test data, the test type being a second-stage performance test. The test apparatus 1600 may further include:

[0321] The second execution module is used to call the target testing tool in the second container to execute the target test cases in order to determine the second difference between the response results of the second phase and the expected results of the second phase marked on the test data.

[0322] The second parsing module is used to parse the monitoring results of the second phase to determine the first function invoked by at least one service response test data in the first container.

[0323] The fourth determination module is used to determine the third difference between the first function and the second function of the second stage marked on the test data.

[0324] The fifth determining module is used to determine the server's performance in the second phase based on the second difference and / or the third difference.

[0325] In one possible implementation of this disclosure, the test type is a stress test, and the test apparatus 1600 may further include:

[0326] The third parsing module is used to parse the monitoring results to determine whether at least one service in the first container has experienced a runtime anomaly while responding to test data.

[0327] The sixth determination module is used to determine that the server has failed the stress test in response to a service malfunction occurring in at least one of the first containers.

[0328] The seventh determination module is used to determine that the server has passed the stress test in response to the fact that no service in at least one of the first containers has experienced any operational anomalies.

[0329] In one possible implementation of this disclosure, the testing apparatus 1600 may further include:

[0330] The storage module is used to store the test results in correspondence with the task identifier.

[0331] The testing apparatus of this disclosure obtains task parameters associated with the task identifier based on the task identifier of the task to be tested; loads target test cases based on test case identifiers in the task parameters; loads target test sets based on test set identifiers in the task parameters; sends test data from the target test set to at least one first container of a service deployed on a server based on the target test cases; and obtains test results obtained by performing target tests matching the task to be tested on the services in at least one first container using the test data. In this disclosure, since the test sets and test cases corresponding to different test tasks can be different, when performing target tests matching the task to be tested on the server, obtaining the target test set and target test cases matching the task to be tested, and performing target tests on the services deployed on the server in the first container based on the target test cases and test sets, can achieve targeted and reliable testing. Therefore, based on the test results, the performance, stress resistance, etc., of the server can be accurately and reliably determined.

[0332] To implement the above embodiments, this disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the test method proposed in any of the above embodiments of this disclosure.

[0333] To implement the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the test method proposed in any of the above embodiments of this disclosure.

[0334] To implement the above embodiments, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the testing method proposed in any of the above embodiments of this disclosure.

[0335] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0336] Figure 17 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0337] like Figure 17 As shown, the electronic device 1700 includes a computing unit 1701, which can perform various appropriate actions and processes according to a computer program stored in ROM (Read-Only Memory) 1702 or loaded from storage unit 1708 into RAM (Random Access Memory) 1703. The RAM 1703 may also store various programs and data required for the operation of the electronic device 1700. The computing unit 1701, ROM 1702, and RAM 1703 are interconnected via bus 1704. An I / O (Input / Output) interface 1705 is also connected to bus 1704.

[0338] Multiple components in electronic device 1700 are connected to I / O interface 1705, including: input unit 1706, such as keyboard, mouse, etc.; output unit 1707, such as various types of displays, speakers, etc.; storage unit 1708, such as disk, optical disk, etc.; and communication unit 1709, such as network card, modem, wireless transceiver, etc. Communication unit 1709 allows electronic device 1700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0339] The computing unit 1701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1701 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 1701 performs the various methods and processes described above, such as the test methods described above. For example, in some embodiments, the test methods described above can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1700 via ROM 1702 and / or communication unit 1709. When the computer program is loaded into RAM 1703 and executed by the computing unit 1701, one or more steps of the test methods described above can be performed. Alternatively, in other embodiments, the computing unit 1701 may be configured to perform the above-described test methods by any other suitable means (e.g., by means of firmware).

[0340] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0341] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0342] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0343] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0344] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0345] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0346] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0347] Cloud computing refers to a technology system that provides access to a shared pool of physical or virtual resources via a network. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for applications such as artificial intelligence and blockchain, as well as for model training.

[0348] According to the technical solution of this disclosure, task parameters associated with the task identifier are obtained based on the task identifier of the task to be tested; target test cases are loaded based on the test case identifier in the task parameters; target test sets are loaded based on the test set identifier in the task parameters; test data in the target test set is sent to at least one first container of a service deployed on a server based on the target test cases; and test results are obtained by performing target tests on the services in at least one first container using the test data, matching the task to be tested. In this disclosure, since the test sets and test cases corresponding to different test tasks can be different, when performing target tests on the server matching the task to be tested, obtaining the target test set and target test cases matching the task to be tested, and performing target tests on the services in the server deployed in the first container based on the target test cases and test sets, can achieve targeted and reliable testing. Therefore, based on the test results, the performance, stress resistance, etc., of the server can be accurately and reliably determined.

[0349] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution proposed in this disclosure can be achieved, and this is not limited herein.

[0350] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A testing method, comprising: Obtain the task to be tested, and based on the task identifier of the task to be tested, obtain the task parameters associated with the task identifier; The task parameters include test case identifiers and test set identifiers; Based on the test case identifier, load the target test case that matches the test case identifier; Based on the test set identifier, load the target test set that matches the test set identifier; From the candidate containers that have deployed target testing tools that match the task type of the task to be tested, determine a second container that is in an idle state; The target testing tool in the second container is invoked to construct a test scenario that matches the test type of the task to be tested based on the target test case, and under the test scenario, the test data in the target test set is sent to the first container of the service in at least one deployment server. The test results are the response results generated by the service response in the first container received by the second container in response to the test data, and the monitoring results obtained by monitoring the operation of the first container. In response to the service performing a first-stage processing on the test data, and the test type being a first-stage performance test, the target test tool in the second container is invoked to execute the target test case to determine a first difference between the response result and the expected result corresponding to the first stage. Analyze the monitoring results of the first phase to determine the resources consumed by the service in response to the test data; Determine a first test metric and a second test metric for the performance test in the first phase; the first test metric is negatively correlated with the first difference; the second test metric is negatively correlated with the amount of resources consumed by the service. The performance of the server in the first stage is determined based on the first test metric and the second test metric.

2. The method according to claim 1, wherein, The step of invoking the target testing tool in the second container to construct a test scenario matching the test type of the task to be tested based on the target test case includes: Obtain configuration parameters from the target test case; The target testing tool in the second container is invoked to construct a test scenario that matches the test type of the task to be tested, based on the configuration parameters.

3. The method according to claim 1, wherein, The step of determining a second container in an idle state from candidate containers that have deployed target testing tools matching the task type of the task to be tested includes: Based on the running status of the candidate containers where the target testing tool is deployed, a third container is determined from each of the candidate containers, wherein the running status of the third container is an idle state; The second container is determined from each of the third containers according to the test type.

4. The method according to claim 1, wherein, The task to be tested is associated with at least one already tested task, and determining a second container in an idle state from candidate containers that have deployed target testing tools matching the task type of the task to be tested includes: Based on the running status of the candidate containers where the target testing tool is deployed, a third container is determined from each of the candidate containers, wherein the running status of the third container is an idle state; From each of the third containers, determine the second container invoked by the tested task.

5. The method according to claim 1, wherein, Determining the server's performance in the first phase based on the first test metric and the second test metric includes: The first packet response time, total execution time, and hard latency of the test data are determined; wherein, the first packet response time is the duration between the first moment and the second moment, the total execution time is the duration between the first moment and the third moment, and the hard latency is the duration between the fourth moment and the third moment; the first moment is the moment when the target testing tool in the second container sends the first data packet of the test data, the second moment is the moment when the second container receives the first data packet of the response result of the first stage, the third moment is the moment when the second container receives the last data packet of the response result of the first stage, and the fourth moment is the moment when the target testing tool in the second container sends the last data packet of the test data; Based on at least one of the first frame response time, total execution time, and hard latency of the test data, determine the third test metric corresponding to the performance test of the first stage. The performance of the server in the first stage is determined based on the first test metric, the second test metric, and the third test metric.

6. The method according to claim 1, wherein, The service in at least one first container is used to perform a second-stage processing on the test data, the test type being a second-stage performance test, and the method further includes: The target test case is executed by calling the target testing tool in the second container to determine a second difference between the response result of the second phase and the expected result of the second phase marked on the test data; The monitoring results of the second phase are analyzed to determine the first function invoked by the service in the at least one first container in response to the test data; Determine a third difference between the first function and the second function of the second stage as marked on the test data; The server's performance in the second phase is determined based on the second difference and / or the third difference.

7. The method according to claim 1, wherein, The test type is a stress test, and the method further includes: The monitoring results are analyzed to determine whether any service in the at least one first container experiences a runtime anomaly while responding to the test data. In response to a service malfunction occurring in at least one first container, it is determined that the server failed the stress test; In response to the absence of operational anomalies in the services within the at least one first container, it is determined that the server has passed the stress test.

8. The method according to any one of claims 1-7, wherein, The method further includes: The test results are stored in correspondence with the task identifier.

9. A testing apparatus, comprising: The first acquisition module is used to acquire the task to be tested and, based on the task identifier of the task to be tested, acquire the task parameters associated with the task identifier, wherein the task parameters include test case identifier and test set identifier; The loading module is used to load target test cases that match the test case identifier according to the test case identifier, and to load target test sets that match the test set identifier according to the test set identifier. The sending module is used to send test data from the target test set to a first container of a service in at least one deployment server, based on the target test case. The second acquisition module is used to acquire test results obtained by using the test data to perform a target test on the services in the at least one first container that matches the task to be tested. The sending module includes: The first determining unit is used to determine a target testing tool that matches the task type based on the task type of the task to be tested. The second determining unit is used to determine a second container in an idle state from among the candidate containers based on the running status of the candidate containers where the target testing tool is deployed. The construction unit is used to call the target testing tool in the second container to construct a test scenario that matches the test type of the task to be tested, based on the target test cases; A sending unit is used to, in the test scenario, invoke the target testing tool in the second container to send the test data in the target test set to the at least one first container; The second acquisition module is specifically configured to: receive a response result sent by the at least one first container through the second container, wherein the response result is generated by the service in the at least one first container responding to the test data; monitor the operation of the at least one first container during the service in the at least one first container responding to the test data to obtain monitoring results; and use the response result and the monitoring results as the test result. Wherein, the service in at least one first container is used to perform a first-stage processing on the test data, the test type being a first-stage performance test, and the apparatus further includes: The first execution module is used to call the target testing tool in the second container to execute the target test case, so as to determine the first difference between the response result of the first stage and the expected result of the first stage marked on the test data; The first determining module is used to determine a first test indicator corresponding to the performance test of the first stage based on the first difference; the first test indicator is negatively correlated with the first difference; The first parsing module is used to parse the monitoring results of the first stage to determine the resources occupied by the service response to the test data in the at least one first container; The second determining module is used to determine a second test indicator corresponding to the performance test of the first stage based on the resources occupied by the services in the at least one first container; the second test indicator is negatively correlated with the amount of resources occupied by the services. The third determining module is used to determine the performance of the server in the first stage based on the first test indicator and the second test indicator.

10. The apparatus according to claim 9, wherein, The building unit is specifically used for: Obtain configuration parameters from the target test case; The target testing tool in the second container is invoked to construct a test scenario that matches the test type of the task to be tested, based on the configuration parameters.

11. The apparatus according to claim 9, wherein, The second determining unit is specifically used for: Based on the running status of the candidate containers where the target testing tool is deployed, a third container is determined from each of the candidate containers, wherein the running status of the third container is an idle state; The second container is determined from each of the third containers according to the test type.

12. The apparatus according to claim 9, wherein, The task to be tested is associated with at least one previously tested task, and the second determining unit is specifically used for: Based on the running status of the candidate containers where the target testing tool is deployed, a third container is determined from each of the candidate containers, wherein the running status of the third container is an idle state; From each of the third containers, determine the second container invoked by the tested task.

13. The apparatus according to claim 9, wherein, The third determining module is specifically used for: The first packet response time, total execution time, and hard latency of the test data are determined; wherein, the first packet response time is the duration between the first moment and the second moment, the total execution time is the duration between the first moment and the third moment, and the hard latency is the duration between the fourth moment and the third moment; the first moment is the moment when the target testing tool in the second container sends the first data packet of the test data, the second moment is the moment when the second container receives the first data packet of the response result of the first stage, the third moment is the moment when the second container receives the last data packet of the response result of the first stage, and the fourth moment is the moment when the target testing tool in the second container sends the last data packet of the test data; Based on at least one of the first frame response time, total execution time, and hard latency of the test data, determine the third test metric corresponding to the performance test of the first stage. The performance of the server in the first stage is determined based on the first test metric, the second test metric, and the third test metric.

14. The apparatus according to claim 9, wherein, The service in the at least one first container is used to perform a second-stage processing on the test data, the test type being a second-stage performance test, and the apparatus further includes: The second execution module is used to call the target testing tool in the second container to execute the target test case, so as to determine a second difference between the response result of the second phase and the expected result of the second phase marked on the test data; The second parsing module is used to parse the monitoring results of the second phase to determine the first function called by the service in the at least one first container in response to the test data; The fourth determining module is used to determine a third difference between the first function and the second function of the second stage marked on the test data; The fifth determining module is used to determine the performance of the server in the second stage based on the second difference and / or the third difference.

15. The apparatus according to claim 9, wherein, The test type is a pressure test, and the device further includes: The third parsing module is used to parse the monitoring results to determine whether the service in the at least one first container has experienced a runtime anomaly in the process of responding to the test data. The sixth determining module is used to determine that the server has failed the stress test in response to a service malfunction occurring in the at least one first container. The seventh determination module is used to determine that the server has passed the stress test in response to the fact that no operational abnormality has occurred in the service in the at least one first container.

16. The apparatus according to any one of claims 9-15, wherein, The device further includes: The storage module is used to store the test results in correspondence with the task identifier.

17. An electronic device, wherein, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8.

Images