A method for software stability testing, an electronic device, and a storage medium

Abstract

The present application relates to a method for software stability testing, an electronic device and a storage medium, wherein the processing device of the present application is included in an integrated circuit device, the integrated circuit device includes a general-purpose interconnection interface and a computing device. The computing device interacts with the processing device to jointly complete a user-specified computing operation. The integrated circuit device can further include a storage device connected with the computing device and the processing device respectively for data storage of the computing device and the processing device.

Description

Technical Field

[0001] This invention generally relates to the field of testing. More specifically, this invention relates to a method for software stability testing, an electronic device, and a storage medium. Background Technology

[0002] During the development of artificial intelligence software, a set of plans and processes are generally used to ensure the quality of the released software. Software stability is an important indicator for measuring software quality.

[0003] There are many common testing schemes available, and the most commonly used testing schemes include the following.

[0004] Extreme testing: This involves using multiple identical devices to monitor the CPU, load time, bandwidth, response time, I / O throughput, etc., of both the local server and the cloud server. This stability testing method is generally suitable for testing servers, specifically for stability testing when a local server and a cloud server are interconnected. For example, when connecting a mobile phone to a cloud server, it tests whether data transmission is stable and whether there is latency in response time. Extreme testing is mainly used when network communication and bandwidth are high to test the stability of the interconnection between servers.

[0005] Stress testing: Use Jenkins (Jenkins is an open-source continuous integration tool with a user-friendly interface, mainly used for continuous and automated building / testing of software projects and monitoring of external tasks) or scripts to perform repeated operations over a long period of time to check whether system resources are abnormal.

[0006] Benchmark testing: A standard version is defined, which can be an industry-mature software version or a previous version. This testing approach aims to compare performance with existing metrics. Benchmark testing can only measure the performance difference compared to the standard version, but it cannot test the stability of the software itself. For example, when performing image inference, if the standard version can infer 100 images per second, while the software under test can infer 200 images per second, this only indicates an improvement in the performance of the software under test, not an improvement in its stability.

[0007] In summary, several significant drawbacks of traditional software stability testing methods can be identified: First, they are only applicable to specific scenarios, focusing solely on response time, bandwidth, and throughput system metrics while neglecting factors such as software memory usage, thus rendering them unsuitable for general use. Second, they fail to quantify the results, focusing only on end-to-end performance and outcomes, and cannot directly demonstrate the software's inherent stability. Therefore, none of the current solutions are ideal.

[0008] To address the aforementioned problems, this invention proposes a software stability testing scheme. Summary of the Invention

[0009] To at least partially solve the technical problems mentioned in the background art, the present invention provides a method for software stability testing, a readable storage medium, and an electronic device.

[0010] In one aspect, the present invention discloses a method for software stability testing, the method comprising: receiving a test strategy; running multiple test cases based on the test strategy to obtain multiple test results, each test case generating a parameter indicator, the multiple test results including multiple actual parameter indicators, wherein the parameter indicators include at least one of runtime, memory usage, bandwidth, and I / O; setting a fluctuation range for the multiple actual parameter indicators; calculating the number of times the multiple actual parameter indicators fall within the fluctuation range; determining whether the number is greater than or equal to a threshold; if so, passing the stability test.

[0011] In another aspect, the present invention discloses an electronic device comprising: a processor; a memory for storing executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0012] In another aspect, the present invention discloses a computer-readable storage medium having stored thereon computer program instructions for testing software stability, which, when executed by a server, implement the method described above.

[0013] This invention measures software stability by creating performance metrics, judging stability by whether the results of test cases meet preset parameter indicators. Specifically, this invention quantifies the results by calculating the number of actual parameter indicators for multiple test cases that fall within a reasonable fluctuation range, providing a direct indication of the software's stability. Furthermore, these parameter indicators include at least one of runtime, memory usage, bandwidth, and I / O. During testing, it not only focuses on response time, bandwidth, and throughput system metrics but also considers software memory issues, enabling a more comprehensive and effective assessment of software stability. Attached Figure Description

[0014] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts wherein:

[0015] Figure 1This is a schematic diagram illustrating the structure of a board card according to an embodiment of the present invention;

[0016] Figure 2 This is a structural diagram illustrating an integrated circuit device according to an embodiment of the present invention;

[0017] Figure 3 This is a flowchart illustrating a software stability testing method according to an embodiment of the present invention;

[0018] Figure 4 This is a flowchart illustrating a method for setting the fluctuation range according to an embodiment of the present invention;

[0019] Figure 5 This is a flowchart illustrating a specific software stability testing method according to an embodiment of the present invention; and

[0020] Figure 6 This is a software stability testing apparatus illustrating an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0023] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0024] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0025] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of the present invention is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0027] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0028] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0029] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown...) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204.

[0030] The computing device 201 is configured to execute user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0031] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0032] The processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping the computing device 201. Depending on the implementation, the processing device 203 may be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors. These processors include, but are not limited to, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing device 201 of this invention can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0033] DRAM 204 is used to store data to be processed. It is DDR memory, typically 16G or larger in size, and is used to store data in computing device 201 and / or processing device 203.

[0034] Figure 3This diagram illustrates a flowchart of a software stability testing method according to an embodiment of the present invention. The testing hardware environment includes the aforementioned board 10, combined processing device 20, and different models of operating systems, host-side system memory, CPU, etc. In this embodiment, software stability testing refers to monitoring whether the software's runtime, I / O, memory usage, bandwidth, and other indicators fluctuate within an expected range over a certain period. If the fluctuations in runtime, I / O, memory usage, bandwidth, etc., during the test are all within the expected range, the software stability test is considered passed; if the fluctuations in runtime, I / O, memory usage, and / or bandwidth, etc., during the test exceed the expected range, the software stability test is considered failed. The specific steps of this embodiment's software stability testing method are as follows:

[0035] Step 301: Receive the Test Strategy. A test strategy is the most reasonable approach, method, or process selected (or formulated) to reveal or reduce product quality risks to the greatest extent possible at the lowest cost, or to complete testing as early as possible. Simply put, a test strategy defines what to test, how to test it, and how to coordinate test resources and testing time. The rationality and efficiency of the test strategy will greatly affect the progress of the testing project. To ensure that the testing work is carried out scientifically, accurately, comprehensively, and systematically, a reasonable test strategy must first be formulated. After receiving the test strategy, the test hardware environment will conduct orderly testing according to the test strategy.

[0036] Specifically, a testing strategy is a test plan developed by the tester based on the testing phases, objectives, and scope. Testing phases include unit testing, integration testing, system testing, and regression testing. Testing objectives include functional testing, performance testing, stress testing, capacity testing, installation testing, configuration testing, anomaly testing, stability testing (including testing under normal and abnormal conditions), and boundary testing. The testing scope includes interface testing, partial testing, or overall testing, etc.

[0037] In this embodiment, the testing strategy includes the testing time and the specific testing methods. The testing time refers to the total testing time for running test cases. The specific testing methods define the testing phases and methods. Compared to traditional testing processes that only test software during the runtime phase, the testing strategy of this embodiment includes testing the software during the compilation phase and / or runtime phase.

[0038] In detail, this invention includes stability testing during the compilation phase, employing a strategy of compiling and testing simultaneously, allowing compilation and testing to proceed in parallel—the concept of "the earlier the better." The testing strategy during the compilation phase includes a first time period, which is the total compilation time for the multiple test cases. Furthermore, within this first time period, the multiple test cases are divided into multiple processes and multiple models for simultaneous compilation and testing.

[0039] In this embodiment of the invention, the stability of the software (monitoring response time, memory, bandwidth, etc.) is tested during the compilation phase. The length of the first time period depends on the specific test scenario, ensuring full utilization of system resources and improving resource efficiency during testing. For example, under normal circumstances, the first time period is set to 7×24, where "7" represents 7 days and "24" represents 24 hours, i.e., 24 hours per day, with continuous testing for 7 days. This invention does not impose any limitations on this. The testing method employs multiple processes and multiple models during the compilation phase. The purpose of the compilation process is to generate serialized models. Multi-process compilation of the software generates multiple serialized models each time the software is compiled. Using multiple models for testing serves two purposes: firstly, to verify whether multiple models can still be generated when system resources are limited, thus verifying functionality; secondly, each model can be set with different parameters, and using multiple models for testing also improves resource utilization.

[0040] In this embodiment, testing begins during the compilation phase, employing a strategy of compiling and testing simultaneously. This allows compilation and testing to proceed in parallel, enabling the early detection of issues in stability testing, reducing workload, and improving efficiency. The system's I / O, memory, and bandwidth are monitored during the compilation phase to determine if these metrics exceed expected ranges within a specific timeframe, thus testing software stability. If fluctuations in I / O, memory, and bandwidth remain within the expected range, the software stability test is passed. Conversely, if any of these metrics exceed the expected range, the software stability test fails.

[0041] As mentioned earlier, the testing phase may also include runtime stability testing. During the runtime phase, the testing strategy includes a second time period, which is the total duration for running the multiple test cases. Furthermore, within the second time period, the multiple test cases are divided into multiple processes and multiple threads for testing.

[0042] From a neural network software development perspective, runtime testing involves processing the serialized model before proceeding to the inference phase. However, from a user's perspective, the compilation process isn't always necessary. Besides obtaining the serialized model through compilation, an existing serialized model can be retrieved directly from the database. Therefore, testing the generation of the serialized model is unnecessary at this stage. Similarly, the second time period depends on the specific test scenario, aiming to maximize system resource utilization during testing. In this embodiment, the second time period is longer than the first. For example, under normal circumstances, the second time period is set to 31×24, where "31" represents 31 days and "24" represents 24 hours, meaning 24 hours per day for 31 consecutive days of testing. This invention does not impose any limitations on this.

[0043] The testing methodology also employs multi-process and multi-threaded approaches. Multi-process refers to distributing the same serialized model across multiple different processes to ensure efficient resource utilization. If the test results differ from expectations when the same serialized model is distributed across multiple processes, further analysis is needed to determine if the number of processes is excessive. When the number of processes for the same serialized model exceeds the system's resources, the system will internally prioritize and execute these processes, leading to discrepancies between the test output and expectations. In this case, the number of processes needs to be adjusted. During runtime, all models must be tested to ensure the accuracy of each model.

[0044] Multithreading refers to the simultaneous execution of multiple threads within a single model, accelerating inference efficiency. During runtime, a multi-process and multi-threaded testing approach is employed. Software stability is tested by monitoring for unexpected fluctuations in metrics such as I / O, memory, and bandwidth. If the fluctuations in I / O, memory, and bandwidth remain within the expected range, the software stability test is passed. Conversely, if any of these metrics exceed the expected range, the software stability test fails.

[0045] As mentioned above, the testing phase in this embodiment can be the compilation phase and / or the runtime phase. When the testing in the compilation phase is combined with the testing in the runtime phase, the results of the software stability test will be more accurate and reliable.

[0046] Step 302: Run multiple test cases based on the test strategy to obtain multiple test results. Each test case generates a parameter metric, and the multiple test results include multiple actual parameter metrics. After receiving the test strategy, the hardware environment understands what to test and how to test it, and then runs the test cases according to the test strategy. After the test cases are executed, the actual parameter metrics for that test case are obtained.

[0047] Step 303: Set the fluctuation range for multiple actual parameter indicators. If the actual parameter indicators of a test case fluctuate within the allowable error range, the actual parameter indicators of the test case are considered accurate. If the actual parameter indicators of a test case exceed the allowable error range, the actual parameter indicators of the test case are considered unsatisfactory. These parameter indicators include at least one of runtime, memory usage, I / O, and bandwidth. Runtime refers to the time consumed to run one test case; memory usage refers to the amount of memory used to run one test case; I / O refers to I / O throughput, i.e., the amount of data successfully read and written per unit time; bandwidth represents the data transmission capacity, specifically the total amount of data transmitted by the bus per unit time, which is equal to the product of the bus width and the operating frequency. Figure 4 The flowchart of a method for setting the fluctuation range according to an embodiment of the present invention is shown. The test strategy includes standard parameter indicators for the operation of each test case. The steps for setting the fluctuation range of the actual parameter indicators of the test cases are as follows.

[0048] Step 401: Calculate the variance between the actual parameter index and the standard parameter index for each test case. Step 402: Define the fluctuation range as within three variances based on the standard parameter index.

[0049] Assuming the parameter is runtime, let x be the actual runtime of a test case, x' be the standard runtime of a test case, σ be the variance, and N be the number of test cases run within the total test time. This total time period refers to either the first time period during the compilation phase or the second time period during the runtime phase. Then, the variance of x and x' is calculated as follows: The actual runtime of the test cases fluctuates within the range of x∈[x'-3σ,x'+3σ].

[0050] When this parameter is memory usage, let x be the actual memory usage after running one test case, x' represent the standard memory usage after running one test case, σ represent the variance, and N be the number of test cases run within the total test time. This total time period refers to either the first time period during the compilation phase or the second time period during the runtime phase. Then, the variance between x and x' is calculated as follows: The actual memory usage of the test cases fluctuates within the range of x∈[x'-3σ,x'+3σ].

[0051] When this parameter is bandwidth, let x be the total amount of data actually transmitted after running one test case, x' represent the standard amount of data transmitted after running one test case, σ represent the variance, and N be the number of test cases run within the total test time. This total time period refers to either the first time period during the compilation phase or the second time period during the runtime phase. Then, the variance between x and x' is calculated as follows: The actual bandwidth fluctuation range of the test cases is: x∈[x'-3σ,x'+3σ].

[0052] When the parameter is IO, let x be the number of data successfully read and written in one test case, x' be the standard number of data read and written in one test case, σ be the variance, and N be the number of test cases run within the total test time. This total time period refers to either the first time period during the compilation phase or the second time period during the runtime phase. Then, the variance between x and x' is calculated as follows: The actual I / O fluctuation range of the test cases is: x∈[x'-3σ,x'+3σ].

[0053] When the parameter is another indicator, the calculation method for the fluctuation range is similar. This invention does not limit the type of parameter indicator.

[0054] Step 304: Calculate the number of actual parameter indicators that fall within the fluctuation range. During the total test period, multiple test cases are run. After each test case finishes running, a corresponding actual parameter indicator is generated. It is then determined whether the actual parameter indicator for each test case falls within the aforementioned fluctuation range, and the number of test cases falling within this range is calculated.

[0055] Step 305: Determine whether the number is greater than or equal to the threshold. If the number of test cases falling within the fluctuation range is greater than or equal to the threshold, it means that there are enough test cases within the preset range, then proceed to step 306 to pass the stability test; if the number of test cases falling within the fluctuation range is less than the threshold, it means that there are too few test cases within the preset range, then proceed to step 307 to fail the stability test.

[0056] In this embodiment, the threshold can be determined by the percentage of test cases within the actual parameter fluctuation range to the total number of test cases. When the percentage is set at 99%, it means that no more than 1% of the test results fall outside the fluctuation range, and the stability test is considered to have passed; otherwise, the stability test is not passed.

[0057] Optionally, the threshold can also be determined based on experience. The actual parameter indicators of the test cases are distributed around the standard parameter indicator. Based on experience, it is unacceptable for 10 consecutive test cases to have actual parameter indicators in the same direction as the standard parameter indicator. If 10 consecutive test cases have actual parameter indicators in the same direction as the standard parameter indicator, the test fails, and the cause needs to be investigated. The direction of the standard parameter indicator includes above and below the standard parameter indicator. An actual parameter indicator above the standard parameter indicator means that the actual parameter indicator exceeds the fluctuation range (i.e., the actual parameter indicator is greater than x'+3σ). An actual parameter indicator below the standard parameter indicator means that the actual parameter indicator is below the fluctuation range (i.e., the actual parameter indicator is less than x'-3σ). When 10 consecutive test cases have actual parameter indicators that are all greater than x'+3σ or less than x'-3σ, the test fails.

[0058] In this embodiment of the invention, the stability of the software is quantified by calculating the number of actual parameter indicators that fall within the fluctuation range.

[0059] Figure 5 This is a flowchart illustrating a specific software stability testing method. This example uses runtime as the parameter metric to test software stability; specifically, it assesses software stability by determining whether the actual time required to run a test case falls within a fluctuating range.

[0060] Step 501: The processor receives the strategy for running test cases. Assume that software stability is tested during the compilation phase, with a total test time of 7×24 hours. Within this timeframe, multiple test cases are compiled and tested simultaneously across multiple processes and multiple models. Based on the current hardware system resources, the test is divided into 10 processes. Each compilation generates 10 serialized models, indicating that the current hardware system resources are sufficient for testing 10 processes. Furthermore, the 10 generated serialized models are tested using multiple models, i.e., different parameters are set for each serialized model. For example, the test data for each serialized model may be represented using either fixed-point or floating-point data, or different weight values ​​may be set for the input data, etc.

[0061] Step 502: Execute the test cases. Each test case corresponds to an actual runtime. Specifically, assuming 100 test cases are executed within a total testing time of 7×24, the actual runtime of each test case can be obtained after execution.

[0062] Step 503: Calculate the fluctuation range of actual runtime. Assume that all 100 test cases are of the same type, and their standard runtime is 100 minutes. In different embodiments, the standard runtime of multiple test cases may not be the same; this application does not impose any restrictions on this. The actual runtimes of the 100 test cases are as follows: 98 minutes for 10 test cases, 96 minutes for 50 test cases, 101 minutes for 20 test cases, 102 minutes for 15 test cases, and 120 minutes for 5 test cases. Then, the variance between the actual runtime and the standard runtime of the 100 test cases is calculated. (Keep two significant figures). Therefore, the actual runtime of the above 100 test cases fluctuates within the range of [83.8, 116.2].

[0063] Step 504: Calculate the number of test cases whose actual runtime falls within the above fluctuation range. Out of 100 test cases, 95 test cases have actual runtimes within the range [83.8, 116.2]. Furthermore, 5 test cases have actual runtimes outside the fluctuation range.

[0064] Step 505: Determine if the number of test cases whose actual runtime falls within the fluctuation range is greater than or equal to the threshold. If the number of test cases whose actual runtime falls within the fluctuation range is greater than or equal to the threshold, proceed to step 506, and the software's stability test passes. If the number of test cases whose actual runtime falls within the fluctuation range is less than the threshold, proceed to step 507, and the software's stability test fails. In this embodiment, the threshold is set to 10 test cases whose actual runtime is not within the fluctuation range. Therefore, 5 < 10, and the number of test cases whose actual runtime falls within the fluctuation range is less than the threshold number. Proceed to step 506, and the software's stability test passes.

[0065] Furthermore, during testing, a database can be established to store resource data from the compilation and runtime phases. This resource data includes standard and actual parameter metrics for each test case. The resource data in the database can be visualized as trend charts, allowing testers to make a preliminary assessment of the software's stability based on these charts. To prevent the database from storing too much data and reduce storage pressure, the system will delete previously stored data once the data storage volume reaches a preset value during normal software operation.

[0066] If the test fails, this embodiment further prints software data resources and hardware information, including CPU utilization, operating system, board and driver models, memory utilization, disk space, compiler version, etc., to help testers locate errors based on the printed information.

[0067] Figure 6 Figure 600 illustrates a software stability testing apparatus according to an embodiment of the present invention. The testing apparatus 600 includes a receiving unit 601, an execution unit 602, a setting unit 603, a calculation unit 604, and a judgment unit 605.

[0068] The receiving unit 601 is used to receive the test strategy from the user side. When performing software stability testing, the test strategy includes the test duration and the specific test methods. The test duration refers to the total test time for running test cases, and the specific test methods define the test phases and methods. The device 600 supports testing the software during the compilation phase and / or runtime phase. Based on the test duration and test methods, the testing device 600 can complete the entire testing process.

[0069] In detail, this embodiment of the invention includes stability testing during the compilation phase. After receiving the testing strategy, the receiving unit 601 adopts a strategy of compiling and testing simultaneously, allowing the compilation and testing processes to run in parallel. The testing strategy during the compilation phase includes a first time period, which is the total compilation time for the multiple test cases. Further, during the first time period, the multiple test cases are divided into multiple processes and multiple models for simultaneous compilation and testing. As mentioned above, the testing phase also includes stability testing during the runtime phase. During the runtime phase, the testing strategy further includes a second time period, which is the total runtime for the multiple test cases. Further, during the second time period, the multiple test cases are divided into multiple processes and multiple threads for testing.

[0070] The execution unit 602 is used to run multiple test cases according to the test strategy received by the receiving unit 601 to obtain multiple test results. Each test case generates a parameter metric, and the multiple test results include multiple actual parameter metrics.

[0071] Setting unit 603 is used to set the fluctuation range of multiple actual parameter indicators. If the actual parameter indicators of a test case fluctuate within the allowable error range, the actual parameter indicators of the test case are considered accurate. If the actual parameter indicators of a test case exceed the allowable error range, the actual parameter indicators of the test case are considered unsatisfactory. These parameter indicators include at least one of runtime, memory usage, I / O, and bandwidth. Runtime refers to the time consumed to run one test case; memory usage refers to the amount of memory used to complete one test case; I / O refers to I / O throughput, i.e., the amount of data successfully read or written per unit time; bandwidth represents the data transmission capacity, specifically the total amount of data transmitted by the bus per unit time, which is equal to the product of the bus width and the operating frequency.

[0072] The calculation unit 604 is used to calculate the variance between the actual parameter index and the standard parameter index for each test case, wherein the test strategy includes the standard parameter index for each test case operation, and sends the calculated variance to the setting unit 603. The setting unit 603 defines the fluctuation range as within three variances based on the standard parameter index, according to the variance calculated by the calculation unit 604.

[0073] The calculation unit 604 is also used to calculate the number of actual parameter indicators falling within the fluctuation range based on the multiple test results obtained by the execution unit 602 and the fluctuation range obtained by the setting unit 603. During the total test time period, the execution unit 602 can run multiple test cases, and each test case will correspond to an actual parameter indicator after it is completed.

[0074] The judgment unit 605 is used to receive multiple test results from the execution unit 602 and sequentially judge whether the actual parameter index corresponding to each test case falls within the above fluctuation range. The calculation unit 604 calculates the number of test cases that fall within the above fluctuation range based on the judgment result of the judgment unit 605.

[0075] The judgment unit 605 is further configured to determine whether the number of test cases calculated by the calculation unit 604 is greater than or equal to a threshold, and further determine whether the test passes. If the number of test cases falling within the fluctuation range is greater than or equal to the threshold, indicating that the number within the preset range is sufficient, the stability test passes; if the number of test cases falling within the fluctuation range is less than the threshold, indicating that the number within the preset range is too small, the stability test fails. The threshold can be determined as a percentage of the number of test cases falling within the actual parameter index fluctuation range to the total number of test cases. When this percentage is set at 99%, it means that no more than 1% of the test results fall outside the fluctuation range, and the stability test is considered passed; otherwise, the stability test fails. Furthermore, the threshold can also be determined based on empirical values ​​or obtained through specific analysis of the actual test scenario. This invention does not impose any limitations on the setting of the threshold.

[0076] The testing device 600 can be used for any one of the following tests: memory testing, bandwidth testing, and I / O testing.

[0077] Another embodiment of the present invention is a computer-readable storage medium storing computer program code for software stability testing. When the computer program code is run by a server, the server includes a processor and a memory, the memory storing the aforementioned computer program code, and the processor running the computer program code in the memory. In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the solution of the present invention is embodied in the form of a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, a server, or a network device) to execute some or all of the steps of the method described in the embodiments of the present invention. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0078] Another embodiment of the present invention is an electronic device, comprising: a processor and a memory for storing executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method for software stability testing. Depending on different application scenarios, the electronic device or apparatus of the present invention may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic device or apparatus of the present invention can also be applied to the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical fields. Furthermore, the electronic devices or apparatuses of the present invention can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminals. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present invention can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling, and collaborative work of end-to-cloud or cloud-edge-end integration.

[0079] This invention assesses software stability by determining whether the results of test cases meet preset parameter indicators. Specifically, it quantifies the results by calculating the number of actual parameter indicators for multiple test cases that fall within a reasonable fluctuation range, providing a direct indication of the software's stability. These parameter indicators include at least one of runtime, memory usage, bandwidth, and I / O. During testing, it considers not only response time, bandwidth, and throughput system metrics but also software memory usage, enabling a more comprehensive and effective assessment of software stability.

[0080] It should be noted that, for the sake of brevity, this invention describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solution of this invention is not limited to the order of the described actions. Therefore, based on the disclosure or teachings of this invention, those skilled in the art will understand that some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this invention can be considered as optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions of this invention. In addition, depending on the solution, the description of some embodiments of this invention also has different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this invention can also refer to the relevant descriptions of other embodiments.

[0081] In terms of specific implementation, based on the disclosure and teachings of this invention, those skilled in the art will understand that the several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or device embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection utilizing an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0082] In this invention, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same position or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this invention. Additionally, in some scenarios, multiple units in the embodiments of this invention may be integrated into one unit or each unit may exist physically independently.

[0083] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0084] The foregoing can be better understood in accordance with the following terms:

[0085] Clause A1. A method for software stability testing, the method comprising: receiving a test strategy; running multiple test cases based on the test strategy to obtain multiple test results, each test case generating a parameter indicator, the multiple test results including multiple actual parameter indicators, wherein the parameter indicators include at least one of runtime, memory usage, bandwidth, and I / O; setting a fluctuation range for the multiple actual parameter indicators; calculating the number of times the multiple actual parameter indicators fall within the fluctuation range; determining whether the number is greater than or equal to a threshold; and if so, passing the stability test.

[0086] Clause A2, according to the method described in Clause A1, the test strategy includes standard parameter indicators for the operation of each test case, and the setting steps include: calculating the variance between the actual parameter indicator and the standard parameter indicator for each test case; and defining the fluctuation range as within three of the variances based on the standard parameter indicator.

[0087] Clause A3. According to the method described in Clause A1, the software stability test is a stability test during the compilation phase, and the test strategy includes a first time period during which the compilation phase runs, wherein the first time period is the total time period during which the multiple test cases are compiled.

[0088] Clause A4. According to the method described in Clause A3, the operation steps include: during the first time period, dividing the multiple test cases into multiple processes and multiple models for simultaneous compilation and testing.

[0089] Clause A5, the method described in Clause A1 or A3, wherein the software stability test is a runtime stability test, and the test strategy further includes a second time period of the runtime phase, wherein the second time period is the total time period for which the plurality of test cases are run.

[0090] Clause A6. According to the method described in Clause A5, the operation strategy further includes: during the second time period, dividing the multiple test cases into multiple processes and multiple threads for testing.

[0091] Clause A7. The method described in Clause A6 further includes: establishing a database;

[0092] The multiple test cases and their actual runtime are stored in the database.

[0093] Clause A8. The stability test described in accordance with Clause A1 includes at least one of memory testing, bandwidth testing, and I / O testing.

[0094] Clause A9. An electronic device comprising: a processor; and a memory for storing executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described in any one of Clauses A1 to A8.

[0095] Clause A10. A computer-readable storage medium storing computer program instructions for software stability testing, characterized in that, when the computer program instructions are executed by a server, they implement the method described in any one of Clauses A1 to A8.

[0096] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for stability testing of artificial intelligence software, characterized in that, The method includes: Receive test strategy; Multiple test cases are run based on the test strategy to obtain multiple test results. Each test case generates a parameter indicator. The multiple test results include multiple actual parameter indicators, wherein the parameter indicators include at least one of running time, memory usage, bandwidth, and I / O. Set the fluctuation range of the aforementioned multiple actual parameter indicators; Calculate the number of actual parameter indicators that fall within the fluctuation range; Determine whether the number is greater than or equal to the threshold; and If so, the stability test is passed; The stability test of the artificial intelligence software is a stability test during the compilation phase. The test strategy includes a first time period during which the compilation phase runs. The first time period is the total time period during which the multiple test cases are compiled. The purpose of the compilation is to generate a serialized model.

2. The method according to claim 1, characterized in that, The testing strategy includes standard parameter metrics for each test case execution, and the set fluctuation range of the multiple actual parameter metrics includes: Calculate the variance between the actual parameter metric and the standard parameter metric for each test case; and The fluctuation range is defined as within three variances based on the standard parameter index.

3. The method according to claim 1, characterized in that, Running multiple test cases based on the aforementioned testing strategy includes: During the first time period, the multiple test cases are divided into multiple processes and multiple models and compiled and tested simultaneously.

4. The method according to claim 1, characterized in that, The software stability test is a runtime stability test, and the test strategy also includes a second time period during the runtime phase, wherein the second time period is the total time period during which the multiple test cases are run.

5. The method according to claim 4, characterized in that, The testing strategy also includes: During the second time period, the multiple test cases are divided into multiple processes and multiple threads for testing.

6. The method according to claim 5, characterized in that, The method further includes: Establish a database; The multiple test cases and actual parameter metrics are stored in the database.

7. The method according to claim 1, characterized in that, The stability test includes at least one of memory test, bandwidth test, and I / O test.

8. An electronic device, characterized in that, include: processor; Memory used to store executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing computer program instructions for software stability testing, characterized in that, When the computer program instructions are executed by the server, they implement the method described in any one of claims 1 to 7.

Images