Test case sorting method and device, storage medium and electronic equipment

Abstract

The invention provides a test case sorting method and device, a storage medium and electronic equipment, and relates to the technical field of machine learning. The method comprises the following steps: acquiring feature vectors of a plurality of test cases, and determining a first sequence of the plurality of test cases by utilizing a sorting strategy of a first learning model; executing a software test according to the plurality of test cases of the first sequence, and determining a test result; maximizing a reward function of the first learning model according to a test result to adjust a sorting strategy; according to an adjusted sorting strategy corresponding to the maximized reward function, updating the plurality of test cases from a first sorting to a second sorting; wherein the plurality of test cases in the second sequence realize execution of the most error-detecting test cases in unit duration when executing software testing. According to the method and the device, the test cases which can be mistakenly removed are executed at most in unit time, so that the test cases which can be mistakenly removed and are shorter in time are executed firstly, and the software test quality is improved.

Description

Background Art

[0002] During the software development process, integrating the code of different developers into a shared code repository and performing automated building and testing can ensure the stability and quality of the software.

[0003] However, as the scale of software increases, the scale of test cases used to test the software also increases, and more and more test cases are included. The coverage of test cases, error detection efficiency, etc. continue to change as the software testing cycle is extended. If the test cases are not sorted, the efficiency and quality of software testing will be reduced, resulting in delays in the progress of software development projects.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a test case sorting method and device, a storage medium and an electronic device, which at least to a certain extent overcome the problem of reduced efficiency and quality of software testing due to the non-sorting of test cases in the related art.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a test case sorting method is provided, comprising:

[0008] Acquire feature vectors of a plurality of test cases, and determine a first ranking of the plurality of test cases using a ranking strategy of a first learning model;

[0009] Execute software testing according to the first sorted plurality of test cases and determine the test results;

[0010] Maximizing the reward function of the first learning model according to the test result to adjust the sorting strategy;

[0011] According to the adjusted sorting strategy corresponding to the maximized reward function, the multiple test cases are updated from the first sorting to the second sorting; wherein, the multiple test cases in the second sorting are used to execute the most error-revealing test cases within a unit time when executing software testing.

[0012] In some embodiments, acquiring feature vectors of a plurality of test cases, and determining a first ranking of the plurality of test cases using a ranking strategy of a first learning model includes:

[0013] Obtaining historical information of multiple test cases; the historical information of the test cases includes: estimated execution time of the test cases, execution time of the last software test executed by the test cases, and historical execution results;

[0014] Determining feature vectors of the multiple test cases according to the historical information of the multiple test cases;

[0015] Inputting the feature vectors of the multiple test cases into a first learning model, and calculating the priority of each test case in the multiple test cases using a sorting strategy of the first learning model;

[0016] The multiple test cases are sorted according to the priority of each test case to determine a first sorting of the multiple test cases.

[0017] In some embodiments, performing software testing according to the first-ordered plurality of test cases and determining test results includes:

[0018] Execute software testing according to the first sorted test cases, and determine the execution time and number of errors revealed by each software test case when executing the software test; the number of errors revealed is the number of program errors revealed by the software test case when executing the software test;

[0019] The execution time and number of errors revealed by each software test case during software testing are determined as test results.

[0020] In some embodiments, maximizing the reward function of the first learning model according to the test result to adjust the sorting strategy includes:

[0021] If the execution time of the test case when executing the software test is not of concern, maximizing the first reward function of the first learning model according to the test result and the first ranking;

[0022] If the execution time of the test case when executing the software test is of concern, maximizing the second reward function of the first learning model according to the test result and the first ranking;

[0023] According to the maximized first reward function or second reward function, the parameters of the first learning model are updated using a gradient descent algorithm to adjust the sorting strategy; wherein the sorting strategy is adjusted to achieve the goal of executing the most fault-revealing test cases within a unit time.

[0024] In some embodiments, if the execution time of the test case when executing the software test is not concerned, maximizing the first reward function of the first learning model according to the test result and the first ranking includes:

[0025] If the execution time of the test case when executing the software test is not concerned, then according to the first sorting, the position of each test case in the first sorting is determined;

[0026] According to the test results, determining the test cases that revealed errors during the software test execution;

[0027] Determine a first reward function of the first learning model according to the position of each test case in the first ranking and the test cases that reveal errors during software test execution;

[0028] A first reward function of the first learning model is maximized according to the test result and the first ranking.

[0029] In some embodiments, the first reward function of the first learning model is determined as follows:

[0030]

[0031] Where NAPFD is the index value of the first reward function. The larger the index value, the higher the test case with more errors is in the ranking; rank(t) is the position of test case t in the ranking; TS i A collection of multiple test cases; TS i fail A collection of test cases that reveal errors during software test execution; TS i total,fail It is a set of multiple test cases and error-revealing test cases; p is an intermediate parameter.

[0032] In some embodiments, if the execution time of the test case when executing the software test is of interest, maximizing the second reward function of the first learning model according to the test result and the first ranking includes:

[0033] If the execution time of the test case during the software test is of interest, then according to the first sorting, the position of each test case in the first sorting is determined;

[0034] According to the test results, determine the test cases that revealed errors during the software test execution and the actual execution time of each test case;

[0035] Determine a second reward function of the first learning model according to the position of each test case in the first ranking, the test cases that reveal errors during software test execution, and the actual execution time of each test case;

[0036] A second reward function of the first learning model is maximized according to the test result and the first ranking.

[0037] In some embodiments, the second reward function of the first learning model is determined as follows:

[0038]

[0039] Among them, NAPFDc is the index value of the second reward function. The larger the index value, the higher the error-revealing test case with shorter execution time will be in the ranking; TS i A collection of multiple test cases; TF i is the position of the i-th error-revealing test case in multiple test cases; m is the number of error-revealing test cases in the test cases executed by the software test; n is the number of test cases executed by the software test; t i is the actual execution time of test case i.

[0040] In some embodiments, updating the plurality of test cases from the first sorting to the second sorting according to the adjusted sorting strategy corresponding to the maximized reward function comprises:

[0041] Determining an updated first learning model according to the adjusted sorting strategy corresponding to the maximized reward function;

[0042] Inputting the test results into the updated first learning model, and calculating the update priority of multiple test cases for executing software tests according to the first sorting using the adjusted sorting strategy;

[0043] The multiple test cases are sorted according to the update priority, and the multiple test cases are updated from the first sorting to a second sorting.

[0044] In some embodiments, it also includes:

[0045] Performing a smoke test on the multiple test cases sorted according to the second sorting order, and determining a smoke test result;

[0046] The second sorting is optimized according to the smoke test result.

[0047] According to another aspect of the present disclosure, a test case sorting device is also provided, comprising:

[0048] A first ranking determination module, configured to obtain feature vectors of a plurality of test cases and determine a first ranking of the plurality of test cases using a ranking strategy of a first learning model;

[0049] A test result determination module, used to perform software testing according to the first sorted plurality of test cases and determine the test result;

[0050] A sorting strategy adjustment module, used to maximize the reward function of the first learning model according to the test result to adjust the sorting strategy;

[0051] A sorting update module is used to update the multiple test cases from the first sorting to a second sorting according to the adjusted sorting strategy corresponding to the maximization reward function; wherein, the multiple test cases in the second sorting are used to execute the most error-revealing test cases within a unit time when executing software testing.

[0052] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the test case sorting methods described above by executing the executable instructions.

[0053] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the test case sorting method described in any one of the above is implemented.

[0054] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the computer program implements any of the above test case sorting methods.

[0055] The test case sorting method and device, storage medium and electronic device provided in the embodiments of the present disclosure include: obtaining feature vectors of multiple test cases, determining the first sorting of multiple test cases using the sorting strategy of the first learning model; performing software testing according to the multiple test cases of the first sorting, and determining the test results; maximizing the reward function of the first learning model according to the test results to adjust the sorting strategy; updating the multiple test cases from the first sorting to the second sorting according to the adjusted sorting strategy corresponding to the maximized reward function; wherein, the multiple test cases of the second sorting are used to execute the most error-revealing test cases within a unit time when executing the software test. The embodiment of the present disclosure obtains the feature vectors of multiple test cases and inputs them into the first learning model, determines the first sorting by the sorting strategy, and determines the first sorting by using the feature vectors of all test cases. The information considered in the sorting is more comprehensive, and it is not easy to be misled by the feature vector of a single test case and fall into the local optimum. The multiple test cases perform software testing according to the first sorting, determine the test results, and maximize the reward function of the first learning model according to the test results to adjust the sorting strategy, thereby updating the multiple test cases from the first sorting to the second sorting. By updating the sorting strategy of the first learning model with the test results obtained from executing software tests, the sorting strategy of the model can be adaptively adjusted as the software version evolves, and the final sorting can be directly optimized based on the overall sorting effect feedback, which is not easily misled by the established intermediate goals. In the environment of continuous integration of software testing, the ability of test cases to reveal errors can be effectively improved. By maximizing the reward function to update the sorting of test cases, the most error-revealing test cases can be executed within a unit time, ensuring that the test cases that can reveal errors and take less time are executed first, and the test cases that cannot reveal errors and take longer are executed later, so as to expose software defects in a timely manner, give developers the fastest feedback, and improve the quality of software testing.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0058] Figure 1 A schematic diagram showing the system structure of a test case sorting method in an embodiment of the present disclosure.

[0059] Figure 2 A schematic diagram of a test case sorting method in an embodiment of the present disclosure is shown.

[0060] Figure 3 A schematic diagram of a first sorting process of determining multiple test cases in a test case sorting method according to an embodiment of the present disclosure is shown.

[0061] Figure 4 A schematic diagram of a test result determination process of a test case sorting method in an embodiment of the present disclosure is shown.

[0062] Figure 5 A schematic diagram of a process of adjusting a sorting strategy of a test case sorting method in an embodiment of the present disclosure is shown.

[0063] Figure 6 A schematic diagram of a test case ranking method according to an embodiment of the present disclosure is shown for maximizing a first reward function of a first learning model based on test results and a first ranking.

[0064] Figure 7 A schematic diagram of a test case ranking method according to an embodiment of the present disclosure is shown for maximizing a second reward function of a first learning model based on test results and a first ranking.

[0065] Figure 8 A schematic diagram of a test case sorting method in an embodiment of the present disclosure is shown for updating multiple test cases from a first sorting to a second sorting process.

[0066] Fig. 9 An architectural diagram of a first learning model of a test case ranking method in an embodiment of the present disclosure is shown.

[0067] Fig.10 A comparison diagram of the program code and pseudocode of the first learning model encoding part of a test case sorting method in an embodiment of the present disclosure is shown.

[0068] Fig.11 A comparison diagram of partial program code and pseudocode decoded by a first learning model of a test case sorting method in an embodiment of the present disclosure is shown.

[0069] Fig.12 A flow chart of a test case sorting method in an embodiment of the present disclosure is shown.

[0070] Fig.13 A schematic diagram of a test case sorting device in an embodiment of the present disclosure is shown.

[0071] Fig.14 A structural block diagram of a computer device for a test case sorting method in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0072] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0073] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0074] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0075] CI / CD refers to continuous integration (CI) and continuous delivery or continuous deployment (CD).

[0076] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0077] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the test case sorting method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0078] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and may be a wired network or a wireless network.

[0079] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0080] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0081] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0082] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0083] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0084] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0085] Under the above system architecture, a test case sorting method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.

[0086] In some embodiments, the test case sorting method provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the test case sorting method provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the test case sorting method provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0087] Continuous integration and continuous deployment (CI / CD) technology is a modern technology in computer software engineering and has been widely used. Continuous integration refers to the process of integrating the code of different developers into a shared code repository during the software development process, and performing automated building and testing to ensure the stability and quality of the software. Continuous deployment is based on continuous integration and automatically deploys the tested code to the production environment. The combination of these two technologies has greatly improved the efficiency of software development and the speed of delivery. Through automated continuous integration, developers can submit code frequently, and through automated building and testing tools, quickly discover and resolve code conflicts and defects to avoid integration problems delaying the project progress. Continuous integration and continuous deployment technology allows the development team to quickly obtain feedback and fix problems in a timely manner. Continuous integration and continuous deployment technology encourages teamwork and communication. By sharing a code repository and frequently integrating code, developers can work more closely together and reduce errors and delays in information exchange.

[0088] However, as the scale of software increases, the test case set contains more and more test cases, and the test case coverage, error detection effect and other characteristics also change with the extension of the integration cycle. Traditional test case sorting technology is difficult to apply.

[0089] Figure 2 A schematic diagram of a test case sorting method in an embodiment of the present disclosure is shown. Figure 2 As shown, the test case sorting method provided in the embodiment of the present disclosure includes the following steps:

[0090] Step S202: Acquire feature vectors of multiple test cases, and determine a first ranking of the multiple test cases using a ranking strategy of a first learning model;

[0091] Step S204: executing software testing according to the first sorted plurality of test cases, and determining the test results;

[0092] Step S206: maximizing the reward function of the first learning model according to the test result to adjust the sorting strategy;

[0093] Step S208: Based on the adjusted sorting strategy corresponding to the maximized reward function, multiple test cases are updated from the first sorting to the second sorting; wherein, the multiple test cases in the second sorting execute the most error-revealing test cases within a unit time when executing software testing.

[0094] By obtaining the feature vectors of multiple test cases and inputting them into the first learning model, the first ranking is determined by the sorting strategy, and the feature vectors of all test cases are used to determine the first ranking. The information considered in the sorting is more comprehensive, and it is not easy to be misled by the feature vector of a single test case and fall into the local optimum. Multiple test cases perform software testing according to the first ranking, determine the test results, and maximize the reward function of the first learning model according to the test results to adjust the sorting strategy, so that multiple test cases are updated from the first ranking to the second ranking. By updating the sorting strategy of the first learning model with the test results obtained by executing the software test, the sorting strategy of the model can be adaptively adjusted as the software version evolves, and the final ranking can be directly optimized according to the overall sorting effect feedback. It is not easy to be misled by the established intermediate goals, and the ability of test cases to reveal errors can be effectively improved in the environment of continuous integration of software testing. By maximizing the reward function to update the sorting of test cases, the most error-revealing test cases are executed within a unit time, ensuring that the test cases that can reveal errors and take a shorter time are executed first, and the test cases that cannot reveal errors and take a longer time are executed later, so that the defects of the software are exposed immediately, giving developers the fastest feedback, and improving the quality of software testing.

[0095] In the embodiment, a first learning model is constructed based on reinforcement learning. The first learning model can continuously adjust the sorting strategy according to the test results as the software test proceeds, thereby achieving adaptive adjustment of the first learning model.

[0096] Figure 3 A schematic diagram of a first sorting process of determining multiple test cases in a test case sorting method according to an embodiment of the present disclosure is shown. Figure 3 As shown, in the embodiment, the feature vectors of multiple test cases are obtained, and the first sorting of the multiple test cases is determined using the sorting strategy of the first learning model, including:

[0097] Step S302: Acquire historical information of multiple test cases; the historical information of the test cases includes: estimated execution time of the test cases, execution time of the last software test executed by the test cases, and historical execution results;

[0098] Step S304: determining feature vectors of multiple test cases according to historical information of multiple test cases;

[0099] Step S306: inputting the feature vectors of the multiple test cases into the first learning model, and calculating the priority of each test case in the multiple test cases using the sorting strategy of the first learning model;

[0100] Step S308: Sort the multiple test cases according to the priority of each test case to determine a first sorting of the multiple test cases.

[0101] In the embodiment, the test cases are not sorted before executing the software test. First, they need to be sorted according to the historical information of the test cases to speed up the execution of the test cases. The estimated execution time of the test cases, the execution time of the last software test of the test cases and the historical execution results are obtained, and the estimated execution time of the test cases, the execution time of the last software test of the test cases and the historical execution results are extracted to determine the feature vectors of multiple test cases. The feature vectors of multiple test cases are input into the first learning model, and the first learning model calculates the priority of each test case in the multiple test cases according to the sorting strategy, so as to obtain the attention of each test case; wherein, the shorter the execution time of the last software test of the test case, the higher the priority. The smaller the difference between the estimated execution time of the test case and the execution time of the last software test of the test case, the higher the priority. The test case with more errors in the historical execution results has a higher priority. According to the priority, the test cases with high priority are arranged in front, and the test cases with low priority are arranged in the back, and the multiple test cases are sorted to obtain the first sorting of the multiple test cases.

[0102] The first sorting is determined by obtaining the historical information of the test case and inputting it into the first model, tracing back to the last execution. This method has good robustness to the length of the input historical information. A small amount of historical information can achieve the optimal sorting effect, and can handle regression test cases and new test cases well.

[0103] Figure 4 A schematic diagram of a test result determination process of a test case sorting method according to an embodiment of the present disclosure is shown. Figure 4 As shown, in the embodiment, executing software testing according to the first sorted multiple test cases and determining the test results include:

[0104] Step S402: executing software testing according to the plurality of test cases in the first sorting, and determining the execution time and the number of errors revealed by each software test case when executing the software testing; the number of errors revealed is the number of program errors revealed by the software test case when executing the software testing;

[0105] Step S404: Determine the execution time and number of errors found in each software test case during software testing as the test result.

[0106] In an embodiment, the first sorting is obtained based on the historical information of the test case, and multiple test cases are sorted according to the first sorting to perform software testing. During the software testing process, the execution time of each software test case when performing the software test is obtained, and the number of program errors revealed by each software test case when performing the software test is recorded as the number of revealed errors. Since there are many types of program errors in the software testing process, such as syntax errors, logic errors, runtime errors, performance problems, security vulnerabilities, compatibility problems, external dependency problems, data consistency problems, concurrency problems, and user interface and experience problems, all of the above program errors need to be revealed through test cases in software testing. The execution time and the number of revealed errors of each software test case obtained above when performing the software test are determined as the test result.

[0107] Figure 5 A schematic diagram of the process of adjusting the sorting strategy of a test case sorting method in an embodiment of the present disclosure is shown. Figure 5 As shown, in the embodiment, maximizing the reward function of the first learning model according to the test results to adjust the sorting strategy includes:

[0108] Step S502: if the execution time of the test case when executing the software test is not of concern, maximizing the first reward function of the first learning model according to the test result and the first ranking;

[0109] Step S504: if the execution time of the test case during software testing is of interest, maximizing the second reward function of the first learning model according to the test result and the first ranking;

[0110] Step S506: Based on the maximized first reward function or the second reward function, the parameters of the first learning model are updated using a gradient descent algorithm to adjust the sorting strategy; wherein the sorting strategy is adjusted to achieve the goal of executing the most fault-revealing test cases within a unit time.

[0111] For software testing, it can be classified according to whether the execution time of the test case is concerned. When the number of errors revealed is more concerned, the quality of software testing will be more emphasized. Therefore, if the execution time of the test case is not concerned, the first reward function of the first learning model is maximized according to the test results and the first sorting. According to the maximized first reward function, the parameters of the first learning model are updated using the gradient descent algorithm to adjust the sorting strategy; the adjusted sorting strategy, in order to reveal more program errors, prioritizes the test cases with more errors revealed in order to reveal more program errors, even if it takes a certain amount of time.

[0112] When paying more attention to the efficiency of software testing, it is necessary to give priority to the execution time of software testing, and maximize the second reward function of the first learning model according to the test results and the first sorting; according to the maximized second reward function, use the gradient descent algorithm to update the parameters of the first learning model to adjust the sorting strategy; the adjusted sorting strategy gives priority to the execution time of each bug-uncovering test case, and puts the test cases with shorter execution time in front for priority execution, so that more test cases can be executed in unit time.

[0113] Figure 6 A schematic diagram of a test case ranking method according to an embodiment of the present disclosure is shown, in which a first reward function of a first learning model is maximized according to a test result and a first ranking. Figure 6 As shown, in the embodiment, if the execution time of the test case when executing the software test is not concerned, the first reward function of the first learning model is maximized according to the test result and the first ranking, including:

[0114] Step S602: if the execution time of the test case when executing the software test is not concerned, then according to the first sorting, the position of each test case in the first sorting is determined;

[0115] Step S604: Determine the test cases that revealed errors during the software test execution according to the test results;

[0116] Step S606: determining a first reward function of the first learning model according to the position of each test case in the first ranking and the test cases that reveal errors during software test execution;

[0117] Step S608: Maximize the first reward function of the first learning model according to the test result and the first ranking.

[0118] In an embodiment, during software testing, if one does not pay attention to the execution time of the test case when executing the software test, but tends to pay more attention to the number of errors revealed, then it is necessary to first construct a first reward function and then maximize the first reward function. Specifically, according to the first ranking of multiple test cases, determine the position of each test case in the first ranking; from the test results of multiple test cases, determine the test cases that revealed errors during the execution of software testing; according to the position of each test case in the first ranking and the test cases that revealed errors during the execution of software testing, construct the first reward function of the first learning model. The first reward function is used to feedback and adjust the sorting strategy of the first learning model. Maximize the first reward function of the first learning model based on the test results and the first ranking.

[0119] In the embodiment, the first reward function of the first learning model is determined as follows:

[0120]

[0121] Where NAPFD is the index value of the first reward function. The larger the index value, the higher the test case with more errors is in the ranking; rank(t) is the position of test case t in the ranking; TS i A collection of multiple test cases; TS i fail A collection of test cases that reveal errors during software test execution; TS i total,fail It is a set of multiple test cases and error-revealing test cases; p is an intermediate parameter.

[0122] The NAPFD indicator value is used to back-propagate the first model to update the parameters, and the sorting strategy is continuously adjusted in the process of "sorting-running test-feedback", so as to finally achieve good sorting performance. For a sorting of test cases, when the test cases that reveal errors are put in front and the test cases that pass are put in the back, then this sorting can discover and reveal more errors under the condition of executing fewer test cases. The sorting error-revealing performance is measured by the NAPFD indicator value, and NAPFD is used as the first reward function. The more errors that the sorted and scheduled test cases can reveal, the more the test cases that reveal errors are put in front, and the larger the NAPFD indicator value, indicating that the more successful the sorting of the test cases is, the more positive feedback the reward information will give to the model. The calculation of NAPFD takes into account the ranking of the test cases that reveal errors.

[0123] Figure 7 A schematic diagram of a test case sorting method according to an embodiment of the present disclosure is shown, in which the second reward function of the first learning model is maximized according to the test results and the first sorting. Figure 7As shown, in the embodiment, if the execution time of the test case when executing the software test is concerned, the second reward function of the first learning model is maximized according to the test result and the first ranking, including:

[0124] Step S702: if the execution time of the test case during the software test is of interest, then according to the first sorting, determine the position of each test case in the first sorting;

[0125] Step S704: Determine the test cases that revealed errors during the software test execution and the actual execution time of each test case according to the test results;

[0126] Step S706: determining a second reward function of the first learning model according to the position of each test case in the first ranking, the test cases that reveal errors during software test execution, and the actual execution time of each test case;

[0127] Step S708: Maximize the second reward function of the first learning model according to the test result and the first ranking.

[0128] In an embodiment, during software testing, if attention is paid to the execution time of the test case when executing the software test, errors tend to be revealed in a shorter time, and developers can get feedback faster. In this case, it is necessary to first construct a second reward function and then maximize the second reward function.

[0129] Specifically, according to the first ranking of multiple test cases, the position of each test case in the first ranking is determined; from the test results of multiple test cases, the test cases that reveal errors in the software test execution are determined, and the actual execution time of each test case is recorded. According to the position of each test case in the first ranking, the test cases that reveal errors in the software test execution, and the actual execution time of each test case, a second reward function of the first learning model is constructed. The second reward function is used to feedback and adjust the sorting strategy of the first learning model. The second reward function of the first learning model is maximized according to the test results and the first ranking.

[0130] In the embodiment, the second reward function of the first learning model is determined as follows:

[0131]

[0132] Among them, NAPFDc is the index value of the second reward function. The larger the index value, the higher the error-revealing test case with shorter execution time will be in the ranking; TS i A collection of multiple test cases; TF iis the position of the i-th error-revealing test case in multiple test cases; m is the number of error-revealing test cases in the test cases executed by the software test; n is the number of test cases executed by the software test; t i is the actual execution time of test case i.

[0133] Based on the above NAPFD as the index value of the first reward function, the index value NAPFDc of the second reward function is proposed through further extrapolation and expansion. Compared with NAPFD, which considers the ranking position of the error-revealing test cases, the calculation of NAPFDc further considers the execution cost of the test cases, because for two test cases that can reveal errors, the one that takes less time to execute first can reveal the error quickly, and the error-revealing efficiency is higher. On the whole, the more error-revealing test cases are executed in a shorter time, the larger the NAPFDc index value of the ranking is, and the more rewards are given to the model. Compared with NAPFD, NAPFDc includes the execution time, which can better indicate the time cost of the ranking and motivate the model in a more direction.

[0134] Figure 8 A schematic diagram of a test case sorting method in an embodiment of the present disclosure is shown, in which a plurality of test cases are updated from a first sorting to a second sorting process. Figure 8 As shown, according to the adjusted sorting strategy corresponding to the maximized reward function, multiple test cases are updated from the first sorting to the second sorting, including:

[0135] Step S802: determining an updated first learning model according to the adjusted sorting strategy corresponding to the maximized reward function;

[0136] Step S804: input the test results into the updated first learning model, and calculate the update priority of multiple test cases for executing software tests according to the first sorting using the adjusted sorting strategy;

[0137] Step S806: sorting the multiple test cases according to the update priority, and updating the multiple test cases from the first sorting to the second sorting.

[0138] In an embodiment, the sorting strategy is adjusted by the maximized reward function to obtain the adjusted sorting strategy. The first learning model is updated by using the adjusted sorting strategy to obtain an updated first learning model. The test results are input into the updated first learning model, and the update priority is calculated for multiple test cases that perform software testing according to the first sorting by using the adjusted sorting strategy. The corresponding test cases are sorted by using the update priority, with the test cases with high update priority being placed in front and the test cases with low update priority being placed in the back, thereby forming a second sorting of the multiple test cases.

[0139] In an embodiment, after arranging the multiple test cases according to the second sorting, the method further includes: performing a smoke test on the multiple test cases sorted according to the second sorting to determine a smoke test result; and optimizing the second sorting according to the smoke test result.

[0140] Smoke testing is the process of validating code changes before embedding them into the product's source tree. Smoke testing is the most cost-effective way to identify and fix software defects after the code has been reviewed. Smoke testing is designed to confirm that changes in the code work as expected and do not destabilize the entire release.

[0141] Smoke testing is performed on the sorting results to advance the progress of the pipeline and achieve the effect of continuous integration and continuous deployment; smoke testing is a concise and fast testing process after the software is released. Its main purpose is to test whether the current software can work properly. Usually, the content of the smoke test is concise and clear, and it does not focus on the detailed testing of a single function, which makes its time cost low, and by discovering problems in advance, it also avoids a lot of time wasted in comprehensive testing. If the newly released software cannot pass the smoke test, it means that the software has serious defects. Software that cannot pass the smoke test will not be able to undergo subsequent comprehensive testing, thereby reducing unnecessary test rounds. According to the results of the upper limit analysis, the pipeline parts that are focused on testing are locked, and smoke testing is performed on these parts in the order of test cases, which can produce an optimization effect on the entire pipeline.

[0142] Fig. 9 The architecture diagram of the first learning model of a test case ranking method in the embodiment of the present disclosure is shown as follows: Fig. 9 As shown, the first learning model in this embodiment is constructed and trained based on the idea of ​​reinforcement learning. The first learning model includes an execution network and an evaluation network. Among them, the execution network includes encoding (Bi-LSTM) and decoding (LSTM); the evaluation network includes encoding (Bi-LSTM), processing module and decoding (ReLU). The execution network is used to output the sorting results of the test cases according to the sorting strategy, including the first sorting and the second sorting; the evaluation network is used to update the execution network according to the maximized reward function to reduce the variance.

[0143] Fig.10 A comparison diagram of the first learning model encoding part of a test case sorting method in an embodiment of the present disclosure and the pseudo code is shown, as shown in FIG. Fig.10 As shown in the figure, in the encoding (Bi-LSTM) part, the main logical steps include:

[0144] The Model and related layer definitions (Input, Bidirectional, Dense) are imported from the Keras library.

[0145] A function build_bi_lstm_model is defined to build a bidirectional long short-term memory network (Bi-LSTM) model. The function accepts two parameters: input_shape represents the shape of the input data, and output_shape represents the number of neurons in the output layer.

[0146] Inside the function:

[0147] First, an input layer is created, whose shape is specified by input_shape.

[0148] Then a bidirectional LSTM layer is added with 32 units in each direction.

[0149] The prediction result is then output through a fully connected layer (Dense), and the softmax activation function is used so that the output can be interpreted as a probability distribution, which is suitable for multi-classification tasks.

[0150] Finally, a complete model is defined based on the input layer and the output layer, and it is compiled. The optimizer used is Adam, the loss function is cross entropy (categorical_crossentropy), and the accuracy is monitored as a performance indicator during the training process.

[0151] A specific model instance is constructed using this function, in which the input sequence length is set to be variable, the feature dimension is 100, and there are 5 different classification categories.

[0152] Print out an overview of the model structure.

[0153] Fig.11 A comparison diagram of a first learning model decoding part of a test case ranking method in an embodiment of the present disclosure and a pseudo code is shown, as shown in FIG. Fig.11 As shown in the figure, in the decoding (LSTM) part, the main logical steps include:

[0154] The Sequential model and required layers (LSTM, Dense) are imported from the Keras module of TensorFlow.

[0155] It is assumed that a time series dataset named data is already available and necessary preprocessing has been performed.

[0156] A sequential model was created.

[0157] An LSTM layer is added to the model with 50 units, and the shape of the input data is specified, where timesteps represents the number of time steps and features represents the number of features at each step.

[0158] A fully connected layer is added for the final output. Assuming this is a regression problem, there is only one output node.

[0159] The Adam optimization algorithm was selected when compiling the model, and the mean squared error (mse) was used as the loss function.

[0160] The model is trained using the given data data and its corresponding labels labels for a total of 100 cycles, while controlling the amount of printed information to be medium.

[0161] Use the trained model to predict the validation data validation_data and get the predicted value predictions.

[0162] Fig.12 A flow chart of a test case sorting method according to an embodiment of the present disclosure is shown as follows: Fig.12 As shown, the main process steps include:

[0163] Inputting feature vectors of a plurality of test cases into a first learning model;

[0164] The first learning model sorts multiple test cases according to the sorting strategy;

[0165] performing software testing according to the first ranking;

[0166] Determine whether to pay attention to the execution time of software testing;

[0167] If the execution time is not important, the first reward function of the first learning model is maximized;

[0168] If you care about execution time, maximize the second reward function of the first learning model;

[0169] The first reward function or the second reward function is fed back to the first learning model, the sorting strategy is adjusted, and the second sorting is output.

[0170] Test case sorting technology in a continuous integration environment is a pointer sorting method based on reinforcement learning. It uses reinforcement learning algorithms to build models and can adaptively adjust its own strategies as the neural network evolves. Different from the previous reinforcement learning methods that focus on the characteristics of a single test case, it uses the complete test case set features as model input, uses the attention mechanism to consider the characteristics of all test cases at each step of sorting, and calculates indicators for the overall sorting results, which are fed back to the reinforcement learning model to dynamically adjust its strategy. The purpose of the present invention is to use machine learning methods to screen and sort test cases and complete smoke testing in DevOps.

[0171] The main ideas are: Decoupling: Decoupling of microservice containers. Use case sorting: Sort the use cases in the order of optimization using reinforcement learning. Smoke testing: Continuously test and provide feedback based on the sorting results.

[0172] Use reinforcement learning algorithms to sort the elements in the test case set, execute the test cases that may reveal errors and take less time first, and then execute the test cases that will not reveal errors and take longer, so as to expose software defects in a timely manner and give developers the fastest feedback.

[0173] Here we use a new "pointer sorting method" for test cases in a continuous integration environment based on reinforcement learning: the method uses historical information and other features of the test cases as input (including its estimated execution time, the last execution time, and historical execution results). In each integration cycle, the neural network uses the pointer attention mechanism to obtain the degree of attention to all alternative test cases, thereby obtaining the sorting results, and obtains the direction of strategy update from the feedback of test execution, calculates the NAPFD indicator, which is used for backpropagation of the model to update parameters, and continuously adjusts the sorting strategy in the process of "sorting-running test-feedback", ultimately achieving good sorting performance.

[0174] The reinforcement learning strategy is randomly initialized. A test case set is input, and the model calculates the priority of each test case based on the test case characteristics. The test cases are sorted according to the priority, and then the test is scheduled and executed according to the sorting results. The test results are fed back to the reinforcement learning model. The model dynamically adjusts its strategy based on the feedback to maximize the preset reward function.

[0175] The pointer sorting method can adjust the sorting strategy as the software version evolves, and effectively improve the error detection capability of the test sequence in a continuous integration environment. The pointer sorting method has good robustness to the length of the input historical information, and a small amount of historical information can achieve the optimal effect, and can handle regression test cases and new test cases well. Compared with previous machine learning algorithms, it considers more comprehensive information when sorting, is not easily misled by the characteristics of a single test case and falls into the local optimum, and can directly optimize the final goal based on the overall sorting effect feedback, and is not easily misled by artificially set intermediate goals.

[0176] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations, and various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of this disclosure have all been authorized.

[0177] Based on the same inventive concept, the present disclosure also provides a test case sorting device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0178] Fig.13 A schematic diagram of a test case sorting device in an embodiment of the present disclosure is shown. Fig.13 As shown, the device comprises:

[0179] A first ranking determination module 1301 is used to obtain feature vectors of multiple test cases and determine a first ranking of the multiple test cases using a ranking strategy of a first learning model;

[0180] A test result determination module 1302 is used to perform software testing according to the first sorted plurality of test cases to determine the test result;

[0181] A sorting strategy adjustment module 1303, configured to maximize the reward function of the first learning model according to the test result to adjust the sorting strategy;

[0182] The sorting update module 1304 is used to update the multiple test cases from the first sorting to the second sorting according to the adjusted sorting strategy corresponding to the maximization reward function; wherein, the multiple test cases in the second sorting are used to execute the most error-revealing test cases within a unit time when executing software testing.

[0183] It should be noted that the first sorting determination module 1301, the test result determination module 1302, the sorting strategy adjustment module 1303 and the sorting update module 1304 correspond to S202 to S208 in the method embodiment, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0184] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0185] Refer to the following Fig.14 1400 according to this embodiment of the present disclosure is described. Fig.14The electronic device 1400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0186] like Fig.14 As shown, the electronic device 1400 is in the form of a general computing device. The components of the electronic device 1400 may include but are not limited to: at least one processing unit 1410, at least one storage unit 1420, and a bus 1430 connecting different system components (including the storage unit 1420 and the processing unit 1410).

[0187] The storage unit stores a program code, and the program code can be executed by the processing unit 1410, so that the processing unit 1410 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1410 can execute the following steps of the above method embodiment: obtain feature vectors of multiple test cases, and use the sorting strategy of the first learning model to determine the first sorting of the multiple test cases; perform software testing according to the multiple test cases of the first sorting to determine the test results; maximize the reward function of the first learning model according to the test results to adjust the sorting strategy; update the multiple test cases from the first sorting to the second sorting according to the adjusted sorting strategy corresponding to the maximized reward function; wherein, when executing software testing, the multiple test cases of the second sorting are used to execute the most error-revealing test cases within a unit time.

[0188] The storage unit 1420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 14201 and / or a cache storage unit 14202 , and may further include a read-only storage unit (ROM) 14203 .

[0189] The storage unit 1420 may also include a program / utility 14204 having a set (at least one) of program modules 14205, such program modules 14205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0190] Bus 1430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0191] The electronic device 1400 may also communicate with one or more external devices 1440 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1400, and / or communicate with any device that enables the electronic device 1400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1450. Furthermore, the electronic device 1400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1460. As shown, the network adapter 1460 communicates with other modules of the electronic device 1400 via a bus 1430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0192] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0193] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above test case sorting method when executed by a processor.

[0194] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0195] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0196] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0197] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0198] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0199] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0200] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0201] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0202] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A test case sorting method, characterized in that: include: Acquire feature vectors of a plurality of test cases, and determine a first ranking of the plurality of test cases using a ranking strategy of a first learning model; Execute software testing according to the first sorted plurality of test cases and determine the test results; Maximizing the reward function of the first learning model according to the test result to adjust the sorting strategy; According to the adjusted sorting strategy corresponding to the maximized reward function, the multiple test cases are updated from the first sorting to the second sorting; wherein, the multiple test cases in the second sorting are used to execute the most error-revealing test cases within a unit time when executing software testing.

2. The test case sorting method according to claim 1, characterized in that: Acquiring feature vectors of a plurality of test cases, and determining a first ranking of the plurality of test cases using a ranking strategy of a first learning model, comprises: Obtaining historical information of multiple test cases; the historical information of the test cases includes: estimated execution time of the test cases, execution time of the last software test executed by the test cases, and historical execution results; Determining feature vectors of the multiple test cases according to the historical information of the multiple test cases; Inputting the feature vectors of the multiple test cases into a first learning model, and calculating the priority of each test case in the multiple test cases using a sorting strategy of the first learning model; The multiple test cases are sorted according to the priority of each test case to determine a first sorting of the multiple test cases.

3. The test case sorting method according to claim 1, characterized in that: Execute software testing according to the first-ordered plurality of test cases and determine test results, including: Execute software testing according to the first sorted test cases, and determine the execution time and number of errors revealed by each software test case when executing the software test; the number of errors revealed is the number of program errors revealed by the software test case when executing the software test; The execution time and number of errors revealed by each software test case during software testing are determined as test results.

4. The test case sorting method according to claim 3, characterized in that: Maximizing the reward function of the first learning model according to the test result to adjust the sorting strategy includes: If the execution time of the test case when executing the software test is not of concern, maximizing the first reward function of the first learning model according to the test result and the first ranking; If the execution time of the test case when executing the software test is of concern, maximizing the second reward function of the first learning model according to the test result and the first ranking; According to the maximized first reward function or second reward function, the parameters of the first learning model are updated using a gradient descent algorithm to adjust the sorting strategy; wherein the sorting strategy is adjusted to achieve the goal of executing the most fault-revealing test cases within a unit time.

5. The test case sorting method according to claim 4, characterized in that: If the execution time of the test case when executing the software test is not concerned, maximizing the first reward function of the first learning model according to the test result and the first ranking includes: If the execution time of the test case when executing the software test is not concerned, then according to the first sorting, the position of each test case in the first sorting is determined; According to the test results, determining the test cases that revealed errors during the software test execution; Determine a first reward function of the first learning model according to the position of each test case in the first ranking and the test cases that reveal errors during software test execution; A first reward function of the first learning model is maximized according to the test result and the first ranking.

6. The test case sorting method according to claim 5, characterized in that: The first reward function of the first learning model is determined as follows: Where NAPFD is the index value of the first reward function. The larger the index value, the higher the test case with more errors is in the ranking; rank(t) is the position of test case t in the ranking; TS i A collection of multiple test cases; TS i fail A collection of test cases that reveal errors during software test execution; TS i total,fail It is a set of multiple test cases and error-revealing test cases; p is an intermediate parameter.

7. The test case sorting method according to claim 4, characterized in that: If the execution time of the test case when executing the software test is of concern, maximizing the second reward function of the first learning model according to the test result and the first ranking includes: If the execution time of the test case during the software test is of interest, then according to the first sorting, the position of each test case in the first sorting is determined; According to the test results, determine the test cases that revealed errors during the software test execution and the actual execution time of each test case; Determine a second reward function of the first learning model according to the position of each test case in the first ranking, the test cases that reveal errors during software test execution, and the actual execution time of each test case; A second reward function of the first learning model is maximized according to the test result and the first ranking.

8. The test case sorting method according to claim 7, characterized in that: The second reward function of the first learning model is determined as follows: Among them, NAPFDc is the index value of the second reward function. The larger the index value, the higher the error-revealing test case with shorter execution time will be in the ranking; TS i A collection of multiple test cases; TF i is the position of the i-th error-revealing test case in multiple test cases; m is the number of error-revealing test cases in the test cases executed by the software test; n is the number of test cases executed by the software test; t i is the actual execution time of test case i.

9. The test case sorting method according to claim 1, characterized in that: The method further comprises: updating the plurality of test cases from the first sorting to a second sorting according to the adjusted sorting strategy corresponding to the maximized reward function, comprising: Determining an updated first learning model according to the adjusted sorting strategy corresponding to the maximized reward function; Inputting the test results into the updated first learning model, and calculating the update priority of multiple test cases for executing software tests according to the first sorting using the adjusted sorting strategy; The multiple test cases are sorted according to the update priority, and the multiple test cases are updated from the first sorting to a second sorting.

10. The test case sorting method according to claim 1, characterized in that: Also includes: Performing a smoke test on the multiple test cases sorted according to the second sorting order, and determining a smoke test result; The second sorting is optimized according to the smoke test result.

11. A test case sorting device, characterized in that: include: A first ranking determination module, configured to obtain feature vectors of a plurality of test cases and determine a first ranking of the plurality of test cases using a ranking strategy of a first learning model; A test result determination module, used to perform software testing according to the first sorted plurality of test cases and determine the test result; A sorting strategy adjustment module, used to maximize the reward function of the first learning model according to the test result to adjust the sorting strategy; A sorting update module is used to update the multiple test cases from the first sorting to a second sorting according to the adjusted sorting strategy corresponding to the maximization reward function; wherein, the multiple test cases in the second sorting are used to execute the most error-revealing test cases within a unit time when executing software testing.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the test case sorting method according to any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the test case sorting method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the test case sorting method according to any one of claims 1 to 10 is implemented.

Images