Method for detecting test case and computing equipment

Abstract

The invention provides a method for detecting a test case and a computing device, and the method comprises the steps: statically analyzing whether a function corresponding to each packaging function in the test case obtained from a database meets a rule or not based on a related rule in a rule library, and detecting the test case, thereby judging whether the test case is a flag test or not. Whether the test case is stable or not can be determined by conducting static analysis on the test case, the system performance is guaranteed, and the situation that the system performance is reduced due to the fact that the test case is operated for multiple times can be avoided.

Description

Technical Field

[0001] This application relates to the field of software testing, and more specifically, to a method for detecting test cases and a computing device. Background Art

[0002] In the field of software engineering, "testing" is a core component for ensuring software quality. Currently, the most commonly used "flaky tests" refer to the situation where a test is considered unstable when it can sometimes pass and sometimes fail in multiple retry attempts without any code changes. For example, a test is executed and fails, then the same test is executed again without any code changes, but this time it passes successfully. It can be seen that "flaky tests" are tests that show passing and failing results using the same code. In view of this, a test failure may or may not mean that there is a new problem. Trying to reproduce the failure by re-running the test with the same version of the code may or may not result in the test passing. Therefore, these tests can be considered unreliable and ultimately lose their value. However, ignoring the tests also means ignoring the errors in actual production.

[0003] From the above introduction, it can be seen that to detect flaky tests, the traditional method is to simply re-execute to confirm its instability. For unit tests, this method is acceptable. However, at the system test level, due to its wide coverage, repeated execution will consume a large amount of time and computing resources. Therefore, a method for detecting test cases is needed to ensure system performance. Summary of the Invention

[0004] This application proposes a method for detecting test cases. By using the rules in the rule library to detect the respective function capabilities of each encapsulated function in the test case, it can be determined whether the test case is stable through static analysis, ensuring system performance and avoiding the reduction of system performance caused by running the test case multiple times.

[0005] In a first aspect, a method for detecting test cases is provided. This method can be executed by a computing device, or alternatively, by a component (such as a chip or a circuit) of the computing device. For example, the computing device can be a computer.

[0006] The method includes: receiving first information of a user, where the first information is used to indicate a test case, and the test case includes at least one encapsulated function; determining, according to the test case, function paths respectively corresponding to at least one encapsulated function included in the test case in a database, where the database includes a first mapping relationship between at least one encapsulated function and function paths respectively corresponding to at least one encapsulated function; obtaining at least one encapsulated function according to the function paths; detecting the test case according to at least one encapsulated function and a rule library, where the rule library includes at least one rule, and each rule in the at least one rule is used to detect whether the function function respectively corresponding to at least one encapsulated function included in the test case conforms to the rule; and outputting a detection result to the user.

[0007] In a possible implementation manner, the first information includes a specific test case. Exemplarily, the first information of the user is a file of the test case, and at least one test case is included in the file. In another possible implementation manner, the first information includes address information of a location where the test case is located, and the test case can be obtained based on the first information input by the user. Exemplarily, the first information input by the user is server and file address information in the server, and the test case can be obtained in a corresponding file of the corresponding server based on the address information.

[0008] In this application, the manner of receiving first information from a user may include at least one of the following: for example, the first information may be input by the user in text through a dialog box, and the computing device may also identify the first information input by the user through voice recognition, etc. This application does not limit how the computing device receives the first information from the user.

[0009] In this application, the "function function" can also be understood as the internal logic of the encapsulated function, or the function logic, or it can also be understood as what the encapsulated function is specifically doing or what function it has implemented.

[0010] As a possible scenario, the solution provided in this application can be applied to be executed on an ordinary computing device. For example, the computing device may be a computer, or a module or unit constituting a computer. At this time, the "user" in this application can be understood as a client, and in a cloud scenario, the "user" can also be understood as a "tenant".

[0011] As another possible scenario, the solution provided in this application can be applied to a cloud service scenario. For example, it may be executed by a cloud management platform, or for example, it may be executed by a communication device in the cloud management platform.

[0012] Based on the above technical solution, in this application, it is detected whether each encapsulation function corresponding to the test cases obtained by statically analyzing the database based on the relevant rules in the rule library meets the rule, so as to determine whether the test case is a flaky test. In this application, it can be determined whether the test case is stable by statically analyzing the test case, which guarantees the system performance and can avoid the reduction of the system performance caused by running the test case multiple times.

[0013] Combined with the first aspect, in a possible implementation manner, the rules in the rule library include: the function functions corresponding to at least one encapsulation function cannot conflict with each other, and / or, the function functions corresponding to at least one encapsulation function cannot conflict with each other inside.

[0014] Combined with the first aspect, in a possible implementation manner, when it is detected that the function functions corresponding to at least one encapsulation function included in the test case do not meet the rules, the detection result includes at least one of the following: the name information of the rule, the description information of the rule, and the location information of the code segment, where the description information of the rule is used to indicate the content of the rule, and the location information of the code segment is used to indicate the location of the code segment that does not meet the rule in the code corresponding to at least one encapsulation function, and the code includes code segments.

[0015] For example, the "name information of the rule" may be an identifier or index corresponding to a certain rule. The "description information of the rule" may be a detailed description of a certain rule. For example, the description information can be used to indicate the content specified by the rule (it can also be understood that the description information is used to indicate the implementation logic of the rule). The "location information" is used to indicate the location of the code segment that does not meet the rule in the code corresponding to the encapsulation function. For example, it may include the line number corresponding to the code segment.

[0016] Combined with the first aspect, in a possible implementation manner, when it is detected that the test case meets the rules, for example, the detected result output can be empty, that is, the detected result can include any content.

[0017] Combined with the first aspect, in a possible implementation manner, the method further includes: executing the test case and obtaining the execution log corresponding to the test case, where the execution log is used to record the specific information of the execution process of the test case; determining the actually executed code segments in the code corresponding to each encapsulation function in the test case according to the execution log; and detecting the test case, including: detecting the actually executed code segments.

[0018] In combination with the first aspect, in a possible implementation manner, according to the execution log, determining the code segments actually executed in the code corresponding to each of at least one encapsulated function included in the test case includes: aligning the execution log with the code corresponding to each of at least one encapsulated function included in the test case according to the window alignment algorithm and / or the similarity algorithm to generate a second mapping relationship, where the second mapping relationship is the mapping relationship between the code corresponding to each of at least one encapsulated function and the code in the execution log; and determining the code segments actually executed in the code corresponding to each of at least one encapsulated function according to the second mapping relationship.

[0019] Based on the above technical solution, in the present application, the test case can be dynamically executed to obtain the execution log. By comparing the execution log with the code corresponding to the encapsulated function included in the test case, the actually executed code segments can be determined, and then static analysis can be performed on the actually executed code segments, which can reduce misjudgment and execution costs and ensure more accurate detection results.

[0020] In combination with the first aspect, in a possible implementation manner, the method further includes: analyzing the code corresponding to each of at least one encapsulated function included in the test case, the detection result, and the execution log to generate a feature vector corresponding to the test case, where the feature vector is used to characterize the behavior of the test case; and predicting whether the test case is an unreliable flaky test according to the machine learning model and the feature vector, where the input of the machine learning model is the feature vector, the output of the machine learning model is the prediction result, and the machine learning model is a model for predicting whether the test case is a flaky test according to the feature vector corresponding to the test case.

[0021] Based on the above technical solution, in the present application, the execution log is obtained based on the dynamic execution of the test case, and then the actually executed code segments are determined based on the execution log. By performing static analysis on the actually executed code segments, the extracted feature vector is made more accurate, and when predicting through the trained model later, the prediction result will be more accurate.

[0022] In combination with the first aspect, in a possible implementation manner, the method can also be executed by a cloud management platform in a cloud service system, where the cloud management platform is used to manage the infrastructure for providing cloud services, the infrastructure includes at least one cloud data center, and each of the at least one cloud data centers includes at least one server. In a possible implementation manner, a database and a rule library are included on the at least one server; in a possible implementation manner, test cases are further included on the at least one server.

[0023] Second aspect, the present application proposes a computing device, which is configured to execute the method of the first aspect above. Specifically, the device may include units and / or modules for executing the methods proposed by the present application, such as a transceiver module and / or a processing module. Exemplarily, the computing device may be a computer or a cloud management platform in a cloud service scenario. The cloud management platform is used to manage the infrastructure that provides cloud services, and the infrastructure includes at least one cloud data center, and each cloud data center in the at least one cloud data center includes at least one server. In one possible implementation, a database and a rule library are included on the at least one server; in another possible implementation, test cases are further included on the at least one server.

[0024] Third aspect, the present application provides a computing device, which includes: at least one processor, configured to execute a computer program or instruction stored in a memory to execute the method of the first aspect above. Optionally, the device further includes a memory for storing the computer program or instruction. Optionally, the device further includes a communication interface through which the processor reads the computer program or instruction stored in the memory.

[0025] In one implementation, the computing device is a device for implementing the functions of the above method in a chip.

[0026] In another implementation, the computing device is a chip, a chip system or a circuit for implementing the functions of the above method in a chip.

[0027] Fourth aspect, the present application provides a processor, which includes: an input circuit, an output circuit and a processing circuit. The processing circuit is configured to receive a signal through the input circuit and transmit the signal through the output circuit, so that the processor executes the method of the first aspect above.

[0028] In a specific implementation process, the above processor may be one or more chips, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be transistors, gate circuits, flip-flops and various logic circuits, etc. The input signal received by the input circuit may be received and input by, for example but not limited to, a transceiver, and the signal output by the output circuit may be output to, for example but not limited to, a transmitter and transmitted by the transmitter, and the input circuit and the output circuit may be the same circuit, which is used as the input circuit and the output circuit at different times respectively. The embodiments of the present application do not limit the specific implementation manners of the processor and various circuits.

[0029] For operations such as sending, obtaining / receiving, etc. involved in the processor, if there is no special description, or if it does not conflict with its actual function or internal logic in the relevant description, it can be understood as operations such as the processor outputting and receiving, inputting, etc., and can also be understood as the sending and receiving operations performed by the radio frequency circuit and the antenna. This application does not make any limitations in this regard.

[0030] In a fifth aspect, a processing device is provided, including a processor and a memory. The processor is configured to read instructions stored in the memory, and can receive signals through a transceiver and transmit signals through a transmitter to execute the method of the first aspect described above.

[0031] Optionally, the processor is one or more, and the memory is one or more.

[0032] Optionally, the memory can be integrated with the processor, or the memory is separately provided from the processor.

[0033] In a specific implementation process, the memory can be a non-transitory memory, such as a read only memory (ROM), which can be integrated with the processor on the same chip or can be separately provided on different chips. This application embodiment does not make any limitations on the type of the memory and the setting manner of the memory and the processor.

[0034] It should be understood that relevant data interaction processes, such as sending the first information, can be a process of outputting the first information from the processor, and the receiving ability information can be a process of the processor receiving input ability information. Specifically, the data output by the processor can be output to the transmitter, and the input data received by the processor can come from the transceiver. Among them, the transmitter and the transceiver can be collectively referred to as the transceiver.

[0035] The processing device in the above fifth aspect can be one or more chips. The processor in the processing device can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0036] In a sixth aspect, a computing cluster is provided, including at least one computing device, and each computing device includes a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in any possible implementation manner of the first aspect.

[0037] Optionally, the processor can be a general-purpose processor, which can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0038] In a seventh aspect, a computer-readable storage medium is provided. The computer-readable medium stores program code for a device to execute, and the program code includes the method for executing the first aspect described above.

[0039] In an eighth aspect, a computer program product including instructions is provided. When the computer program product runs on a computer, it causes the computer to execute the method of the first aspect described above.

[0040] In a ninth aspect, a chip system is provided, including a processor for calling and running a computer program from a memory, so that a device installed with the chip system executes the method of the first aspect described above. Description of the Drawings

[0041] Figure 1 is a schematic block diagram of typical steps of the flaky test provided in this application.

[0042] Figure 2 is a schematic flowchart of a method 200 for detecting test cases provided in this application.

[0043] Figure 3 is a schematic diagram of a cloud service scenario used in this application.

[0044] Figure 4 is a schematic block diagram of a computing device 400 provided in this application.

[0045] Figure 5 is a schematic block diagram of a computing device 500 provided in this application. Detailed Embodiments

[0046] Next, the technical solutions in this application will be described with reference to the drawings.

[0047] To facilitate understanding of the technical solutions of this application, the following first briefly introduces some individual technical terms involved in this application.

[0048] 1. Test case

[0049] A "test case" is a test input, execution conditions, and expected results prepared for a specific purpose to test a program path or verify whether a specific requirement is met. Generally speaking, it is to describe the operation steps of our test system in text according to a certain format.

[0050] Standard test cases usually consist of the following modules:

[0051] (1) Test case number: The unique identifier of the test case.

[0052] (2) Module: Indicates which module the tested requirement specifically belongs to, mainly for better identification and maintenance of test cases.

[0053] (3) Test case title: Also known as the test point, it is to describe the focus of the test case in one sentence. Each test case corresponds to a test purpose.

[0054] (4) Priority: Defined according to the priority level of the requirement. High-priority requirements should cover core business, important features, and parts with high usage frequency.

[0055] (5) Preconditions: Some conditions that need to be met before the test case is executed, otherwise the test case cannot be executed. For example, some test environments or operations that need to be performed in advance.

[0056] (6) Test data: When executing the test, some external data need to be input to complete the test. These data are determined according to the statistical situation of the test case, including parameters, files, or database records, etc.

[0057] (7) Test steps: The step description of the test case, and the executor can complete the execution of the test according to the test steps.

[0058] (8) Expected result: The most important part of the test case, mainly used to judge whether the object under test is normal. The user's expectations should be described according to the requirements.

[0059] (9) Actual result: PASS for passed, FAIL for failed, N / A for not executed.

[0060] 2. Function encapsulation

[0061] Function encapsulation is a function feature that encapsulates the functions and classes written by a programmer to implement a certain function, and only provides a simple function interface to the outside. When a programmer needs to perform the same operation during programming, the programmer (caller) does not need to write the same function for invocation, but can directly call it from the function library. Programmers can also download functional functions from the Internet and then encapsulate them into the library functions of the compiler. When a function that needs to execute this function is required, it can be directly called. The programmer does not need to know how the function is implemented internally, but only needs to know what functions or classes this function provides.

[0062] In this application, the function after function encapsulation is called an "encapsulated function". Usually, an encapsulated function may include multiple simple functions, and these multiple simple functions together implement a certain function. At least one encapsulated function can be included in "a test case" in this application.

[0063] 3. Flaky tests

[0064] In the field of software engineering, "testing" is a core component to ensure software quality. The currently used flaky test refers to an unreliable and unstable test phenomenon, that is, under the same software code and configuration environment, uncertain test results (for example, sometimes successful and sometimes failed) are obtained. If a unit test returns different results (for example, pass or fail) in any two executions and no underlying modifications are made to the source code, then it is considered unstable. A flaky test may be caused by program-level uncertainties in the test code or the code under test (for example, thread ordering and other concurrency issues), or it may be caused by the variability of the test environment (for example, the machine on which the test is executed, the test set executed simultaneously, etc.). The former requires fixing the code, while the latter requires identifying the causes of the uncertainty and resolving these causes to eliminate the phenomenon of flakiness. Tests on code patterns and infrastructure must aim to reduce the likelihood of flaky tests occurring.

[0065] Figure 1 It is a schematic block diagram of the typical steps of the flaky test in the current solution, as Figure 1As shown in the figure, during flaky testing, feature extraction is first performed on the test cases. "Feature extraction" can be understood as retrieving the sample library twice to build a bag-of-words dictionary, which can be used to count the word frequencies of each sample to build feature vectors. Usually, this feature vector is used to characterize the performance and behavior patterns of the test cases. Then, dimensionality reduction is performed on the feature vectors. "Dimensionality reduction" can be understood as the fact that based on the bag-of-words and word frequency methods, feature vectors with thousands of dimensions will be generated, and techniques such as principal component analysis or sparse projection are applied to the feature vectors to avoid dimensionality explosion. Finally, ensemble learning is performed. "Ensemble learning" can be understood as training a classifier based on ensemble learning for the dimensionality-reduced feature vectors, thereby improving the accuracy and stability of the model. Exemplarily, the trained model can be used to predict whether a test case is a flaky test.

[0066] As can be seen from the above introduction, to conduct flaky testing, the traditional method is to simply repeat the execution of test cases to confirm their instability. For unit testing, this method is acceptable. However, at the system testing level, due to its wide coverage, repeated execution will consume a large amount of time and computing resources. Therefore, a method for detecting test cases is needed to ensure system performance.

[0067] In view of this, the present application proposes a method for detecting test cases, which detects the respective function functions of each encapsulated function in the test case through the rules in the rule library, so that it can be determined whether the test case is stable through static analysis, ensuring system performance, and avoiding the reduction of system performance caused by multiple runs of test cases.

[0068] Figure 2 is a schematic flowchart of a method 200 for detecting test cases proposed by the present application. As Figure 2 shown, the method includes:

[0069] 210, receiving the first information input by the user, where the first information is used to indicate the test case.

[0070] In a possible implementation manner, the first information includes the specific test case. Exemplarily, the first information input by the user is a file of the test case, and at least one test case is included in the file. In another possible implementation manner, the first information includes the address information of the location where the test case is located, and the test case can be obtained based on the first information input by the user. Exemplarily, the first information input by the user is the file address information, and the test case can be obtained based on this file address information.

[0071] As a possible scenario, the solution provided by the present application can be applied to be executed on a general computing device. For example, the computing device can be a computer, or a module or unit that constitutes a computer. At this time, the "user" in the present application can be understood as the client.

[0072] As another possible scenario, the solution provided in this application can be applied to a cloud service scenario. Figure 3 It is a schematic block diagram of a cloud scenario shown in this application. As Figure 2 shown, this cloud scenario may include: a cloud management platform 310, the Internet 320, and a client 330. As Figure 3 shown, the cloud management platform 310 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each cloud data center includes multiple servers, and each server includes cloud service resources respectively, providing corresponding cloud services for tenants. In the embodiments of this application, the cloud service resources may be cloud databases.

[0073] The cloud management platform 310 may be located in the cloud data center, and it may provide access interfaces (such as interfaces or application program interfaces (APIs)). Tenants can operate the client 330 to remotely access the access interface to register cloud account numbers and passwords on the cloud management platform 110 and log in to the cloud management platform 310. After the cloud management platform 310 authenticates the cloud account number and password successfully, tenants can further pay on the cloud management platform 110 to select and purchase virtual machines of specific specifications (processors, memory, disks). After the successful payment and purchase, the cloud management platform 310 provides the remote login account number and password of the purchased virtual machine, and the client 330 can remotely log in to this virtual machine and install and run the applications of the tenant in this virtual machine. Therefore, tenants can create, manage, log in to, and operate virtual machines in the cloud data center through the cloud management platform 310.

[0074] The functions of the cloud management platform 310 include but are not limited to a tenant console, computing management services, network management services, storage management services, authentication services, and image management services. The tenant console provides interfaces or APIs to interact with tenants. The computing management services are used to manage the servers running virtual machines and containers and bare metal servers. The network management services are used to manage network services (such as gateways, firewalls, etc.). The storage management services are used to manage storage services (such as data bucket services). The authentication services are used to manage the account numbers and passwords of tenants. The image management services are used to manage virtual machine images. Tenants use the client 330 and can log in to the cloud management platform 310 through the Internet 320 to manage the rented cloud services.

[0075] In a cloud scenario, the "tenant" mentioned above is a logical concept. For example, after each person or an enterprise registers an account on a public cloud platform, the platform considers us as a "tenant", and the cloud system will allocate cloud resources with the "tenant" as the basic unit. Usually, a tenant can lease the services provided by a cloud service provider on a cloud computing platform. The tenant needs to provide identity information, contact person and contact information, and a deduction account information for paying the rent of the cloud service. It should be understood that the tenants of cloud services can be individuals, enterprises, schools, hospitals, administrative organs, etc.

[0076] In this scenario, the execution subject of the method provided in this application can be understood as the "cloud management platform" (or a module in the cloud management platform), and the "user" in this application can be understood as the "tenant".

[0077] 220. According to this test case, determine the function path corresponding to each of the at least one encapsulated function included in the test case in the database.

[0078] In this application, the database includes the function path corresponding to each encapsulated function. For example, the database includes a first mapping relationship between each encapsulated function and its corresponding function path. For example, this mapping relationship can be in the form of a table, as shown in Table 1 below.

[0079] Table 1

[0080] Encapsulation function Path Encapsulation function #1 Function path #A Encapsulation function #2 Function path #B Encapsulation function #3 Function path #C

[0081] 230. According to the function path, obtain the at least one encapsulated function.

[0082] It should be understood that based on the path corresponding to each encapsulated function, the specific code corresponding to the encapsulated function can be obtained, so as to determine the specific content and / or function represented by the encapsulated function.

[0083] Exemplarily, for example, based on Table 1, it can be found that the path of encapsulated function #1 is Path #A, and based on Path #A, the specific function included in the encapsulated function can be found; exemplarily, based on Table 1, it is found that the path of encapsulated function #2 is Path #B, and based on Path #B, the code of the encapsulated function can be determined to be obtained, and it is determined that the encapsulated function is a function for initializing the program running environment.

[0084] 240. Detect the test case according to the at least one encapsulated function and the rule library.

[0085] The "rule library" in this application includes at least one rule, and each of the at least one rule is used to detect whether the function features corresponding to each of the at least one encapsulated function included in the test case comply with the rule. Exemplarily, each rule in the rule library can be determined based on historical experience and expert experience. For example, the rule library can be pre-configured.

[0086] In this application, in a possible implementation manner, the rule library includes the following rule: The function features corresponding to each of the at least one encapsulated function cannot conflict with each other.

[0087] For example, test case #1 includes two encapsulated functions, and the function features corresponding to each of the two encapsulated functions are both parallel processing functions. Exemplarily, the rule can specifically be rule #1, and rule #1 stipulates that the function features corresponding to each of the two encapsulated functions should be functions with a sequential order. Exemplarily, assuming that this test case is for the scenario of writing an email and sending an email, and this test case includes two encapsulated functions. Generally, the encapsulated function for writing an email should be executed first, and then the encapsulated function for sending an email should be executed. However, the two encapsulated functions in this test case do not have a sequential order and are parallel processing functions, which can also be understood as that the encapsulated function for writing an email and the encapsulated function for sending an email are executed simultaneously. Therefore, the function features corresponding to each of the two encapsulated functions included in this test case do not comply with the rule.

[0088] Another example is that test case #2 includes three encapsulated functions. Exemplarily, the rule can specifically be rule #2, and what rule #2 stipulates is that the test case needs to include an encapsulated function for initializing the program running environment. Assume that the three encapsulated functions are encapsulated function #a, encapsulated function #b, and encapsulated function #c respectively. Among them, when encapsulated function #a is executed, it modifies the running environment of the program initial configuration. Encapsulated function #b needs to continue to be executed in the running environment modified by encapsulated function #a, while encapsulated function #c needs to be executed on the running environment of the program initial configuration. However, in this test case, there is no function for initializing the program running environment. Therefore, the function features corresponding to each of the three encapsulated functions included in this test case do not comply with the rule.

[0089] In another possible implementation manner, the rule library can further include the following rule: The function features corresponding to each of the at least one encapsulated function cannot conflict with each other internally.

[0090] For example, Test Case #3 includes four encapsulated functions. The function of Encapsulated Function #d is to send an access request to a remote server. After the remote server returns a response, the response result is processed. However, for the code corresponding to Encapsulated Function #d, there is only the code for sending the request and the code for processing the response result. Usually, there is a waiting period between sending the request and processing the response result, that is, waiting for the remote server to query and return the response result. However, there is no code corresponding to the "waiting" function in Function #d. Therefore, the function corresponding to Function #d does not conform to this rule.

[0091] Based on the above technical solution, in the present application, based on the relevant rules in the rule library, statically analyze whether the function functions corresponding to each encapsulated function in the test cases obtained from the database meet the rule, and detect the test case, so as to determine whether the test case is a flaky test. In the present application, by statically analyzing the test case, it can be determined whether the test case is stable, which guarantees the system performance and can avoid the reduction of system performance caused by running the test case multiple times.

[0092] In a possible implementation manner, when detecting a test case, the following solution can be specifically adopted: execute the test case and obtain the execution log corresponding to the test case. According to the execution log, determine the code segments actually executed in the code corresponding to each encapsulated function in the test case. Therefore, the code segments actually executed can be detected. Exemplarily, in a possible implementation manner, the execution log can be aligned with the code corresponding to each encapsulated function in the test case according to the window alignment algorithm and / or the similarity algorithm to generate a second mapping relationship, where the second mapping relationship is the mapping relationship between the code corresponding to each of the at least one encapsulated function and the code in the execution log; according to the second mapping relationship, determine the code segments actually executed by each encapsulated function. Exemplarily, the second mapping relationship can be in the form of a table.

[0093] It should be understood that the "execution log" can be used to record the execution process of the test case, and is used to record the running status, abnormal status, error information, etc. of the code corresponding to each encapsulated function in the test case. For example, the execution log includes specific information about the execution process of the test case, such as the execution time (i.e., timestamp), level (info), requested path, request parameters, and output results.

[0094] Based on the above technical solution, the present application proposes to dynamically execute the test case in order to obtain the execution log. By comparing the execution log with the code corresponding to the encapsulated functions included in the test case, the code segments actually executed can be determined, and then statically analyze the code segments actually executed, which can reduce misjudgment and execution costs and ensure more accurate detection results.

[0095] Optionally, the method further includes analyzing the code, detection results, and execution logs corresponding to each encapsulated function in the test case to generate a feature vector corresponding to the test case, where the feature vector is used to characterize the performance and behavior of the test case; predicting whether the test case is an unreliable test (flaky test) based on the machine learning model and the feature vector, where the input of the machine learning model is the feature vector, the output of the machine learning model is the prediction result, and the machine learning model is a model for predicting whether the test case is a flaky test based on the feature vector corresponding to the test case. Exemplarily, when the output of the machine model is "0", it indicates that the test case is not a flaky test, and when the output of the machine model is "1", it indicates that the test case is a flaky test.

[0096] Exemplarily, the "machine learning model" in this application can be a decision tree model, random forest, neural network model, etc., without limitation. Exemplarily, the machine learning model can also be obtained by training the model based on the previously extracted feature vectors. For example. The dimensions of the feature vector can be added together to obtain the feature vector for a single execution of the entire test case, and a classifier can be trained to form a flaky test detection model that can be used for system-level test cases.

[0097] Based on the above technical solution, in this application, the execution log is obtained based on the dynamic execution of the test case, and then the actually executed code snippet is determined based on the execution log. By performing static analysis on the actually executed code snippet, the extracted feature vector can be made more accurate, and when predicting through the trained model later, the prediction result will be more accurate.

[0098] 250. Output the detection result to the user.

[0099] In one possible scenario, when it is detected that the test case does not conform to the rule, the detection result may include at least one of the following: the name information of the rule, the description information of the rule, and the location information of the code snippet. For example, the "name information of the rule" can be the identifier or index corresponding to a certain rule. The "description information of the rule" can be a detailed description of a certain rule. For example, the description information may include the content specified by the rule. The "location information" is used to indicate the location of the code snippet that does not conform to the rule in the code corresponding to the encapsulated function. For example, it may include the line number corresponding to the code snippet.

[0100] In another possible scenario, when it is detected that the test case conforms to the rule, for example, the output detection result may be empty, that is, the detection result may not include any content.

[0101] It should be noted that in this application, the method 200 can also be used in a cloud service system. For example, it can be executed by a cloud management platform in the cloud service system, where the cloud management platform is used to manage the infrastructure that provides cloud services. The infrastructure includes at least one cloud data center, and each cloud data center in the at least one cloud data center includes at least one server.

[0102] In a possible implementation, the at least one server includes a database and the rule library. At this time, it can also be understood that the database and the rule library are pre-stored in the servers on the cloud.

[0103] In a possible implementation, the at least one server further includes the test case. For example, a certain server or some of the at least one server may include a file library, and the file library includes multiple test cases. The user indicates the address in the file library where the test case is located through the first information, and at this time, the test case can be obtained.

[0104] It can be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the front and back associated objects.

[0105] Those skilled in the art should be able to realize that, combining the units and algorithm steps of each example described in the embodiments disclosed in this article, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described function for each specific application, but this implementation should not be considered to exceed the scope of this application.

[0106] The embodiments of this application can divide the functional modules of the computing device according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of this application is illustrative, only a logical function division, and there can be other division methods in actual implementation. The following will be described by taking the division of each functional module corresponding to each function as an example.

[0107] Figure 4 It is a schematic block diagram of the computing device 400 provided by the embodiments of this application. As shown in the figure, the computing device 400 may include: a transceiver module 410 and a processing module 420.

[0108] Among them, each of the above modules is respectively used to execute each step of the foregoing various methods, which will not be elaborated here.

[0109] It should also be understood that the computing device 400 here is embodied in the form of functional units. The term "unit" here may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a proprietary processor, or a group of processors, etc.) for executing one or more software or firmware programs, and a memory, a combined logic circuit, and / or other suitable components that support the described functions.

[0110] The computing device 400 of each of the above solutions has the function of implementing the corresponding steps of the foregoing method 200. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions; for example, the determination unit and the like can be replaced by a processor to respectively execute the transceiver operations and related processing operations in each method embodiment. In addition, the determination unit can be a processing circuit.

[0111] It should be noted that Figure 4 the computing device in may be the computing device (or cloud management platform, or communication device used in the cloud management platform) in the foregoing method embodiment, or a chip or chip system corresponding to the computing device (or cloud management platform, or communication device used in the cloud management platform), for example: a system on chip (SoC). Among them, the processing module is a processor or microprocessor or integrated circuit integrated on the chip. It is not limited here.

[0112] Figure 5 is a schematic block diagram of another computing device 500 provided by an embodiment of the present application. As shown in the figure, the device 500 includes: at least one processor 520. The processor 520 is coupled to the memory and is used to execute instructions stored in the memory to send signals and / or receive signals. Optionally, the device 500 further includes a memory 530 for storing instructions. Optionally, the device 500 further includes a transceiver 510, and the processor 520 controls the transceiver 510 to send signals and / or receive signals.

[0113] It should be understood that the above-mentioned processor 520 and memory 530 can be combined into a processing device, and the processor 520 is used to execute the program code stored in the memory 530 to implement the above functions. Specifically, the memory 530 can also be integrated in the processor 520 or independent of the processor 520.

[0114] It should also be understood that the transceiver 510 may include a transceiver (or receiver) and a transmitter (or transmitter). The transceiver may further include an antenna, and the number of antennas may be one or more. The transceiver 510 may be a communication interface or interface circuit.

[0115] Specifically, the processor 520 in the device 500 may correspond to the processing module 420 in the device 400. The transceiver 510 in the device 500 may correspond to the transceiver module 410 in the device 400.

[0116] As a solution, the device 500 is used to implement the steps corresponding to the computing device (or cloud management platform, or communication device in the cloud management platform) in the above-mentioned method 200 embodiment.

[0117] For example, the processor 520 is used to execute the computer programs or instructions stored in the memory 530 to implement the respective steps in the above-mentioned method 200.

[0118] In this embodiment, a computer program product containing instructions is also provided. The computer program product may be software or program product containing instructions that can run on a computing device cluster or be stored in any available medium. When it runs on the computing device cluster, it causes the computing device cluster to execute the above-provided method, or causes the computing device cluster to implement the functions of the above-provided device.

[0119] In this embodiment, a computer-readable storage medium is also provided. The computer-readable storage medium may be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive), etc. The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to execute the above-provided method.

[0120] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0121] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0122] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] In addition, in each embodiment of the present application, the functional units can be integrated in a processing module, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0125] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0126] As described above, it is only the specific implementation manner of the present application. However, the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.

Claims

1. A method for detecting test cases, characterized in that, Including: Receiving first information of a user, where the first information is used to indicate a test case, and the test case includes at least one encapsulated function; Determining, according to the test case, function paths respectively corresponding to the at least one encapsulated function included in the test case in a database, where the database includes a first mapping relationship between the at least one encapsulated function and the function paths respectively corresponding to the at least one encapsulated function; Obtaining the at least one encapsulated function according to the function paths; Detecting the test case according to the at least one encapsulated function and a rule library, where the rule library includes at least one rule, and each rule in the at least one rule is used to detect whether the function functions respectively corresponding to the at least one encapsulated function included in the test case conform to the rule; Outputting a detection result to the user.

2. The method according to claim 1, wherein The rule includes: the function functions respectively corresponding to the at least one encapsulated function cannot conflict with each other, and / or, the function functions respectively corresponding to the at least one encapsulated function cannot conflict with each other inside.

3. The method according to claim 1 or 2, characterized in that, When it is detected that the function functions respectively corresponding to the at least one encapsulated function included in the test case do not conform to the rule, The detection result includes at least one of the following: name information of the rule, description information of the rule, position information of a code segment, where the description information of the rule is used to indicate the content of the rule, and the position information of the code segment is used to indicate the position of the code segment that does not conform to the rule in the code respectively corresponding to the at least one encapsulated function, and the code includes the code segment.

4. The method according to any one of claims 1 to 3, characterized in that, Before detecting the test case, the method further includes: Executing the test case and obtaining an execution log corresponding to the test case, where the execution log is used to record specific information of the execution process of the test case; Determining, according to the execution log, the code segments actually executed in the code respectively corresponding to the at least one encapsulated function included in the test case; The detecting the test case includes: Detecting the actually executed code segments.

5. The method according to claim 4, characterized in that, The determining, according to the execution log, the code segments actually executed in the code respectively corresponding to the at least one encapsulated function included in the test case includes: Aligning the execution log with the code respectively corresponding to the at least one encapsulated function included in the test case according to a window alignment algorithm and / or a similarity algorithm to generate a second mapping relationship, where the second mapping relationship is a mapping relationship between the code respectively corresponding to the at least one encapsulated function and the code in the execution log; Determining, according to the second mapping relationship, the code segments actually executed in the code respectively corresponding to the at least one encapsulated function.

6. The method according to claim 4 or 5, characterized in that The method further includes: Analyzing the code respectively corresponding to the at least one encapsulated function included in the test case, the detection result, and the execution log to generate a feature vector corresponding to the test case, where the feature vector is used to characterize the behavior of the test case. Predict whether the test case is an unreliable flaky test according to the machine learning model and the feature vector, where the input of the machine learning model is the feature vector, the output of the machine learning model is the prediction result, and the machine learning model is a model for predicting whether the test case is a flaky test according to the feature vector corresponding to the test case.

7. The method according to any one of claims 1 to 6, characterized in that, The method is executed by a cloud management platform in a cloud service system, where the cloud management platform is used to manage the infrastructure that provides cloud services, the infrastructure includes at least one cloud data center, and each cloud data center in the at least one cloud data center includes at least one server, and the database and the rule library are included on the at least one server.

8. The method according to claim 7, characterized in that, The at least one server also includes the test case.

9. A computing device for detecting test cases, characterized in that, Includes: A transceiver module and a processing module, where The transceiver module is used to receive the first information of the user, and the first information is used to indicate a test case, and the test case includes at least one encapsulated function; The processing module is used to determine, according to the test case, the paths corresponding to the at least one encapsulated function included in the test case in the database, where the database includes a first mapping relationship between the at least one encapsulated function and the function paths corresponding to the at least one encapsulated function respectively; The processing module is used to obtain the at least one encapsulated function according to the function path; The processing module is used to detect the test case according to the at least one encapsulated function and the rule library, where the rule library includes at least one rule, and each rule in the at least one rule is used to detect whether the function functions corresponding to the at least one encapsulated function included in the test case conform to the rule; The transceiver module is used to output the detection result to the user.

10. The computing device according to claim 9, wherein The rule includes: the function functions corresponding to the at least one encapsulated function cannot conflict with each other, and / or, the function functions corresponding to the at least one encapsulated function cannot conflict with each other internally.

11. The computing device according to claim 9 or 10, characterized in that, When it is detected that the function functions corresponding to the at least one encapsulated function included in the test case do not conform to the rule The detection result includes at least one of the following: the name information of the rule, the description information of the rule, and the position information of the code segment, where the description information of the rule is used to indicate the content of the rule, and the position information of the code segment is used to indicate the position of the code segment that does not conform to the rule in the code corresponding to the at least one encapsulated function respectively, and the code includes the code segment.

12. The computing device according to any one of claims 9 to 11, characterized in that The processing module is used to execute the test case and obtain the execution log corresponding to the test case, where the execution log is used to record the specific information of the execution process of the test case; The processing module is used to determine, according to the execution log, the actually executed code segments in the code corresponding to the at least one encapsulated function included in the test case; The processing module is used to detect the test case, including: The processing module is used to detect the actually executed code segments.

13. The computing device according to claim 12, wherein The processing module is used to determine, according to the execution log, the actually executed code segments in the code corresponding to each of the at least one encapsulated function included in the test case, including: The processing module is used to align the execution log with the code corresponding to each of the at least one encapsulated function included in the test case according to the window alignment algorithm and / or the similarity algorithm, to generate a second mapping relationship, where the second mapping relationship is the mapping relationship between the code corresponding to each of the at least one encapsulated function and the code in the execution log; The processing module is used to determine the actually executed code segments of each encapsulated function according to the second mapping relationship table.

14. The computing device according to claim 12 or 13, wherein The processing module is used to analyze the code corresponding to each of the at least one encapsulated function included in the test case, the detection result, and the execution log, to generate a feature vector corresponding to the test case, where the feature vector is used to characterize the behavior of the test case; The processing module is used to predict whether the test case is an unreliable flaky test according to the machine learning model and the feature vector, where the input of the machine learning model is the feature vector, the output of the machine learning model is the prediction result, and the machine learning model is a model used to predict whether the test case is a flaky test according to the feature vector corresponding to the test case.

15. The computing device according to any one of claims 9 to 14, characterized in that, The computing device is a communication device for a cloud management platform in a cloud service system, where the cloud management platform is used to manage the infrastructure for providing cloud services, the infrastructure includes at least one cloud data center, each cloud data center in the at least one cloud data center includes at least one server, and the database and the rule library are included on the at least one server.

16. The computing device according to claim 15, wherein The at least one server also includes the test case.

17. A computing device, characterized in that, It includes a processor and a memory, and the processor runs the instructions in the memory, so that the processor executes the method according to any one of claims 1 to 8.

18. A computer program product comprising instructions, characterized in that, When the instructions are run by a cluster of computing devices, the cluster of computing devices executes the method according to any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, It includes computer program instructions, and when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices executes the method according to any one of claims 1 to 8.

Images