Search software testing method and device and search software testing system

Abstract

The invention provides a test method and device for search software and a test system for the search software, and the method comprises the steps: obtaining an original test case of the search software, the original test case being a test case for testing a search function of the search software; setting a metamorphic relation corresponding to a search function of the search software, wherein the metamorphic relation is a relation which needs to be met between search input and output of the search software; generating a plurality of derivative test cases of the original test case according to the metamorphic relationship, wherein the input of the derivative test cases and the input of the original test case meet the metamorphic relationship; testing the search function of the search software according to the original test case and the plurality of derivative test cases to obtain the output of the original test case and the output of each derivative test case; under the condition that the output of the original test case and the output of each derivative test case meet the metamorphic relationship, the search function of the search software is determined to pass the test, and the problem that the output result of the search software cannot be predicted and is difficult to test is solved.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a method and device for testing search software, a computer program product and a search software testing system. Background Art

[0002] In the context of massive data storage, the operation process of conditional screening functions or search engine tools in information technology systems often faces Oracle problems, that is, the execution output results of the tested system are complex or unpredictable, making it difficult to test the software system. Summary of the invention

[0003] The main purpose of the present application is to provide a method, device, computer program product and search software testing system for searching software, so as to at least solve the problem in the prior art that the output results of searching software cannot be predicted, making it difficult to test.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a testing method for search software is provided, comprising: obtaining an original test case of the search software, wherein the original test case is a test case for testing the search function of the search software; setting a transformation relationship corresponding to the search function of the search software, wherein the transformation relationship is a relationship that needs to be satisfied between the input and output of the search of the search software; generating multiple derivative test cases of the original test case according to the transformation relationship, wherein the input of the derivative test case satisfies the transformation relationship with the input of the original test case; testing the search function of the search software according to the original test case and the multiple derivative test cases to obtain the output of the original test case and the output of each of the derivative test cases; and determining that the search function of the search software has passed the test when the output of the original test case and the output of each of the derivative test cases both satisfy the transformation relationship.

[0005] Optionally, when the output of the original test case and the output of each of the derived test cases satisfy the metamorphosis relationship, determining that the search function of the search software has passed the test includes: determining that the derived test case that satisfies the metamorphosis relationship has passed the test; calculating the ratio of the number of the derived test cases that failed the test to the total number of the derived test cases to obtain the metamorphosis relationship violation rate; and when the metamorphosis relationship violation rate is less than a first predetermined threshold, determining that the search function of the search software has failed the test.

[0006] Optionally, the metamorphosis relationship corresponding to the search function of the search software is set, including: setting a first metamorphosis relationship, the first metamorphosis relationship is a relationship in which the source input and the subsequent input are equal, the source output and the subsequent output are equal, and the order of the source output and the subsequent output is the same or different; setting a second metamorphosis relationship, the second metamorphosis relationship is a relationship in which the source input and the subsequent input are equal, the source output and the subsequent output are equal and the order is also the same; setting a third metamorphosis relationship, the third metamorphosis relationship is a relationship in which, based on the source input, the search conditions are continuously increased to form multiple subsequent inputs, and the output corresponding to each subsequent input is a subset of the previous output; setting a fourth metamorphosis relationship, the fourth metamorphosis relationship is a relationship in which the source input is the union of all the subsequent inputs, and the source output is equal to the union of all the subsequent outputs; setting a fifth metamorphosis relationship, the fifth metamorphosis relationship is a relationship in which the subsequent outputs are not intersected with each other, and the subsequent outputs are all subsets of the source output.

[0007] Optionally, generating multiple derivative test cases of the original test case according to the metamorphosis relationship includes: when the metamorphosis relationship is the first metamorphosis relationship or the second metamorphosis relationship, using the input equal to the input of the original test case as the input of the derivative test case, and using the output of the search software executing the derivative test case as the output of the derivative test case; when the metamorphosis relationship is the third metamorphosis relationship, based on the input of the original test case, the search conditions are continuously increased to form multiple inputs of the derivative test case, and the multiple outputs of the search software executing the derivative test case are used as the multiple outputs of the derivative test case; In the case where the metamorphosis relationship is the fourth metamorphosis relationship, multiple subsets of the input of the original test case are used as multiple inputs of the derived test case and the union of the multiple inputs of the derived test case is equal to the input of the original test case, and the multiple outputs of the search software executing the derived test case are used as the multiple outputs of the derived test case; in the case where the metamorphosis relationship is the fifth metamorphosis relationship, multiple subsets of the input of the original test case are used as multiple inputs of the derived test case and the multiple inputs of the derived test case are not intersecting, and the multiple outputs of the search software executing the derived test case are used as the multiple outputs of the derived test case.

[0008] Optionally, when the output of the original test case and the output of each of the derived test cases satisfy the metamorphosis relationship, before determining that the search function of the search software passes the test, the method further includes: when the input of the original test case is equal to the input of the derived test case, and the output of the original test case is equal to the output of the derived test case and the order is the same or different, determining that the derived test case satisfies the first metamorphosis relationship; when the input of the original test case is equal to the input of the derived test case, and the output of the original test case is equal to the output of the derived test case and the order is the same, determining that the derived test case satisfies the second metamorphosis relationship; When the input of the original test case is the union of multiple inputs of the derived test case, and the multiple outputs of the derived test case are respectively subsets of the previous output, the derived test case is determined to satisfy the third metamorphosis relationship; when the input of the original test case is the union of multiple inputs of the derived test case, and the output of the original test case is equal to the union of the multiple outputs of the derived test case, the derived test case is determined to satisfy the fourth metamorphosis relationship; when the multiple outputs of the derived test case are not intersected with each other, and the multiple outputs of the derived test case are all subsets of the output of the original test case, the derived test case is determined to satisfy the fifth metamorphosis relationship.

[0009] Optionally, when the metamorphosis relationship is the first metamorphosis relationship or the second metamorphosis relationship, the method further includes: when the derived test case does not satisfy the first metamorphosis relationship or the second metamorphosis relationship, calculating the similarity between the output of the derived test case and the output of the original test case to obtain the Jaccard coefficient of the derived test case; and when the Jaccard coefficient is greater than a predetermined threshold, determining that the derived test case has passed the test.

[0010] Optionally, the similarity between the output of the derived test case and the input of the original test case is calculated to obtain the Jaccard coefficient, including: calculating the intersection of the output of the derived test case and the output of the original test case to obtain a test intersection; calculating the union of the output of the derived test case and the output of the original test case to obtain a test union; calculating the ratio of the number of elements in the test intersection to the number of elements in the test union to obtain the Jaccard coefficient of the derived test case.

[0011] According to another aspect of the present application, a testing device for search software is provided, comprising: an acquisition unit, used to acquire an original test case of the search software, wherein the original test case is a test case for testing the search function of the search software; a setting unit, used to set a transformation relationship corresponding to the search function of the search software, wherein the transformation relationship is a relationship that needs to be satisfied between the input and output of multiple searches of a test case of the search software; a generation unit, used to generate multiple derivative test cases of the original test case according to the transformation relationship; a testing unit, used to test the search function of the search software according to the original test case and the multiple derivative test cases, and obtain the output of the original test case and the output of each of the derivative test cases; a first determination unit, used to determine that the search function of the search software has passed the test when the output of the original test case and the output of each of the derivative test cases satisfy the transformation relationship.

[0012] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, any one of the methods described above is implemented.

[0013] According to another aspect of the present application, a search software testing system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0014] By applying the technical solution of the present application, in the testing method of the above-mentioned search software, by setting the transformation relationship that the input and output of the search software need to satisfy, setting appropriate inputs according to the transformation relationship, forming an original test case and multiple derived test cases, the search software executes the original test case and multiple derived test cases, and when the output of the original test case and the output of each derived test case satisfy the transformation relationship, it is determined that the search function of the search software has passed the test. There is no need to compare the output of the test case with the predicted output to determine whether the test has passed, that is, there is no need to predict the output of the test case, which solves the problem in the prior art that the output result of the search software cannot be predicted, making it difficult to test. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A hardware structure block diagram of a mobile terminal for executing a search software testing method provided in an embodiment of the present application is shown;

[0016] Figure 2 A schematic diagram of a flow chart of a search software testing method provided according to an embodiment of the present application is shown;

[0017] Figure 3 A schematic diagram of a process of constructing, identifying a metamorphosis relationship and performing a metamorphosis test according to an embodiment of the present application is shown;

[0018] Figure 4 A schematic diagram of a process of a metamorphosis test provided according to an embodiment of the present application is shown;

[0019] Figure 5 A schematic diagram of a flow chart of generating a derivative test case by MR and its execution process according to an embodiment of the present application is shown;

[0020] Figure 6 A structural block diagram of a search software testing device provided according to an embodiment of the present application is shown.

[0021] The above drawings include the following reference numerals:

[0022] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0027] Metamorphic Relation: A rule used for model testing that can describe the relationship between input and output in a model algorithm; it checks the relationship between input and output in multiple execution results of the application under test, namely the metamorphic relation (MR), which can be used to determine whether the model test passes, and can also be used to generate derived use cases from source use cases.

[0028] Oracle problem: refers to the difficulty in verifying the test results of a given test case during the testing process when the system or application under test is complex.

[0029] Metamorphosis testing: A testing method that uses the inherent properties of model algorithms. It is a typical representative of model-based software testing methods. It can weaken the Oracle problem and does not focus on the verification of each individual test case, that is, the output of a single test. Its idea is to assume that the inputs related to the attributes satisfy the given metamorphosis relationship in some way. By verifying whether the metamorphosis relationship remains unchanged, it determines whether the model test passes, thereby verifying the correctness of the program software design. It is a method of testing using metamorphosis relationships.

[0030] ES search engine: ES search engine (Elasticsearch, ES) is a distributed full-text search engine that can efficiently store and index various types of data, such as structured, unstructured text, digital data, etc. It is often used in system retrieval functions, but it is different from database and other retrieval methods and is widely used.

[0031] Big data search engine testing: A method of testing search engines in systems that carry big data. It usually uses the ES search engine or connects to a database for query. The method of testing and evaluating the accuracy, precision, and recognition efficiency of the above query results is called big data search testing.

[0032] Metamorphosis relationship violation rate: also known as MR violation rate, which is the ratio of test results that violate the metamorphosis relationship to all test results during the metamorphosis test process, and is used to reflect the pass status of the metamorphosis test case.

[0033] Jaccard coefficient: A calculation coefficient used to measure the similarity between two sample sets. Its value range is [0,1]. The closer the value is to 1, the more similar the two sample sets are, and vice versa.

[0034] As introduced in the background technology, the output results of search software in the prior art are unpredictable, which makes it difficult to test. To solve this technical problem, the embodiments of the present application provide a test method, device, computer program product and search software test system for search software.

[0035] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for testing search software according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0037] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the test method of the search software in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The specific example of the above-mentioned network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] In this embodiment, a testing method for search software running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Figure 2 FIG. 1 is a flow chart of a method for testing search software according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0040] Step S201, obtaining an original test case of the search software, wherein the original test case is a test case for testing the search function of the search software;

[0041] Step S202, setting a metamorphosis relationship corresponding to the search function of the search software, wherein the metamorphosis relationship is a relationship that needs to be satisfied between the input and output of the search of the search software;

[0042] Step S203, generating a plurality of derivative test cases of the original test case according to the transformation relationship, wherein the input of the derivative test case and the input of the original test case satisfy the transformation relationship;

[0043] Step S204, testing the search function of the search software according to the original test case and the plurality of derived test cases, and obtaining the output of the original test case and the output of each of the derived test cases;

[0044] Step S205: When the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship, it is determined that the search function of the search software has passed the test.

[0045] In the testing method for the above-mentioned search software, by setting the transformation relationship that the input and output of the search software need to satisfy, setting appropriate inputs according to the transformation relationship, forming an original test case and multiple derived test cases, the search software executes the original test case and multiple derived test cases, and when the output of the original test case and the output of each derived test case satisfy the transformation relationship, it is determined that the search function of the search software has passed the test. There is no need to compare the output of the test case with the predicted output to determine whether the test passes, that is, there is no need to predict the output of the test case, which solves the problem in the prior art that the output results of the search software cannot be predicted, making it difficult to test.

[0046] In order to check whether the search function of the search software is qualified, in an optional implementation, the above step S205 includes:

[0047] Step S2051, determine that the above-derived test cases satisfying the above metamorphic relationship pass the test;

[0048] Step S2052, calculate the ratio of the number of the above-derived test cases that fail the test to the total number of the above-derived test cases to obtain the metamorphic relationship violation rate;

[0049] Step S2053, in the case where the above metamorphic relationship violation rate is less than the first predetermined threshold, determine that the search function of the above search software fails the test.

[0050] In the above embodiment, the metamorphic relationship violation rate is used as an evaluation index. For each metamorphic relationship MR, the metamorphic relationship violation rate refers to the ratio between the number of derived test cases violating the metamorphic relationship MR and the total number. Its calculation formula is as follows: Ratio MR = Num abnormal / Total, where Ratio MR represents the MR violation ratio, Num abnormal represents the number of derived test cases violating the metamorphic relationship MR, Total represents the total number of derived test cases. If the above metamorphic relationship violation rate is less than the first predetermined threshold, then the search function of the above search software fails the test. Modify the input parameters of the source use case according to the constructed MR to generate the input of the derived test case, record the output results of the original test case and the derived test case, and compare whether the two violate the metamorphic relationship. If they match, the test passes; otherwise, it fails. The specific process of metamorphic testing is as Figure 3 shown.

[0051] In order to set the metamorphic relationship that meets the user's requirements, in an alternative embodiment, the above step S202 includes:

[0052] Step S2021, set the first metamorphic relationship, where the first metamorphic relationship is the relationship that the source input and the subsequent input are equal, the source output and the subsequent output are equal, and the order of the above source output and the above subsequent output is the same or different;

[0053] Step S2022, set the second metamorphic relationship, where the second metamorphic relationship is the relationship that the above source input and the above subsequent input are equal, the above source output and the above subsequent output are equal and the order is also the same;

[0054] Step S2023, set the third metamorphic relationship, where the third metamorphic relationship is the relationship that on the basis of the above source input, the search conditions are continuously increased to form multiple above subsequent inputs, and the output corresponding to each above subsequent input is a subset of the previous output;

[0055] Step S2024, setting a fourth transformation relationship, wherein the fourth transformation relationship is a relationship in which the source input is the union of all the subsequent inputs, and the source output is equal to the union of all the subsequent outputs;

[0056] Step S2025, setting a fifth transformation relationship, wherein the fifth transformation relationship is a relationship in which the subsequent outputs are not intersected with each other and the subsequent outputs are all subsets of the source outputs.

[0057] In the above implementation, under the engine search function, an ES search engine auxiliary testing tool is developed, which is required to meet the following five metamorphosis relations: the first metamorphosis relation (MR1 equivalence), the second metamorphosis relation (MR2 equivalence), the third metamorphosis relation (MR3 merging condition), the fourth metamorphosis relation (MR4 complete partitioning) and the fifth metamorphosis relation (MR5 partitioning difference), metamorphosis relation MR1: equivalence, indicating that the source input and the subsequent input are equal, the source output and the subsequent output are equal but the order is the same or different. Metamorphosis relation MR2: equivalence, refers to the relationship that the source input and the subsequent input are equal, the source output and the subsequent output are equal and the order is also the same. This relationship model is a subset of the previous relationship model (equivalence). Metamorphosis relation MR3: merging condition, indicating that on the basis of the source input, the search conditions are continuously added as multiple subsequent inputs, and the corresponding outputs are all subsets of the previous output. Metamorphosis relation MR4: complete partitioning, indicating that the source input is the union of all subsequent inputs, and the source output is equal to the union of all subsequent outputs. Transformation relation MR5: Partition difference, which means that the subsequent outputs do not intersect with each other but all belong to the source output subset. The appropriate transformation relation can be selected according to user needs.

[0058] In order to test different transformation relationships, in an optional implementation, the above step S203 includes:

[0059] Step S2031, when the above-mentioned metamorphosis relationship is the above-mentioned first metamorphosis relationship or the above-mentioned second metamorphosis relationship, using the input equal to the input of the above-mentioned original test case as the input of the above-mentioned derived test case, and using the output of the above-mentioned search software executing the above-mentioned derived test case as the output of the above-mentioned derived test case;

[0060] Step S2032, when the metamorphosis relationship is the third metamorphosis relationship, based on the input of the original test case, the search conditions are continuously increased to form multiple inputs of the derived test case, and the multiple outputs of the derived test case executed by the search software are used as multiple outputs of the derived test case;

[0061] Step S2033, when the above-mentioned metamorphosis relationship is the above-mentioned fourth metamorphosis relationship, multiple subsets of the input of the above-mentioned original test case are used as multiple inputs of the above-mentioned derived test case, and the union of the multiple inputs of the above-mentioned derived test case is equal to the input of the above-mentioned original test case, and the multiple outputs of the above-mentioned search software executing the above-mentioned derived test case are used as the multiple outputs of the above-mentioned derived test case;

[0062] Step S2034, when the above-mentioned metamorphosis relationship is the above-mentioned fifth metamorphosis relationship, multiple subsets of the input of the above-mentioned original test case are used as multiple inputs of the above-mentioned derived test case and the multiple inputs of the above-mentioned derived test case are not intersecting, and the multiple outputs of the above-mentioned search software executing the above-mentioned derived test case are used as multiple outputs of the above-mentioned derived test case.

[0063] In the above implementation, an example is given as shown in Table 1, where the original input is the input of the original test case, and the subsequent input is the input of the derived test case, wherein the MR4 complete partition (tested with asset tags) and the MR5 partition difference (tested with asset types) differ as follows: When testing the MR4 complete partition of the metamorphosis relationship, a large condition needs to be input for the first input, and all subsequent outputs are subsets of the first output result set, and there may be intersections between the result sets obtained by the subsequent input conditions. When testing the MR5 partition difference of the metamorphosis relationship, a large condition needs to be input for the first input, and all subsequent outputs are mutually exclusive subsets of the first output result set, that is, there may be no intersections between the result sets obtained by the subsequent input conditions. Based on the source test case designed according to the MR1-MR5 metamorphosis relationship, a script is written to automatically generate a derived test case, whose generation rules each satisfy the five metamorphosis relationships of MR1-MR5, modify the input-output pairs, and perform repeated automated tests, returning the query results that violate the metamorphosis relationship after each execution. For the metamorphosis relationship of the input text type of MR1-MR3, only the input needs to be changed to automatically generate derivative test cases in batches. For the metamorphosis relationship of MR4-MR5, additional query conditions need to be selected. The specific generation rules are as follows: (1) Select 20 asset types and asset labels respectively to generate 4 sets Stype1, Stype2, Slabel1, and Slabel2. Each set of Stype1 and Stype2 contains 10 keywords of asset types without duplication; each set of Slabel1 and Slabel2 contains 10 keywords of asset labels without duplication; (2) Calculate Stype1×Stype2 and Slabel1×Slabel2 respectively, where × represents the Cartesian product. This calculation method can obtain 10×10, i.e., 100 derivative test cases of asset type and asset label combinations.

[0064] Table 1

[0065]

[0066]

[0067]

[0068] In order to determine whether the derived test case satisfies the metamorphosis relationship, in an optional implementation manner, when the output of the original test case and the output of each of the derived test cases satisfy the metamorphosis relationship, before determining that the search function of the search software passes the test, the method further includes:

[0069] Step S301, when the input of the original test case is equal to the input of the derived test case, and the output of the original test case is equal to the output of the derived test case and the order is the same or different, determine that the derived test case satisfies the first transformation relationship;

[0070] Step S302, when the input of the original test case is equal to the input of the derived test case, and the output of the original test case is equal to the output of the derived test case and the order is the same, determining that the derived test case satisfies the second transformation relationship;

[0071] Step S303, when the input of the original test case is the union of multiple inputs of the derived test case, and multiple outputs of the derived test case are subsets of the previous output, determining that the derived test case satisfies the third metamorphosis relationship;

[0072] Step S304, when the input of the original test case is the union of multiple inputs of the derived test case, and the output of the original test case is equal to the union of multiple outputs of the derived test case, determining that the derived test case satisfies the fourth transformation relationship;

[0073] Step S305: When the multiple outputs of the derived test case are not intersecting with each other and the multiple outputs of the derived test case are subsets of the outputs of the original test case, it is determined that the derived test case satisfies the fifth transformation relationship.

[0074] In the above implementation, as shown in Table 1, the expected result describes the conditions that satisfy each transformation relationship, for example, the result subset Result 1 -Result 6 The collection is consistent with the full set, and the subset Result 1 -Result 6Mutually exclusive, that is, the multiple outputs of the derived test cases do not intersect with each other, and the multiple outputs of the derived test cases are subsets of the outputs of the original test cases. Then it is determined that the derived test cases satisfy the fifth transformation relationship.

[0075] In order to examine the search function from multiple aspects, in an optional implementation manner, when the above-mentioned metamorphosis relationship is the above-mentioned first metamorphosis relationship or the above-mentioned second metamorphosis relationship, the above-mentioned method further includes:

[0076] Step S401, when the derived test case does not satisfy the first metamorphosis relationship or the second metamorphosis relationship, calculating the similarity between the output of the derived test case and the output of the original test case to obtain the Jaccard coefficient of the derived test case;

[0077] Step S402: When the Jaccard coefficient is greater than a predetermined threshold, it is determined that the derived test case has passed the test.

[0078] In the above implementation, in addition to calculating the MR violation ratio, i.e., the test pass rate, it is also necessary to calculate the sample similarity used to evaluate the Jaccard coefficient. For each query result obtained by the ES search engine based on the query conditions of the original test case and the derived test case, the finite sample result sets obtained by the two queries are compared. By measuring the similarity and difference of the elements in the set, the stability of the query results is measured, thereby reflecting the query stability and validity of the metamorphosis test for the ES search engine test. The larger the value of the Jaccard coefficient, the higher the degree of repetition between the retrieval results of the original case and the derived case, which indirectly reflects the stronger ranking stability of the tested ES search engine. In the case where the output result order requirement is not high, even if the above-mentioned derived test case does not meet the above-mentioned first metamorphosis relationship or the above-mentioned second metamorphosis relationship, the above-mentioned Jaccard coefficient is greater than the predetermined threshold value, and it is determined that the above-mentioned derived test case has passed the test.

[0079] In order to calculate the Jaccard coefficient, in an optional implementation, the above step S401 includes:

[0080] Step S4011, calculating the intersection of the output of the derived test case and the output of the original test case to obtain a test intersection;

[0081] Step S4012, calculating the union of the output of the derived test case and the output of the original test case to obtain a test union;

[0082] Step S4013, calculating the ratio of the number of elements in the test intersection to the number of elements in the test union, and obtaining the Jaccard coefficient of the derived test case.

[0083] In the above implementation, the Jaccard coefficient is calculated as follows: for the sample set A obtained from the original use case query result and the sample set B obtained from the derived use case query, the ratio of the size of the intersection of A and B to the size of the union of A and B is calculated, and the formula is as follows: J(A, B) = |A∩B| / |A∪B| = |A∩B| / (|A|+|B|-|A∩B|), wherein |A∩B| represents the number of elements in the intersection of A and B, and |A∪B| represents the number of elements in the union of A and B. When both sets A and B are empty, the Jaccard coefficient is defined as 1, and the value range of Jaccard(A,B) is as follows: J(A,B)∈[0,1].

[0084] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the test method for the search software of the present application will be described in detail below in conjunction with specific embodiments.

[0085] This embodiment relates to a specific method for testing search software. Figure 4 and Figure 5 As shown, the following steps are included:

[0086] SUT represents the system under testing (SUT) with m input parameters and p output results. 1 represents the source test case, A i (i=2,…,n) represents the subsequent test case, that is, the test case after the input parameters of the source test case are modified according to the transformation relationship, also known as the derived test case. j1 , A j2 , ..., A jm , (j=1,…,n) represents the m input parameters of the original test case and the derived test case, O(A 1 ) and O(A i ) is the n outputs corresponding to the original test case and the derived test case, B j1 , B j2 , ..., B jp represents O(A j ) output results. The degradation test is performed by comparing O(A 1 ) and O(A i ) satisfies MR to test the functional correctness or reliability of the SUT.

[0087] The MR (transformation relation) of a function f is expressed as a series of relations between the function inputs I 0 , I 1 , ..., I n , (n>=1) and its corresponding output value f(I 0 ), f(I1 ), ..., f(I n ), that is, the transformation relationship R={(I 0 ,I 1 ,...,I n ),(f(I 0 ),f(I 1 ),..,f(I n ))n>=1}where I 0 is the source input, f(I 0 ) is the source output, I 1 , I 2 , ..., I n For subsequent input, f(I 1 ), f(I 2 ), ..., f(I n ) is the subsequent output.

[0088] Suppose the original test case is S. After executing S, the search engine returns a non-empty S execution result set (s 1 ,s 2 ,…,s i ,…,s n ), assuming that the result set s i The text title is t i , i is any integer between 1 and n. The number of result sets returned by the ES search engine is huge, so it is necessary to construct a metamorphosis relationship to generate derived use cases. Assume that the original use case S is in the metamorphosis relationship MR i Under the action of i , abnormal i Indicates whether an exception occurs in the test case. The initial value is 0. When the execution results of the original test case and the derived test case do not satisfy the transformation relationship MR i When, abnormal i The value is 1, which means that the test results of this group of use cases violate the metamorphosis relationship MR i , an exception occurred.

[0089] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0090] The present application embodiment also provides a kind of testing device of search software, it should be noted that the testing device of search software of the present application embodiment can be used to execute the testing method for search software provided by the present application embodiment. The device is used to implement the above-mentioned embodiment and preferred implementation mode, and the description has been made no further details. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented with software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0091] The following is an introduction to the testing device for the search software provided in the embodiment of the present application.

[0092] Figure 6 is a structural block diagram of a test device for search software according to an embodiment of the present application. Figure 6 As shown, the device comprises:

[0093] The acquisition unit 10 is used to acquire the original test case of the search software, where the original test case is a test case for testing the search function of the search software;

[0094] A setting unit 20, used to set a metamorphosis relationship corresponding to the search function of the search software, wherein the metamorphosis relationship is a relationship that needs to be satisfied between input and output of multiple searches of a test case of the search software;

[0095] A generating unit 30, configured to generate a plurality of derivative test cases of the original test case according to the transformation relationship;

[0096] The testing unit 40 is used to test the search function of the search software according to the original test case and the plurality of derived test cases, and obtain the output of the original test case and the output of each of the derived test cases;

[0097] The first determination unit 50 is used to determine whether the search function of the search software passes the test when the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship.

[0098] In the testing device of the above-mentioned search software, by setting the transformation relationship that the input and output of the search software need to satisfy, setting appropriate inputs according to the transformation relationship, forming an original test case and multiple derived test cases, the search software executes the original test case and multiple derived test cases, and when the output of the original test case and the output of each derived test case satisfy the transformation relationship, it is determined that the search function of the search software has passed the test. There is no need to compare the output of the test case with the predicted output to determine whether the test passes, that is, there is no need to predict the output of the test case, which solves the problem in the prior art that the output results of the search software cannot be predicted, making it difficult to test.

[0099] In order to check whether the search function of the search software is qualified, in an optional implementation manner, the first determination unit includes:

[0100] A first determination module is used to determine that the above-mentioned derivative test cases satisfying the above-mentioned metamorphosis relationship pass the test;

[0101] A first calculation module is used to calculate the ratio of the number of the above-mentioned derivative test cases that failed the test to the total number of the above-mentioned derivative test cases to obtain a metamorphosis relationship violation rate;

[0102] The second determination module is used to determine that the search function of the search software fails the test when the degradation relationship violation rate is less than a first predetermined threshold.

[0103] In the above implementation, the metamorphosis relationship violation rate is used as an evaluation index. For each metamorphosis relationship MR, the metamorphosis relationship violation rate refers to the ratio between the number of derived test cases that violate the metamorphosis relationship MR and the total number. The calculation formula is as follows: Ratio MR =Num abnormal / Total, where Ratio MR Indicates the MR violation ratio, Num abnormal It indicates the number of derived test cases that violate the metamorphosis relationship MR, and Total indicates the total number of derived test cases. If the metamorphosis relationship violation rate is less than the first predetermined threshold, the search function of the search software fails the test. The input parameters of the source test case are modified according to the constructed MR to generate the input of the derived test case. The output of the original test case and the output results of the derived test case are recorded, and the two are compared to see if they violate the metamorphosis relationship. If they do, the test passes, otherwise it fails. The specific process of the metamorphosis test is as follows: Figure 3 shown.

[0104] In order to set a transformation relationship that meets user needs, in an optional implementation, the setting unit includes:

[0105] A first setting module is used to set a first transformation relationship, wherein the first transformation relationship is a relationship in which a source input is equal to a subsequent input, a source output is equal to a subsequent output, and an order of the source output and the subsequent output is the same or different;

[0106] A second setting module is used to set a second transformation relationship, wherein the second transformation relationship is a relationship in which the source input is equal to the subsequent input, and the source output is equal to the subsequent output and the order is the same;

[0107] A third setting module is used to set a third metamorphosis relationship, wherein the third metamorphosis relationship is a relationship in which, based on the source input, the search conditions are continuously increased to form multiple subsequent inputs, and the output corresponding to each subsequent input is a subset of the previous output;

[0108] a fourth setting module, used for setting a fourth transformation relationship, wherein the fourth transformation relationship is a relationship in which the source input is the union of all the subsequent inputs, and the source output is equal to the union of all the subsequent outputs;

[0109] The fifth setting module is used to set a fifth transformation relationship, wherein the fifth transformation relationship is a relationship in which the subsequent outputs are not intersected with each other and the subsequent outputs are all subsets of the source outputs.

[0110] In the above implementation, under the engine search function, an ES search engine auxiliary testing tool is developed, which is required to meet the following five metamorphosis relations: the first metamorphosis relation (MR1 equivalence), the second metamorphosis relation (MR2 equivalence), the third metamorphosis relation (MR3 merging condition), the fourth metamorphosis relation (MR4 complete partitioning) and the fifth metamorphosis relation (MR5 partitioning difference), metamorphosis relation MR1: equivalence, indicating that the source input and the subsequent input are equal, the source output and the subsequent output are equal but the order is the same or different. Metamorphosis relation MR2: equivalence, refers to the relationship that the source input and the subsequent input are equal, the source output and the subsequent output are equal and the order is also the same. This relationship model is a subset of the previous relationship model (equivalence). Metamorphosis relation MR3: merging condition, indicating that on the basis of the source input, the search conditions are continuously added as multiple subsequent inputs, and the corresponding outputs are all subsets of the previous output. Metamorphosis relation MR4: complete partitioning, indicating that the source input is the union of all subsequent inputs, and the source output is equal to the union of all subsequent outputs. Transformation relation MR5: Partition difference, which means that the subsequent outputs do not intersect with each other but all belong to the source output subset. The appropriate transformation relation can be selected according to user needs.

[0111] In order to test different transformation relationships, in an optional implementation, the generation unit includes:

[0112] A first processing module is used for, when the above-mentioned metamorphosis relationship is the above-mentioned first metamorphosis relationship or the above-mentioned second metamorphosis relationship, using the input equal to the input of the above-mentioned original test case as the input of the above-mentioned derived test case, and using the output of the above-mentioned search software executing the above-mentioned derived test case as the output of the above-mentioned derived test case;

[0113] A second processing module is used for, when the above-mentioned metamorphosis relationship is the above-mentioned third metamorphosis relationship, based on the input of the above-mentioned original test case, continuously adding search conditions to form multiple inputs of the above-mentioned derivative test case, and using the multiple outputs of the above-mentioned derivative test case executed by the above-mentioned search software as the multiple outputs of the above-mentioned derivative test case;

[0114] A third processing module is used for, when the above-mentioned metamorphosis relationship is the above-mentioned fourth metamorphosis relationship, using multiple subsets of the input of the above-mentioned original test case as multiple inputs of the above-mentioned derived test case and the union of the multiple inputs of the above-mentioned derived test case is equal to the input of the above-mentioned original test case, and using the multiple outputs of the above-mentioned derived test case executed by the above-mentioned search software as the multiple outputs of the above-mentioned derived test case;

[0115] The fourth processing module is used to use multiple subsets of the input of the above-mentioned original test case as multiple inputs of the above-mentioned derived test case when the above-mentioned metamorphosis relationship is the above-mentioned fifth metamorphosis relationship, and the multiple inputs of the above-mentioned derived test case are not intersecting, and use the multiple outputs of the above-mentioned search software executing the above-mentioned derived test case as multiple outputs of the above-mentioned derived test case.

[0116] In the above implementation, an example is given as shown in Table 1, where the original input is the input of the original test case, and the subsequent input is the input of the derived test case, wherein the MR4 complete partition (tested with asset tags) and the MR5 partition difference (tested with asset types) differ as follows: When testing the MR4 complete partition of the metamorphosis relationship, a large condition needs to be input for the first input, and all subsequent outputs are subsets of the first output result set, and there may be intersections between the result sets obtained by the subsequent input conditions. When testing the MR5 partition difference of the metamorphosis relationship, a large condition needs to be input for the first input, and all subsequent outputs are mutually exclusive subsets of the first output result set, that is, there may be no intersections between the result sets obtained by the subsequent input conditions. Based on the source test case designed according to the MR1-MR5 metamorphosis relationship, a script is written to automatically generate a derived test case, whose generation rules each satisfy the five metamorphosis relationships of MR1-MR5, modify the input-output pairs, and perform repeated automated tests, returning the query results that violate the metamorphosis relationship after each execution. For the metamorphosis relationship of the input text type of MR1-MR3, only the input needs to be changed to automatically generate derivative test cases in batches. For the metamorphosis relationship of MR4-MR5, additional query conditions need to be selected. The specific generation rules are as follows: (1) Select 20 asset types and asset labels respectively to generate 4 sets Stype1, Stype2, Slabel1, and Slabel2. Each set of Stype1 and Stype2 contains 10 keywords of asset types without duplication; each set of Slabel1 and Slabel2 contains 10 keywords of asset labels without duplication; (2) Calculate Stype1×Stype2 and Slabel1×Slabel2 respectively, where × represents the Cartesian product. This calculation method can obtain 10×10, i.e., 100 derivative test cases of asset type and asset label combinations.

[0117] In order to determine whether the derived test case satisfies the metamorphosis relationship, in an optional implementation manner, the above-mentioned device further includes:

[0118] A second determination unit is used to determine that the search function of the search software passes the test before the output of the original test case and the output of each of the derived test cases satisfy the metamorphosis relationship, and to determine that the derived test case satisfies the first metamorphosis relationship when the input of the original test case is equal to the input of the derived test case, and the output of the original test case is equal to the output of the derived test case and the order is the same or different;

[0119] A third determining unit is used to determine that the derived test case satisfies the second transformation relationship when the input of the original test case is equal to the input of the derived test case, and the output of the original test case is equal to the output of the derived test case and the order is the same;

[0120] A fourth determination unit, configured to determine that the derived test case satisfies the third transformation relationship when the input of the original test case is a union of multiple inputs of the derived test case and multiple outputs of the derived test case are subsets of the previous output;

[0121] a fifth determining unit, configured to determine that the derived test case satisfies the fourth transformation relationship when the input of the original test case is the union of multiple inputs of the derived test case and the output of the original test case is equal to the union of multiple outputs of the derived test case;

[0122] The sixth determination unit is used to determine that the derived test case satisfies the fifth transformation relationship when the multiple outputs of the derived test case do not intersect with each other and the multiple outputs of the derived test case are all subsets of the output of the original test case.

[0123] In the above implementation, as shown in Table 1, the expected result describes the conditions that satisfy each transformation relationship, for example, the result subset Result 1 -Result 6 The collection is consistent with the full set, and the subset Result 1 -Result 6 Mutually exclusive, that is, the multiple outputs of the derived test cases do not intersect with each other, and the multiple outputs of the derived test cases are subsets of the outputs of the original test cases. Then it is determined that the derived test cases satisfy the fifth transformation relationship.

[0124] In order to examine the search function from multiple aspects, in an optional implementation, the above device further includes:

[0125] A calculation unit, used for calculating the similarity between the output of the derived test case and the output of the original test case when the derived test case does not satisfy the first metamorphosis relationship or the second metamorphosis relationship, to obtain the Jaccard coefficient of the derived test case;

[0126] The seventh determination unit is used to determine whether the derived test case passes the test when the Jaccard coefficient is greater than a predetermined threshold.

[0127] In the above implementation, in addition to calculating the MR violation ratio, i.e., the test pass rate, it is also necessary to calculate the sample similarity used to evaluate the Jaccard coefficient. For each query result obtained by the ES search engine based on the query conditions of the original test case and the derived test case, the finite sample result sets obtained by the two queries are compared. By measuring the similarity and difference of the elements in the set, the stability of the query results is measured, thereby reflecting the query stability and validity of the metamorphosis test for the ES search engine test. The larger the value of the Jaccard coefficient, the higher the degree of repetition between the retrieval results of the original case and the derived case, which indirectly reflects the stronger ranking stability of the tested ES search engine. In the case where the output result order requirement is not high, even if the above-mentioned derived test case does not meet the above-mentioned first metamorphosis relationship or the above-mentioned second metamorphosis relationship, the above-mentioned Jaccard coefficient is greater than the predetermined threshold value, and it is determined that the above-mentioned derived test case has passed the test.

[0128] In order to calculate the Jaccard coefficient, in an optional implementation, the calculation unit includes:

[0129] A second calculation module is used to calculate the intersection of the output of the derived test case and the output of the original test case to obtain a test intersection;

[0130] A third calculation module is used to calculate the union of the output of the derived test case and the output of the original test case to obtain a test union;

[0131] The fourth calculation module is used to calculate the ratio of the number of elements in the test intersection to the number of elements in the test union to obtain the Jaccard coefficient of the derived test case.

[0132] In the above implementation, the Jaccard coefficient is calculated as follows: for the sample set A obtained from the original use case query result and the sample set B obtained from the derived use case query, the ratio of the size of the intersection of A and B to the size of the union of A and B is calculated, and the formula is as follows: J(A, B) = |A∩B| / |A∪B| = |A∩B| / (|A|+|B|-|A∩B|), wherein |A∩B| represents the number of elements in the intersection of A and B, and |A∪B| represents the number of elements in the union of A and B. When both sets A and B are empty, the Jaccard coefficient is defined as 1, and the value range of Jaccard(A,B) is as follows: J(A,B)∈[0,1].

[0133] The above-mentioned test device for the search software includes a processor and a memory. The above-mentioned units are stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0134] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of unpredictable output results of search software leading to difficulty in testing in the prior art can be solved by adjusting kernel parameters.

[0135] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0136] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the test method of the search software.

[0137] Specifically, the testing method of the search software includes:

[0138] Step S201, obtaining an original test case of the search software, wherein the original test case is a test case for testing the search function of the search software;

[0139] Step S202, setting a metamorphosis relationship corresponding to the search function of the search software, wherein the metamorphosis relationship is a relationship that needs to be satisfied between the input and output of the search of the search software;

[0140] Step S203, generating a plurality of derivative test cases of the original test case according to the transformation relationship, wherein the input of the derivative test case and the input of the original test case satisfy the transformation relationship;

[0141] Step S204, testing the search function of the search software according to the original test case and the plurality of derived test cases, and obtaining the output of the original test case and the output of each of the derived test cases;

[0142] Step S205: When the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship, it is determined that the search function of the search software has passed the test.

[0143] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the test method of the search software is executed when the program is running.

[0144] Specifically, the testing method of the search software includes:

[0145] Step S201, obtaining an original test case of the search software, wherein the original test case is a test case for testing the search function of the search software;

[0146] Step S202, setting a metamorphosis relationship corresponding to the search function of the search software, wherein the metamorphosis relationship is a relationship that needs to be satisfied between the input and output of the search of the search software;

[0147] Step S203, generating a plurality of derivative test cases of the original test case according to the transformation relationship, wherein the input of the derivative test case and the input of the original test case satisfy the transformation relationship;

[0148] Step S204, testing the search function of the search software according to the original test case and the plurality of derived test cases, and obtaining the output of the original test case and the output of each of the derived test cases;

[0149] Step S205: When the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship, it is determined that the search function of the search software has passed the test.

[0150] An embodiment of the present invention provides a search software testing system, the search software testing system includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0151] Step S201, obtaining an original test case of the search software, where the original test case is a test case for testing the search function of the search software;

[0152] Step S202, setting a metamorphosis relationship corresponding to the search function of the search software, wherein the metamorphosis relationship is a relationship that needs to be satisfied between the input and output of the search of the search software;

[0153] Step S203, generating a plurality of derivative test cases of the original test case according to the transformation relationship, wherein the input of the derivative test case and the input of the original test case satisfy the transformation relationship;

[0154] Step S204, testing the search function of the search software according to the original test case and the plurality of derived test cases, and obtaining the output of the original test case and the output of each of the derived test cases;

[0155] Step S205: When the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship, it is determined that the search function of the search software has passed the test.

[0156] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0157] Step S201, obtaining an original test case of the search software, wherein the original test case is a test case for testing the search function of the search software;

[0158] Step S202, setting a metamorphosis relationship corresponding to the search function of the search software, wherein the metamorphosis relationship is a relationship that needs to be satisfied between the input and output of the search of the search software;

[0159] Step S203, generating a plurality of derivative test cases of the original test case according to the transformation relationship, wherein the input of the derivative test case and the input of the original test case satisfy the transformation relationship;

[0160] Step S204, testing the search function of the search software according to the original test case and the plurality of derived test cases, and obtaining the output of the original test case and the output of each of the derived test cases;

[0161] Step S205: When the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship, it is determined that the search function of the search software has passed the test.

[0162] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0163] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0168] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0169] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0170] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0171] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0172] 1) In the testing method of the search software of the present application, by setting the transformation relationship that the input and output of the search software need to satisfy, setting appropriate inputs according to the transformation relationship, forming an original test case and multiple derived test cases, the search software executes the original test case and multiple derived test cases, and when the output of the original test case and the output of each derived test case satisfy the transformation relationship, it is determined that the search function of the search software has passed the test. There is no need to compare the output of the test case with the predicted output to determine whether the test has passed, that is, there is no need to predict the output of the test case, which solves the problem in the prior art that the output results of the search software cannot be predicted, making it difficult to test.

[0173] 2) In the test device of the search software of the present application, by setting the transformation relationship that the input and output of the search software need to satisfy, setting appropriate inputs according to the transformation relationship, forming an original test case and multiple derived test cases, the search software executes the original test case and multiple derived test cases, and when the output of the original test case and the output of each derived test case satisfy the transformation relationship, it is determined that the search function of the search software has passed the test. There is no need to compare the output of the test case with the predicted output to determine whether the test has passed, that is, there is no need to predict the output of the test case, which solves the problem in the prior art that the output results of the search software cannot be predicted, making it difficult to test.

[0174] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for testing search software, characterized in that: include: Obtaining an original test case of the search software, wherein the original test case is a test case for testing a search function of the search software; Setting a metamorphosis relationship corresponding to the search function of the search software, wherein the metamorphosis relationship is a relationship that needs to be satisfied between the input and output of the search of the search software; Generate multiple derivative test cases of the original test case according to the metamorphosis relationship, where the input of the derivative test case and the input of the original test case satisfy the metamorphosis relationship; Testing the search function of the search software according to the original test case and the plurality of derived test cases to obtain the output of the original test case and the output of each of the derived test cases; When the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship, it is determined that the search function of the search software passes the test.

2. The method according to claim 1, characterized in that When the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship, determining that the search function of the search software passes the test includes: Determining that the derived test case that satisfies the transformation relationship passes the test; Calculating the ratio of the number of the derived test cases that failed the test to the total number of the derived test cases to obtain a metamorphosis relationship violation rate; When the degradation relationship violation rate is less than the first predetermined threshold, it is determined that the search function of the search software has failed the test.

3. The method according to claim 1, characterized in that Setting the transformation relationship corresponding to the search function of the search software includes: Setting a first transformation relationship, wherein the first transformation relationship is a relationship in which a source input is equal to a subsequent input, a source output is equal to a subsequent output, and an order of the source output and the subsequent output is the same or different; Setting a second transformation relationship, wherein the second transformation relationship is a relationship in which the source input and the subsequent input are equal, and the source output and the subsequent output are equal and in the same order; Setting a third metamorphosis relationship, wherein the third metamorphosis relationship is a relationship in which, based on the source input, the search conditions are continuously increased to form multiple subsequent inputs, and the output corresponding to each subsequent input is a subset of the previous output; Setting a fourth transformation relationship, wherein the fourth transformation relationship is a relationship in which the source input is the union of all the subsequent inputs, and the source output is equal to the union of all the subsequent outputs; A fifth transformation relationship is set, wherein the fifth transformation relationship is a relationship in which the subsequent outputs are disjoint in pairs and all the subsequent outputs are subsets of the source outputs.

4. The method according to claim 3, characterized in that Generating multiple derivative test cases of the original test case according to the transformation relationship includes: In the case where the metamorphosis relationship is the first metamorphosis relationship or the second metamorphosis relationship, using an input equal to an input of the original test case as an input of the derived test case, and using an output of the search software executing the derived test case as an output of the derived test case; In the case where the metamorphosis relationship is the third metamorphosis relationship, based on the input of the original test case, the search conditions are continuously increased to form multiple inputs of the derived test case, and the multiple outputs of the derived test case executed by the search software are used as multiple outputs of the derived test case; In the case where the metamorphosis relationship is the fourth metamorphosis relationship, multiple subsets of the input of the original test case are used as multiple inputs of the derived test case and the union of the multiple inputs of the derived test case is equal to the input of the original test case, and multiple outputs of the derived test case executed by the search software are used as multiple outputs of the derived test case; When the transformation relationship is the fifth transformation relationship, multiple subsets of the input of the original test case are used as multiple inputs of the derived test case and the multiple inputs of the derived test case are not intersecting, and multiple outputs of the search software executing the derived test case are used as multiple outputs of the derived test case.

5. The method according to claim 3, characterized in that: When the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship, before determining that the search function of the search software passes the test, the method further includes: When the input of the original test case is equal to the input of the derived test case, and the output of the original test case is equal to the output of the derived test case and the order is the same or different, determine that the derived test case satisfies the first transformation relationship; When the input of the original test case is equal to the input of the derived test case, and the output of the original test case is equal to the output of the derived test case and the order is the same, determine that the derived test case satisfies the second transformation relationship; In the case where the input of the original test case is the union of multiple inputs of the derived test case, and multiple outputs of the derived test case are respectively subsets of the previous output, determining that the derived test case satisfies the third metamorphosis relationship; When the input of the original test case is the union of multiple inputs of the derived test case, and the output of the original test case is equal to the union of multiple outputs of the derived test case, determining that the derived test case satisfies the fourth transformation relationship; In the case where multiple outputs of the derived test case are disjoint with each other and multiple outputs of the derived test case are subsets of the output of the original test case, it is determined that the derived test case satisfies the fifth transformation relationship.

6. The method according to claim 3, characterized in that In a case where the transformation relationship is the first transformation relationship or the second transformation relationship, the method further includes: When the derived test case does not satisfy the first metamorphosis relationship or the second metamorphosis relationship, calculating the similarity between the output of the derived test case and the output of the original test case to obtain the Jaccard coefficient of the derived test case; When the Jaccard coefficient is greater than a predetermined threshold, it is determined that the derived test case passes the test.

7. The method according to claim 6, characterized in that Calculating the similarity between the output of the derived test case and the input of the original test case to obtain the Jaccard coefficient includes: Calculate the intersection of the output of the derived test case and the output of the original test case to obtain a test intersection; Calculating the union of the output of the derived test case and the output of the original test case to obtain a test union; The ratio of the number of elements in the test intersection to the number of elements in the test union is calculated to obtain the Jaccard coefficient of the derived test case.

8. A testing device for search software, characterized in that: include: An acquisition unit, used for acquiring an original test case of the search software, wherein the original test case is a test case for testing a search function of the search software; A setting unit, used to set a metamorphosis relationship corresponding to the search function of the search software, wherein the metamorphosis relationship is a relationship that needs to be satisfied between input and output of multiple searches of a test case of the search software; A generating unit, configured to generate a plurality of derivative test cases of the original test case according to the transformation relationship; A testing unit, used for testing the search function of the search software according to the original test case and the plurality of derived test cases, and obtaining the output of the original test case and the output of each of the derived test cases; The first determination unit is used to determine whether the search function of the search software passes the test when the output of the original test case and the output of each of the derived test cases satisfy the transformation relationship.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A search software testing system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images