Test method and device for artificial intelligence application software, equipment and product

Abstract

The invention discloses a test method and device for artificial intelligence application software, equipment and a product, and the method comprises the steps: inputting a test question into the artificial intelligence application software which is currently tested, and obtaining a first answer returned by the artificial intelligence application software; manually determining a second answer to the test question; establishing a judgment criterion, wherein the judgment criterion is used for evaluating the quality score of the first answer; evaluating the first answer by using a large language model according to the second answer and an evaluation criterion, and generating a quality score corresponding to the first answer; and if the quality score is greater than a set threshold value, determining that a first answer returned by the artificial intelligence application software passes the test. The invention discloses a test method, device, equipment and product for artificial intelligence application software, which can test the artificial intelligence application software with an uncertain output result by using a large language model, and has the advantages of good test effect, wide application range and reliable performance.

Description

Technical Field

[0001] The present invention belongs to the field of software testing, and in particular, relates to a testing method, device, equipment and product for artificial intelligence application software. Background Art

[0002] With the rapid development of artificial intelligence technology, the update speed of artificial intelligence application software is getting faster and faster, and the performance is also more excellent. How to test artificial intelligence application software has become a generally concerned issue. Under the existing technical conditions, traditional software products rely more on preset logic and rules to complete tasks, so the output results are usually relatively clear, and traditional UI automation testing tools can achieve relatively satisfactory software testing effects. However, the output results of artificial intelligence application software depend on the capabilities of artificial intelligence models, and the output results have variability and uncertainty. Therefore, traditional UI automation testing tools cannot fully test artificial intelligence application software. Summary of the Invention

[0003] In view of this, the present invention aims to overcome the defects in the prior art and proposes a testing method, device, equipment and product for artificial intelligence application software.

[0004] To achieve the above object, the technical solution of the present invention is realized as follows: In a first aspect, the present invention discloses a testing method for artificial intelligence application software, including: inputting a test question to the currently tested artificial intelligence application software to obtain a first answer returned by the artificial intelligence application software; Manually determining a second answer to the test question, where the second answer is the standard answer to the test question; Establishing a judgment criterion, which is used to evaluate the quality score of the first answer; Inputting the first answer, the second answer and the judgment criterion into a large language model together, and using the large language model to evaluate the first answer according to the second answer and the judgment criterion to generate a quality score corresponding to the first answer; If the quality score is greater than a set threshold, the first answer returned by the artificial intelligence application software passes the test.

[0005] In an embodiment of the present invention, the method further includes: recording a test case of clicking a button to open a corresponding page in the application software, inserting a checkpoint at a specified position on the opened corresponding page, and the checkpoint is used to obtain the control name; playing back the test case, if the control name obtained by the checkpoint is the same as the expected control name, the operation of opening the corresponding page by the application software passes the test.

[0006] In an embodiment of the present invention, the large language model is deployed on a server.

[0007] In one embodiment of the present invention, the evaluation criteria include scoring the first answer respectively from four dimensions: richness, logical coherence, fluency, and consistency.

[0008] In one embodiment of the present invention, the quality score is the average value after summing the scores of the four dimensions.

[0009] In one embodiment of the present invention, inputting a test question to the currently tested artificial intelligence application software to obtain the first answer returned by the artificial intelligence application software includes: obtaining the first answer by inserting a checkpoint into the reply box of the artificial intelligence application software.

[0010] In a second aspect, the present invention discloses a test device for an artificial intelligence application software, and the device includes: A first answer module, configured to input a test question to the currently tested artificial intelligence application software to obtain the first answer returned by the artificial intelligence application software; A second answer module, configured to manually determine the second answer to the test question, where the second answer is the standard answer to the test question; A standard establishment module, configured to establish evaluation criteria for evaluating the quality score of the first answer; A quality scoring module, configured to input the first answer, the second answer, and the evaluation criteria into a large language model together, and use the large language model to evaluate the first answer according to the second answer and the evaluation criteria to generate the quality score corresponding to the first answer; A test evaluation module, configured to determine that the first answer returned by the artificial intelligence application software passes the test if the quality score is greater than a set threshold.

[0011] In a third aspect, the present invention discloses an electronic device, including: one or more processors; a storage device for storing one or more programs, where when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the above method.

[0012] In a fourth aspect, the present invention discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0013] In a fifth aspect, the present invention discloses a computer program product, including a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0014] Compared with the prior art, the present invention has the following advantages: The present invention discloses a testing method, device, equipment and product for artificial intelligence application software, including inputting a test question into the artificial intelligence application software under current test to obtain a first answer returned by the artificial intelligence application software; manually determining a second answer to the test question; establishing a judgment criterion, which is used to evaluate the quality score of the first answer; using a large language model to evaluate the first answer according to the second answer and the judgment criterion to generate a quality score corresponding to the first answer; if the quality score is greater than a set threshold, the first answer returned by the artificial intelligence application software passes the test. The present invention discloses a testing method, device, equipment and product for artificial intelligence application software, which can utilize a large language model to realize the testing of artificial intelligence application software with uncertain output results, and has the advantages of good testing effect, wide application range and reliable performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0016] In the drawings: Figure 1 is a schematic diagram of an application scenario of a testing method for artificial intelligence application software according to an embodiment of the present invention; Figure 2 is a schematic diagram of a testing method for artificial intelligence application software according to an embodiment of the present invention; Figure 3 is a schematic diagram of artificial intelligence application software testing of a testing method for artificial intelligence application software according to an embodiment of the present invention; Figure 4 is a schematic diagram of traditional testing of a testing method for artificial intelligence application software according to an embodiment of the present invention; Figure 5 is a schematic diagram of a testing device for artificial intelligence application software according to an embodiment of the present invention; Figure 6 is a schematic diagram of an electronic device for testing artificial intelligence application software according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other.

[0018] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention.

[0019] In the description of the present invention, it should be noted that unless otherwise clearly defined and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood through specific situations.

[0020] In the description of the present invention, it should be further noted that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise stated, the meaning of "a plurality" is two or more.

[0021] The application scenarios of a testing method, device, equipment, and product for artificial intelligence application software disclosed by the present invention are as Figure 1 shown. Under the existing technical conditions, the output results of artificial intelligence application software depend on the capabilities of the artificial intelligence model, and the output results have variability and uncertainty. Therefore, traditional UI automation testing tools cannot fully test artificial intelligence application software. The present invention discloses a testing method, device, equipment, and product for artificial intelligence application software. By manually determining the standard answers to test questions and establishing evaluation criteria, it is possible to use large language models to test artificial intelligence application software with uncertain output results, which has the advantages of good testing effects, wide application ranges, and reliable performance.

[0022] The present invention will be described in detail below with reference to the drawings and in combination with embodiments.

[0023] In an embodiment of the present invention, as Figure 2 and Figure 3 shown, a testing method for artificial intelligence application software includes: Step S201: Input a test question into the artificial intelligence application software under current test, and obtain a first answer returned by the artificial intelligence application software; Exemplarily, in this embodiment, the artificial intelligence application software based on the Linux desktop operating system is tested. The artificial intelligence application software is an application software developed based on the Qt framework. Its core lies in using technologies such as machine learning to enable the software to perform corresponding tasks, such as speech recognition, image recognition, etc. Its output results usually have uncertainty, while ordinary application software relies more on preset logics and rules to complete tasks, and the output results are usually clear. Existing software testing methods cannot test artificial intelligence application software whose output results usually have uncertainty.

[0024] Exemplarily, obtain the first answer by inserting a checkpoint in the reply box of the artificial intelligence application software.

[0025] Step S202: Manually determine a second answer to the test question, where the second answer is the standard answer to the test question; Step S203: Establish a judgment criterion, which is used to evaluate the quality score of the first answer; Step S204: Input the first answer, the second answer, and the judgment criterion into a large language model together. Use the large language model to evaluate the first answer according to the second answer and the judgment criterion, and generate a quality score corresponding to the first answer; In this embodiment, exemplarily, the large language model is the deepseek large language model; Step S205: If the quality score is greater than the set threshold, the first answer returned by the artificial intelligence application software passes the test.

[0026] In this embodiment, the large language model is deployed on the server, reducing the occupation of system memory, thereby reducing the impact on test performance, and improving the flexibility and convenience of deployment.

[0027] In another embodiment of the present invention, as Figure 4 shown, the method further includes: recording a test case of clicking a button in the application software to open a corresponding page, inserting a checkpoint at a specified position on the opened corresponding page, and the checkpoint is used to obtain the control name; playing back the test case. If the control name obtained by the checkpoint is the same as the expected control name, the operation of the application software to open the corresponding page passes the test.

[0028] In this embodiment, this method can be applicable to both artificial intelligence application software and traditional software at the same time to test software functions and improve test efficiency.

[0029] In another embodiment of the present invention, the evaluation criteria include scoring the first answer separately from four dimensions: richness, logical coherence, fluency, and consistency.

[0030] In this embodiment, the quality score is the average value after summing the scores of the four dimensions.

[0031] In an embodiment of the present invention, an application example of the present method is as follows: The hardware environment is a desktop computer, using the Galaxy Kylin operating system, and the artificial intelligence application software for testing is the AI assistant; After entering the test question "How to use the picture viewer" in the text box of the AI assistant and sending the question, the first answer returned by the AI assistant is obtained; Exemplarily, the specific process of obtaining the user input test question "How to use the picture viewer" is as follows: The insertion checkpoint locates the user input according to the component with the kylin-aiassistant_QLineEdit_AuthorAttribute property of User, so as to obtain the question issued by the user when asking the question, that is, the content in the kylin-aiassistant_QLineEdit_UserMessage control, as the input test question; After the AI assistant finishes answering, insert a checkpoint, and locate the AI assistant output according to the content with the kylin-aiassistant_QLineEdit_AuthorAttribute property of Bot, so as to obtain the reply when the AI assistant answers, that is, the content in the kylin-aiassistant_QLineEdit_botMessage control, as the first answer; Manually determine the second answer to the test question. For example: the second answer is to output an introduction to the use of the picture viewer tool; Exemplarily, at this time, manually input the second answer in the pop-up window of the insertion checkpoint, that is, the standard answer to the test question "How to use the picture viewer" is "An introduction to the use of the picture viewer tool"; Input the first answer returned by the AI assistant, the manually input second answer, and the preset evaluation criteria into the large language model; Exemplarily, input the prompt words in the large language model: use the second answer as the standard answer, and at the same time score the quality of the first answer according to the following evaluation criteria, with a full score of 100 points: ① When the content of the first answer has diverse perspectives and sufficient details, the logic is strictly progressive without breaks, the language is natural and fluent, and the theme is highly focused without deviation, the score is 80 to 100 points; ②When the first answer is substantial, logically clear with occasional minor jumps, the sentences are smooth with a small amount of redundancy but are easy to understand, the theme is clear and the secondary content is slightly redundant, the score ranges from 60 to 80 points; ③When the first answer is thin and lacks depth, the logic is loose and the connection is insufficient, the sentences are intermittent and the expression is rigid, and the core part of the theme deviates, the score ranges from 30 to 60 points; ④When the first answer has empty content and lacks information, the logic is chaotic and self-contradictory, the language is fragmented and there are frequent errors, and the content is completely off-topic, the score ranges from 10 to 30 points.

[0032] Furthermore, set the threshold to 60 points. When the large language model generates a quality score for the first answer based on the input scoring criteria with the second answer as the standard answer, if the quality score is greater than or equal to 60 points, the first answer returned by the artificial intelligence application software passes the test; if it is less than 60 points, the first answer returned by the artificial intelligence application software fails the test.

[0033] In this embodiment, after the test is completed, the large language model generates a test report, which summarizes the test conclusions of the artificial intelligence application software and provides solid data support for the subsequent software evaluation.

[0034] In another embodiment of the present invention, the test process for the application software to open the corresponding page is as follows. For example, in the AI assistant, after clicking the settings button, test whether the settings page can be opened normally: Record test cases: In the AI assistant settings page, obtain the position (location['x'], location['y']) of the settings button in the upper right corner through the Qt control name kylin-aiassistant_QMenu_ButtonSettings, and call the click public method to click; Judge whether the settings page is opened by whether there is the word "Settings" in the window title in the upper left corner after opening; Insert a checkpoint at the upper left corner position of the opened page. This checkpoint includes multiple selectable fixed attribute content labels, which can obtain the actual results of attributes such as "control name", "control length", and "control height" at the current insertion position; After selecting the "control name" as the checkpoint, an assertion (using the assert statement) will be automatically added to judge whether there is the word "Settings" at the upper left corner position; After the process of recording test cases is completed, the corresponding code script will be generated according to the case steps.

[0035] Locate the specific recorded test case according to the test case ID, and replay the recorded automation script; Playback test case: When the playback test case is executed, the actual result will be compared with the expected result. Through assertions, it is judged whether the current execution meets the expectations. If the control name obtained by the checkpoint is the same as the expected control name, the operation of opening the corresponding page by the application software passes the test; After the execution is completed, the final test result is output, and a test report is generated by the test result generation service.

[0036] In this embodiment, the control positioning only depends on the control name and does not depend on the stability of the page position. The positioning is performed through the control attributes rather than coordinate positioning, which solves the disadvantage that the coordinates strictly depend on the page stability. Therefore, this method is applicable to the test tasks in the entire software development life cycle, such as module testing, integration testing, and script maintenance phase testing.

[0037] As Figure 5 shown, the present invention also discloses a test device for an artificial intelligence application software, including: A first answering module 501, configured to input a test question to the artificial intelligence application software under current test, and obtain a first answer returned by the artificial intelligence application software; A second answering module 502, configured to manually determine a second answer to the test question, where the second answer is the standard answer to the test question; A standard establishment module 503, configured to establish a judgment standard, and the judgment standard is used to evaluate the quality score of the first answer; A quality scoring module 504, configured to input the first answer, the second answer, and the judgment standard into a large language model together, and use the large language model to evaluate the first answer according to the second answer and the judgment standard, and generate a quality score corresponding to the first answer; A test evaluation module 505, configured to determine that if the quality score is greater than a set threshold, the first answer returned by the artificial intelligence application software passes the test.

[0038] The present invention also discloses an electronic device. As Figure 6 shown, a block diagram of an embodiment applicable to the above-mentioned electronic device for testing artificial intelligence application software is disclosed.

[0039] The electronic device 60 in this embodiment includes a processor 601, which can perform various appropriate actions and processes according to the program stored in the ROM 602 or the program loaded from the storage section 608 into the RAM 603. The processor 601 may include, for example, a general microprocessor, an instruction set processor, and / or a related chipset and / or a dedicated microprocessor, etc. The processor 601 may also include on-board memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiments of the present invention.

[0040] In the RAM 603, various programs and data required for the operation of the electronic device 60 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present invention by executing the programs in the ROM 602 and / or the RAM 603. It should be noted that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603, and the processor 601 can also perform various operations of the method flow according to the embodiments of the present invention by executing the programs stored in one or more memories.

[0041] According to an embodiment of the present invention, the electronic device 60 may further include an I / O interface 605, and the I / O interface 605 is also connected to the bus 604. The electronic device 60 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube, a liquid crystal display, a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 6010 is also connected to the I / O interface 605 as needed. A removable medium 6011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 6010 as needed so that a computer program read from it can be installed into the storage portion 608 as needed.

[0042] The present invention also provides a computer-readable storage medium.

[0043] The computer-readable storage medium may be included in the electronic device / device system described in the above embodiments; or it may exist separately without being assembled into the electronic device / device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present invention is implemented.

[0044] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory RAM, a read-only memory ROM, an erasable programmable read-only memory EPROM or a flash memory, a portable compact disk read-only memory CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device, or device.

[0045] Embodiments of the present invention also include a computer program product.

[0046] The computer program product includes a computer program, which contains program code for executing the methods provided in the embodiments of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of the present invention.

[0047] In one embodiment, the computer program can rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program can also be transmitted and distributed in the form of signals on a network medium. The program code included in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0048] According to the embodiments of the present invention, the program code for executing the computer program provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedural and / or object-oriented programming languages. Programming languages include but are not limited to, for example, Java, C++, Python, C language, or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network or a wide area network, or can be connected to an external computing device.

[0049] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and combinations of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly recited in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recited in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0050] The embodiments of the present invention have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although the embodiments have been described separately above, this does not mean that the measures in the respective embodiments cannot be used advantageously in combination. The scope of the present invention is defined by the appended claims and their equivalents, and without departing from the scope of the present invention, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present invention.

Claims

1. A testing method for artificial intelligence application software, characterized in that including: Input a test question to the currently tested artificial intelligence application software, and obtain a first answer returned by the artificial intelligence application software; Manually determine a second answer to the test question, where the second answer is the standard answer to the test question; Establish a judgment criterion, which is used to evaluate the quality score of the first answer; Input the first answer, the second answer, and the judgment criterion into a large language model together, and use the large language model to evaluate the first answer according to the second answer and the judgment criterion, and generate the quality score corresponding to the first answer; If the quality score is greater than the set threshold, the first answer returned by the artificial intelligence application software passes the test.

2. The test method for an artificial intelligence application software according to claim 1, wherein The method further includes: recording a test case in the application software, clicking a button to open a corresponding page, inserting a checkpoint at a specified position on the opened corresponding page, where the checkpoint is used to obtain the control name; playing back the test case, if the control name obtained by the checkpoint is the same as the expected control name, the operation of the application software to open the corresponding page passes the test.

3. A test method for an artificial intelligence application software according to claim 1, characterized in that The large language model is deployed on a server.

4. A test method for an artificial intelligence application software according to claim 1, characterized in that, The judgment criterion includes scoring the first answer from four dimensions: richness, logical coherence, fluency, and consistency.

5. The test method for an artificial intelligence application software according to claim 4, characterized in that, The quality score is the average value after summing the scores of the four dimensions.

6. The test method for an artificial intelligence application software according to claim 1, characterized in that, The step of inputting a test question to the currently tested artificial intelligence application software and obtaining a first answer returned by the artificial intelligence application software includes: obtaining the first answer by inserting a checkpoint in the reply box of the artificial intelligence application software.

7. A testing device for artificial intelligence application software, characterized in that: The device includes: A first answer module, configured to input a test question to the currently tested artificial intelligence application software, and obtain a first answer returned by the artificial intelligence application software; A second answer module, configured to manually determine a second answer to the test question, where the second answer is the standard answer to the test question; A standard establishment module, configured to establish a judgment criterion, which is used to evaluate the quality score of the first answer; A quality scoring module, configured to input the first answer, the second answer, and the judgment criterion into a large language model together, and use the large language model to evaluate the first answer according to the second answer and the judgment criterion, and generate the quality score corresponding to the first answer; A test evaluation module, configured to if the quality score is greater than the set threshold, the first answer returned by the artificial intelligence application software passes the test.

8. An electronic device, characterized in that, including: One or more processors; A storage device, configured to store one or more programs, where when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that , when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, , when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 6.

Images