Artificial intelligence service test method and related device

Abstract

The invention discloses an artificial intelligence service test method and a related device, and relates to the technical field of artificial intelligence, and the method comprises the steps: pre-training to obtain a generative large model with a test case generation capability as a test case generation model, and presetting a test data generator with a multi-modal test data generation capability, firstly, an interface document of a to-be-tested artificial intelligence service is obtained; converting the interface document into a file in a preset format; then calling a test case generation model to generate a test case according to the file in the preset format, and calling a preset test data generator to automatically generate test data according to the test case; and finally, based on the test case and the test data, testing the artificial intelligence service to be tested. In the scheme, the test case and the test data can be automatically generated, the dependence on testers is reduced, and the preparation period of the test case and the test data can be shortened, so that the artificial intelligence service test efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an artificial intelligence service testing method and related devices. Background Art

[0002] With the continuous advancement of science and technology, the field of artificial intelligence (AI) has also ushered in rapid development. Various types of artificial intelligence services have emerged one after another. Its wide application in various fields has become a reality and has gradually become a key component of social infrastructure. For artificial intelligence services with rapid iterations in open environments and massive users, users have higher requirements for the reliability, stability and efficiency of artificial intelligence services. Testing is an important part of the software production process. Testers use high-quality and efficient testing methods to comprehensively evaluate the performance of artificial intelligence services in different scenarios, identify potential problems and solve them in a timely manner, thereby ensuring their reliability, stability and efficiency in actual applications.

[0003] The traditional AI service testing method requires professional testers to manually design test cases, construct test data, and implement tests. This method relies heavily on the professionalism of testers, resulting in uneven quality of test cases, difficulty in constructing and maintaining test data, and a long test implementation preparation cycle, which in turn leads to inefficient AI service testing.

[0004] Therefore, how to provide an artificial intelligence service testing method to improve the efficiency of artificial intelligence service testing has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] In view of the above problems, this application provides an artificial intelligence service testing method and related devices to achieve the purpose of improving the efficiency of artificial intelligence service testing. The specific scheme is as follows:

[0006] The first aspect of the present application provides an artificial intelligence service testing method, comprising:

[0007] Obtain the interface documentation of the AI ​​service to be tested;

[0008] Convert the interface document of the AI ​​service to be tested into a file in a preset format;

[0009] According to the file in a preset format, a test case generation model is called to generate a test case, wherein the test case generation model is a pre-trained generative large model with the ability to generate test cases;

[0010] According to the test case, calling a preset test data generator to automatically generate test data; the preset test data generator has the ability to generate multi-modal test data;

[0011] Based on the test case and the test data, the artificial intelligence service to be tested is tested.

[0012] In a possible implementation, converting the interface document of the artificial intelligence service to be tested into a file in a preset format includes:

[0013] Determine a document conversion prompt; the document conversion prompt includes a document conversion task description and an interface document of the artificial intelligence service to be tested;

[0014] The document conversion prompt is provided to a document conversion model to obtain a file in a preset format generated by the document conversion model, wherein the document conversion model is a pre-trained generative large model with document conversion capabilities.

[0015] In a possible implementation, the step of calling a test case generation model to generate a test case according to a file in a preset format includes:

[0016] Determine a test case generation prompt; the test case generation prompt includes a test case generation task description and the file in the preset format;

[0017] The test case generation prompt is provided to the test case generation model to obtain the test case generated by the test case generation model.

[0018] In a possible implementation, after calling the test case generation model to generate the test case according to the file in the preset format, the method further includes:

[0019] Returning the test case to the tester for correction, and obtaining the corrected test case and usage suggestions for the test case;

[0020] The test case generation model is optimized using the revised test case and / or the usage suggestion of the test case.

[0021] In a possible implementation, calling a preset test data generator to automatically generate test data according to the test case includes:

[0022] Parsing the test case to obtain key information, where the key information is used to indicate basic information of the artificial intelligence service to be tested, where the basic information of the artificial intelligence service to be tested includes a business type, an input data type, and predefined semantic rules of the artificial intelligence service;

[0023] Determine test data generation prompt based on the key information;

[0024] Performing semantic understanding classification on the test data generation prompt to obtain a classification result; the classification result is used to indicate the type of test data to be generated;

[0025] Based on the classification result, the test data generator is called to automatically generate test data.

[0026] In one possible implementation, the test data generator includes a pre-trained generative large model with test data generation capabilities and artificial intelligence services in other vertical fields.

[0027] In a possible implementation, if the classification result indicates that the type of the test data to be generated is text, then based on the classification result, calling the test data generator to automatically generate the test data includes:

[0028] Providing the test data generation prompt to the generative large model to obtain a generation result of the generative large model;

[0029] The test data is determined according to the generation result of the generative large model.

[0030] In a possible implementation, if the classification result indicates that the type of the test data to be generated is non-text, then based on the classification result, calling a test data generator to automatically generate test data includes:

[0031] Providing the test data generation prompt to the generative large model to obtain a generation result of the generative large model;

[0032] The generation result of the generative large model is re-edited by utilizing the artificial intelligence services in other vertical fields to obtain the test data.

[0033] A second aspect of the present application provides an artificial intelligence service testing device, comprising:

[0034] An acquisition unit, used to acquire the interface document of the artificial intelligence service to be tested;

[0035] A document conversion unit, used to convert the interface document of the artificial intelligence service to be tested into a file in a preset format;

[0036] A test case generation unit, used to generate a test case by calling a test case generation model according to a file in a preset format, wherein the test case generation model is a pre-trained generative large model with test case generation capability;

[0037] A test data generating unit, configured to call a preset test data generator to automatically generate test data according to the test case; the preset test data generator has a multi-modal test data generating capability;

[0038] A testing unit is used to test the artificial intelligence service to be tested based on the test case and the test data.

[0039] In a possible implementation, the document conversion unit is specifically used to:

[0040] Determine a document conversion prompt; the document conversion prompt includes a document conversion task description and an interface document of the artificial intelligence service to be tested;

[0041] The document conversion prompt is provided to a document conversion model to obtain a file in a preset format generated by the document conversion model, wherein the document conversion model is a pre-trained generative large model with document conversion capabilities.

[0042] In a possible implementation, the test case generating unit is specifically used to:

[0043] Determine a test case generation prompt; the test case generation prompt includes a test case generation task description and the file in the preset format;

[0044] The test case generation prompt is provided to the test case generation model to obtain the test case generated by the test case generation model.

[0045] In a possible implementation, the device also includes: a test case generation model optimization unit, the test case generation model optimization unit is used to return the test case to the tester for correction after calling the test case generation model to generate the test case according to the file in a preset format, and obtain the corrected test case and the usage suggestion of the test case; use the corrected test case and / or the usage suggestion of the test case to optimize the test case generation model.

[0046] In a possible implementation, the test data generating unit includes:

[0047] A parsing unit, used to parse the test case to obtain key information, where the key information is used to indicate basic information of the artificial intelligence service to be tested, where the basic information of the artificial intelligence service to be tested includes a business type, an input data type, and predefined semantic rules of the artificial intelligence service;

[0048] A test data generation prompt determining unit, configured to determine a test data generation prompt based on the key information;

[0049] A classification unit, used for performing semantic understanding classification on the test data generation prompt to obtain a classification result; the classification result is used to indicate the type of test data to be generated;

[0050] The test data generating subunit is used to call the test data generator to automatically generate test data based on the classification result.

[0051] In one possible implementation, the test data generator includes a pre-trained generative large model with test data generation capabilities and artificial intelligence services in other vertical fields.

[0052] In a possible implementation, if the classification result indicates that the type of the test data to be generated is text, the test data generating subunit is specifically configured to:

[0053] The test data generation prompt is provided to the generative large model to obtain a generation result of the generative large model; and the test data is determined according to the generation result of the generative large model.

[0054] In a possible implementation, if the classification result indicates that the type of the test data to be generated is non-text, the test data generating subunit is specifically configured to:

[0055] The test data generation prompt is provided to the generative big model to obtain the generation result of the generative big model; and the generation result of the generative big model is re-edited by using the artificial intelligence services in other vertical fields to obtain the test data.

[0056] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the artificial intelligence service testing method of the above-mentioned first aspect or any implementation method of the first aspect.

[0057] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0058] The memory is used to store computer programs;

[0059] The processor is used to execute the computer program so that the electronic device can implement the artificial intelligence service testing method of the above-mentioned first aspect or any implementation method of the first aspect.

[0060] A fifth aspect of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the artificial intelligence service testing method of the above-mentioned first aspect or any implementation of the first aspect.

[0061] By means of the above technical scheme, the present application provides an artificial intelligence service testing method and related devices, in which a pre-trained generative large model with the ability to generate test cases is used as a test case generation model, and a test data generator with the ability to generate multimodal test data is preset. When the artificial intelligence service to be tested needs to be tested, the interface document of the artificial intelligence service to be tested is first obtained; then the interface document of the artificial intelligence service to be tested is converted into a file in a preset format; then, according to the file in the preset format, the test case generation model is called to generate test cases, and according to the test cases, the preset test data generator is called to automatically generate test data; finally, based on the test cases and test data, the artificial intelligence service to be tested is tested. In this scheme, both test cases and test data can be automatically generated, which reduces the dependence on testers and can reduce the preparation cycle of test cases and test data, thereby improving the efficiency of artificial intelligence service testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0063] Figure 1 A flowchart of an artificial intelligence service testing method provided in an embodiment of the present application;

[0064] Figure 2 A schematic diagram of an example JSON Schema file provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of parameters in a JSON Schema file example provided in an embodiment of the present application;

[0066] Figure 4 A schematic diagram of an example of generating text test data provided in an embodiment of the present application;

[0067] Figure 5 A schematic diagram of an example of generating speech test data provided in an embodiment of the present application;

[0068] Figure 6 A schematic diagram of an example of generating image test data provided in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of the structure of an artificial intelligence service testing device provided in an embodiment of the present application;

[0070] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0072] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0073] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0074] With the continuous advancement of science and technology, the field of artificial intelligence (AI) has also ushered in rapid development. Various types of artificial intelligence services have emerged one after another. Its wide application in various fields has become a reality and has gradually become a key component of social infrastructure. For artificial intelligence services with rapid iterations in open environments and massive users, users have higher requirements for the reliability, stability and efficiency of artificial intelligence services. Testing is an important part of the software production process. Testers use high-quality and efficient testing methods to comprehensively evaluate the performance of artificial intelligence services in different scenarios, identify potential problems and solve them in a timely manner, thereby ensuring their reliability, stability and efficiency in actual applications.

[0075] Artificial intelligence services generally have strong business relevance. In the traditional artificial intelligence service research and development process, testers carry out the assurance work for artificial intelligence services. Generally speaking, the focus is on test case design and test data construction.

[0076] Among them, the design of test cases is usually done by testers who try to understand the actual use scenarios of the product based on the product functional specification documents and R&D interfaces at the beginning of the project, and then use the relevant methodologies for use case design in the test field (such as equivalence class partitioning, boundary value analysis, causal diagram testing, etc.) to list the use cases and convert them into interface parameters to be executed to form artificial intelligence service test cases. This process is heavily dependent on the professionalism of the testers, and the quality of the test cases is also uneven.

[0077] The use of multimodal test data is generally interspersed in the process of artificial intelligence service interaction. For example, dictation and multimodality require audio and video, synthesis and translation require text, and OCR (Optical Character Recognition) requires pictures. After the test case design is completed, the test data of different modalities is manually customized, collected and edited by testers according to user needs, which makes the construction and maintenance of test data more difficult.

[0078] In addition, whether it is the test cases designed manually by testers or the test data constructed manually, during the test implementation preparation stage, it may generally take about 2 rounds (3 to 5 days per round) of review and use to meet the delivery quality, which leads to a long test implementation preparation cycle.

[0079] To sum up, the traditional AI service testing method requires professional testers to manually design test cases, construct test data, and implement tests. This method relies heavily on the professionalism of testers, resulting in uneven quality of test cases, difficult construction and maintenance of test data, and a long test implementation preparation cycle, which in turn leads to low efficiency in AI service testing.

[0080] In order to solve the above problems, the embodiment of the present application provides an artificial intelligence service testing method, which can improve the efficiency of artificial intelligence service testing. The artificial intelligence service testing method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0081] Reference Figure 1 , Figure 1 A flowchart of an artificial intelligence service testing method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, an artificial intelligence service testing method provided in an embodiment of the present application may include the following steps, which are described in detail below.

[0082] S101: Obtain the interface document of the artificial intelligence service to be tested;

[0083] When an artificial intelligence service provides an open interface for external use, an API (Application Programming Interface) document is generally provided. It is an interface description that complies with the open API specification and generally has several presentation formats: markdown, JSON, or YAML format files. The key parts of the API document will define the request protocol, method, request and response parameter descriptions, etc. A complete request or response parameter generally contains the following information: field name, variable name, data type, whether it is required, maximum length, and description. Multiple parameters are combined to form a complete set of API documents. In this application, the interface document can be an API document.

[0084] S102: Converting the interface document of the artificial intelligence service to be tested into a file in a preset format;

[0085] In one feasible way, the interface document of the AI ​​service to be tested can be converted into a file in a preset format using a large model. The file in the preset format contains AI service capability description information, function parameters, and uplink and downlink data segments. The detailed description of the uplink and downlink data segments includes field name, variable name, data type, whether it is required, maximum length, description, etc. In this application, the preset format file can be a JSON Schema file. For ease of understanding, refer to Figure 2 , Figure 2 A schematic diagram of an example of a JSON Schema file provided in an embodiment of the present application. Figure 2 As shown in the JSON Schema file example, the left side contains AI service capability description information, function parameters, and uplink and downlink data segments, and the right side contains detailed descriptions of the uplink and downlink data segments.

[0086] S103: According to the file in a preset format, calling a test case generation model to generate a test case, wherein the test case generation model is a pre-trained generative large model with test case generation capability;

[0087] In this application, the test case generation model can be based on the generalization ability of the generative large model, and the parameters in the preset format file are arranged and matched to construct a test case generation module to achieve the generation of test cases. The established rules of the test case generation module can include upper and lower boundary values ​​of numeric types, values ​​of different length ranges of string types, special character constructions, traversal type enumeration values ​​or random values, etc. For ease of understanding, refer to Figure 3 , Figure 3 A parameter diagram in a JSON Schema file example provided in an embodiment of the present application, such as Figure 3As shown, the left side is the JSON Schema file structure, where the "parameter" part in the "schemainput" is the relevant description of the enumeration parameter example, and the enumeration parameter example can be an enumeration parameter in the HTTP request scenario body.

[0088] It should be noted that when the test case generation model generates a test case, it can also simultaneously generate a description of the key points of the test case, that is, the basis for the test case generation model to generate the test case, such as setting a parameter value beyond the specified range for abnormal testing to evaluate system stability.

[0089] S104: According to the test case, calling a preset test data generator to automatically generate test data; the preset test data generator has a multi-modal test data generation capability;

[0090] The use of multimodal test data is generally interspersed in the process of artificial intelligence service interaction. For example, dictation and multimodality require audio and video, synthesis and translation require text, and OCR (Optical Character Recognition) requires pictures, etc. Therefore, in this application, a test data generator with the ability to generate multimodal test data can be pre-built.

[0091] Considering that the generative big model performs well in text content creation, it may still be behind vertical AI services in other fields, such as speech synthesis, video generation, and image rendering. Based on this situation, the generative big model and other vertical AI services (including AI algorithms and AI tools) can be uniformly engineered and integrated to provide users with a test data generator with multimodal test data generation capabilities. The test data generator provides users with a unified standard interface, shielding users from the interaction differences between different AI services and the complexity of multimodal data processing.

[0092] S105: Testing the artificial intelligence service to be tested based on the test case and the test data.

[0093] In this application, the artificial intelligence service to be tested can be automatically tested based on the test case and the test data. In one possible implementation, a set of customized universal test frameworks can be preset to encapsulate common functions such as artificial intelligence service data processing, gateway authentication, data interaction, concurrent scheduling, and data statistics. The universal test framework can implement common functions such as cohesive service gateway authentication, audio and video codec processing, universal storage, data result processing, and session management, open standardized calling interfaces, and realize request-response interactions for different capabilities, different interaction scenarios, and different protocols of artificial intelligence services. At the same time, custom plug-in modules are introduced to support users in personalized result processing and performance statistics.

[0094] The test cases and the test data are completely decoupled from the general test framework and are interactively implemented based on a set of descriptions.

[0095] The present embodiment provides an artificial intelligence service testing method, in which a pre-trained generative large model with the ability to generate test cases is used as a test case generation model, and a test data generator with the ability to generate multimodal test data is preset. When the artificial intelligence service to be tested needs to be tested, the interface document of the artificial intelligence service to be tested is first obtained; then the interface document of the artificial intelligence service to be tested is converted into a file in a preset format; then, according to the file in the preset format, the test case generation model is called to generate test cases, and according to the test cases, the preset test data generator is called to automatically generate test data; finally, based on the test cases and test data, the artificial intelligence service to be tested is tested. In this solution, both test cases and test data can be automatically generated, which reduces the dependence on testers and can reduce the preparation cycle of test cases and test data, thereby improving the efficiency of artificial intelligence service testing.

[0096] In another embodiment of the present application, a specific implementation method of converting the interface document of the artificial intelligence service to be tested into a file in a preset format is described, and the method may include the following steps:

[0097] S201: Determine a document conversion prompt; the document conversion prompt includes a document conversion task description and an interface document of the artificial intelligence service to be tested;

[0098] In an achievable manner, the user can customize the document conversion prompt, and the determining of the document conversion prompt includes obtaining the document conversion prompt customized by the user. To further facilitate the user, in the present application, at least one document conversion prompt template may be preset, each of which includes a document conversion task description, a document filling slot, and a preset format file example filling slot; the at least one document conversion prompt template is provided to the user, and the user selects a target document conversion prompt template from it; the user may edit the target document conversion prompt template to obtain the edited target document conversion prompt template as the document conversion prompt.

[0099] In another achievable method, determining the document conversion prompt includes: obtaining an edited target document conversion prompt template obtained by the user editing the target document conversion prompt template as the document conversion prompt, the target document conversion prompt template is selected by the user from at least one preset document conversion prompt template, and each of the document conversion prompt templates includes a document conversion task description, a document filling slot, and a preset format file example filling slot.

[0100] For ease of understanding, the present application embodiment provides an example of a document conversion prompt template, which is as follows:

[0101] You are a R&D engineer. Now you need to parse the protocol according to the API document I provided to you [API document can be uploaded here] and generate a standard JSON Schema file. You can refer to the protocol specification (https: / / spec.openapis.org / oas / latest.html). The expected JSON Schema file will contain the following information. You need to fill in the parsed content as much as possible according to the rules.

[0102] mate: Description of the interaction process, mainly used to store key information of the AI ​​service interaction process, and contains a partial description of the schema (input, output).

[0103] Schema (input, output): Description of request and response parameters and data, and compliance with schema syntax validation.

[0104] Here is a sample JSON Schema file parsed by the AI ​​service [You can upload a sample JSON Schema file here] for your reference. Please parse the API document and return the response to me according to the sample file.

[0105] Among them, [You can upload API documents here] is the document filling slot, [You can upload JSON Schema sample file here] is the preset format file sample filling slot, and the rest is the document conversion task description.

[0106] After providing the document conversion prompt template example to the user, the user can fill the interface document of the artificial intelligence service to be tested into the document filling slot to obtain a document conversion prompt.

[0107] S202: providing the document conversion prompt to a document conversion model to obtain a file in a preset format generated by the document conversion model, wherein the document conversion model is a pre-trained generative large model with document conversion capability.

[0108] In another embodiment of the present application, a specific implementation method of calling a test case generation model to generate a test case according to a file in a preset format is described, and the method may include the following steps:

[0109] S301: Determine a test case generation prompt; the test case generation prompt includes a test case generation task description and the file in the preset format;

[0110] In an achievable manner, the user can customize the test case generation prompt, and the determination of the test case generation prompt includes: obtaining the test case generation prompt customized by the user. To further facilitate the user, in the present application, at least one test case generation prompt template can also be preset, each of which includes a test case generation task description and a preset format file filling slot; the at least one test case generation prompt template is provided to the user, and the user selects a target test case generation prompt template from it, and the user can edit the target test case generation prompt template to obtain the edited target test case generation prompt template; the edited target test case generation prompt template is obtained as the test case generation prompt.

[0111] In another achievable method, determining the test case generation prompt includes: obtaining an edited target test case generation prompt template obtained by a user editing the target test case generation prompt template, as the test case generation prompt; the target test case generation prompt template is selected by the user from at least one preset test case generation prompt template, and each of the test case generation prompt templates includes a test case generation task description and a preset format file filling slot.

[0112] For ease of understanding, the present application embodiment provides an example of a test case generation prompt template, which is as follows:

[0113] I want you to be a test engineer for AI services. I will be a product user, and I will provide you with some product documentation and JSON Schema files [JSON Schema files can be uploaded here]. I hope you can generate test cases based on professional methods in the testing field (equivalence class partitioning method, boundary value analysis method, causal graph testing method, etc.) as much as possible to ensure that the test cases cover a wide range and are not redundant. Each test case can be specific request data, which is returned to me in a JSON serialized string, with one test case per line.

[0114] Among them, [JSON Schema file can be uploaded here] is the preset format file to fill the slot, and the rest is the test case generation task description.

[0115] After the test case generation prompt template example is provided to the user, the user can fill the file in the preset format into the preset format file filling slot to obtain the test case generation prompt.

[0116] S302: providing the test case generation prompt to the test case generation model to obtain a test case generated by the test case generation model.

[0117] In another embodiment of the present application, after calling the test case generation model to generate the test case according to the file in a preset format, the method further includes: returning the test case to the tester for correction, obtaining a corrected test case and usage suggestions for the test case; and optimizing the test case generation model using the corrected test case and / or the usage suggestions for the test case.

[0118] In the present application, the test case can be returned to the tester for correction, and the correction method includes but is not limited to addition, deletion, modification and query. In the present application, after the test case generation model is called according to the file in the preset format to generate the test case, the test case is returned to the tester for correction, and the corrected test case and the usage suggestion of the test case are obtained. The test case generation model is optimized using the corrected test case and / or the usage suggestion of the test case, so that the generative large model can generate random test cases under limited conditions while retaining the content customized by the tester. While covering the basic functions of artificial intelligence services, it has fuzzy testing characteristics of some simple scenarios. While ensuring the quality of artificial intelligence service test cases, the test case design time is greatly shortened, and the test case design level of testers is improved in practice.

[0119] In another embodiment of the present application, calling a preset test data generator to automatically generate test data according to the test case includes:

[0120] S401: Parse the test case to obtain key information, where the key information is used to indicate basic information of the artificial intelligence service to be tested, where the basic information of the artificial intelligence service to be tested includes a business type, an input data type, and predefined semantic rules of the artificial intelligence service;

[0121] In the test case, the mate field contained therein will be associated with the basic information of the AI ​​service, including business type, interaction mode, input data type, routing parameters and predefined semantic rules. The test platform can parse the test case according to the established rules and obtain the basic information of the AI ​​service to be tested as key information for the generation of subsequent test cases.

[0122] S402: Determine test data generation prompt based on the key information;

[0123] In one achievable manner, the user can customize the test data generation prompt, and the determining the test data generation prompt based on the key information includes obtaining the test data generation prompt customized by the user based on the key information. In another achievable manner, in the present application, at least one test data generation prompt can also be preset, and the at least one test data generation prompt is provided to the user, and the user selects a target test data generation prompt from the prompt based on the key information as the test data generation prompt.

[0124] For ease of understanding, the present application embodiment provides a test data generation prompt example, which is as follows:

[0125] Example 1: Text Generator: Please act as a text generator. You will only reply to me with a text of 50 characters. I will tell you what you actually want to describe, and you will only reply with the result in the form of a markdown table, nothing else. Please note that your response should be concise and to the point, without any additional explanation. You will only reply with the result of the markdown table.

[0126] Example 2: Please provide English pronunciation assistance to people who speak different languages. I will give you sentences in a specific language and you need to answer with the correct English pronunciation. Answer only with pronunciation, no translation or explanation is required.

[0127] S403: performing semantic understanding classification on the test data generation prompt to obtain a classification result; the classification result is used to indicate the type of test data to be generated;

[0128] S404: Based on the classification result, calling the test data generator to automatically generate test data.

[0129] In the present application, the test data generator includes a pre-trained generative large model with test data generation capabilities and artificial intelligence services in other vertical fields.

[0130] In one possible implementation, if the classification result indicates that the type of test data to be generated is text, then based on the classification result, the test data generator is called to automatically generate test data, including: providing the test data generation prompt to the generative big model to obtain the generation result of the generative big model; and determining the test data according to the generation result of the generative big model.

[0131] In this application, considering that the generative big model performs well in text content creation, the generation result of the generative big model can be directly determined as the test data. Figure 4 , Figure 4 This is a schematic diagram of an example of generating text test data provided by an embodiment of the present application. Figure 4As shown, the left side is the mate field in the test case. By parsing it, it can be determined that the business type of the AI ​​service is "its", corresponding to the translation business, the interaction mode is "http", corresponding to the http one-time interaction, and the key input parameters of the predefined semantic rules are the "from" and "to" fields, corresponding to the source language "en" and the target language "cn" respectively. Based on this, it can be determined that the test data that matches it generates a prompt: "The sun always comes out after the wind and rain", please help me return the text content in English. The prompt generated by the test data is provided to the generative large model, and the text generated by the generative large model is obtained as the test data: "The sun always comes out after the wind and rain".

[0132] In another possible implementation, if the classification result indicates that the type of test data to be generated is non-text, then based on the classification result, the test data generator is called to automatically generate test data, including: providing the test data generation prompt to the generative big model to obtain the generation result of the generative big model; using the artificial intelligence services in other vertical fields to perform secondary editing on the generation result of the generative big model to obtain the test data.

[0133] Considering that the generative large model performs well in text content creation, it may still be behind the artificial intelligence services in vertical fields in other fields, such as speech synthesis, video generation, and image rendering. Therefore, in this application, the artificial intelligence services in other vertical fields can be used to perform secondary editing on the generation results of the generative large model to obtain the test data to improve the quality of the test data. In this application, the secondary editing of the generation results of the generative large model mainly involves some post-processing of the generation results of the generative large model, such as encoding and decoding, resampling, scaling, etc.

[0134] For ease of understanding, refer to Figure 5 , Figure 5 This is a schematic diagram of an example of generating voice test data provided by an embodiment of the present application. Figure 5As shown, the left side is the mate field in the test case. By parsing it, it can be determined that the business type of the AI ​​service is "iat", corresponding to the voice recognition business, the interaction mode is "ws", corresponding to the ws full-duplex interaction, and the key input parameters of the predefined semantic rules are fields such as "language", which represent the language of the voice to be recognized. Based on this, it can be determined that the test data generation prompt that matches it is: Please help me generate a Japanese daily greeting, which contains a 2s silent audio in the middle. It can be seen that in addition to Japanese, there are also key descriptions such as daily greetings and silent audio in the middle. The parameters corresponding to these key descriptions can be obtained from the mate field of the test case to construct the above test data generation prompt. The test data generation prompt is provided to the generative large model to obtain a PCM format audio generated by the generative large model, and then the PCM format audio is edited by artificial intelligence services in other vertical fields to obtain the edited PCM format audio segment as test data.

[0135] Refer to Figure 6 , Figure 6 This is a schematic diagram of an example of generating image test data provided by an embodiment of the present application. Figure 6 As shown, the left side is the mate field in the test case. By parsing it, it can be determined that the business type of the AI ​​service is "ocr", the interaction mode is "http", corresponding to the HTTP one-time interaction, and the key input parameters of the predefined semantic rules are fields such as "language", which represent the language to be identified in the picture. Based on this, it can be determined that the test data generation prompt that matches it is: Please help me generate a picture, the content of the picture is a piece of letter paper, and there is a handwritten Mongolian blessing on the paper. It can be seen that key descriptions such as "handwriting" are added here. The parameters corresponding to these key descriptions can be obtained from the mate field of the test case to construct the above test data generation prompt. The test data generation prompt is provided to the generative large model to obtain a handwritten picture containing Mongolian generated by the generative large model, and then the handwritten picture containing Mongolian is edited by artificial intelligence services in other vertical fields to obtain an edited handwritten picture containing Mongolian as test data.

[0136] The above introduces an artificial intelligence service testing method provided by an embodiment of the present application. The following will introduce a device for executing the above artificial intelligence service testing method.

[0137] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of an artificial intelligence service testing device provided in an embodiment of the present application. Figure 7 As shown, the artificial intelligence service testing device includes:

[0138] An acquisition unit 11 is used to acquire an interface document of the artificial intelligence service to be tested;

[0139] A document conversion unit 12, used to convert the interface document of the artificial intelligence service to be tested into a file in a preset format;

[0140] A test case generation unit 13 is used to generate a test case by calling a test case generation model according to a file in a preset format, wherein the test case generation model is a pre-trained generative large model with test case generation capability;

[0141] The test data generating unit 14 is used to call a preset test data generator to automatically generate test data according to the test case; the preset test data generator has a multi-modal test data generating capability;

[0142] The testing unit 15 is used to test the artificial intelligence service to be tested based on the test case and the test data.

[0143] In a possible implementation, the document conversion unit is specifically used to:

[0144] Determine a document conversion prompt; the document conversion prompt includes a document conversion task description and an interface document of the artificial intelligence service to be tested;

[0145] The document conversion prompt is provided to a document conversion model to obtain a file in a preset format generated by the document conversion model, wherein the document conversion model is a pre-trained generative large model with document conversion capabilities.

[0146] In a possible implementation, the test case generating unit is specifically used to:

[0147] Determine a test case generation prompt; the test case generation prompt includes a test case generation task description and the file in the preset format;

[0148] The test case generation prompt is provided to the test case generation model to obtain the test case generated by the test case generation model.

[0149] In a possible implementation, the device also includes: a test case generation model optimization unit, the test case generation model optimization unit is used to return the test case to the tester for correction after calling the test case generation model to generate the test case according to the file in a preset format, and obtain the corrected test case and the usage suggestion of the test case; use the corrected test case and / or the usage suggestion of the test case to optimize the test case generation model.

[0150] In a possible implementation, the test data generating unit includes:

[0151] A parsing unit, used to parse the test case to obtain key information, where the key information is used to indicate basic information of the artificial intelligence service to be tested, where the basic information of the artificial intelligence service to be tested includes a business type, an input data type, and predefined semantic rules of the artificial intelligence service;

[0152] A test data generation prompt determining unit, configured to determine a test data generation prompt based on the key information;

[0153] A classification unit, used for performing semantic understanding classification on the test data generation prompt to obtain a classification result; the classification result is used to indicate the type of test data to be generated;

[0154] The test data generating subunit is used to call the test data generator to automatically generate test data based on the classification result.

[0155] In one possible implementation, the test data generator includes a pre-trained generative large model with test data generation capabilities and artificial intelligence services in other vertical fields.

[0156] In a possible implementation, if the classification result indicates that the type of the test data to be generated is text, the test data generating subunit is specifically configured to:

[0157] The test data generation prompt is provided to the generative large model to obtain a generation result of the generative large model; and the test data is determined according to the generation result of the generative large model.

[0158] In a possible implementation, if the classification result indicates that the type of the test data to be generated is non-text, the test data generating subunit is specifically configured to:

[0159] The test data generation prompt is provided to the generative big model to obtain the generation result of the generative big model; and the generation result of the generative big model is re-edited by using the artificial intelligence services in other vertical fields to obtain the test data.

[0160] The present application also provides an electronic device in an embodiment. Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0161] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0162] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0163] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the artificial intelligence service testing methods provided in the embodiments of the present application.

[0164] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the artificial intelligence service testing methods provided in the embodiment of the present application.

[0165] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0166] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0167] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0168] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. An artificial intelligence service testing method, characterized in that: include: Obtain the interface documentation of the AI ​​service to be tested; Convert the interface document of the AI ​​service to be tested into a file in a preset format; According to the file in a preset format, a test case generation model is called to generate a test case, wherein the test case generation model is a pre-trained generative large model with the ability to generate test cases; According to the test case, calling a preset test data generator to automatically generate test data; the preset test data generator has the ability to generate multi-modal test data; Based on the test case and the test data, the artificial intelligence service to be tested is tested.

2. The method according to claim 1, characterized in that The step of converting the interface document of the artificial intelligence service to be tested into a file in a preset format includes: Determine a document conversion prompt; the document conversion prompt includes a document conversion task description and an interface document of the artificial intelligence service to be tested; The document conversion prompt is provided to a document conversion model to obtain a file in a preset format generated by the document conversion model, wherein the document conversion model is a pre-trained generative large model with document conversion capabilities.

3. The method according to claim 1, characterized in that The step of calling a test case generation model to generate a test case according to a file in a preset format includes: Determine a test case generation prompt; the test case generation prompt includes a test case generation task description and the file in the preset format; The test case generation prompt is provided to the test case generation model to obtain the test case generated by the test case generation model.

4. The method according to claim 1, characterized in that: After calling the test case generation model to generate the test case according to the file in the preset format, the method further includes: Returning the test case to the tester for correction, and obtaining the corrected test case and usage suggestions for the test case; The test case generation model is optimized using the revised test case and / or the usage suggestion of the test case.

5. The method according to claim 1, characterized in that The step of calling a preset test data generator to automatically generate test data according to the test case includes: Parsing the test case to obtain key information, where the key information is used to indicate basic information of the artificial intelligence service to be tested, where the basic information of the artificial intelligence service to be tested includes a business type, an input data type, and predefined semantic rules of the artificial intelligence service; Determine test data generation prompt based on the key information; Performing semantic understanding classification on the test data generation prompt to obtain a classification result; the classification result is used to indicate the type of test data to be generated; Based on the classification result, the test data generator is called to automatically generate test data.

6. The method according to claim 5, characterized in that The test data generator includes a pre-trained generative large model with test data generation capabilities and other vertical field artificial intelligence services.

7. The method according to claim 6, characterized in that If the classification result indicates that the type of the test data to be generated is text, then based on the classification result, calling the test data generator to automatically generate the test data includes: Providing the test data generation prompt to the generative large model to obtain a generation result of the generative large model; The test data is determined according to the generation result of the generative large model.

8. The method according to claim 6, characterized in that If the classification result indicates that the type of the test data to be generated is non-text, then based on the classification result, calling a test data generator to automatically generate test data includes: Providing the test data generation prompt to the generative large model to obtain a generation result of the generative large model; The generation result of the generative large model is re-edited by utilizing the artificial intelligence services in other vertical fields to obtain the test data.

9. An artificial intelligence service testing device, characterized in that: include: An acquisition unit, used to acquire the interface document of the artificial intelligence service to be tested; A document conversion unit, used to convert the interface document of the artificial intelligence service to be tested into a file in a preset format; A test case generation unit, used to generate a test case by calling a test case generation model according to a file in a preset format, wherein the test case generation model is a pre-trained generative large model with test case generation capability; A test data generating unit, used to call a preset test data generator to automatically generate test data according to the test case; The preset test data generator has the ability to generate multimodal test data; A testing unit is used to test the artificial intelligence service to be tested based on the test case and the test data.

10. A computer program product, characterized in that It includes computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the artificial intelligence service testing method as described in any one of claims 1 to 8.

11. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the artificial intelligence service testing method as described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the artificial intelligence service testing method as described in any one of claims 1 to 8.

Images