Test method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a testing method, device, electronic device and storage medium. Among them, the testing method includes: determining at least one target AI model that needs to be called by the application to be tested; configuring a test script for collaborative testing of the at least one target AI model based on the business operation process of the application to be tested; executing the test script, and determining the operating status of the at least one target AI model based on the test data generated during the execution of the test script. The embodiments of the present application can automatically test the AI ​​model in combination with the application, and automatically perform collaborative testing on at least one target AI model that needs to be called by the application to be tested by executing a test script configured based on the business operation process of the application to be tested. The testing process is simpler and more efficient, and the collaborative testing of multiple target AI models takes into account the influencing factors between different target AI models, so the accuracy of the test results is higher.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a testing method, device, electronic equipment and storage medium. Background Art

[0002] AI (Artificial Intelligence) is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It mainly includes the principles of computer intelligence and the manufacture of computers similar to human brain intelligence, so that computers can achieve higher-level applications. AI can be applied to various scenarios such as fingerprint recognition, voiceprint recognition, face recognition, speech recognition, and semantic recognition.

[0003] At present, AI models are generally deployed on cloud resource pools, and applications can call AI models on cloud resource pools to perform related operations during operation. However, as time changes, the operation status of AI models may become abnormal, resulting in reduced recognition accuracy of AI models.

[0004] The existing technology can only monitor the operating status of the hardware of the cloud resource pool, while the AI ​​model deployed on the cloud resource pool is an application process, and the operating status of the AI ​​model is usually determined by regular manual analysis. However, this method has a complex processing process and low efficiency. Summary of the invention

[0005] In view of the above problems, the embodiments of the present application propose a testing method, device, electronic device and storage medium, which can automatically test the operating status of the AI ​​model.

[0006] According to one aspect of an embodiment of the present application, a testing method is provided, the method comprising:

[0007] Determine at least one target artificial intelligence (AI) model that needs to be called by the application to be tested;

[0008] Based on the business operation process of the application to be tested, configure a test script for collaborative testing of the at least one target AI model;

[0009] Execute the test script, and determine the operating status of the at least one target AI model based on the test data generated during the execution of the test script.

[0010] Optionally, after configuring a test script for collaborative testing of the at least one target AI model, it also includes: configuring an execution strategy for the test script, the execution strategy including at least one of the following: a start execution time point, an execution cycle, and a number of executions; executing the test script includes: executing the test script according to the execution strategy.

[0011] Optionally, when the execution strategy includes the execution start time, the execution start time point is a time point when the application to be tested is in an idle state.

[0012] Optionally, before configuring the test script for collaborative testing of the at least one target AI model, it also includes: configuring the target sample required by the at least one target AI model; the target sample includes sample input data, expected output data of each target AI model and expected output data of the application to be tested; based on the business operation process of the application to be tested, configuring the test script for collaborative testing of the at least one target AI model, including: based on the business operation process of the application to be tested, configuring the execution order of each target AI model, information on the calling interface, information on the input data, information on the actual output data and information on the expected output data, as well as information on the actual output data and information on the expected output data of the application to be tested; the input data of the first target AI model executed is the sample input data, and the input data of the next target AI model executed is the output data of the previous target AI model executed.

[0013] Optionally, configuring the target sample required to be used by the at least one target AI model includes: determining the first target AI model to be executed among the at least one target AI model based on the business operation process of the application to be tested; configuring the sample required to be used by the first executed target AI model as the target sample; and the target sample traverses multiple business scenarios of the first executed target AI model.

[0014] Optionally, determining the operating status of at least one target AI model based on the test data generated during the execution of the test script includes: determining the operating status of at least one target AI model based on the test data of the target AI model generated during the execution of the test script.

[0015] Optionally, based on the test data of the target AI model generated during the execution of the test script, the operating status of at least one target AI model is determined, including: for each target AI model, obtaining the interface call data of the current target AI model generated during the execution of the test script, the interface call data being used to indicate whether the interface call is successful; when the interface call data of the current target AI model indicates that the interface call has failed, determining that the operating status of the current target AI model is abnormal.

[0016] Optionally, based on the test data of the target AI model generated during the execution of the test script, the operating status of the at least one target AI model is determined, including: obtaining the actual output data of the first executed target AI model generated during the execution of the test script, and determining that the operating status of the first executed target AI model is abnormal when the error between the actual output data and the expected output data of the first executed target AI model exceeds a first preset error; for each target AI model except the first executed target AI model, based on the error between the actual output data and the expected output data of the current target AI model generated during the execution of the test script, and the error between the actual output data and the expected output data of the target AI model executed before the current target AI model, a comprehensive error is calculated, and when the comprehensive error exceeds a second preset error, it is determined that the operating status of the current target AI model is abnormal.

[0017] Optionally, based on the test data generated during the execution of the test script, the running status of the at least one target AI model is determined, including: based on the test data of the application to be tested generated during the execution of the test script, the running status of the at least one target AI model is determined.

[0018] Optionally, based on the test data of the application to be tested generated during the execution of the test script, the operating status of the at least one target AI model is determined, including: obtaining the actual output data of the application to be tested generated during the execution of the test script; when the error between the actual output data of the application to be tested and the expected output data exceeds a third preset error, determining that there is a target AI model with an abnormal operating status among the at least one target AI model.

[0019] Optionally, the method also includes: determining the recognition accuracy of the at least one target AI model based on test data generated during multiple executions of the test script.

[0020] According to another aspect of an embodiment of the present application, a testing device is provided, the device comprising:

[0021] A determination module, used to determine at least one target artificial intelligence AI model that needs to be called by the application to be tested;

[0022] A first configuration module, configured to configure a test script for collaborative testing of the at least one target AI model based on the business operation process of the application to be tested;

[0023] A testing module is used to execute the test script and determine the operating status of the at least one target AI model based on the test data generated during the execution of the test script.

[0024] Optionally, the device also includes: a second configuration module, used to configure the execution strategy of the test script, the execution strategy including at least one of the following: the start time point of execution, the execution cycle, and the number of executions; the test module is specifically used to execute the test script according to the execution strategy.

[0025] Optionally, when the execution strategy includes the execution start time, the execution start time point is a time point when the application to be tested is in an idle state.

[0026] Optionally, the device also includes: a third configuration module, used to configure the target samples required to be used by the at least one target AI model, the target samples including sample input data, expected output data of each target AI model and expected output data of the application to be tested; the first configuration module is specifically used to configure the execution order of each target AI model, calling interface information, input data information, actual output data information and expected output data information, as well as actual output data information and expected output data information of the application to be tested based on the business operation process of the application to be tested; the input data of the first target AI model executed is the sample input data, and the input data of the next target AI model executed is the output data of the previous target AI model executed.

[0027] Optionally, the third configuration module is specifically used to determine the first executed target AI model among the at least one target AI model based on the business operation process of the application to be tested; configure the sample required by the first executed target AI model as the target sample; and the target sample traverses multiple business scenarios of the first executed target AI model.

[0028] Optionally, the test module includes: a first test unit, used to determine the operating status of the at least one target AI model based on the test data of the target AI model generated during the execution of the test script.

[0029] Optionally, the first test unit includes: a first acquisition subunit, used to obtain, for each target AI model, interface call data of the current target AI model generated during the execution of the test script, wherein the interface call data is used to indicate whether the interface call is successful; and a first determination subunit, used to determine that the running status of the current target AI model is abnormal when the interface call data of the current target AI model indicates that the interface call has failed.

[0030] Optionally, the first test unit includes: a second determination subunit, used to obtain the actual output data of the first executed target AI model generated in the process of executing the test script, and determine that the operating state of the first executed target AI model is abnormal when the error between the actual output data and the expected output data of the first executed target AI model exceeds a first preset error; a third determination subunit, used to calculate a comprehensive error for each target AI model except the first executed target AI model, based on the error between the actual output data and the expected output data of the current target AI model generated in the process of executing the test script, and the error between the actual output data and the expected output data of the target AI model executed before the current target AI model, and determine that the operating state of the current target AI model is abnormal when the comprehensive error exceeds a second preset error.

[0031] Optionally, the test module includes: a second test unit, used to determine the operating status of the at least one target AI model based on the test data of the application to be tested generated during the execution of the test script.

[0032] Optionally, the second test unit includes: a second acquisition subunit, used to acquire actual output data of the application to be tested generated during the execution of the test script; and a fourth determination subunit, used to determine that there is a target AI model with an abnormal operating status in the at least one target AI model when the error between the actual output data and the expected output data of the application to be tested exceeds a third preset error.

[0033] Optionally, the testing module is also used to determine the recognition accuracy of the at least one target AI model based on the test data generated during multiple executions of the test script.

[0034] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; and one or more computer-readable storage media having instructions stored thereon; when the instructions are executed by the one or more processors, the processors execute the test method as described in any one of the above items.

[0035] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the test method as described in any one of the above items.

[0036] In an embodiment of the present application, at least one target AI model that needs to be called by the application to be tested is determined; based on the business operation process of the application to be tested, a test script for collaborative testing of the at least one target AI model is configured; the test script is executed; based on the test data generated during the execution of the test script, the operating status of the at least one target AI model is determined. It can be seen from this that in an embodiment of the present application, the AI ​​model can be automatically tested in combination with the application, and based on the business operation process of the application to be tested, a test script for collaborative testing of at least one target AI model that needs to be called by the application to be tested can be configured, so that by executing the test script, the at least one target AI model that needs to be called by the application to be tested is automatically collaboratively tested, and the testing process is simpler and more efficient, and the collaborative testing of multiple target AI models takes into account the influencing factors between different target AI models, so the accuracy of the test results is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the embodiment of the present application, the drawings required for use in the description of the embodiment of the present application will be briefly introduced below. Obviously, the drawings described below are only some drawings of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a schematic diagram of a system structure of an embodiment of the present application.

[0039] Figure 2 It is a step flow chart of a testing method of an embodiment of the present application.

[0040] Figure 3 It is a step flow chart of another testing method of an embodiment of the present application.

[0041] Figure 4 It is a schematic diagram of a configuration test script according to an embodiment of the present application.

[0042] Figure 5 It is a structural block diagram of a testing device according to an embodiment of the present application.

[0043] Figure 6 It is a structural block diagram of another testing device according to an embodiment of the present application.

[0044] Figure 7 It is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] Reference Figure 1 , showing a schematic diagram of a system structure of an embodiment of the present application.

[0047] like Figure 1 As shown, in the embodiment of the present application, the test device can communicate with the AI ​​model (also referred to as the AI ​​capability engine) and the intelligent application. The test device can call the corresponding AI model and intelligent application according to the business operation process of the intelligent application.

[0048] Among them, AI models can include Figure 1 The speech recognition models shown (such as ASR (Automatic Speech Recognition) model), semantic recognition models (such as NLP (Natural Language Processing) model), voiceprint recognition models (such as SPR (Speaker Recognition) model), of course, face recognition models, fingerprint recognition models, etc. can also be included. Intelligent applications can include Figure 1 Shown are smart application 1, smart application 2, ... smart application n.

[0049] The test method in the embodiment of the present application is Figure 1 The test setup shown is executed and described in detail below.

[0050] Reference Figure 2 , showing a step flow chart of a testing method in an embodiment of the present application.

[0051] like Figure 2 As shown, the test method may include the following steps:

[0052] Step 201: determine at least one target AI model that needs to be called by the application to be tested.

[0053] The application to be tested can be Figure 1 Any smart application shown. The smart application can be any APP (application program) that needs to call the AI ​​model.

[0054] Schematically, for the application to be tested, at least one target AI model that the application to be tested needs to call can be determined based on the business operation process of the application to be tested. Specifically, the AI ​​model included in the business operation process of the application to be tested is the target AI model that the application to be tested needs to call. Depending on the actual situation, there are one or more target AI models.

[0055] For example, the business operation process of an application to be tested includes the process of first calling a speech recognition model to perform speech recognition on speech data, and then calling a semantic recognition model to perform semantic recognition on the results of speech recognition. Therefore, the target AI models that the application to be tested needs to call are the speech recognition model and the semantic recognition model.

[0056] Step 202: Based on the business operation process of the application to be tested, configure a test script for collaborative testing of the at least one target AI model.

[0057] This embodiment takes the application to be tested as a whole and configures a test script for collaborative testing of at least one target AI model that needs to be called by the application to be tested, so that at least one target AI model can be tested together. The specific process will be discussed in detail in the following embodiments.

[0058] Step 203: execute the test script, and determine the operating status of the at least one target AI model based on the test data generated during the execution of the test script.

[0059] By executing the above test script, at least one target AI model that needs to be called by the application to be tested is collaboratively tested. Based on the test data generated during the test process, the operating status of at least one target AI model can be determined. The specific process will be discussed in detail in the following embodiments.

[0060] In the embodiments of the present application, the AI ​​model can be automatically tested in combination with the application. Based on the business operation process of the application to be tested, a test script can be configured for collaborative testing of at least one target AI model that needs to be called by the application to be tested, so that by executing the test script, at least one target AI model that needs to be called by the application to be tested can be automatically collaboratively tested. The testing process is simpler and more efficient, and the collaborative testing of multiple target AI models takes into account the influencing factors between different target AI models, so the test results are more accurate.

[0061] Reference Figure 3 , shows a step flow chart of another testing method of an embodiment of the present application.

[0062] like Figure 3 As shown, the test method may include the following steps:

[0063] Step 301: determine at least one target AI model that needs to be called by the application to be tested.

[0064] Step 302: configure the target samples required to be used by the at least one target AI model.

[0065] Illustratively, the process of configuring the target sample required to be used by the at least one target AI model may include: determining the first executed target AI model among the at least one target AI model based on the business operation process of the application to be tested; configuring the sample required to be used by the first executed target AI model as the target sample. The test device may provide a sample database, and the target sample may be stored in the sample database.

[0066] Based on the business operation process of the application to be tested, at least one target AI model that needs to be called by the application to be tested and the execution order of each target AI model in the business operation process can be determined, and the sample that needs to be used by the first target AI model executed in the business operation process can be determined as the target sample that needs to be used by the at least one target AI model.

[0067] The target samples can traverse various business scenarios of the first executed target AI model, including the business types and leaf nodes of the application to be tested, thereby increasing the diversity of the target samples and improving the accuracy of the test.

[0068] The target sample may include sample input data, expected output data of each target AI model, and expected output data of the application to be tested. The sample input data is different depending on the type of the first AI model executed. Schematically, if the first AI model executed is a speech recognition model, the sample input data is speech data (such as recording data, etc.); if the first AI model executed is a semantic recognition model, the sample input data is text data; if the first AI model executed is a face recognition model, the sample input data is image data, and so on.

[0069] Step 303: Based on the business operation process of the application to be tested, configure a test script for collaborative testing of the at least one target AI model.

[0070] Illustratively, the process of configuring a test script for collaborative testing of at least one target AI model based on the business operation process of the application to be tested may include: configuring the execution order of each target AI model, information on the calling interface, information on the input data, information on the actual output data, and information on the expected output data, as well as information on the actual output data and expected output data of the application to be tested, based on the business operation process of the application to be tested. Among them, the input data of the first executed target AI model is the sample input data, and starting from the second executed target AI model, the input data of the latter executed target AI model is the output data of the previous executed target AI model.

[0071] First, define the data template according to the interface of each AI model and the interface of each application. The content of the data template may include: information about the calling interface of the AI ​​model (including the IP (Internet Protocol) address of the calling interface of the AI ​​model, etc.), information about the input data (including the location and file name of the input data of the AI ​​model, etc.), information about the actual output data (including the location and file name of the actual output data of the AI ​​model, etc.) and information about the expected output data (including the location and file name of the expected output data of the AI ​​model, etc.), as well as information about the calling interface of the application (including the IP address of the calling interface of the application, etc.), information about the actual output data (including the location and file name of the actual output data of the application, etc.) and information about the expected output data (including the location and file name of the expected output data of the application, etc.).

[0072] Then, the test script is configured as a graphical interface and implemented in the form of module construction. In the test script configuration interface, the process of the test script is defined based on the business operation process of the application to be tested, and each node in the process of the test script is configured according to the above data template. The process of the test script may include: the execution order of each target AI model, the information of the calling interface, the information of the input data, the information of the actual output data and the expected output data, as well as the information of the actual output data and the expected output data of the application to be tested.

[0073] In schematic form, taking into account the differences in interfaces of AI models of different types and from different manufacturers, the testing device may provide an interface calling module, which may encapsulate the calling interface of the AI ​​model into a standard calling interface and provide a visual standard calling interface.

[0074] The following examples illustrate this. Figure 4 Schematic diagram of a configuration test script in an embodiment of the present application. Figure 4As shown, the application process includes speech recognition → semantic recognition → application output. Among them, the content of the data template used for speech recognition includes speech recognition model IP, recording data (that is, the input data of the speech recognition model), speech recognition return result (that is, the actual output data of the speech recognition model), and speech recognition expected result (that is, the expected output data of the speech recognition model); the content of the data template used for semantic recognition includes semantic recognition model IP, speech recognition return result (that is, the input data of the semantic recognition model), semantic recognition return result (that is, the actual output data of the semantic recognition model), and semantic recognition expected result (that is, the expected output data of the semantic recognition model); the content of the data template used for application output includes the actual output data of the application and the expected output data of the application.

[0075] Step 304: configure the execution strategy of the test script.

[0076] Illustratively, the execution strategy of the test script may include but is not limited to at least one of the following: execution mode (including manual execution, automatic execution), execution start time point, execution cycle, execution times, and the like.

[0077] Wherein, when the execution strategy includes the execution start time, the execution start time is the time when the application to be tested is in an idle state. For example, the execution start time can be set to 2 a.m., 3 a.m., etc. This method can avoid the peak usage period of the application to be tested and reduce system pressure.

[0078] Step 305: execute the test script according to the execution strategy, and determine the operating status of the at least one target AI model based on the test data generated during the execution of the test script.

[0079] The test script is automatically executed according to the execution strategy configured above, the AI ​​model is called through the AI ​​model's calling interface, the target samples to be used are loaded, and the test data generated during the execution of the test script is obtained and stored.

[0080] In an optional implementation, the AI ​​model may be automatically tested at the device level. In this manner, the operating state of the at least one target AI model is determined based on the test data of the target AI model generated during the execution of the test script.

[0081] Illustratively, the process of determining the operating state of the at least one target AI model based on the test data of the target AI model generated during the execution of the test script may include the following steps A1-A2:

[0082] Step A1, for each target AI model, obtain the interface call data of the current target AI model generated during the execution of the test script.

[0083] Among them, the interface call data of the current target AI model is used to indicate whether the interface call of the current target AI model is successful.

[0084] Step A2: When the interface call data of the current target AI model indicates that the interface call fails, determine that the operating status of the current target AI model is abnormal.

[0085] For any target AI model, if the interface call of the target AI model fails, it means that the running status of the target AI model is abnormal.

[0086] Illustratively, the process of determining the operating state of at least one target AI model based on the test data of the target AI model generated during the execution of the test script may include the following steps B1 to B2:

[0087] Step B1, obtaining the actual output data of the first executed target AI model generated during the execution of the test script, and determining that the operating status of the first executed target AI model is abnormal when the error between the actual output data and the expected output data of the first executed target AI model exceeds a first preset error.

[0088] Since the first target AI model executed is not affected by other target AI models, the operating status of the first target AI model executed can be determined based on the error between the actual output data and the expected output data of the first target AI model executed.

[0089] The error between the actual output data and the expected output data of the first executed target AI model may be the difference between the actual output data of the first executed target AI model and the expected output data of the first executed target AI model. When the error corresponding to the first executed target AI model exceeds the first preset error, it can be determined that the operating state of the first executed target AI model is abnormal. The specific value of the first preset error can be set according to the actual situation, and this embodiment does not limit this.

[0090] Step B2, for each target AI model except the first executed target AI model, calculate the comprehensive error based on the error between the actual output data and the expected output data of the current target AI model generated during the execution of the test script, and the error between the actual output data and the expected output data of the target AI model executed before the current target AI model, and when the comprehensive error exceeds the second preset error, determine that the operating state of the current target AI model is abnormal.

[0091] Since starting from the second executed target AI model, the input data of the target AI model executed later is the output data of the target AI model executed previously, so starting from the second executed target AI model, the current target AI model will be affected by the target AI model executed previously. Therefore, starting from the second executed target AI model, based on the error between the actual output data and the expected output data of the current target AI model, and the error between the actual output data and the expected output data of the target AI model executed before the current target AI model (which can be the previous one or multiple previously executed ones), a comprehensive error is calculated, and the operating status of the current target AI model is determined based on the comprehensive error.

[0092] The error between the actual output data and the expected output data of the current target AI model may be the difference between the actual output data of the current target AI model and the expected output data of the current target AI model.

[0093] In schematic form, the comprehensive error may be a weighted calculation result of the error between the actual output data and the expected output data of the current target AI model and the error between the actual output data and the expected output data of the target AI model executed before the current target AI model. In the weighted calculation process, the weight of the error corresponding to each target AI model may be set according to the actual situation.

[0094] When the comprehensive error corresponding to the current target AI model exceeds the second preset error, it can be determined that the operating state of the current target AI model is abnormal. The specific value of the second preset error can be set according to actual conditions, and this embodiment does not limit this.

[0095] In another optional implementation, the AI ​​model may be automatically tested at the business level. In this manner, the operating state of the at least one target AI model is determined based on the test data of the application to be tested generated during the execution of the test script.

[0096] Illustratively, the process of determining the running state of the at least one target AI model based on the test data of the application to be tested generated during the execution of the test script may include the following steps C1-C2:

[0097] Step C1, obtaining actual output data of the application to be tested generated during the execution of the test script.

[0098] Step C2: When the error between the actual output data and the expected output data of the application to be tested exceeds a third preset error, determining that there is a target AI model with an abnormal operating status in the at least one target AI model.

[0099] When the error between the actual output data of the application to be tested and the expected output data exceeds the third preset error, it can be determined that there is a target AI model with an abnormal operating state. As to which specific target AI model has an abnormal operating state, the test data of each target AI model as described above can be obtained and analyzed and determined in the above manner.

[0100] The specific value of the third preset error can be set according to actual conditions, and this embodiment does not limit this.

[0101] Step 306: Generate an alarm if there is a target AI model with an abnormal operating status.

[0102] The alarm information here may include the identifier of the application to be tested, the identifier of the target AI model with abnormal running status, the abnormality type, the alarm level, etc.

[0103] Illustratively, the abnormality type may include but is not limited to interface call failure, high model recognition error, high application output error, etc. The alarm level includes but is not limited to general alarm, major alarm, etc. For example, interface call failure generates a major alarm, high model recognition error generates a general alarm, high application output error generates a major alarm, and so on.

[0104] Step 307: Determine the recognition accuracy of the at least one target AI model based on the test data generated during multiple executions of the test script.

[0105] For each target AI model, based on the test data of the target AI model generated during multiple executions of the test script, the recognition accuracy of the target AI model can be calculated. The accuracy can be the ratio of the number of accurate recognition results to the total number of recognitions.

[0106] Step 308: Generate an alarm when the recognition accuracy of any target AI model is less than a preset accuracy.

[0107] The alarm information of the alarm here may include the identifier of the application to be tested, the identifier of the target AI model whose recognition accuracy is less than the preset accuracy, the recognition accuracy of the target AI model whose recognition accuracy is less than the preset accuracy, the alarm level, etc. In principle, the lower the recognition accuracy of the target AI model, the higher the alarm level generated.

[0108] The following examples illustrate this.

[0109] Taking the application scenario of an intelligent voice customer service robot querying a communication package service as an example, its main business operation process is as follows:

[0110] (1) The user calls in through the call center.

[0111] (2) When a user inquires about a certain communication package, the speech recognition model is called to perform speech recognition on the user's voice and convert the user's voice into text.

[0112] (3) The semantic recognition model is called to perform semantic recognition on the converted text to determine whether the user is inquiring about a certain communication package. The semantic recognition model returns the Code (identification code) corresponding to the communication package.

[0113] (4) Query the business system according to the Code to obtain the user's package usage, and call TTS (Text To Speech) to broadcast.

[0114] The two key AI models, speech recognition model and semantic recognition model, are used in the business operation process of the above application as key nodes for key process monitoring. The service quality of the TTS engine generally does not fluctuate and does not need to be used as a monitoring node.

[0115] The testing process is as follows:

[0116] (1) Based on the business operation process of the application, determine that the target AI models to be called are the speech recognition model and the semantic recognition model.

[0117] (2) The speech recognition model is the first target AI model to be executed. Configure the target samples required by the speech recognition model.

[0118] The target sample can be recording data, which is recording data of different questions, different backgrounds, and different voices selected from the accumulated user communication package query recording training samples. The expected output data of the speech recognition model is the correct transcription result corresponding to the recording data, and the expected output data of the semantic recognition model is the Code code of the communication package.

[0119] (3) Configure the test script.

[0120] That is, choreographing the execution process, including setting the execution order, setting the calling interface information, setting the target sample information to be loaded, setting the actual output data information, setting the expected output data information, and so on.

[0121] (4) Configure the execution policy.

[0122] For example, configure it to execute automatically at 2:00 am every day.

[0123] (5) During the execution of the test script, if the interface call of the target AI model fails, an alarm will be generated; if the recognition error of the target AI model exceeds the preset error, an alarm will be generated.

[0124] (6) After executing the test script for one cycle (i.e., multiple times), the recognition accuracy of the target AI model is counted, and an alarm is generated if the recognition accuracy is lower than the preset accuracy.

[0125] In the embodiment of the present application, an automatic testing device independent of the AI ​​model is constructed to provide dynamic arrangement of intelligent application processes and dynamic configuration of data samples. By automatically executing test scripts regularly, the operating status and recognition accuracy of each AI model in the intelligent application process are monitored. The test of the AI ​​model supports both input and output judgment based on the end-to-end application process and input and output judgment based on the AI ​​model.

[0126] The embodiments of the present application solve the problem of being unable to actively monitor the operating status and recognition accuracy of the AI ​​model, and can be applied to actively and automatically monitor the AI ​​models deployed on the existing network, reducing the workload of manual periodic evaluation. The test device is independent of the existing AI model and intelligent application system, and there is no need to build a data acquisition module into these systems, which enables flexible orchestration and dynamic adjustment of the business test process to better adapt to the development and changes of business processes. It supports multiple automatic and manual execution modes, and can also be provided to operation and maintenance personnel. When a user complains, by configuring the actual output data and expected output data of the user complaint scenario, and then executing the test script, the execution data of key process nodes can be collected, thereby quickly locating the problem.

[0127] Reference Figure 5 , shows a structural block diagram of a testing device in an embodiment of the present application.

[0128] like Figure 5 As shown, the test device may include the following modules:

[0129] A determination module 501 is used to determine at least one target artificial intelligence AI model that needs to be called by the application to be tested;

[0130] A first configuration module 502 is used to configure a test script for collaborative testing of the at least one target AI model based on the business operation process of the application to be tested;

[0131] The testing module 503 is used to execute the test script and determine the operating status of the at least one target AI model based on the test data generated during the execution of the test script.

[0132] Reference Figure 6 , shows a structural block diagram of another testing device in an embodiment of the present application.

[0133] like Figure 6 As shown, the test device may include the following modules:

[0134] A determination module 601 is used to determine at least one target artificial intelligence AI model that needs to be called by the application to be tested;

[0135] A first configuration module 602 is used to configure a test script for collaborative testing of the at least one target AI model based on the business operation process of the application to be tested;

[0136] The testing module 603 is used to execute the test script and determine the operating status of the at least one target AI model based on the test data generated during the execution of the test script.

[0137] Optionally, the device also includes: a second configuration module 604, used to configure the execution strategy of the test script, the execution strategy including at least one of the following: start execution time point, execution cycle, and execution times; the test module 603 is specifically used to execute the test script according to the execution strategy.

[0138] Optionally, when the execution strategy includes the execution start time, the execution start time point is a time point when the application to be tested is in an idle state.

[0139] Optionally, the device also includes: a third configuration module 605, used to configure the target samples required to be used by the at least one target AI model, the target samples including sample input data, expected output data of each target AI model and expected output data of the application to be tested; the first configuration module 602, specifically used to configure the execution order of each target AI model, calling interface information, input data information, actual output data information and expected output data information, as well as actual output data information and expected output data information of the application to be tested based on the business operation process of the application to be tested; the input data of the first executed target AI model is the sample input data, and the input data of the next executed target AI model is the output data of the previous executed target AI model.

[0140] Optionally, the third configuration module 605 is specifically used to determine the first executed target AI model among the at least one target AI model based on the business operation process of the application to be tested; configure the sample required by the first executed target AI model as the target sample; and the target sample traverses multiple business scenarios of the first executed target AI model.

[0141] Optionally, the testing module 603 includes: a first testing unit, used to determine the operating status of the at least one target AI model based on the test data of the target AI model generated during the execution of the test script.

[0142] Optionally, the first test unit includes: a first acquisition subunit, used to obtain, for each target AI model, interface call data of the current target AI model generated during the execution of the test script, wherein the interface call data is used to indicate whether the interface call is successful; and a first determination subunit, used to determine that the running status of the current target AI model is abnormal when the interface call data of the current target AI model indicates that the interface call has failed.

[0143] Optionally, the first test unit includes: a second determination subunit, used to obtain the actual output data of the first executed target AI model generated in the process of executing the test script, and determine that the operating state of the first executed target AI model is abnormal when the error between the actual output data and the expected output data of the first executed target AI model exceeds a first preset error; a third determination subunit, used to calculate a comprehensive error for each target AI model except the first executed target AI model, based on the error between the actual output data and the expected output data of the current target AI model generated in the process of executing the test script, and the error between the actual output data and the expected output data of the target AI model executed before the current target AI model, and determine that the operating state of the current target AI model is abnormal when the comprehensive error exceeds a second preset error.

[0144] Optionally, the testing module 603 includes: a second testing unit, used to determine the running status of the at least one target AI model based on the test data of the application to be tested generated during the execution of the test script.

[0145] Optionally, the second test unit includes: a second acquisition subunit, used to acquire actual output data of the application to be tested generated during the execution of the test script; and a fourth determination subunit, used to determine that there is a target AI model with an abnormal operating status in the at least one target AI model when the error between the actual output data and the expected output data of the application to be tested exceeds a third preset error.

[0146] Optionally, the testing module 603 is also used to determine the recognition accuracy of the at least one target AI model based on the test data generated during multiple executions of the test script.

[0147] Optionally, the device further includes: a sample database 606 for storing samples.

[0148] Optionally, the device further includes: an interface calling module 607, which is used to encapsulate a standard calling interface of the AI ​​model.

[0149] Optionally, the device also includes: an alarm module 608, which is used to generate an alarm when there is a target AI model with an abnormal operating status, and generate an alarm when the recognition accuracy of any target AI model is less than a preset accuracy.

[0150] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0151] In an embodiment of the present application, an electronic device is also provided. The electronic device may include one or more processors, and one or more computer-readable storage media storing instructions thereon, the instructions being, for example, application programs. When the instructions are executed by the one or more processors, the processors execute the test method of any of the above embodiments.

[0152] Reference Figure 7 , shows a schematic diagram of an electronic device structure according to an embodiment of the present application. Figure 7 As shown, the electronic device includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704. The processor 701, the communication interface 702 and the memory 703 communicate with each other through the communication bus 704.

[0153] The memory 703 is used to store computer programs.

[0154] The processor 701 is used to implement the testing method of any of the above embodiments when executing the program stored in the memory 703.

[0155] The communication interface 702 is used for communication between the electronic device and other devices.

[0156] The communication bus 704 mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0157] The processor 701 mentioned above may include but is not limited to: a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0158] The memory 703 mentioned above may include but is not limited to: Read Only Memory (ROM), Random Access Memory (RAM), Compact Disc Read Only Memory (CD-ROM), Electronic Erasable Programmable Read Only Memory (EEPROM), hard disk, floppy disk, flash memory, etc.

[0159] In an embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program is stored. The program can be executed by a processor of an electronic device. When the computer program is executed by the processor, the processor executes the information processing method described in any of the above embodiments.

[0160] The various embodiments in this specification are interrelated, and each embodiment is described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referenced to each other.

[0161] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0162] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0163] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0164] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0166] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0169] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0170] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A testing method, characterized in that: The method comprises: Determine at least one target artificial intelligence (AI) model that needs to be called by the application to be tested; Configure a target sample to be used by the at least one target AI model; the target sample includes sample input data, expected output data of each target AI model, and expected output data of the application to be tested; Based on the business operation process of the application to be tested, configure a test script for collaborative testing of the at least one target AI model; Executing the test script, and determining the operating status of the at least one target AI model based on the test data generated during the execution of the test script; Based on the business operation process of the application to be tested, configuring a test script for collaborative testing of the at least one target AI model, including: based on the business operation process of the application to be tested, configuring the execution order of each target AI model, information on the calling interface, information on the input data, information on the actual output data, and information on the expected output data, as well as information on the actual output data and the expected output data of the application to be tested; The input data of the first target AI model executed is the sample input data, and the input data of the next target AI model executed is the output data of the previous target AI model executed.

2. The method according to claim 1, characterized in that: After configuring a test script for collaborative testing of the at least one target AI model, the method further includes: configuring an execution strategy for the test script, wherein the execution strategy includes at least one of the following: a start execution time point, an execution cycle, and an execution number; Executing the test script includes: executing the test script according to the execution strategy.

3. The method according to claim 2, characterized in that In the case where the execution strategy includes the execution start time, the execution start time point is a time point when the application to be tested is in an idle state.

4. The method according to claim 1, characterized in that: Configuring the target sample required to be used by the at least one target AI model includes: Based on the business operation process of the application to be tested, determine a first target AI model to be executed among the at least one target AI model; The samples required for the first executed target AI model are configured as the target samples; the target samples traverse various business scenarios of the first executed target AI model.

5. The method according to claim 1, characterized in that Determining the operating state of the at least one target AI model based on the test data generated during the execution of the test script includes: Based on the test data of the target AI model generated during the execution of the test script, the running status of the at least one target AI model is determined.

6. The method according to claim 5, characterized in that Determining the operating state of the at least one target AI model based on the test data of the target AI model generated during the execution of the test script includes: For each target AI model, obtain interface call data of the current target AI model generated during the execution of the test script, where the interface call data is used to indicate whether the interface call is successful; When the interface call data of the current target AI model indicates that the interface call fails, it is determined that the running state of the current target AI model is abnormal.

7. The method according to claim 5, characterized in that Determining the operating state of the at least one target AI model based on the test data of the target AI model generated during the execution of the test script includes: Acquire actual output data of the first executed target AI model generated during the execution of the test script, and determine that the operating state of the first executed target AI model is abnormal when an error between the actual output data and the expected output data of the first executed target AI model exceeds a first preset error; For each target AI model except the first executed target AI model, a comprehensive error is calculated based on the error between the actual output data and the expected output data of the current target AI model generated during the execution of the test script, and the error between the actual output data and the expected output data of the target AI model executed before the current target AI model. When the comprehensive error exceeds the second preset error, it is determined that the operating state of the current target AI model is abnormal.

8. The method according to claim 1, characterized in that Determining the operating state of the at least one target AI model based on the test data generated during the execution of the test script includes: Based on the test data of the application to be tested generated during the execution of the test script, the running status of the at least one target AI model is determined.

9. The method according to claim 8, characterized in that Determining the running state of the at least one target AI model based on the test data of the application to be tested generated during the execution of the test script includes: Acquire actual output data of the application to be tested generated during the execution of the test script; When an error between actual output data and expected output data of the application to be tested exceeds a third preset error, it is determined that there is a target AI model with an abnormal operating status in the at least one target AI model.

10. The method according to claim 1, characterized in that The method further comprises: Based on the test data generated during multiple executions of the test script, the recognition accuracy of the at least one target AI model is determined.

11. A testing device, characterized in that: The device comprises: A determination module, used to determine at least one target artificial intelligence AI model that needs to be called by the application to be tested; A third configuration module, used to configure a target sample to be used by the at least one target AI model, wherein the target sample includes sample input data, expected output data of each target AI model, and expected output data of the application to be tested; A first configuration module, configured to configure a test script for collaborative testing of the at least one target AI model based on the business operation process of the application to be tested; A testing module, configured to execute the test script and determine the operating status of the at least one target AI model based on test data generated during the execution of the test script; The first configuration module is specifically used to configure the execution order of each target AI model, calling interface information, input data information, actual output data information and expected output data information, as well as actual output data information and expected output data information of the application to be tested based on the business operation process of the application to be tested; the input data of the first target AI model executed is the sample input data, and the input data of the next target AI model executed is the output data of the previous target AI model executed.

12. An electronic device, characterized in that: include: one or more processors; and one or more computer-readable storage media having instructions stored thereon; When the instructions are executed by the one or more processors, the processors are enabled to perform the testing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the processor is caused to execute the testing method according to any one of claims 1 to 10.

Images