AI outbound call evaluation method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides an AI outbound call evaluation method, device, electronic device and storage medium, the method comprising: obtaining outbound voice data; determining speech synthesis evaluation indicators based on the broadcast error rate of the TTS process; determining speech recognition evaluation indicators based on the word error rate and sentence error rate of the ASR process; determining single model evaluation indicators for each single model in the NLP process; determining multi-model evaluation indicators corresponding to a combination of multiple models; determining language processing evaluation indicators based on multiple multi-model evaluation indicators; determining comprehensive indicators based on speech synthesis evaluation indicators, speech recognition evaluation indicators and language processing evaluation indicators to obtain evaluation results. The present application performs a separate evaluation on each process of the AI ​​outbound call, and further conducts a comprehensive evaluation on the evaluation indicators obtained for each process, to obtain a comprehensive and systematic evaluation result for the entire AI outbound call process, thereby improving the accuracy of the evaluation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an AI outbound call evaluation method, device, electronic device and storage medium. Background Art

[0002] Artificial Intelligence (AI) is a science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. It is a new branch of computer science that attempts to produce a new type of intelligent machine that responds in a similar way to human intelligence by exploring the essence of intelligence. Research in this field includes robots, speech synthesis / recognition, image recognition, natural language processing and expert systems, etc.

[0003] In recent years, the application of artificial intelligence in the field of multi-intent recognition has become more and more extensive, and the field technologies and model types involved are also more diverse and complex. In the field of AI outbound calling (an intelligent service platform that uses robots to help platform users automatically make calls and communicate with platform users' customers), in the process of real-time communication between robots and customers (real people), at least artificial intelligence technologies such as (simulated human) speech synthesis, (real) speech recognition, natural language processing and multi-intent recognition are needed to use robots to meet customer needs.

[0004] At present, the evaluation of model effects in various research fields of artificial intelligence has become increasingly mature, and there are many evaluation indicators from different angles for researchers to refer to. However, not only is there a difference between the effect of a pure model and the effect of the model applied to real life, but there is also a huge difference between the effect of any single model in a complex mixed application and the effect of the overall application. Therefore, it often gives people a "distorted" evaluation result of the model effect, that is, there are situations where users feel that the robot has not correctly identified their ideas. Therefore, how to accurately improve the accuracy of AI outbound call evaluation has become an urgent problem to be solved. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose an AI outbound call evaluation method, device, electronic device and storage medium, which can improve the accuracy of AI outbound call evaluation.

[0006] To achieve the above objectives, a first aspect of an embodiment of the present application proposes an AI outbound call evaluation method, the method comprising:

[0007] Acquire outbound voice data, wherein the outbound voice data includes synthesized voice broadcast information and target object voice input information;

[0008] Determine a broadcast error rate of the synthesized speech broadcast information, and determine a speech synthesis evaluation index according to the broadcast error rate;

[0009] Determine a word error rate and a sentence error rate of the target object's speech input information, and determine a speech recognition evaluation index according to the word error rate and the sentence error rate;

[0010] Using multiple single models to respectively determine the intent classification corpus data of the target object's speech input information, and determining the single model evaluation index of each single model according to each of the intent classification corpus data, wherein the single model evaluation index is used to characterize the intent classification accuracy of the single model, wherein the multiple single models are different models used for intent classification in the natural language processing process;

[0011] Determine a plurality of model combinations according to the plurality of single models, and determine a multi-model evaluation index corresponding to the plurality of model combinations one by one according to the plurality of model combinations and the plurality of single model evaluation indexes;

[0012] Determining a language processing evaluation index according to a plurality of the multi-model evaluation indexes;

[0013] Determine a comprehensive index based on the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index;

[0014] An evaluation result is obtained according to the comprehensive index.

[0015] In some embodiments, obtaining the evaluation result according to the comprehensive indicator includes:

[0016] Determine a speech intention evaluation index according to the speech recognition evaluation index and the language processing evaluation index;

[0017] An evaluation result is obtained according to the comprehensive index and the speech intention evaluation index.

[0018] In some embodiments, determining a language processing evaluation index according to a plurality of the multi-model evaluation indexes includes:

[0019] Determine the proportion information corresponding to each of the plurality of model combinations;

[0020] The plurality of multi-model evaluation indicators are weighted according to the proportion information to obtain the language processing evaluation indicator.

[0021] In some embodiments, determining the comprehensive index according to the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index includes:

[0022] Determine the weight factors corresponding to the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index respectively;

[0023] The speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index are weighted according to the weight factor to obtain the comprehensive index.

[0024] In some embodiments, each of the intent classification corpus data includes a plurality of model intent classification corpus data, and the plurality of model intent classification corpus data are obtained by classifying a plurality of different intent classifications by the single model;

[0025] The single model evaluation index corresponding to each single model is obtained by the following steps:

[0026] Determine the number of correct corpora for intent classification according to the plurality of said model intent classification corpora and the plurality of preset correct expected classification numbers, wherein the plurality of said correct expected classification numbers correspond one to one with the plurality of said model intent classification corpora;

[0027] The accuracy of intent classification is determined according to the number of correct corpora for intent classification, and a single model evaluation index is obtained.

[0028] In some embodiments, determining a speech synthesis evaluation index according to the broadcast error rate includes:

[0029] When the broadcast error rate is less than a preset first error rate threshold, determining the broadcast accuracy rate of the synthesized voice broadcast information;

[0030] The speech synthesis evaluation index is determined according to the broadcast accuracy rate.

[0031] In some embodiments, determining the speech recognition evaluation index according to the word error rate and the sentence error rate includes:

[0032] When the word error rate is less than a preset second error rate threshold, a speech recognition evaluation index is determined according to the sentence error rate.

[0033] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes an AI outbound call evaluation device, the device comprising:

[0034] A data acquisition module, used to acquire outbound voice data, wherein the outbound voice data includes synthesized voice broadcast information and target object voice input information;

[0035] A broadcast evaluation module, used to determine a broadcast error rate of the synthesized speech broadcast information, and determine a speech synthesis evaluation index according to the broadcast error rate;

[0036] A language recognition module, used to determine a word error rate and a sentence error rate of the target object's voice input information, and determine a voice recognition evaluation index according to the word error rate and the sentence error rate;

[0037] A single model evaluation module, used to use multiple single models to respectively determine the intent classification corpus data of the target object speech input information, and determine the single model evaluation index of each single model according to each of the intent classification corpus data, wherein the single model evaluation index is used to characterize the intent classification accuracy of the single model, wherein the multiple single models are different models used for intent classification in the natural language processing process;

[0038] A multi-model evaluation module, used to determine a plurality of model combinations according to the plurality of single models, and to determine a multi-model evaluation index corresponding to the plurality of model combinations one by one according to the plurality of model combinations and the plurality of single model evaluation indexes;

[0039] A natural language processing comprehensive evaluation module, used to determine a language processing evaluation index based on a plurality of said multi-model evaluation indicators;

[0040] Applying a comprehensive evaluation module, for determining a comprehensive index according to the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index;

[0041] The evaluation result generating module is used to obtain the evaluation result according to the comprehensive index.

[0042] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the AI ​​outbound call evaluation method described in the first aspect above is realized, or the AI ​​outbound call evaluation method described in the second aspect is realized.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the AI ​​outbound call evaluation method described in the first aspect above, or the AI ​​outbound call evaluation method described in the second aspect.

[0044] The AI ​​outbound call evaluation method, device, electronic device and storage medium proposed in the present application obtain outbound call voice data, wherein the outbound call voice data includes synthesized voice broadcast information and target object voice input information; determine the broadcast error rate of the synthesized voice broadcast information, and determine the voice synthesis evaluation index according to the broadcast error rate; determine the word error rate and sentence error rate of the target object voice input information, and determine the voice recognition evaluation index according to the word error rate and sentence error rate; use multiple single models to respectively determine the intent classification corpus data of the target object voice input information, and determine the single model evaluation index of each single model according to each intent classification corpus data, the single model evaluation index is used to characterize the intent classification accuracy of the single model, and the multiple single models are different models used for intent classification in the natural language processing process; determine multiple model combinations according to the multiple single models, and determine multi-model evaluation indicators corresponding to the multiple model combinations according to the multiple model combinations and the multiple single model evaluation indicators; determine the language processing evaluation index according to the multiple multi-model evaluation indicators; determine the comprehensive index according to the voice synthesis evaluation index, the voice recognition evaluation index and the language processing evaluation index; and obtain the evaluation result according to the comprehensive index. This application conducts a separate evaluation of each process of AI outbound calls, and further conducts a comprehensive evaluation of the evaluation indicators obtained for each process, to obtain a comprehensive and systematic evaluation result of the entire AI outbound call process, thereby improving the accuracy of the evaluation and facilitating the subsequent use of the evaluation results to optimize the model and algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of the AI ​​outbound call evaluation method provided in an embodiment of the present application;

[0046] Figure 2 yes Figure 1 A schematic diagram of a sub-process of step S800 in FIG.

[0047] Figure 3 yes Figure 1 A schematic diagram of a sub-process of step S600 in FIG.

[0048] Figure 4 yes Figure 1 A schematic diagram of a sub-process of step S700 in FIG.

[0049] Figure 5 It is a structural diagram of an AI outbound call evaluation device provided in an embodiment of the present application;

[0050] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] In order to better describe the embodiment of the present application, an AI outbound call evaluation method, device, electronic device and storage medium are provided. Here is a brief description of the key technologies involved in the AI ​​outbound call process. These technologies mainly include text to speech synthesis (Text To Speech, TTS) technology, speech to text (Acceleration Slip Regulation, ASR) technology, natural language processing (Natural Language Processing, NLP) technology, etc.

[0055] TTS speech synthesis technology is used in the robot voice broadcast stage, ASR speech recognition technology is used in the text analysis stage of voice messages input by customers, and NLP natural language processing technology is used to analyze the intent of voice messages input by customers. Among them, natural language processing technology involves technical means such as word segmentation / keyword extraction, sentence model, rule engine, ES engine, ESIM model, TextCNN model and Fasttext model. When analyzing the intent of voice messages input by customers, the technical means in natural language processing technology are used to complete the intent analysis and processing.

[0056] Therefore, to evaluate the entire AI outbound calling process, it is necessary to consider the results of the comprehensive application and coordination of TTS speech synthesis technology, ASR speech recognition technology and NLP natural language processing technology. The effect evaluation results of any single field technology and model cannot fully and truly represent the performance of the entire application.

[0057] Based on this, the embodiments of the present application provide an AI outbound call evaluation method, device, electronic device and storage medium, which can comprehensively consider TTS speech synthesis technology, ASR speech recognition technology and NLP natural language processing technology, so that the evaluation results have better accuracy.

[0058] The AI ​​outbound call evaluation method, device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the AI ​​outbound call evaluation method in the embodiments of the present application is described.

[0059] The AI ​​outbound call evaluation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The AI ​​outbound call evaluation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the AI ​​outbound call evaluation method, etc., but is not limited to the above forms.

[0060] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0061] Reference Figure 1 , Figure 1 is an optional flow chart of the AI ​​outbound call evaluation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S800.

[0062] Step S100, obtaining outbound voice data, the outbound voice data including synthesized voice broadcast information and target object voice input information;

[0063] Step S200, determining the error rate of the synthesized speech broadcast information, and determining the speech synthesis evaluation index according to the error rate of the broadcast;

[0064] Step S300, determining the word error rate and sentence error rate of the target object's voice input information, and determining a voice recognition evaluation index according to the word error rate and sentence error rate;

[0065] Step S400, using multiple single models to respectively determine the intent classification corpus data of the target object's speech input information, and determining the single model evaluation index of each single model according to each intent classification corpus data, the single model evaluation index is used to characterize the intent classification accuracy of the single model, wherein the multiple single models are different models used for intent classification in the natural language processing process;

[0066] Step S500, determining multiple model combinations according to multiple single models, and determining multi-model evaluation indicators corresponding to the multiple model combinations one by one according to the multiple model combinations and the multiple single model evaluation indicators;

[0067] Step S600, determining a language processing evaluation index according to a plurality of multi-model evaluation indexes;

[0068] Step S700, determining a comprehensive index according to the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index;

[0069] Step S800, obtaining evaluation results according to comprehensive indicators.

[0070] Steps S100 to S800 of the embodiment of the present application can obtain the outbound voice data in the AI ​​outbound call process, and then analyze the outbound voice data to determine the broadcast error rate of the synthesized voice broadcast information, and determine the word error rate and sentence error rate of the target object voice input information, so that the voice synthesis evaluation index can be determined according to the broadcast error rate, and the voice recognition evaluation index can be determined according to the word error rate and sentence error rate; at the same time, multiple single models are used to determine the intention classification corpus data of the target object voice input information respectively, and then the single model evaluation index of each single model can be determined according to each intention classification corpus data, and then, according to multiple model combinations and multiple single model evaluation indicators, a multi-model evaluation index corresponding to multiple model combinations can be determined, and then the language processing evaluation index can be determined according to multiple multi-model evaluation indicators, and finally the comprehensive index is determined according to the voice synthesis evaluation index, the voice recognition evaluation index and the language processing evaluation index to obtain the evaluation result. The present application evaluates each process of the AI ​​outbound call separately, and further comprehensively evaluates the evaluation indicators obtained for each process, so as to obtain a comprehensive and systematic evaluation result of the entire AI outbound call process, thereby improving the accuracy of the evaluation and facilitating the subsequent optimization of the model and algorithm using the evaluation results.

[0071] Specifically, the synthesized speech broadcast information is directly generated based on the TTS speech synthesis technology. The broadcast error rate of the text-to-speech conversion can be used as the evaluation basis for the speech synthesis evaluation index of the TTS speech synthesis process. The lower the broadcast error rate, the better the result of the corresponding speech synthesis evaluation index. The target object's speech input information is recognized as text through ASR speech recognition technology. Recognition errors may occur during the recognition process. These errors can finally be summarized as word error rate and sentence error rate. It can be understood that the lower the word error rate and sentence error rate, the better the corresponding speech recognition evaluation index determined based on the word error rate and sentence error rate. NLP natural language processing technology can perform intent analysis on the target object's speech input information, and thus determine the language processing evaluation index based on the results of intent classification.

[0072] For NLP natural language processing technology, since the processing process involves many single models, it is necessary to first evaluate each single model separately, and then use multiple models to complete the combined evaluation, and finally determine the language processing evaluation indicators of the entire NLP natural language processing process based on the combined evaluation results. It should be noted that the single models in NLP natural language processing include at least emotion recognition model, sentence model, rule engine model, ES recall model, TextCNN model and Fasttext model.

[0073] For the evaluation of a single model, the intent classification accuracy of the single model can be directly used as the evaluation standard, and the evaluation process can be completed using the intent classification corpus data obtained by each single model itself for intent classification. The combination of multiple models needs to consider the recognition effect of the combination of various models that have a mutual influence relationship. In this embodiment, at least the combination of emotion recognition model and rule engine model, emotion recognition model and sentence model and ES recall model, emotion recognition model and sentence model and TextCNN model, emotion recognition and sentence model and Fasttext model, the specific number of combined models and the combination can be adaptively adjusted according to actual needs; it should be noted that each combined model determines the multi-model evaluation index directly based on the single model used in the combination. For the specific acquisition method, it can be completed directly using the corresponding single model evaluation index based on the principle of Cartesian product. Taking the emotion recognition model and the rule engine model as an example, the multi-model evaluation index is the product of the single model evaluation index corresponding to the emotion recognition model and the single model evaluation index corresponding to the rule engine model. After multiple multi-model evaluation indicators are determined, the impact of multiple model combinations can be directly integrated to obtain the language processing evaluation indicators of the NLP natural language processing process using multiple multi-model evaluation indicators.

[0074] It should be noted that when evaluating the overall effect, there is a close relationship between ASR speech recognition and NLP natural language processing. The effect of ASR speech recognition will directly affect the intention recognition of NLP natural language processing. At the same time, the performance of TTS speech synthesis will directly affect the customer's emotions and answers. That is, if the robot's broadcast is unclear or wrong, it is likely to cause the customer's emotions to fluctuate or answer questions incorrectly / unwillingly, or even hang up the phone. Therefore, for the overall application, in addition to considering the ASR speech recognition process and the NLP natural language processing process, it is also necessary to comprehensively consider the effect of TTS speech synthesis. Then, after determining the speech synthesis evaluation index, speech recognition evaluation index, and language processing evaluation index, the impact of each process in actual processing can be comprehensively considered to determine the final comprehensive index, which will be used as the evaluation result output for R&D personnel to consult, so as to optimize the overall algorithm or model architecture in the future.

[0075] See also Figure 2 In some embodiments, step S800 includes:

[0076] Step S810, determining a speech intention evaluation index according to the speech recognition evaluation index and the language processing evaluation index;

[0077] Step S820, obtaining an evaluation result according to the comprehensive index and the speech intention evaluation index.

[0078] In addition to the comprehensive evaluation of the entire process, the evaluation of the ASR speech recognition process and the NLP natural language processing process also has a relatively prominent practical significance, which can help R&D personnel better understand the intent recognition effect. Therefore, when generating the evaluation results, the speech intent evaluation index will be determined based on the speech recognition evaluation index and the language processing evaluation index, and then the comprehensive index and the speech intent evaluation index will be output as the evaluation results at the same time. It should be noted that the determination of the speech intent evaluation index can be obtained by directly multiplying the speech recognition evaluation index and the language processing evaluation index.

[0079] See also Figure 3 In some embodiments, step S600 includes:

[0080] Step S610, determining the proportion information corresponding to each of the multiple model combinations;

[0081] Step S620: weighting the multiple multi-model evaluation indicators according to the proportion information to obtain a language processing evaluation indicator.

[0082] Different single models have different accuracies and performances. Therefore, in order to further ensure the accuracy of the evaluation results, it is necessary to add proportion information to distinguish the influence of different model combinations. Therefore, when setting the proportion information, it is necessary to consider factors such as the accuracy and performance of the model itself, and then determine the proportion information corresponding to each model combination. Usually, different model combinations will correspond to different proportion information. After determining the praise ratio information, the language processing evaluation index can be calculated by weighted calculation. In this embodiment, the calculation formula of the language processing evaluation index is as follows:

[0083] Language processing evaluation index = rule engine proportion * (emotion recognition model + rule engine model) multi-model evaluation index + ES recall statistical proportion * (emotion recognition model + sentence model + ES recall model) multi-model evaluation index + TextCNN statistical proportion * (emotion recognition model + sentence model + TextCNN model) multi-model evaluation index + Fasttext statistical proportion * (emotion recognition model + sentence model + Fasttext model) multi-model evaluation index,

[0084] In the formula, the rule engine proportion, ES recall statistics proportion, TextCNN statistics proportion, and Fasttext statistics proportion are the proportion information of the corresponding model combinations in the formula.

[0085] See also Figure 4 In some embodiments, step S700 includes:

[0086] Step S710, determining weight factors corresponding to the speech synthesis evaluation index, the speech recognition evaluation index, and the language processing evaluation index;

[0087] Step S720, weighting the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index according to the weight factor to obtain a comprehensive index.

[0088] The TTS speech synthesis process, ASR speech recognition process and NLP natural language processing process have different effects on the intent recognition effect. It is understandable that the TTS speech synthesis process has a relatively smaller impact on the intent recognition result. Therefore, a weight factor can be added to perform weighted operations on the speech synthesis evaluation index, speech recognition evaluation index and language processing evaluation index to obtain a comprehensive index. In this embodiment, the calculation formula of the comprehensive index is as follows:

[0089] Comprehensive index = (w(T)*speech synthesis evaluation index)*(w(A)*speech recognition evaluation index)*(w(N)*language processing evaluation index),

[0090] Where w(T), w(A), and w(N) are the weight factors corresponding to the speech synthesis evaluation index, speech recognition evaluation index, and language processing evaluation index, respectively.

[0091] In some embodiments, each intent classification corpus data includes multiple model intent classification corpus numbers, and the multiple model intent classification corpus numbers are obtained by classifying multiple different intent classifications by a single model;

[0092] The single model evaluation indicators corresponding to each single model are obtained by the following steps:

[0093] Determine the number of correct corpora for intent classification according to the number of model intent classification corpora and the number of preset correct expected classification numbers, wherein the number of correct expected classification numbers corresponds to the number of model intent classification corpora one by one;

[0094] The accuracy of intent classification is determined based on the number of correct corpora for multiple intent classifications, and a single model evaluation index is obtained.

[0095] For a single model, the single model evaluation index can be directly obtained based on the results of intent classification. Taking a single model as an example, each single model needs to perform multiple different intent classifications, thereby obtaining multiple model intent classification corpora; assuming that the total amount of the test set corpus is P, the number of correct expected classifications belonging to intent classification X is PX, the number of model intent classification corpora recognized by the robot as intent classification X is AX, and the number of correct intent classification corpora recognized by the robot as intent classification X is TX = PX ∩ AX. The calculation formula for intent classification accuracy is as follows:

[0096]

[0097] In the formula, n is the n different intent classifications corresponding to each single model, and the intent classification accuracy is the single model evaluation indicator.

[0098] In addition, when calculating the intent classification accuracy, the single classification accuracy and single classification correct proportion of each intent classification of the single model are also calculated. The specific calculation formula is as follows:

[0099] Single classification accuracy = TX / PX,

[0100] The correct proportion of single classification = TX / AX,

[0101] The single classification accuracy rate represents the recognition capability of a certain intent classification robot, which can be provided to researchers so that they can collect more corpus and types of the intent classification in a targeted manner and perform data annotation, and finally use it for model training to improve the recognition effect. The single classification correct ratio also represents the recognition capability of a certain intent classification robot. Unlike the single classification accuracy rate, it is provided to researchers for direct model analysis and optimization to improve the recognition effect. By obtaining the single classification accuracy rate and the single classification correct ratio, in the entire evaluation process, in addition to knowing the effect of AI outbound calls, researchers can also obtain data that can be used more effectively. It should be noted that for each single model, the single model evaluation indicators as well as multiple single classification accuracy rates and multiple single classification correct ratios must be calculated.

[0102] In some embodiments, step 200 includes:

[0103] When the broadcast error rate is less than a preset first error rate threshold, determining the broadcast accuracy rate of the synthesized voice broadcast information;

[0104] Determine the speech synthesis evaluation index based on the broadcast accuracy.

[0105] First of all, it should be noted that the evaluation of the TTS speech synthesis process can be obtained through the MOS value test method. Directly using the MOS automatic scoring system can save human resources and time expenses. The MOS automatic scoring system can choose MOSNET, AutoNET, etc. However, considering that the robot only needs to broadcast the script normally and clearly, it will not have a significant impact on the subsequent intention recognition process, and you only need to care about the correct / wrong ratio of the robot's broadcast. Therefore, the MOS value test result can be converted into a probability value to determine that the robot's broadcast is correct. Therefore, this application directly uses the broadcast error rate to determine the speech synthesis evaluation index.

[0106] Specifically, during actual broadcasting, if the broadcasting error rate is too high, it will lead to the inability to understand the meaning of the text. Therefore, before determining the speech synthesis evaluation index, it is necessary to first determine whether the broadcasting error rate is less than the first error rate threshold. When it is less than the first error rate threshold, it means that this broadcast is effective. After that, determine the broadcasting accuracy rate and use the broadcasting accuracy rate as the final speech synthesis evaluation index.

[0107] In some embodiments, step 200 further includes:

[0108] When the broadcast error rate is greater than the first error rate threshold, a first test failure flag is returned.

[0109] When the broadcast error rate exceeds the first error rate threshold, it means that the meaning of the text cannot be understood, and the result of this test has no practical significance. In this case, the first test failure flag can be directly fed back. At the same time, this test can be ended to avoid wasting computing power in subsequent processes.

[0110] It should be noted that it is generally believed that a broadcast error rate exceeding 15% will make it difficult to understand the text content. Therefore, in some embodiments, the first error rate threshold may be set to 15%.

[0111] In some embodiments, step 200 further includes:

[0112] When the broadcast error rate is less than a preset third error rate threshold, the broadcast accuracy rate is determined to be completely correct, and the speech synthesis evaluation index is determined according to the broadcast accuracy rate.

[0113] When the error rate is less than the third error rate threshold, it can be basically understood as no error for automatic recognition, and the understanding of the text will not be affected at all. Therefore, in the case where the broadcast accuracy rate is less than the third error rate threshold, it can be directly assumed to be completely correct, and the speech synthesis evaluation index represents 100% correctness. It should be noted that it is generally believed that the broadcast error rate is lower than 3% to completely not affect the understanding of the text. Therefore, in some embodiments, the third error rate threshold can be set to 3%.

[0114] In some embodiments, step S300 includes:

[0115] When the word error rate is less than a preset second error rate threshold, the speech recognition evaluation index is determined according to the sentence error rate.

[0116] First of all, it should be noted that the effect of ASR speech recognition technology can be evaluated by word error rate (WER) and sentence error rate (SER). Assuming S is the number of replaced words, D is the number of deleted words, I is the number of inserted words, C is the number of correct words, N is S replaced + D deleted + C correct words, that is, N is the total number of words in the source text, then the calculation formula of WER is:

[0117] WER=(S+D+I) / N=(S+D+I) / (S+D+C),

[0118] It should be noted that N does not include I, so WER may be greater than 100%.

[0119] The calculation formula of SER is:

[0120] SER = number of error sentences / total number of sentences;

[0121] In the formula, the number of error sentences is determined based on the presence or absence of error words. Usually, if there are error words in a sentence, it can be determined that the sentence is an error sentence. Therefore, the total number of error sentences in the entire ASR speech recognition result can be determined based on this method.

[0122] Through the above process, we can know that the impact of the number of wrong words is greater than that of the number of wrong sentences. In speech recognition, if the error rate of the recognized words is too high, the recognition result will be unreadable. Therefore, before determining the speech synthesis evaluation index, it is necessary to first determine whether the sentence error rate is less than the second error rate threshold. When it is less than the second error rate threshold, the sentence accuracy rate will be determined based on the sentence error rate, that is, the speech synthesis evaluation index will be obtained.

[0123] In some embodiments, step S300 further includes:

[0124] When the word error rate is greater than the second error rate threshold, a second test failure flag is returned.

[0125] When the word error rate exceeds the third error rate threshold, it means that the text of the speech recognition is not readable at all. Then the result of this test has no practical significance, and the second test failure flag can be directly fed back. At the same time, this test can be ended to avoid wasting computing power in subsequent processes.

[0126] It should be noted that it is generally believed that a word error rate exceeding 15% will result in poor readability. Therefore, in some embodiments, the second error rate threshold may be set to 15%.

[0127] In some embodiments, step S300 further includes:

[0128] When the sub-error rate is less than a preset fourth error rate threshold, the sentence accuracy is determined to be completely correct, and the speech recognition evaluation index is determined according to the sentence accuracy.

[0129] When the word error rate is less than the fourth error rate threshold, the text recognized by speech will not be affected at all in terms of understanding of the text. Therefore, in the case where the word error rate is less than the fourth error rate threshold, it can be directly assumed to be completely correct. At this time, the speech recognition evaluation index represents 100% correctness. It should be noted that it is generally believed that the broadcast error rate is less than 3% and will not affect the understanding of the text. Therefore, in some embodiments, the fourth error rate threshold can be set to 3%.

[0130] In order to better illustrate the AI ​​outbound call evaluation method of the embodiment of the present application, it is described here in the form of a minimum implementation example.

[0131] It should be noted that for the entire evaluation process, the test is meaningful only after all the test processes in the entire process are completed normally. If an error occurs in the intermediate process or the processing fails, the test will be stopped and the corresponding flag information will be returned. Specifically, when the representative call is not connected, the unconnected flag is returned; when the call is connected but the TTS speech synthesis judgment test fails, the first test failure flag is returned; when the call is connected, the TTS speech synthesis is successful, but the ASR speech recognition test fails, the second test failure flag information is returned; when the call is connected, the TTS speech synthesis is successful and the ASR speech recognition is successful, but the NLP natural language processing recognition fails, the third test failure flag is returned; when the call is connected, the TTS speech synthesis is successful and the ASR speech recognition is successful, and the NLP natural language processing recognition fails, the third test failure flag is returned; when the call is connected, the TTS speech synthesis is successful and the ASR speech recognition is successful, and the NLP recognition is successful, the AI ​​outbound call evaluation method of the embodiment of the present application can be fully executed at this time. Under this premise, the evaluation result obtained is the valid evaluation result. Under the above premise, the AI ​​outbound call evaluation method of the minimum embodiment of the present application is briefly described below.

[0132] Obtain outbound voice data. The outbound voice data is obtained by recording the AI ​​outbound call process. The outbound voice data at least includes synthesized voice broadcast information and target object voice input information;

[0133] Determine the broadcast error rate of the synthesized voice broadcast information, and determine the voice synthesis evaluation index according to the broadcast error rate; when the broadcast error rate is less than 15%, use the broadcast error rate to determine the broadcast accuracy of the synthesized voice broadcast information, and determine the broadcast accuracy rate as the voice synthesis evaluation index; when the broadcast error rate is less than 3%, it does not affect the text understanding at all, and the broadcast accuracy rate can be equated with being completely correct, and the broadcast accuracy rate can be determined as the voice synthesis evaluation index; it should be noted that when the broadcast error rate is greater than 15%, it means that the test has failed, and the first test failure flag can be directly returned;

[0134] Determine the word error rate and sentence error rate of the target object's voice input information, and determine the speech recognition evaluation index based on the word error rate and sentence error rate; when the word error rate is less than 15%, it means that the recognition result can be basically understood, and the sentence accuracy rate is determined based on the sentence error rate, and the accuracy rate can be used as the speech recognition evaluation index; when the word error rate is less than 3%, the recognition result can be completely recognized normally, and the sentence accuracy rate is determined as completely correct, and the sentence accuracy rate is determined as the speech recognition evaluation index; it should be noted that when the word error rate is greater than 15%, it means that the recognition result cannot be understood, and the second test failure flag is directly returned;

[0135] Multiple single models are used to respectively determine the intent classification corpus data of the target object's speech input information, each intent classification corpus data includes multiple model intent classification corpus numbers AX, and the multiple model intent classification corpus numbers AX are obtained by classifying multiple different intent classifications by a single model; the intent classification correct corpus number TX is determined according to the multiple model intent classification corpus numbers AX and the preset multiple correct expected classification numbers PX, based on which the intent classification correct corpus number TX of multiple intent classifications of each single model can be determined, so as to complete the calculation of the single model evaluation index using the total intent classification correct corpus number and the total corpus amount P in the corpus set; it should be noted that when determining the single model evaluation index, the single classification accuracy rate and the single classification correct ratio will also be determined at the same time for subsequent use by R&D personnel; it should also be noted that multiple single models are models used for intent classification in the natural language processing process;

[0136] Determine multiple model combinations based on multiple single models. The choice of model combinations needs to be determined based on actual needs, usually including a combination of emotion recognition model and rule engine model, a combination of emotion recognition model and sentence model and ES recall model, a combination of emotion recognition model and sentence model and TextCNN model, a combination of emotion recognition and sentence model and Fasttext model, etc.; the multi-model evaluation index of each model combination can be calculated based on the Cartesian product principle using the single-model evaluation index of the corresponding single model. Based on the same principle, multiple multi-model evaluation indexes can be determined in turn;

[0137] Determine the corresponding proportion information of the above multiple model combinations, including the rule engine proportion, ES recall statistics proportion, TextCNN statistics proportion, and Fasttext statistics proportion; weight multiple multi-model evaluation indicators according to these proportion information to obtain the language processing evaluation indicator;

[0138] Determine the speech intention evaluation index based on the speech recognition evaluation index and the language processing evaluation index. The level of the speech intention evaluation index represents the closeness to the customer's "real" feelings, and can better reflect the "real" effect of intent recognition.

[0139] Determine the weight factors corresponding to the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index respectively; weight the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index according to the weight factors to obtain a comprehensive index;

[0140] The comprehensive indicators and speech intent evaluation indicators can be used as the final evaluation results.

[0141] See also Figure 5 The embodiment of the present application further provides an AI outbound call evaluation device, which can implement the above-mentioned AI outbound call evaluation method, and the device includes:

[0142] The data acquisition module 911 is used to acquire outbound voice data, which includes synthesized voice broadcast information and target object voice input information;

[0143] A broadcast evaluation module 912, used to determine the broadcast error rate of the synthesized speech broadcast information, and determine the speech synthesis evaluation index according to the broadcast error rate;

[0144] A language recognition module 913, used to determine the word error rate and sentence error rate of the target object's voice input information, and determine the voice recognition evaluation index according to the word error rate and sentence error rate;

[0145] A single model evaluation module 914 is used to use multiple single models to respectively determine the intent classification corpus data of the target object speech input information, and determine the single model evaluation index of each single model according to each intent classification corpus data, and the single model evaluation index is used to characterize the intent classification accuracy of the single model, wherein the multiple single models are different models used for intent classification in the natural language processing process;

[0146] A multi-model evaluation module 915 is used to determine a plurality of model combinations according to a plurality of single models, and to determine a multi-model evaluation index corresponding to the plurality of model combinations according to the plurality of model combinations and the plurality of single model evaluation indexes;

[0147] A natural language processing comprehensive evaluation module 916, for determining a language processing evaluation index based on a plurality of multi-model evaluation indexes;

[0148] Applying a comprehensive evaluation module 917, for determining a comprehensive index based on the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index;

[0149] The evaluation result generation module 918 is used to obtain the evaluation result according to the comprehensive index.

[0150] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned AI outbound call evaluation method. The electronic device can be a computer, a server, and other devices.

[0151] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0152] The processor 921 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0153] The memory 922 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 922 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 922, and the processor 921 calls and executes the AI ​​outbound call evaluation method of the embodiment of this application;

[0154] Input / output interface 923, used to implement information input and output;

[0155] Communication interface 924, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0156] A bus 925 that transmits information between various components of the device (e.g., the processor 921, the memory 922, the input / output interface 923, and the communication interface 924);

[0157] The processor 921 , the memory 922 , the input / output interface 923 and the communication interface 924 are connected to each other in communication within the device via a bus 925 .

[0158] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned AI outbound call evaluation method.

[0159] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The AI ​​outbound call evaluation method, device, electronic device and storage medium proposed in the present application obtain outbound call voice data, which includes synthesized voice broadcast information and target object voice input information; determine the broadcast error rate of the synthesized voice broadcast information, and determine the voice synthesis evaluation index according to the broadcast error rate; determine the word error rate and sentence error rate of the target object voice input information, and determine the voice recognition evaluation index according to the word error rate and sentence error rate; use multiple single models to respectively determine the intent classification corpus data of the target object voice input information, and determine the single model evaluation index of each single model according to each intent classification corpus data, the single model evaluation index is used to characterize the intent classification accuracy of the single model, and the multiple single models are different models used for intent classification in the natural language processing process; determine multiple model combinations according to the multiple single models, and determine multi-model evaluation indicators corresponding to the multiple model combinations according to the multiple model combinations and the multiple single model evaluation indicators; determine the language processing evaluation index according to the multiple multi-model evaluation indicators; determine the comprehensive index according to the voice synthesis evaluation index, the voice recognition evaluation index and the language processing evaluation index; and obtain the evaluation result according to the comprehensive index. This application conducts a separate evaluation of each process of AI outbound calls, and further conducts a comprehensive evaluation of the evaluation indicators obtained for each process, to obtain a comprehensive and systematic evaluation result of the entire AI outbound call process, thereby improving the accuracy of the evaluation and facilitating the subsequent use of the evaluation results to optimize the model and algorithm.

[0161] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0162] It can be understood by those skilled in the art that Figure 1-4 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0163] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0165] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0166] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0171] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An AI outbound call evaluation method, characterized in that: The method comprises: Acquire outbound voice data, wherein the outbound voice data includes synthesized voice broadcast information and target object voice input information; Determine a broadcast error rate of the synthesized speech broadcast information, and determine a speech synthesis evaluation index according to the broadcast error rate; Determine a word error rate and a sentence error rate of the target object's speech input information, and determine a speech recognition evaluation index according to the word error rate and the sentence error rate; Using multiple single models to respectively determine the intent classification corpus data of the target object's speech input information, and determining the single model evaluation index of each single model according to each of the intent classification corpus data, wherein the single model evaluation index is used to characterize the intent classification accuracy of the single model, wherein the multiple single models are different models used for intent classification in the natural language processing process; Determine a plurality of model combinations according to the plurality of single models, and determine a multi-model evaluation index corresponding to the plurality of model combinations one by one according to the plurality of model combinations and the plurality of single model evaluation indexes; Determining a language processing evaluation index according to a plurality of the multi-model evaluation indexes; Determine a comprehensive index based on the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index; An evaluation result is obtained according to the comprehensive index.

2. The AI ​​outbound call evaluation method according to claim 1, characterized in that: The evaluation result obtained according to the comprehensive indicator includes: Determine a speech intention evaluation index according to the speech recognition evaluation index and the language processing evaluation index; An evaluation result is obtained according to the comprehensive index and the speech intention evaluation index.

3. The AI ​​outbound call evaluation method according to claim 1 or 2, characterized in that: The determining of the language processing evaluation index according to the plurality of multi-model evaluation indexes comprises: Determine the proportion information corresponding to each of the plurality of model combinations; The plurality of multi-model evaluation indicators are weighted according to the proportion information to obtain the language processing evaluation indicator.

4. The AI ​​outbound call evaluation method according to claim 1 or 2, characterized in that: The determining of the comprehensive index according to the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index comprises: Determine the weight factors corresponding to the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index respectively; The speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index are weighted according to the weight factor to obtain the comprehensive index.

5. The AI ​​outbound call evaluation method according to claim 1, characterized in that: Each of the intent classification corpus data includes a plurality of model intent classification corpus numbers, and the plurality of model intent classification corpus numbers are obtained by classifying a plurality of different intent classifications by the single model; The single model evaluation index corresponding to each single model is obtained by the following steps: Determine the number of correct corpora for intent classification according to the plurality of said model intent classification corpora and the plurality of preset correct expected classification numbers, wherein the plurality of said correct expected classification numbers correspond one to one with the plurality of said model intent classification corpora; The accuracy of intent classification is determined according to the number of correct corpora for intent classification, and a single model evaluation index is obtained.

6. The AI ​​outbound call evaluation method according to claim 1, characterized in that: The determining of the speech synthesis evaluation index according to the broadcast error rate includes: When the broadcast error rate is less than a preset first error rate threshold, determining the broadcast accuracy rate of the synthesized voice broadcast information; The speech synthesis evaluation index is determined according to the broadcast accuracy rate.

7. The AI ​​outbound call evaluation method according to claim 1, characterized in that: The step of determining the speech recognition evaluation index according to the word error rate and the sentence error rate includes: When the word error rate is less than a preset second error rate threshold, a speech recognition evaluation index is determined according to the sentence error rate.

8. An AI outbound call evaluation device, characterized in that: The device comprises: A data acquisition module, used to acquire outbound voice data, wherein the outbound voice data includes synthesized voice broadcast information and target object voice input information; A broadcast evaluation module, used to determine a broadcast error rate of the synthesized speech broadcast information, and determine a speech synthesis evaluation index according to the broadcast error rate; A language recognition module, used to determine a word error rate and a sentence error rate of the target object's voice input information, and determine a voice recognition evaluation index according to the word error rate and the sentence error rate; A single model evaluation module, used to use multiple single models to respectively determine the intent classification corpus data of the target object speech input information, and determine the single model evaluation index of each single model according to each of the intent classification corpus data, wherein the single model evaluation index is used to characterize the intent classification accuracy of the single model, wherein the multiple single models are different models used for intent classification in the natural language processing process; A multi-model evaluation module, used to determine a plurality of model combinations according to the plurality of single models, and to determine a multi-model evaluation index corresponding to the plurality of model combinations one by one according to the plurality of model combinations and the plurality of single model evaluation indexes; A natural language processing comprehensive evaluation module, used to determine a language processing evaluation index based on a plurality of said multi-model evaluation indicators; Applying a comprehensive evaluation module, for determining a comprehensive index according to the speech synthesis evaluation index, the speech recognition evaluation index and the language processing evaluation index; The evaluation result generating module is used to obtain the evaluation result according to the comprehensive index.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the AI ​​outbound call evaluation method as described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the AI ​​outbound call evaluation method as described in any one of claims 1 to 7.

Images