Method for Evaluating Service Effect of Intent Recognition Based on Natural Language Processing (NLP)

Abstract

The present application relates to the field of artificial intelligence technology, and provides a method for evaluating the effect of intent recognition service based on natural language processing (NLP), including: obtaining multiple customer corpora within a preset time interval; dividing multiple customer corpora according to preset dimensional rules to obtain customer corpora of multiple different dimensions; for each customer corpus corresponding to each dimension, calculating the hit mode proportion distribution data according to the customer corpus; constructing a corpus test set corresponding to each dimension according to the hit mode proportion distribution data corresponding to each dimension; using the corpus test sets corresponding to different dimensions to test the intent recognition model, and obtaining the intent recognition accuracy of the intent recognition model in different dimensions. The intent recognition effect in different dimensions can be obtained through multi-dimensional refined classification, and the intent recognition model can be adjusted in a targeted manner to ensure the working effect of the intent recognition model, which solves the disadvantage of distorted intent recognition effect due to excessive randomness of the corpus.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for evaluating the effectiveness of intent recognition services based on natural language processing (NLP). Background Art

[0002] At present, artificial intelligence (AI) models in the field of multi-intention recognition are increasingly widely used in the real world. In order to achieve better recognition and prediction effects, engineers often combine and use multiple AI models with different implementation principles to complete extremely difficult recognition and prediction tasks. Although the research on a single AI model is relatively complete nowadays, there is still no good evaluation scheme and evaluation index for the overall effect of AI applications after the combination of multiple models, which can comprehensively, fairly and objectively reflect its recognition and prediction effects. It is particularly important that the uneven distribution of customer corpora in the test sample set will directly affect the effect evaluation results of the entire AI application. One of the typical problems is the significant impact of the different distribution ratios of customer corpora on each model on the evaluation effect. In other words, if the corpus of the test sample set is not well selected, the final effect evaluation results will be significantly distorted, which is far from the real feelings of humans. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method for evaluating the effectiveness of intent recognition services based on natural language processing (NLP), which can realize the overall effectiveness evaluation of AI applications after combining multiple models to solve the problem of distorted evaluation results.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a method for evaluating the effect of an intent recognition service based on natural language processing (NLP), the method comprising:

[0005] Obtain multiple customer corpora within a preset time interval;

[0006] Dividing the plurality of customer corpora according to a preset dimension rule to obtain a plurality of customer corpora sets of different dimensions;

[0007] For the customer corpus corresponding to each dimension, hit method proportion distribution data is calculated according to the customer corpus, and the hit method proportion distribution data indicates the proportion distribution of the number of customer corpora hit by the multiple hit methods included in the preset intent recognition model;

[0008] According to the hit ratio distribution data corresponding to each dimension, build the corpus test set corresponding to each dimension;

[0009] Testing the intent recognition model using the corpus test sets corresponding to different dimensions to obtain the intent recognition accuracy rates of the intent recognition model under different dimensions.

[0010] In some embodiments, the dividing the multiple customer corpuses according to a preset dimension rule to obtain multiple customer corpus sets of different dimensions includes at least one of the following:

[0011] Classifying the customer corpuses according to the full time period to obtain a customer corpus set corresponding to the full time period;

[0012] Classifying the customer corpuses according to a plurality of preset specific time periods to obtain customer corpus sets corresponding to different specific time periods;

[0013] Classifying the customer corpuses according to a plurality of preset specific days of each month to obtain customer corpus sets corresponding to different specific days of each month;

[0014] Classifying the customer corpuses according to a plurality of preset specific days of each week to obtain customer corpus sets corresponding to different specific days of each week;

[0015] Classifying the customer corpuses according to a plurality of preset business types to obtain customer corpus sets corresponding to different business types;

[0016] Classifying the customer corpuses according to a plurality of business scenarios of a specified business type to obtain customer corpus sets corresponding to different business scenarios of the specified business type;

[0017] Classifying the customer corpuses according to a plurality of business departments and a plurality of business scenarios of a specified business type to obtain customer corpus sets corresponding to different business departments and different business scenarios of the specified business type.

[0018] In some embodiments, the calculating the proportion distribution data of the hit methods according to the customer corpus set includes:

[0019] Inputting each customer corpus included in the customer corpus set into the intent recognition model respectively to determine the hit method corresponding to each customer corpus;

[0020] Determining the proportion distribution data of the hit methods according to the hit methods corresponding to each customer corpus included in the customer corpus set; wherein, the multiple hit methods include: rule engine hit, ES recall hit, TextCNN model hit, Fasttext model hit.

[0021] In some embodiments, characterized in that the determining process of the hit method corresponding to the customer corpus includes the following steps:

[0022] Regularly match the customer corpus through the rule engine of the intention recognition model. When the regular match is successful, determine that the hit method corresponding to the customer corpus is rule engine hit;

[0023] When the regular match fails, traverse the ES corpus of the customer corpus through the ES recall model of the intention recognition model and perform relevance calculation and analysis. When the ES corpus traversal and relevance calculation and analysis are successful, determine that the hit method corresponding to the customer corpus is ES recall hit;

[0024] When the ES corpus traversal and relevance calculation and analysis fail, perform the first fast text analysis on the customer corpus through the TextCNN model of the intention recognition model. When the first fast text analysis is successful, determine that the hit method corresponding to the customer corpus is TextCNN model hit;

[0025] When the first fast text analysis fails, perform the second fast text analysis on the customer corpus through the Fasttext model of the intention recognition model. When the second fast text analysis is successful, determine that the hit method corresponding to the customer corpus is Fasttext model hit.

[0026] In some embodiments, constructing a corpus test set corresponding to each dimension according to the hit method proportion distribution data corresponding to each dimension includes at least one of the following:

[0027] Construct a corresponding corpus test set for the whole time period according to the hit method proportion distribution data determined by the customer corpus set of the whole time period;

[0028] Construct a corresponding corpus test set for different specific time periods according to the hit method proportion distribution data determined by the customer corpus sets of different specific time periods;

[0029] Construct a corresponding corpus test set for different specific days of each month according to the hit method proportion distribution data determined by the customer corpus sets of different specific days of each month;

[0030] Construct a corresponding corpus test set for different specific days of each week according to the hit method proportion distribution data determined by the customer corpus sets of different specific days of each week;

[0031] Construct a corresponding corpus test set for different business types according to the hit method proportion distribution data determined by the customer corpus sets of different business types;

[0032] Construct a corresponding corpus test set for different business scenarios of the specified business type according to the hit method proportion distribution data determined by the customer corpus sets of different business scenarios of the specified business type;

[0033] Construct a corresponding corpus test set for different business departments and different business scenarios of the specified business type according to the hit mode proportion distribution data determined by the customer corpus sets of different business departments and different business scenarios of the specified business type.

[0034] In some embodiments, the corpus test set includes a plurality of customer corpus samples and an intent label corresponding to each customer corpus sample; testing the intent recognition model using the corpus test sets corresponding to different dimensions to obtain the intent recognition accuracy of the intent recognition model under different dimensions includes:

[0035] Traverse the corpus test sets corresponding to each dimension, and for the corpus test set corresponding to the currently traversed dimension, perform the following steps:

[0036] Input each customer corpus sample included in the corpus test set into the intent recognition model to obtain an intent recognition result corresponding to each customer corpus sample;

[0037] Determine the intent recognition accuracy of the intent recognition model under the currently traversed dimension according to the intent recognition results and intent labels corresponding to each customer corpus sample.

[0038] In some embodiments, after obtaining the intent recognition accuracy of the intent recognition model under different dimensions, the method further includes:

[0039] Determine the application scenario of the intent recognition model according to the intent recognition accuracy of the intent recognition model under different dimensions.

[0040] To achieve the above object, a second aspect of the embodiments of the present application proposes an evaluation device for the service effect of intent recognition based on natural language processing (NLP), which is characterized by including:

[0041] A corpus acquisition module, configured to acquire a plurality of customer corpora within a preset time interval;

[0042] A multi-dimensional corpus acquisition module, configured to divide the plurality of customer corpora according to a preset dimension rule to obtain a plurality of customer corpus sets with different dimensions;

[0043] A corpus distribution acquisition module, configured to calculate hit mode proportion distribution data for each customer corpus set corresponding to a dimension, where the hit mode proportion distribution data indicates the proportion distribution of the number of customer corpora hit by a plurality of hit modes included in a preset intent recognition model;

[0044] A test set construction module, configured to construct a corpus test set corresponding to each dimension according to the hit mode proportion distribution data corresponding to each dimension;

[0045] The recognition effect acquisition module is used to test the intention recognition model using the corpus test set corresponding to different dimensions to obtain the intention recognition accuracy of the intention recognition model in different dimensions.

[0046] To achieve the above objective, a third aspect of an embodiment of the present application provides a computer device, including:

[0047] at least one memory;

[0048] at least one processor;

[0049] at least one computer program;

[0050] The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement:

[0051] A method for evaluating the effectiveness of intent recognition services based on natural language processing (NLP) as described in the first aspect above.

[0052] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method for evaluating the effect of intention recognition service based on natural language processing (NLP) described in the first aspect above.

[0053] The embodiment of the present application obtains customer corpora of different dimensions according to multiple dimensional rules determined by actual working conditions; by determining the hit mode results of all customer corpora in each customer corpus in the intent recognition model, the hit mode proportion distribution data of the customer corpus under different dimensional rules is obtained; according to the hit mode proportion distribution data, the corresponding corpus test set is prepared and the intent recognition model test is called to obtain the intent recognition accuracy of the intent recognition model in different dimensions to reflect the effect of intent recognition. Therefore, for the effect evaluation method of the embodiment of the present invention, the recognition effect of the NLP-based intent recognition service in different situations or scenarios is obtained by multi-dimensional refinement classification of the actual working conditions, and the version of the intent recognition service in different dimensions can be adjusted in a targeted manner later to better ensure the working effect of the NLP-based intent recognition service in a specific dimension, thereby solving the shortcoming that the intention recognition effect is too distorted due to the excessive randomness of the received customer corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of a method for evaluating the effect of an intent recognition service based on natural language processing (NLP) provided in an embodiment of the present application;

[0055] Figure 2 It is the flowchart of the operation of the intent recognition model provided by the embodiments of the present application;

[0056] Figure 3 It is the flowchart of the operation of the Elasticsearch (ES) recall model provided by the embodiments of the present application;

[0057] Figure 4 It is Figure 1 the flowchart of the specific steps included in step S120 in

[0058] Figure 5 It is Figure 1 the flowchart of the specific steps included in step S130 in

[0059] Figure 6 It is Figure 5 the flowchart of the specific steps included in step S310 in

[0060] Figure 7 It is Figure 1 the flowchart of the specific steps included in step S140 in

[0061] Figure 8 It is Figure 1 the flowchart of the specific steps included in step S150 in

[0062] Figure 9 It is the flowchart of the method for evaluating the effect of the intent recognition service based on natural language processing (NLP) that further includes step S160;

[0063] Figure 10 It is the schematic structural diagram of the device for evaluating the effect of the intent recognition service based on natural language processing (NLP) provided by the embodiments of the present application;

[0064] Figure 11 It is the schematic hardware structure diagram of the computer device provided by the embodiments of the present application. Detailed implementation manners

[0065] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] It should be noted that although functional module division is performed in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different module division in the device or a different order in the flowchart. Terms such as "first" and "second" in the specification, claims and the above accompanying drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are for the purpose of describing embodiments of this application only and are not intended to limit this application.

[0068] First, some terms involved in this application are explained as follows:

[0069] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence; artificial intelligence is a branch of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. The research in this field includes robots, speech recognition, image recognition, natural language processing, and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence also refers to the theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0070] Natural Language Processing (NLP): It refers to the technology of using the natural language used by humans for communication to interact with machines. Through artificial processing of natural language, the computer can read and understand it. The related research on natural language processing began with the exploration of machine translation by humans. Although natural language processing involves multi-dimensional operations such as speech, grammar, semantics, and pragmatics, simply speaking, the basic task of natural language processing is to segment the corpus to be processed based on methods such as ontology dictionaries, word frequency statistics, and context semantic analysis, forming word item units with the smallest part of speech as the unit and rich in semantics.

[0071] Intent recognition service based on natural language processing NLP: It has the ability to recognize intents, can recognize user intents, and is mainly applied in intelligent question-and-answer customer service systems. By recognizing the intent of the user's consulting questions, it matches the corresponding answers for the user to view. Intent recognition is essentially a classification problem, classifying the intent represented by the user's expression, that is, dividing the user's question into one of the classifications.

[0072] Intent recognition model: Combining multiple AI algorithm models to complement each other to jointly complete the intent recognition of customers, maximizing the guarantee of the effect of intent recognition. For example, an intent recognition model includes: a rule engine model, an Elasticsearch (ES) recall model, a TextCNN model, and a Fasttext model.

[0073] The intention recognition model will be described in detail below. Refer to Figure 2 For the specific workflow of the intention recognition model, in the entire workflow of the NLP-based intention recognition model, step S001: the rule engine model will be used first. By judging whether the result is matched in step S002 through regular matching, the first round of intention recognition is realized, that is, step S003: obtain the first matched intention. The recognition accuracy of this technology has reached 100% at present. However, the matching rule set of the rule engine model is a finite set. That is to say, no matter how the matching rule set is expanded, it must have a clear boundary. However, the richness and diversity of human language determine that the answers of customers in the work scenario are highly random, that is, the combination of customer corpora must be an infinite set. Therefore, the rule engine model cannot fully cover all customer answer scenarios. Therefore, next, the Elasticsearch (ES) recall model will be used as a supplement to the rule engine model to further identify the intention of customer corpora outside the corpus of the rule engine model.

[0074] Step S004: The ES recall model is completed based on the search and relevance calculation of the customer corpus. The specific process is as follows. Refer to Figure 3 As shown. The ES recall model first executes step S041: use the Elasticsearch engine to traverse all the corpora in its corpus, calculate the first relevance score for all the corpora according to the BM25 model, commonly known as the rough ranking score, and sort them to obtain the top 100 corpora ranked by the first relevance score, that is, step S042: find the top 100 corpora in the corpus; then, according to step S051: the ESIM model (Enhanced Sequential Inference Model) calculates a more refined and complex second relevance score for the top 100 corpora, that is, step S005, commonly known as the fine ranking score, and sorts them to obtain the top 5 corpora ranked by the second relevance score, that is, step S052: find the top 5 corpora in the corpus. In actual application, a threshold range is usually set according to experience. If the corpus ranked first among the top 5 corpora meets step S006: the fine ranking score of the top 1 corpus meets the threshold range, it is considered that the intention corresponding to this corpus is the true intention of the customer, that is, step S053 is completed: map the corpus in the library to the intention, obtain the intention represented by the top 5 corpora, that is, step S007, and return the output of the top 5 corpus intentions, that is, step S054. However, the corpus of the ES model is also a finite set and cannot guarantee to fully cover all customer answer scenarios, that is, the situation where the fine ranking scores of some top 1 corpora do not meet the threshold range.

[0075] Continue to refer to Figure 2 , as a supplement to the ES recall model,Figure 2 The next two models, step S008: TextCNN model and step S010: Fasttext model, mainly use fast text analysis technology to calculate the probability that the customer corpus belongs to each intent classification, and determine the customer's true intent based on the probability. Currently, dozens of intent classifications can be set in the AI ​​outbound call application system. With the continuous and refined development of the business, the number of intent classifications will continue to grow. It should be noted that the accuracy of the Fasttext model will decay with the increase of intent classifications, which is particularly obvious. Relatively speaking, the TextCNN model is based on the convolutional neural network (CNN) deep learning algorithm, and is significantly better than the Fasttext model in terms of decay performance. Therefore, when the rule engine model and the ES recall model cannot identify the intent of the customer corpus, the TextCNN model is usually used first to determine whether the intent is successfully returned under this model, that is, step S009, and then the Fasttext model is used as a backup. After being processed by the Fasttext model, the similarity score of the Top1 corpus is compared with the threshold, i.e., step S006. If the threshold requirement is met, a descending result of the intent type that meets the requirement is obtained, i.e., step S011. If the threshold requirement is not met, the entire process fails to identify the intent represented by the customer corpus, i.e., step S012.

[0076] At present, with the development of artificial intelligence, more and more industries, such as the financial industry, such as the insurance field, choose to use artificial intelligence outbound call application systems to gain more economic benefits. For traditional outbound call systems, business personnel are required to answer the phone, and the main body of work is manpower. The AI ​​outbound call application system adopts a human-machine cooperation model, and most of the time AI completes the work. It is responsible for the initial screening of potential customers, and business personnel are responsible for subsequent follow-up customers. With the AI ​​outbound call application system, the daily call volume is several times that of the traditional outbound call system, and industry terms can be set to intelligently communicate with customers, judge customer intentions, and accurately screen out potential customers. The AI ​​outbound call application system can replace the previous customer development work, reduce manual pressure, and better serve customers; it can also answer questions raised by customers in a targeted manner, so that customers can get the answers they want in a timely manner.

[0077] By applying the currently adopted intent recognition model to the AI outbound call application system, the intent recognition of customer conversations can be achieved. However, the current intent recognition model lacks a complete and systematic effect evaluation scheme, and there are the following defects when the intent recognition model is actually called: Since the strength of the single model in terms of intent recognition accuracy is: rule engine model > ES recall model > TextCNN model > Fasttext model, if some test corpora are simply prepared to call the intent recognition model, then all these corpora may be completed by the rule engine model for intent recognition, that is, the intent recognition accuracy reaches 100%, or they may all be completed by the Fasttext model for intent recognition, that is, the intent recognition accuracy is only 60%. That is to say, if a large number of corpora in the customer corpus are hit by the rule engine model or the ES recall model, and very few corpora are hit by the Fasttext model, then a falsely high accuracy result will be obtained; conversely, a falsely low accuracy result will be obtained. However, due to the relative complexity of the actual work, a large number of customers and business personnel do not feel this way. Therefore, there is such a distortion phenomenon in the intent recognition model in reality.

[0078] Based on this, the embodiments of the present application provide a method, device, equipment and medium for evaluating the effect of intent recognition service based on natural language processing NLP, which can realize the scheme evaluation of the overall effect of the intent recognition model after combining multiple AI algorithm models, so as to solve the problem of distorted evaluation results.

[0079] The method, device, equipment and medium for evaluating the effect of intent recognition service based on natural language processing NLP provided by the embodiments of the present application are specifically described through the following embodiments. First, a method for evaluating the effect of intent recognition service based on natural language processing NLP in the embodiments of the present application is described.

[0080] The embodiments of the present application can obtain and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results of theory, method, technology and application system.

[0081] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0082] An intention recognition service effect evaluation method based on natural language processing (NLP) provided by an embodiment of the present application relates to the field of artificial intelligence technology. The intention recognition service effect evaluation method based on natural language processing (NLP) provided by an embodiment of the present application can be applied to a terminal, or to a server side, or can be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application for implementing an information extraction method, etc., but is not limited to the above forms.

[0083] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet-type devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0084] Please refer to Figure 1 , Figure 1 which is an optional flowchart of the intention recognition service effect evaluation method based on natural language processing (NLP) provided by an embodiment of the present application. Figure 1 The method in [[ ]] can specifically include but is not limited to steps S110 to S150.

[0085] Step S110: Obtain multiple customer corpora within a preset time interval;

[0086] Step S120: Divide the multiple customer corpora according to a preset dimension rule to obtain multiple customer corpus sets of different dimensions;

[0087] Step S130: for each customer corpus corresponding to each dimension, hit method proportion distribution data is calculated according to the customer corpus, where the hit method proportion distribution data indicates the proportion distribution of the number of customer corpora hit by the multiple hit methods included in the preset intent recognition model;

[0088] Step S140: constructing a corpus test set corresponding to each dimension according to the hit mode proportion distribution data corresponding to each dimension;

[0089] Step S150: Test the intent recognition model using corpus test sets corresponding to different dimensions to obtain the intent recognition accuracy of the intent recognition model in different dimensions.

[0090] In steps S110 to S150 of some embodiments, customer corpora of different dimensions are obtained according to multiple dimensional rules determined by the actual working situation; by determining the hit mode results of all customer corpora in each customer corpus in the intent recognition model, the hit mode proportion distribution data of the customer corpus under different dimensional rules are obtained; according to the hit mode proportion distribution data, the corresponding corpus test set is prepared and the intent recognition model test is called to obtain the intent recognition accuracy of the intent recognition model in different dimensions to reflect the effect of intent recognition. Therefore, for the effect evaluation method of the embodiment of the present invention, the recognition effect of the NLP-based intent recognition service under different situations or scenarios is obtained by multi-dimensional refinement classification of the actual working conditions, and the version of the intent recognition service under different dimensions can be adjusted in a targeted manner later to better ensure the working effect of the NLP-based intent recognition service under specific dimensions, thereby solving the shortcoming that the intention recognition effect is too distorted due to the excessive randomness of the received customer corpus.

[0091] In step S110 of some embodiments, the preset time interval is generally a relatively long period of time, which can be set to one year or several years. By collecting customer corpus over a long period of time, the comprehensiveness of the customer corpus content can be ensured as much as possible, so that the expression scenarios reflected in the customer corpus cover a wide range, which is conducive to the subsequent intention recognition effect to achieve the expected effect.

[0092] It should be noted that customer corpus may include text records, voice records, and question records of conversations with customers in various business scenarios. It may also be pictures and video information transmitted during the communication process, or suggestions and feedback provided by customers through autonomous channels.

[0093] In step S120 of some embodiments, since all customer corpora in the preset time interval are complex in content and reflect various scenarios, it is necessary to divide them into multiple dimensional rules preset in the preset time interval to specifically fit various situations in actual work so as to facilitate specific analysis of specific situations.

[0094] It can be understood that the preset dimension rules may include time dimension rules and space dimension rules. Specifically, the time dimension may include various common working time periods, or some important time nodes such as every year, month, week, and day; the space dimension may include various business types, various working scenarios, and the working departments for each business process, etc.

[0095] In step S130 of some embodiments, in the intention recognition model, for a large number of different customer corpora, the hit mode results under multiple AI algorithm models are determined and distributed differently. Therefore, for several customer corpora under different dimensions, it can be determined under which model in the intention recognition model these customer corpora are hit, so as to obtain the proportion distribution of the customer corpus hit mode, which is specifically reflected by the proportion distribution data of the hit mode.

[0096] In step S140 of some embodiments, according to the obtained proportion distribution data of the hit mode, multiple corpus test sets of corresponding different dimensions are prepared to make test preparations for subsequent simulation of the actual interaction scenario with customers.

[0097] In step S150 of some embodiments, the intention recognition model is called to test multiple corpus test sets. According to the test results, the test effects of the intention recognition model under different dimension types can be determined. From a quantitative perspective, the test effects are reflected by the intention recognition accuracy.

[0098] It should be noted that the method for evaluating the service effect of intention recognition based on natural language processing (NLP) in the embodiments of the present application can be applied to different fields. For example, in the insurance field of the financial industry, when business personnel need to communicate with customers non-face-to-face, the intention recognition model is used to complete the intention recognition of customer words. By using the method for evaluating the service effect of intention recognition based on natural language processing (NLP), a more refined usage arrangement can be made for the intention recognition model. And the method in the embodiments of the present application can also be applied to other related application scenarios.

[0099] It should be noted that in each specific embodiment of the present application, when it comes to obtaining chat text records, voice records, question records of the other party during a call with a customer in various business scenarios, or picture and video information transmitted during the communication process, or suggestions and feedback provided by the customer through independent channels, the customer's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiments of the present application need to obtain sensitive personal information of the customer, the customer's separate permission or separate consent will be obtained through methods such as pop-up windows or jumping to a confirmation page. After clearly obtaining the customer's separate permission or separate consent, the necessary customer-related data for the normal operation of the embodiments of the present application will be obtained.

[0100] Please refer to Figure 4 , Figure 4 which is a flowchart of the specific method included in step S120 of some embodiments of the present application. In some embodiments of the present application, step S120 includes at least one of steps S210 to S270. The following will introduce these seven steps in detail in conjunction with Figure 4 this.

[0101] Step S210: Classify the customer corpus according to the whole time period to obtain a customer corpus set corresponding to the whole time period;

[0102] Step S220: Classify the customer corpus according to a plurality of preset specific time periods to obtain customer corpus sets corresponding to different specific time periods;

[0103] Step S230: Classify the customer corpus according to a plurality of preset specific days of each month to obtain customer corpus sets corresponding to different specific days of each month;

[0104] Step S240: Classify the customer corpus according to a plurality of preset specific days of each week to obtain customer corpus sets corresponding to different specific days of each week;

[0105] Step S250: Classify the customer corpus according to a plurality of preset business types to obtain customer corpus sets corresponding to different business types;

[0106] Step S260: Classify the customer corpus according to a plurality of business scenarios of a specified business type to obtain customer corpus sets corresponding to different business scenarios of the specified business type;

[0107] Step S270: Classify the customer corpus according to a plurality of business departments and a plurality of business scenarios of a specified business type to obtain customer corpus sets corresponding to different business departments and different business scenarios of the specified business type.

[0108] It can be understood that for the category of dividing the entire time period according to rules in the time dimension, it is for the subsequent overall analysis of all customer corpora in the preset time interval; for a specific time period, specifically, a day can be divided into the following 7-hour intervals: [(0,6), (6,9), (9,12), (12,14), (14,18), (18,20), (20,24)]. These intervals can be understood as dividing a day into working and resting time periods in the morning, afternoon, and evening according to people's work and rest characteristics; for a specific day of each month, specifically, it can be set as one day in the first ten days, the middle ten days, or the last ten days of each month, or it can be set as the 1st, 5th, and 10th days in the first ten days of each month. The set nodal days can be adjusted according to the specific business content; for a specific day of each week, specifically, it can be Sunday. That is, the customer corpora on the rest day are also important in some work operations, so certain analysis needs to be carried out subsequently.

[0109] It can be understood that the rules in the space dimension can include various business types, various work scenarios, and the work departments for each business process. For the financial industry, various customer operations include interview invitations, premium collection reminders, business follow-ups, winning notifications, etc. For these customer operations, relevant professional departments will be configured to follow up on the business. During the process of handling customer operations, the professional staff in each department will communicate and interact with customers in various scenarios.

[0110] Specifically, taking the premium collection reminder business as an example, it includes three scenarios: before premium receivable, during premium receivable, and policy reinstatement reminder. Among them, before premium receivable means that it has not entered the policy renewal period, during premium receivable means that it is in the policy renewal period, and the policy reinstatement reminder means urging users who have passed the policy renewal period to pay the premium to restore the policy effectiveness. In this case, a certain fine or rate may need to be paid, and there is also a time limit, such as 15 days.

[0111] Furthermore, in the time dimension of the preset time interval, the specific situation of the premium collection reminder business is as follows: Suppose the renewal period of a certain policy is from May 25th to May 31st, 2022, and the validity period of the reinstatement reminder is 15 days. Then, on May 24th and before, it is in the scenario of before premium receivable, from May 25th to May 31st, it is in the scenario of during premium receivable, from June 1st to 15th, it is in the scenario of policy reinstatement reminder, and after June 16th, the policy will expire and cannot be restored.

[0112] It should be noted that for the time - dimension rules and space - dimension rules in the embodiments of the present invention, including but not limited to the full time period, specific time periods, specific days of each month, specific days of each week in the embodiments of the present invention, as well as multiple business types, multiple business departments, and multiple business scenarios. With the development of related technologies, the time - dimension rules and space - dimension rules can be extended to more types and applied to the method for evaluating the effect of intent recognition service based on natural language processing (NLP) in the embodiments of the present invention.

[0113] Please refer to Figure 5 , Figure 5 which is a flowchart of the specific method included in step S130 of some embodiments of this application. In some embodiments of this application, step S130 specifically includes but is not limited to step S310 and step S320. The following will introduce these two steps in detail with reference to Figure 5 .

[0114] Step S310: Input each customer corpus included in the customer corpus set into the intent recognition model respectively to determine the hit method corresponding to each customer corpus.

[0115] Step S320: Determine the proportion distribution data of the hit methods according to the hit methods corresponding to each customer corpus included in the customer corpus set; among them, multiple hit methods include: rule - engine hit, ES recall hit, TextCNN model hit, Fasttext model hit.

[0116] In steps S310 and S320 of some embodiments, according to the combined work of each AI algorithm model in the intent recognition model, the AI algorithm model hit by different customer corpora can be determined, that is, the hit method can be determined. According to the hit methods of each customer corpus in the customer corpus set of different dimensions, the proportion distribution of the hit methods can be obtained, that is, the proportion distribution data of the hit methods can be obtained. Specifically, the process of the combined work of each AI algorithm model can refer to Figure 2 and Figure 3 , and in combination with Figure 2 and Figure 3 the relevant processes have been described above, so they will not be elaborated here.

[0117] It should be noted that the present invention only elaborates on a currently adoptable implementation manner, that is, the intent recognition model includes a rule - engine model, an elastic search (ES) recall model, a TextCNN model, and a Fasttext model, and the hit methods corresponding to its customer corpora are rule - engine hit, ES recall hit, TextCNN model hit, and Fasttext model hit respectively. For combining more AI algorithm models into the intent recognition model, the method for evaluating the effect of intent recognition service based on natural language processing (NLP) in the embodiments of the present invention is still applicable.

[0118] Please refer to Figure 6 , Figure 6 which is a flowchart of the specific method included in step S310 of some embodiments of the present application. In some embodiments of the present application, step S310 includes at least one of steps S311 to S314. The following will introduce these four steps in detail in combination with Figure 6 .

[0119] Step S311: Perform regular matching on the customer corpus through the rule engine of the intent recognition model. When the regular matching is successful, determine that the hit method corresponding to the customer corpus is rule engine hit;

[0120] Step S312: When the regular matching fails, perform ES corpus traversal and relevance calculation analysis on the customer corpus through the ES recall model of the intent recognition model. When the ES corpus traversal and relevance calculation analysis are successful, determine that the hit method corresponding to the customer corpus is ES recall hit;

[0121] Step S313: When the ES corpus traversal and relevance calculation analysis fail, perform the first fast text analysis on the customer corpus through the TextCNN model of the intent recognition model. When the first fast text analysis is successful, determine that the hit method corresponding to the customer corpus is TextCNN model hit;

[0122] Step S314: When the first fast text analysis fails, perform the second fast text analysis on the customer corpus through the Fasttext model of the intent recognition model. When the second fast text analysis is successful, determine that the hit method corresponding to the customer corpus is Fasttext model hit.

[0123] Combined with reference to Figure 2 and Figure 3 , in an intent recognition model, the call priority rule of its AI algorithm model is to call the rule engine model, call the elastic search ES recall model, call the TextCNN model, and call the Fasttext model. Therefore, for a certain customer corpus, the rule engine model will be called first for regular matching. If the regular matching is successful, it is determined that the hit method of the customer corpus is rule engine hit. If the regular matching is not successful, the next AI algorithm model will be called according to the call priority until the hit method of the customer corpus is confirmed. Therefore, for each customer corpus in a certain dimension customer corpus set, after determining the corresponding hit method, the proportion distribution of the hit methods can be obtained, which can provide a basis for preparing the corpus test set under this dimension later.

[0124] Please refer to Figure 7 , Figure 7It is a flowchart of the specific method included in step S140 of some embodiments of the present application. In some embodiments of the present application, step S140 includes at least one of steps S410 to S470. The following will describe these seven steps in detail in combination with Figure 7 to introduce these seven steps in detail.

[0125] Step S410: Construct a corresponding full-time corpus test set according to the hit mode proportion distribution data determined from the customer corpus set of the full time period;

[0126] Step S420: Construct a corresponding corpus test set for different specific time periods according to the hit mode proportion distribution data determined from the customer corpus sets of different specific time periods;

[0127] Step S430: Construct a corresponding corpus test set for different specific days of each month according to the hit mode proportion distribution data determined from the customer corpus sets of different specific days of each month;

[0128] Step S440: Construct a corresponding corpus test set for different specific days of each week according to the hit mode proportion distribution data determined from the customer corpus sets of different specific days of each week;

[0129] Step S450: Construct a corresponding corpus test set for different business types according to the hit mode proportion distribution data determined from the customer corpus sets of different business types;

[0130] Step S460: Construct a corresponding corpus test set for different business scenarios of the specified business type according to the hit mode proportion distribution data determined from the customer corpus sets of different business scenarios of the specified business type;

[0131] Step S470: Construct a corresponding corpus test set for different business departments and different business scenarios of the specified business type according to the hit mode proportion distribution data determined from the customer corpus sets of different business departments and different business scenarios of the specified business type.

[0132] In steps S410 to S470 of some embodiments, by inputting each customer corpus of the customer corpus sets under different dimensions into the intent recognition model, the AI algorithm model hit by the customer corpus is determined, thereby determining the hit mode. Therefore, the corresponding hit mode proportion distribution data can be obtained for the customer corpus sets under different dimensions. From the hit mode proportion distribution data, the hit proportion distribution of the customer corpus in each AI algorithm model can be analyzed. Based on this, corpus test sets under different dimensions are constructed, providing preparation conditions for the subsequent testing of the intent recognition model.

[0133] Please refer to Figure 8 , Figure 8It is a flowchart of the specific method included in step S150 of some embodiments of the present application. In some embodiments of the present application, step S150 at least includes step S510 and steps S511 and S512 included in step S510. The following will combine Figure 7 to introduce these three steps in detail.

[0134] In step S150 of some embodiments, the corpus test set includes multiple customer corpus samples and the intent labels corresponding to each customer corpus sample;

[0135] Step S510: Traverse the corpus test set corresponding to each dimension. For the corpus test set corresponding to the currently traversed dimension, execute step S511 and step S512;

[0136] Step S511: Input each customer corpus sample included in the corpus test set into the intent recognition model respectively to obtain the intent recognition results corresponding to each customer corpus sample;

[0137] Step S512: Determine the intent recognition accuracy rate of the intent recognition model under the currently traversed dimension according to the intent recognition results corresponding to each customer corpus sample and the intent labels.

[0138] In steps S510, S511, and S512 of some embodiments, it can be understood that for a specific intent, it can be expressed by multiple expression methods. Therefore, for each customer corpus sample, there is a corresponding objective intent. Therefore, the intent label is specifically used to complete the link corresponding to each customer corpus sample. The corpus test set in one dimension is called to test the intent recognition model, and the intent recognition model will output the intent recognition results. According to the intent recognition results obtained by using AI technology, a comparative analysis is carried out with the objective intent represented by the intent label, so as to obtain the intent recognition accuracy rate in this dimension, so as to reflect the recognition effect of the intent recognition model in this dimension.

[0139] It can be understood that the intent recognition effect is reflected by the intent recognition accuracy rate. Different intent recognition accuracy rates reflect the performance of the intent recognition model in different time dimensions. The intent recognition model may perform well in some dimensions and average in some dimensions. Therefore, it provides a reference for adjusting the intent recognition model according to the actual situation in the future.

[0140] Please refer to Figure 9 , Figure 9 It is an optional flowchart of the method of the present application further including step S160. In some embodiments of the present application, the method of the present application further includes step S160. The following will combine Figure 9 to introduce this step in detail.

[0141] Step S160: Determine the application scenario of the intent recognition model according to the intent recognition accuracy rates of the intent recognition model in different dimensions.

[0142] In step S160 of some embodiments, after obtaining the intent recognition accuracy rate according to step S150, the intent recognition model in different dimensions can be adjusted according to the intent recognition accuracy rate. It can be understood that when the intent recognition accuracy rate in a certain dimension is high, the corresponding intent recognition model is adopted in actual work; when the intent recognition accuracy rate in a certain dimension type is low, the version of the intent recognition model can be adjusted and retested until a version with relatively good intent recognition accuracy rate is obtained. Therefore, generally speaking, it can be understood that according to the effects of intent recognition in different dimensions, the version of the intent recognition model with relatively better performance and response is selected.

[0143] It should be noted that the version of the intent recognition model can be understood as selectively combining several AI algorithm models to maximize the performance and effects of the intent recognition service. For the intent recognition model, the present invention only elaborates on a currently adoptable implementation manner, that is, it includes a rule engine model, an Elasticsearch (ES) recall model, a TextCNN model, and a Fasttext model. For combining more models into the intent recognition model, the method for evaluating the effect of the intent recognition service based on natural language processing (NLP) in the embodiments of the present invention is still applicable.

[0144] The embodiment of the present application provides a method for evaluating the effect of intention recognition service based on natural language processing (NLP). According to the multiple dimensional rules determined by the actual working situation, customer corpora of different dimensions are obtained. The dimensional rules may include time dimension rules and space dimension rules. The time dimension rules may specifically include all time periods, specific time periods, specific days of each month, and specific days of each week. The space dimension rules may specifically include multiple business types, multiple business departments, and multiple business scenarios. By determining which AI algorithm model the customer corpora of different dimensions hit in the intention recognition model, the distribution of corpora in different dimensions is obtained. The AI ​​algorithm model in the intention recognition model may include a rule engine model, an elastic search ES recall model, a TextCNN model, and a Fasttext model. According to the distribution of the corpus, a corresponding corpus test set is prepared, and the intention recognition model is tested on each corpus test set in different dimensions. By analyzing whether the intent obtained by the AI ​​technology is consistent with the objective intent expressed by the actual customer corpus, the accuracy of intention recognition in different dimensions is obtained to reflect the effect of the intention recognition model on intention recognition. Therefore, for the effect evaluation method of the embodiment of the present invention, the recognition effect of the NLP-based intent recognition service under different situations or scenarios is obtained through multi-dimensional refinement of the actual working conditions. Subsequently, the version of the intent recognition service under different dimensions can be adjusted in a targeted manner to better ensure the working effect of the NLP-based intent recognition service under specific dimensions, thereby solving the shortcoming of excessive distortion of the intent recognition effect due to the excessive randomness of the received customer corpus.

[0145] See also Figure 10 The embodiment of the present application also provides an intention recognition service effect evaluation device based on natural language processing (NLP), including a corpus acquisition module 610, a multi-dimensional corpus acquisition module 620, a corpus distribution acquisition module 630, a test set construction module 640, and a recognition effect acquisition module 650.

[0146] The corpus acquisition module 610 is used to acquire multiple customer corpora within a preset time interval;

[0147] The multi-dimensional corpus acquisition module 620 is used to divide the multiple customer corpora according to the preset dimension rules to obtain multiple customer corpus sets of different dimensions;

[0148] The corpus distribution acquisition module 630 is used to calculate the hit method ratio distribution data according to the customer corpus corresponding to each dimension, where the hit method ratio distribution data indicates the ratio distribution of the number of customer corpora hit by the multiple hit methods included in the preset intent recognition model;

[0149] The test set construction module 640 is used to construct a corpus test set corresponding to each dimension according to the hit mode proportion distribution data corresponding to each dimension;

[0150] The recognition effect acquisition module 650 is used to test the intention recognition model using corpus test sets corresponding to different dimensions to obtain the intention recognition accuracy of the intention recognition model in different dimensions.

[0151] It should be noted that the intention recognition service effect evaluation device based on natural language processing NLP in the embodiment of the present application is used to implement the above-mentioned intention recognition service effect evaluation method based on natural language processing NLP. The effect evaluation device in the embodiment of the present application corresponds to the aforementioned effect evaluation method. Please refer to the aforementioned effect evaluation method for the specific processing process, which will not be repeated here.

[0152] An intention recognition service effect evaluation device based on natural language processing (NLP) provided in an embodiment of the present application can implement the above-mentioned intention recognition service effect evaluation method based on natural language processing (NLP), by utilizing the corpus acquisition module 610 to acquire the required customer corpus, and by utilizing the multi-dimensional corpus acquisition module 620 to further acquire customer corpus sets of different dimensions according to multiple dimensional rules determined by actual working conditions; by utilizing the corpus distribution acquisition module 630 to determine which AI algorithm model in the intention recognition model the customer corpus under different dimensional types hits, thereby obtaining the distribution of the proportion of customer corpus hit methods under the dimension, which is specifically reflected by the hit method proportion distribution data; by utilizing the test set construction module 640 to prepare the corresponding corpus test set according to the distribution of the proportion of customer corpus hit methods, and by utilizing the recognition effect acquisition module 650 to perform intent recognition model tests on each corpus test set under different dimensions, thereby obtaining the accuracy of intent recognition under different dimensions to reflect the effect of intent recognition. Therefore, for the effect evaluation method of the embodiment of the present invention, the recognition effect of the NLP-based intent recognition service under different situations or scenarios is obtained through multi-dimensional refinement of the actual working conditions. Subsequently, the version of the intent recognition service under different dimensions can be adjusted in a targeted manner to better ensure the working effect of the NLP-based intent recognition service under specific dimensions, thereby solving the shortcoming of excessive distortion of the intent recognition effect due to the excessive randomness of the received customer corpus.

[0153] The embodiment of the present application also provides a computer device, which includes: at least one memory, at least one processor, and at least one computer program, wherein the at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement any of the effect evaluation methods in the above embodiments. The computer device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0154] Please refer to Figure 11 , Figure 11 which illustrates the hardware structure of a computer device according to another embodiment. The computer device includes:

[0155] A processor 710, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0156] A memory 720, which can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 720 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 720 and are called by the processor 710 to execute a text information extraction method according to an embodiment of the present application;

[0157] An input / output interface 730, which is used to implement information input and output;

[0158] A communication interface 740, which is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0159] A bus 750, which transmits information between various components of the device (such as the processor 710, the memory 720, the input / output interface 730, and the communication interface 740);

[0160] Among them, the processor 710, the memory 720, the input / output interface 730, and the communication interface 740 achieve communication connections with each other inside the device through the bus 750.

[0161] The embodiments of the present application also provide a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is used to make a computer execute the effect evaluation method in any one of the above embodiments.

[0162] A memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0163] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0164] Those skilled in the art can understand that Figures 1 to 11 the technical solutions shown in [the figure] do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figure, or combine certain steps, or different steps.

[0165] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0167] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0168] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (one)" or its similar expression below refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0169] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above-mentioned division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0170] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, the functional units in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0172] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The foregoing storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0173] The foregoing has described the preferred embodiments of the embodiments of this application with reference to the accompanying drawings, and does not thereby limit the scope of the rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall fall within the scope of the rights of the embodiments of this application.

Claims

1. A method for evaluating the service effect of intent recognition based on natural language processing (NLP), characterized in that, the method includes: Obtaining a plurality of customer corpora within a preset time interval; Dividing the plurality of customer corpora according to preset dimension rules to obtain a plurality of customer corpus sets with different dimensions; For each customer corpus set corresponding to a dimension, inputting each customer corpus included in the customer corpus set into a preset intent recognition model respectively, performing regular matching on the customer corpus through the rule engine of the intent recognition model, when the regular matching is successful, determining that the hit method corresponding to the customer corpus is rule engine hit; when the regular matching fails, performing ES corpus traversal and relevance calculation analysis on the customer corpus through the ES recall model of the intent recognition model, when the ES corpus traversal and relevance calculation analysis is successful, determining that the hit method corresponding to the customer corpus is ES recall hit; when the ES corpus traversal and relevance calculation analysis fails, performing first fast text analysis on the customer corpus through the TextCNN model of the intent recognition model, when the first fast text analysis is successful, determining that the hit method corresponding to the customer corpus is TextCNN model hit; when the first fast text analysis fails, performing second fast text analysis on the customer corpus through the Fasttext model of the intent recognition model, when the second fast text analysis is successful, determining that the hit method corresponding to the customer corpus is Fasttext model hit; according to the hit methods corresponding to each customer corpus included in the customer corpus set, determining the hit method proportion distribution data, and the hit method proportion distribution data indicates the proportion distribution of the number of customer corpora hit by each of the multiple hit methods included in the intent recognition model; Constructing a corpus test set corresponding to each dimension according to the hit method proportion distribution data corresponding to each dimension; Testing the intent recognition model by using the corpus test sets corresponding to different dimensions to obtain the intent recognition accuracy of the intent recognition model under different dimensions.

2. The method for evaluating the service effect of intent recognition based on natural language processing (NLP) according to claim 1, characterized in that, the dividing the plurality of customer corpora according to preset dimension rules to obtain a plurality of customer corpus sets with different dimensions includes at least one of the following: Classifying the customer corpora according to the whole time period to obtain a customer corpus set corresponding to the whole time period; Classifying the customer corpora according to a plurality of preset specific time periods to obtain customer corpus sets corresponding to different specific time periods; Classifying the customer corpora according to a plurality of preset specific days of each month to obtain customer corpus sets corresponding to different specific days of each month; Classifying the customer corpora according to a plurality of preset specific days of each week to obtain customer corpus sets corresponding to different specific days of each week; Classifying the customer corpora according to a plurality of preset business types to obtain customer corpus sets corresponding to different business types; Classify the customer corpus according to multiple business scenarios of a specified business type to obtain customer corpus sets corresponding to different business scenarios of the specified business type; Classify the customer corpus according to multiple business departments and multiple business scenarios of a specified business type to obtain customer corpus sets corresponding to different business departments and different business scenarios of the specified business type.

3. The method for evaluating the service effect of intent recognition based on natural language processing (NLP) according to claim 1, characterized in that constructing a corpus test set corresponding to each dimension according to the hit method proportion distribution data corresponding to each dimension, including at least one of the following: constructing a corresponding corpus test set for the whole time period according to the hit method proportion distribution data determined by the customer corpus set for the whole time period; constructing a corresponding corpus test set for different specific time periods according to the hit method proportion distribution data determined by the customer corpus sets for different specific time periods; constructing a corresponding corpus test set for different specific days of each month according to the hit method proportion distribution data determined by the customer corpus sets for different specific days of each month; constructing a corresponding corpus test set for different specific days of each week according to the hit method proportion distribution data determined by the customer corpus sets for different specific days of each week; constructing a corresponding corpus test set for different business types according to the hit method proportion distribution data determined by the customer corpus sets for different business types; constructing a corresponding corpus test set for different business scenarios of a specified business type according to the hit method proportion distribution data determined by the customer corpus set for different business scenarios of the specified business type; constructing a corresponding corpus test set for different business departments and different business scenarios of a specified business type according to the hit method proportion distribution data determined by the customer corpus set for different business departments and different business scenarios of the specified business type.

4. The method for evaluating the service effect of intent recognition based on natural language processing (NLP) according to claim 1, characterized in that the corpus test set includes multiple customer corpus samples and intent labels corresponding to each customer corpus sample; testing the intent recognition model with the corpus test sets corresponding to different dimensions to obtain the intent recognition accuracy of the intent recognition model under different dimensions, including: traversing the corpus test sets corresponding to each dimension, and for the corpus test set corresponding to the currently traversed dimension, performing the following steps: inputting each customer corpus sample included in the corpus test set into the intent recognition model to obtain intent recognition results corresponding to each customer corpus sample; determining the intent recognition accuracy of the intent recognition model under the currently traversed dimension according to the intent recognition results and intent labels corresponding to each customer corpus sample.

5. The method for evaluating the service effect of intent recognition based on natural language processing (NLP) according to claim 1, characterized in that after obtaining the intent recognition accuracy of the intent recognition model under different dimensions, the method further includes: determining the application scenario of the intent recognition model according to the intent recognition accuracy of the intent recognition model under different dimensions.

6. An apparatus for evaluating the effect of intent recognition service based on natural language processing (NLP), characterized in that, it includes: A corpus acquisition module, configured to acquire a plurality of customer corpora within a preset time interval; A multi-dimensional corpus acquisition module, configured to divide the plurality of customer corpora according to preset dimensional rules to obtain a plurality of customer corpus sets with different dimensions; A corpus distribution acquisition module, for each customer corpus set corresponding to a dimension, input each customer corpus included in the customer corpus set into a preset intent recognition model, perform regular matching on the customer corpus through the rule engine of the intent recognition model, and when the regular matching is successful, determine that the hit method corresponding to the customer corpus is rule engine hit; When the regular matching fails, perform ES corpus traversal and relevance calculation analysis on the customer corpus through the ES recall model of the intent recognition model. When the ES corpus traversal and relevance calculation analysis are successful, determine that the hit method corresponding to the customer corpus is ES recall hit; When the ES corpus traversal and relevance calculation analysis fail, perform a first fast text analysis on the customer corpus through the TextCNN model of the intent recognition model. When the first fast text analysis is successful, determine that the hit method corresponding to the customer corpus is TextCNN model hit; when the first fast text analysis fails, perform a second fast text analysis on the customer corpus through the Fasttext model of the intent recognition model. When the second fast text analysis is successful, determine that the hit method corresponding to the customer corpus is Fasttext model hit; according to the hit methods corresponding to each customer corpus included in the customer corpus set, determine the hit method proportion distribution data, and the hit method proportion distribution data indicates the proportion distribution of the number of customer corpora hit by each of the multiple hit methods included in the intent recognition model; A test set construction module, configured to construct a corpus test set corresponding to each dimension according to the hit method proportion distribution data corresponding to each dimension; An identification effect acquisition module, configured to test the intent recognition model by using the corpus test sets corresponding to different dimensions, and obtain the intent recognition accuracy of the intent recognition model under different dimensions.

7. A computer device, characterized in that, it includes: At least one memory; At least one processor; At least one computer program; The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement: a method for evaluating the effect of intent recognition service based on natural language processing (NLP) as described in any one of claims 1 to 5.

8. A storage medium, which is a computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and the computer program is used to cause a computer to execute: a method for evaluating the effect of intent recognition service based on natural language processing (NLP) as described in any one of claims 1 to 5.

Images