Model construction method, apparatus, device, and medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a model construction method, device, equipment and medium. The application obtains training corpus for constructing a model; based on a pre-trained clustering model, the training corpus is subjected to clustering processing to obtain a corresponding clustering result, wherein the clustering result comprises a clustering label and clustering corpus corresponding to the clustering label; based on the clustering label in the clustering result and the corresponding clustering corpus, model training and prediction are performed, and a target intent recognition model is determined according to a model training and prediction result. The above method of automatically generating the target intent recognition model reduces the time input in the process of familiarizing with business points and data labeling, accelerates the process of combing business points and labeling business corpus, improves the efficiency of constructing the target intent recognition model, and reduces the labor cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model construction method, device, equipment and medium. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, the application of dialogue system is more and more widely, such as unmanned customer service system, in the unmanned customer service system, intent recognition is an important part, a common algorithm of intent recognition is to recognize the user's intent through text classification, specifically, the user's intent is divided into several categories, and the corresponding response scheme is matched under the category.

[0003] When building a dialogue system, text classification is usually the simplest and most effective means to build a user intent recognition model. The existing text classification model for intent recognition is mainly based on the business to be undertaken in the dialogue system. The development process often requires operation personnel to understand the business points to be undertaken by the dialogue system, sort out the related business point information, and then annotate a large amount of corpus provided by the business party. After annotation, the classification is adjusted with the help of classification adjustment tools. If the corpus is not sufficient, the tool expansion will be considered. Finally, the text classification model for recognizing user intent is obtained. In this process, the operation personnel spend a lot of time on familiarizing with the business points and data annotation. The efficiency of the operation personnel in constructing the text classification model by sorting out the business points and annotating the business corpus is low, and a large amount of manpower is consumed. SUMMARY

[0004] The main purpose of the present application is to provide a model construction method, device, equipment and medium, which aims to reduce the labor cost of manual sorting and annotation, and improve the efficiency of classification model construction.

[0005] To achieve the above purpose, the present application provides a model construction method, which comprises the following steps:

[0006] Obtain training corpus for constructing a model;

[0007] Based on the pre-trained clustering model, the training corpus is clustered to obtain the corresponding clustering result, wherein the clustering result includes a clustering label and a clustering corpus corresponding to the clustering label;

[0008] Based on the clustering label and the corresponding clustering corpus in the clustering result, the model is trained and predicted, and the target intent recognition model is determined according to the model training and prediction result.

[0009] Preferably, the step of clustering the training corpus based on the pre-trained clustering model to obtain the corresponding clustering result comprises:

[0010] sequentially disordered and segmented to obtain a clustering sample corpus;

[0011] The clustering sample corpus is clustered based on a hierarchical agglomerative clustering algorithm HAC to obtain a clustering label and a clustering corpus corresponding to the clustering label.

[0012] Preferably, the step of clustering the clustering sample corpus based on the hierarchical agglomerative clustering algorithm HAC to obtain a clustering label and a clustering corpus corresponding to the clustering label comprises:

[0013] The clustering sample corpus is classified, and clustering sample corpora of the same type after classification are divided into a cluster to obtain clustering labels corresponding to clusters of different types;

[0014] Based on the clustering label corresponding to the cluster, determine the cluster-in corpus corresponding to the clustering label of different types;

[0015] If the number of cluster-in corpora is greater than a preset threshold N1, the cluster-in corpus is taken as the clustering label, and the cluster-in corpus corresponding to the clustering label is the corresponding clustering corpus;

[0016] If the number of cluster-in corpora is not greater than a preset threshold N1, the cluster-in corpus is taken as the clustering label, and the cluster-in corpus corresponding to the clustering label is the corresponding clustering corpus.

[0017] Preferably, the step of training and predicting a model based on the clustering label and the corresponding clustering corpus in the clustering result, and determining a target intent recognition model according to the model training and prediction result comprises:

[0018] The clustering corpus in the clustering result is divided into training corpora and prediction corpora according to the clustering label;

[0019] Based on the training corpus, a trained initial classification model is obtained;

[0020] The prediction corpus is input into the trained initial classification model for prediction to obtain a prediction score value;

[0021] Based on the clustering label in the clustering result and the prediction score value, the precision recall rate PRF value of the clustering result is determined;

[0022] Based on the PRF value, a corresponding target intent recognition model is determined.

[0023] Preferably, the PRF value is used to determine whether the clustering result is reasonable.

[0024] Determine whether the PRF value reaches a preset threshold;

[0025] If the PRF value reaches a preset threshold, the clustering result is reasonable, and the initial classification model is output as a target intent recognition model.

[0026] If the PRF value does not reach the preset threshold, the clustering result is unreasonable, the clustering result is classified and adjusted to obtain a classified and adjusted clustering result.

[0027] The classified and adjusted clustering result is taken as a current clustering result.

[0028] The clustering corpus in the clustering result is divided into training corpus and prediction corpus according to the clustering label.

[0029] Model training is performed based on the training corpus to obtain a trained initial classification model.

[0030] The prediction corpus is input into the trained initial classification model for prediction to obtain a prediction score value.

[0031] Based on the clustering label in the clustering result and the prediction score value, a precision recall F value (PRF value) of the clustering result is determined.

[0032] Until the PRF value reaches a preset threshold, the clustering result is reasonable, and the initial classification model is output as a target intent recognition model.

[0033] Preferably, the clustering result includes other clustering corpus with a clustering label of other and non-other clustering corpus with a clustering label of non-other,

[0034] The step of classifying and adjusting the clustering result to obtain a classified and adjusted clustering result includes:

[0035] Adjusting the other clustering corpus and the non-other clustering corpus to obtain clustering corpus corresponding to an adjusted clustering label;

[0036] Calculating the confusion degree of the clustering corpus corresponding to the adjusted clustering label;

[0037] When the confusion degree is greater than a preset threshold T2, the non-other clustering corpus before and after the adjustment is merged as the non-other clustering corpus of the current clustering result, and other corpus is taken as the other clustering corpus with the clustering label of other.

[0038] Preferably, the step of adjusting the other clustering corpus and the non-other clustering corpus to obtain clustering corpus corresponding to an adjusted clustering label includes:

[0039] Obtaining a prediction score value of the non-other clustering corpus;

[0040] if the prediction score value of the non-other clustered corpus is lower than a preset threshold T1, changing the clustered label of the non-other clustered corpus to other;

[0041] if the number of other clustered corpora with the clustered label of other exceeds a preset threshold N2, obtaining the adjusted other clustered corpora and the adjusted non-other clustered corpora.

[0042] Preferably, the step of obtaining the training corpus for model construction comprises:

[0043] obtaining original corpora from a business end;

[0044] preprocessing the original corpora to obtain the training corpus for model construction;

[0045] Preferably, the preprocessing method comprises one or more of the following: eliminating stop words, full-width half-width conversion, eliminating emoticons, eliminating greetings and meaningless questions, unifying punctuation marks, and eliminating rarely used punctuation marks.

[0046] In addition, to achieve the above-mentioned purposes, the present application also provides a model construction device, which comprises:

[0047] an obtaining module for obtaining a training corpus for model construction;

[0048] a clustering module for clustering the training corpus based on a pre-trained clustering model to obtain a corresponding clustering result, wherein the clustering result comprises a clustered label and clustered corpora corresponding to the clustered label;

[0049] a determining module for model training and prediction based on the clustered label and the corresponding clustered corpora in the clustering result, and determining a target intent recognition model according to the model training and prediction results.

[0050] Preferably, the obtaining module is further configured to:

[0051] obtain original corpora from a business end;

[0052] preprocess the original corpora to obtain the training corpus for model construction;

[0053] Preferably, the preprocessing method comprises one or more of the following: eliminating stop words, full-width half-width conversion, eliminating emoticons, eliminating greetings and meaningless questions, unifying punctuation marks, and eliminating rarely used punctuation marks.

[0054] Preferably, the clustering module is further configured to:

[0055] sequentially reorder and segment the training corpus to obtain clustered sample corpora;

[0056] The clustering sample corpus is clustered based on a hierarchical agglomerative clustering algorithm HAC to obtain clustering labels and clustering corpora corresponding to the clustering labels.

[0057] Preferably, the clustering module is further configured to:

[0058] The clustering sample corpus is classified, and clustering sample corpora of the same category after classification are divided into a cluster to obtain clustering labels corresponding to clusters of different categories.

[0059] Based on the clustering labels corresponding to the clusters, cluster-in corpora corresponding to clustering labels of different categories are determined.

[0060] If the number of cluster-in corpora is greater than a preset threshold N1, the cluster-in corpora are taken as clustering labels, and cluster-in corpora corresponding to the clustering labels are taken as corresponding clustering corpora.

[0061] If the number of cluster-in corpora is not greater than the preset threshold N1, the cluster-in corpora are taken as clustering labels, and cluster-in corpora corresponding to the clustering labels are taken as corresponding clustering corpora.

[0062] Preferably, the determination module is further configured to:

[0063] The clustering corpora in the clustering result are divided into training corpora and prediction corpora according to the clustering labels.

[0064] A trained initial classification model is obtained based on the training corpora.

[0065] The prediction corpora are input into the trained initial classification model for prediction to obtain a prediction score value.

[0066] Based on the clustering labels in the clustering result and the prediction score value, a precision recall F value PRF of the clustering result is determined.

[0067] Based on the PRF value, a corresponding target intent recognition model is determined.

[0068] Preferably, the determination module is further configured to:

[0069] It is judged whether the PRF value reaches a preset threshold.

[0070] If the PRF value reaches the preset threshold, the clustering result is reasonable, and the initial classification model is output as a target intent recognition model.

[0071] If the PRF value does not reach the preset threshold, the clustering result is unreasonable, and the clustering result is classified and adjusted to obtain a classified and adjusted clustering result.

[0072] adjust the clustering result according to the classification result to obtain an adjusted clustering result;

[0073] divide the clustering corpus in the adjusted clustering result into training corpus and prediction corpus according to the clustering label;

[0074] train a model based on the training corpus to obtain a trained initial classification model;

[0075] input the prediction corpus into the trained initial classification model for prediction to obtain a prediction score value;

[0076] determine a precision recall F value of the clustering result based on the clustering label in the clustering result and the prediction score value;

[0077] until the PRF value reaches a preset threshold value, the clustering result is reasonable, and output the initial classification model as a target intent recognition model.

[0078] Preferably, the determining module is further configured to:

[0079] adjust the other clustering corpus and the non-other clustering corpus to obtain an adjusted clustering corpus corresponding to the adjusted clustering label;

[0080] calculate a confusion degree of the adjusted clustering corpus corresponding to the adjusted clustering label;

[0081] when the confusion degree is greater than a preset threshold value T2, merge the non-other clustering corpus before and after the adjustment as the non-other clustering corpus of the current clustering result, and the other corpus is the other clustering corpus with the clustering label as other.

[0082] Preferably, the determining module is further configured to:

[0083] obtain a prediction score value of the non-other clustering corpus;

[0084] if the prediction score value of the non-other clustering corpus is lower than a preset threshold value T1, change the clustering label of the non-other clustering corpus to other;

[0085] when the number of the other clustering corpus with the clustering label as other exceeds a preset threshold value N2, obtain the adjusted other clustering corpus and the adjusted non-other clustering corpus.

[0086] In addition, to achieve the above object, the present application also provides a model construction device, comprising a memory, a processor and a model construction program stored on the memory and executable on the processor, wherein the model construction program implements the steps of the model construction method as described above when executed by the processor.

[0087] In addition, to achieve the above object, the present application also provides a medium, which is a computer readable storage medium, and the computer readable storage medium stores a model construction program, wherein the model construction program implements the steps of the model construction method as described above when executed by a processor.

[0088] The model construction method, device, equipment and medium provided by the present application obtain training corpus for constructing a model; perform clustering processing on the training corpus based on a pre-trained clustering model to obtain corresponding clustering results, wherein the clustering results include clustering labels and clustering corpus corresponding to the clustering labels; perform model training and prediction based on the clustering labels and the corresponding clustering corpus in the clustering results, and determine a target intent recognition model according to the model training and prediction results.

[0089] The training corpus for constructing an intent recognition model is clustered to obtain clustering results corresponding to the training corpus, the clustering results include clustering labels for classifying clustering corpus and clustering corpus corresponding to the clustering labels, the clustering results including the clustering labels and the clustering corpus corresponding to the clustering labels are subjected to model training and prediction to obtain PRF values corresponding to the clustering results, and a target intent recognition model is determined according to the PRF values. The method for automatically generating a target intent recognition model reduces the time spent in familiarizing with business points and data labeling, improves the efficiency of sorting business points and labeling business corpus, and reduces the labor cost. BRIEF DESCRIPTION OF DRAWINGS

[0090] Figure 1 is a device structure schematic diagram of a hardware running environment related to an embodiment of the model construction method of the present application;

[0091] Figure 2 is a flowchart of the first embodiment of the model construction method of the present application;

[0092] Figure 3 is a specific flowchart of the first embodiment of the model construction method of the present application;

[0093] Figure 4 is a flowchart of the second embodiment of the model construction method of the present application;

[0094] Figure 5 is a sub-flowchart of step S22 in the second embodiment of the model construction method of the present application;

[0095] Figure 6 A flowchart of a third embodiment of the model construction method of the present application;

[0096] Figure 7 A flowchart of a fourth embodiment of the model construction method of the present application;

[0097] Figure 8 A flowchart of a sub-process of step B3 in the fourth embodiment of the model construction method of the present application;

[0098] Figure 9 A functional module diagram of the model construction device involved in the first embodiment of the model construction method of the present application.

[0099] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0100] It should be understood that the specific embodiments described herein merely serve to explain the present application and do not limit the present application.

[0101] As shown in Figure 1 , the device structure diagram of the hardware running environment involved in the embodiment of the present application. Figure 1

[0102] The device of the embodiment of the present application can be a mobile terminal or a server device.

[0103] As shown in Figure 1 , the device can include a processor 1001 such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 can include a display screen (Display), an input unit such as a keyboard (Keyboard), and optionally a user interface 1003 can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory) such as a magnetic disk memory. The memory 1005 can optionally also be a storage device independent of the aforementioned processor 1001.

[0104] Those skilled in the art can understand that Figure 1 the device structure shown in the above description does not constitute a limitation on the device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0105] As shown in Figure 1 ​As shown, the memory 1005 as a computer storage medium can include an operating system, a network communication module, a user interface module, and a model building program.

[0106] The operating system is a program for managing and controlling software resources of the model building device, supports the running of the network communication module, the user interface module, the model building program, and other programs or software; the network communication module is used for managing and controlling the network interface 1002; and the user interface module is used for managing and controlling the user interface 1003.

[0107] In Figure 1 In the model building device shown, the model building device calls the model building program stored in the memory 1005 through the processor 1001, and performs the operations in each embodiment of the model building method described below.

[0108] Based on the hardware structure described above, embodiments of the model building method of the present application are proposed.

[0109] Referring to Figure 2 , Figure 2 The flowchart of the first embodiment of the model building method of the present application is shown. The method comprises the following steps.

[0110] Step S10, obtaining training corpus for building a model;

[0111] Obtaining original corpus from a business end; preprocessing the original corpus to obtain training corpus for model building; wherein the preprocessing method includes one or more of the following: eliminating stop words, converting full-width to half-width, eliminating emoticons, eliminating greetings and meaningless questions, unifying punctuation, and eliminating rarely used punctuation.

[0112] In a specific embodiment, dialog corpus from a business scenario is collected for model building, and a large number of user chat records of intelligent customer service business are used as original corpus for training target model. The original corpus is standardized and preprocessed, and the processing method can include one or more of the following: eliminating stop words, converting full-width to half-width, eliminating emoticons, eliminating greetings and meaningless questions, unifying punctuation, and eliminating rarely used punctuation. By standardizing and preprocessing the overall original corpus from the client, the effect of reducing noise in the original corpus, improving the purity of the original corpus, and obtaining training corpus for building a model is achieved.

[0113] Step S20, based on a pre-trained clustering model, performing clustering processing on the training corpus to obtain a corresponding clustering result, wherein the clustering result includes a clustering label and clustering corpus corresponding to the clustering label;

[0114] The principle of clustering the training corpus to obtain a clustering result corresponding to the training corpus is that the clustering process can use morphological operators to cluster and merge similar classification regions close to each other. After clustering and merging, different clusters are generated for each category. The clusters generated by clustering are a set of data objects, which are similar to each other in the same cluster and different from objects in other clusters.

[0115] In existing clustering algorithms, most clustering algorithms have the ability to process "noise" data. Some clustering algorithms are very sensitive to noise data and can obtain accurate clustering results after clustering audio data.

[0116] In a specific embodiment, the corpus data collected from the service end is preprocessed to obtain preprocessed training corpus. The training corpus is input into a pre-trained clustering model. The preprocessed training corpus is shuffled, and part of the corpus in the training corpus is extracted for clustering. The clustering algorithm can use a hierarchical agglomerative clustering algorithm HAC to obtain different clusters for each category. These clusters have their respective corpus, and a clustering result including clustering labels and clustering corpus corresponding to the clustering labels is obtained. In the above clustering result, if the number of corpus in the cluster is greater than a preset threshold N1 (for example, 50), the cluster id is used as the clustering label, and the remaining corpus is used as the clustering label.

[0117] Step S30, based on the clustering label and the corresponding clustering corpus in the clustering result, model training and prediction are performed, and a target intent recognition model is determined according to the model training and prediction results.

[0118] In the existing technology, an initial pre-training model is used to divide and label artificial data to obtain a data set that can be used for model training. The initial pre-training model is used for deep learning to obtain a target classification model. In the model construction method of the present embodiment, the training corpus of the service end is directly obtained, and the training corpus is clustered to obtain a clustering result including clustering labels and clustering corpus corresponding to the clustering labels. The clustering result is trained and predicted, and a target intent recognition model is determined according to the model training and prediction results.

[0119] In a specific embodiment, according to the clustering result including the clustering label and the clustering corpus corresponding to the clustering label, the clustering result is evenly divided into 5 parts according to the clustering label, for example, the number of the corpus of the clustering label (cluster id) is label1 is 200, and the number of the clustering label of each part of the clustering corpus after division is 40, 4 parts of the clustering corpus are taken each time to train the pre-trained classification model, and the remaining 1 part of the clustering corpus is predicted to obtain the prediction score value of all 5 parts of the clustering corpus. Through the clustering label and the corresponding prediction score value, the PRF value, that is, the Precision (precision), Recall (recall) and F1 value (F1) corresponding to the clustering result can be obtained, and the PRF value is used to evaluate whether the clustering result is reasonable, if the clustering result is reasonable, the classification model trained based on the training data is output to obtain the corresponding target intent recognition model.

[0120] In the embodiment, the preprocessed training corpus is clustered to obtain clustering labels with different feature categories and clustering corpora corresponding to the clustering labels, the clustering labels and the clustering corpora corresponding to the clustering labels are used as the clustering result for model training and prediction, and the target intent recognition model is determined according to the model training and prediction result. By automatically constructing the intent recognition model, the labor cost of manual labeling is greatly reduced, and the operation pressure of the operation personnel is reduced. In addition, the automatic model construction method can help the operation personnel to understand the business, find and sort different business types involved, and improve the efficiency of constructing the intent recognition model.

[0121] Further, based on the first embodiment of the model construction method of the application, the second embodiment of the model construction method of the application is proposed.

[0122] The second embodiment of the model construction method is different from the first embodiment of the model construction method in that the embodiment is a refinement of step S20, that is, the clustering model is trained based on the pre-trained clustering model, and the clustering result corresponding to the training corpus is obtained, which is described with reference to Figure 4 The step specifically includes:

[0123] In step S21, the training corpus is sequentially disordered and segmented to obtain a clustering sample corpus.

[0124] In a specific embodiment, the preprocessed training corpus is input into the pre-trained clustering model, the arrangement order of the training corpus is disordered, and part of the training corpus is extracted for clustering. The way of extracting part of the training corpus for clustering specifically includes: synchronously clustering part of the training corpus, then gradually adding the remaining training corpus, and then obtaining the clustering result corresponding to the training corpus. Compared with the way of directly clustering all the training corpus, the way of extracting part of the training corpus for clustering finally obtains better classification effect.

[0125] In an embodiment, the preprocessed training corpus is disordered, and a part of the training corpus is extracted as a clustering sample corpus, for example, 20% of the total corpus is extracted as a clustering sample corpus for clustering processing.

[0126] In step S22, the clustering sample corpus is processed based on the hierarchical agglomerative clustering algorithm HAC to obtain clustering labels and clustering corpus corresponding to the clustering labels.

[0127] There are many ways to cluster, including division methods, hierarchical methods, density-based methods, grid-based methods, model-based methods, transitive closure methods, Boolean matrix methods, direct clustering methods, correlation analysis clustering, and statistical-based clustering methods. Among different methods, there are various clustering algorithms, and corresponding clustering results can be obtained according to these clustering algorithms.

[0128] In an embodiment, the clustering sample corpus is processed based on the hierarchical agglomerative clustering algorithm HAC to obtain clustering sample corpus belonging to different types of clustering labels. The clustering label is a different cluster corresponding to each category, and the clusters have their respective corpus. These clusters are referred to as clustering labels, and the corpus corresponding to each different cluster is referred to as clustering corpus corresponding to the clustering label. Thus, the clustering result including the clustering label and the clustering corpus corresponding to the clustering label is obtained after clustering processing.

[0129] Referring to Figure 5 , step S22 specifically includes:

[0130] In step A1, the clustering sample corpus is classified, and the clustering sample corpus of the same type is divided into a cluster after classification to obtain clustering labels corresponding to different types of clusters.

[0131] In an embodiment, the clustering sample corpus is classified to obtain clustering labels belonging to different types of clusters, and the clustering sample corpus is classified according to the clustering labels to obtain clustering labels and clustering corpus corresponding to the clustering labels.

[0132] In step A2, based on the clustering labels corresponding to the clusters, the corpus within the clusters corresponding to different types of clustering labels is determined.

[0133] In an embodiment, the clustering sample corpus is classified based on the hierarchical agglomerative clustering algorithm HAC to obtain different clusters corresponding to each category, and the clusters have their respective corpus. These clusters are referred to as clustering labels, and the corpus corresponding to each different cluster is referred to as clustering corpus corresponding to the clustering label.

[0134] Step A3, if the number of the intra-cluster corpus is greater than the preset threshold N1, the intra-cluster corpus is labeled with the cluster as the clustering label, and the intra-cluster corpus corresponding to the clustering label is the corresponding clustering corpus;

[0135] Step A4, if the number of the intra-cluster corpus is not greater than the preset threshold N1, the intra-cluster corpus is labeled with other as the clustering label, and the intra-cluster corpus corresponding to the clustering label is the corresponding clustering corpus.

[0136] In a specific embodiment, the clustering sample corpus can be divided into five different categories of label1, label2, label3, label4 and label5, and the clustering corpus corresponding to the five different clustering labels is included. If the clusters with the number of intra-cluster corpus greater than the preset threshold 50 include label1, label2, label3 and label4, the clustering corpus with the clustering labels of label1, label2, label3 and label4 is labeled with the cluster id, label1, label2, label3 and label4 as the clustering label, and the remaining clustering sample corpus label5 is labeled with other as the clustering label.

[0137] In this embodiment, the clustering sample corpus is obtained by shuffling and dividing the training corpus, and the additional clustering non-sample corpus is also retained. The clustering sample corpus is clustered based on the hierarchical agglomerative clustering algorithm HAC to obtain the clustering result after clustering. The clustering result includes the clustering label and the clustering corpus corresponding to the clustering label. The clustering result obtained by the above method is used to train the target intent recognition model, and the clustering algorithm is introduced to process the training data, thereby reducing the human input in the process of constructing the intent recognition text classification, and improving the efficiency of constructing the intent recognition model.

[0138] Further, based on the first and second embodiments of the model construction method of the present application, a third embodiment of the model construction method of the present application is proposed.

[0139] The third embodiment of the model construction method is different from the first and second embodiments of the model construction method in that the step S30 is performed in the third embodiment. The model is trained and predicted based on the clustering label and the corresponding clustering corpus in the clustering result, and the refinement of the target intent recognition model is determined according to the model training and prediction result, as described in the first and second embodiments. Figure 6 The step specifically includes:

[0140] Step S31, the clustering corpus in the clustering result is divided into training corpus and prediction corpus according to the clustering label;

[0141] The clustering corpus in the clustering result is evenly divided into n parts according to the clustering label, m parts of the n parts of clustering corpus are taken each time to train the initial model to obtain a corresponding classification model, and the remaining clustering corpus in the n parts of clustering corpus is predicted to obtain the prediction score of all clustering corpus.

[0142] In a specific embodiment, the clustering corpus in the clustering result is evenly divided into 5 parts according to the clustering label label1 and label2. In the clustering corpus with the clustering label label1, when the number of clustering corpus is 200, the number of clustering corpus with the label label1 after division is 40, 4 parts of clustering corpus are taken each time for training, and the remaining one part is used for prediction, and the prediction score of all clustering corpus with the clustering label label1 can be obtained. In the clustering corpus with the clustering label label2, when the number of clustering corpus is 400, the number of clustering corpus with the label label2 after division is 80, 4 parts of clustering corpus are taken each time for training, and the remaining one part is used for prediction, and the prediction score of all clustering corpus with the clustering label label2 can be obtained.

[0143] Step S32, model training based on the training corpus is performed to obtain a trained initial classification model;

[0144] Step S33, the prediction corpus is input into the trained initial classification model for prediction to obtain a prediction score value;

[0145] In a specific embodiment, the training corpus in the divided clustering corpus is used to train the pre-trained classification model to obtain a corresponding classification model, and the prediction corpus in the above divided clustering corpus is input into the above classification model, and the corresponding classification result is obtained after classification. The classification result is tested to obtain the prediction score value of the classification result corresponding to the above classification model.

[0146] Step S34, based on the clustering label in the clustering result and the prediction score value, the precision recall rate PRF value of the clustering result is determined;

[0147] The prediction score value corresponding to the classification result of the classification model, the clustering label in the clustering result, and the Precision (precision), Recall (recall), F1 value (F1), and PRF value are obtained, and the PRF value is used to evaluate whether the classification model is reasonable, and then the training data of the classification model is evaluated, and further, whether the clustering result is reasonable is obtained.

[0148] Step S35, based on the PRF value, a corresponding target intent recognition model is determined.

[0149] In a specific embodiment, according to the PRF value of the clustering result, a corresponding target intent recognition model of the output can be determined, and specifically, the PRF value is determined by the following rules:

[0150] If the PRF value reaches a preset threshold, the clustering result is reasonable, and all corpora in the clustering result are used to train the pre-trained model to obtain a corresponding classification model.

[0151] If the PRF value does not reach the preset threshold, the clustering result is not reasonable, and the clustering label corresponding to the clustering result and the clustering corpus corresponding to the clustering label need to be classified and adjusted to obtain the clustering label after classification adjustment and the clustering corpus corresponding to the clustering label, and the clustering result of the clustering label after classification adjustment and the clustering corpus corresponding to the clustering label is retrained and predicted until the PRF value corresponding to the clustering result reaches the preset threshold, and a corresponding classification model is output.

[0152] In the embodiment, by training and predicting the clustering result after clustering processing, a classification output result of the classification model corresponding to the clustering result and a prediction score value corresponding to the clustering result are obtained, according to the classification output result and the prediction score value, a PRF value of the clustering result is obtained, whether the PRF value reaches a preset threshold is determined to determine whether the clustering result is reasonable, and a corresponding target intent recognition model is finally determined, which can improve the accuracy of automatically creating a target intent recognition model, set a fault tolerance mechanism, predict and determine the classification result of the classification model, and improve the accuracy of the classification result of the classification model.

[0153] Further, based on the first, second and third embodiments of the model construction method of the present application, a fourth embodiment of the model construction method of the present application is proposed.

[0154] The fourth embodiment of the model construction method is different from the first, second and third embodiments of the model construction method in that the embodiment is a refinement of step S35, which determines a corresponding target intent recognition model based on the PRF value, and refers to Figure 7 , which specifically includes:

[0155] Step B1, determining whether the PRF value reaches a preset threshold;

[0156] Step B2, if the PRF value reaches the preset threshold, the clustering result is reasonable, and the initial classification model is output as a target intent recognition model;

[0157] In a specific embodiment, if the PRF value of the classification result output by the classification model reaches a preset threshold, it is determined whether all the clustered training corpus is added to the process of training the pre-trained model, if all the clustered training corpus is added, the classification model is directly output as the target intent recognition model, if not all the clustered training corpus is added, the clustered training corpus is added to further train the classification model until all the clustered training corpus is added to the training, and the classification model is output as the target intent recognition model.

[0158] In step B3, if the PRF value does not reach the preset threshold, the clustering result is unreasonable, and the clustering result is classified and adjusted to obtain a classified and adjusted clustering result.

[0159] Referring to Figure 8 , step B3 specifically includes:

[0160] The clustering result includes other clustering corpus with a clustering label of other and non-other clustering corpus with a clustering label of non-other, and the step of classifying and adjusting the clustering result to obtain a classified and adjusted clustering result includes:

[0161] In step b1, the other clustering corpus and the non-other clustering corpus are adjusted to obtain clustering corpus corresponding to an adjusted clustering label.

[0162] In step b2, the confusion degree of the clustering corpus corresponding to the adjusted clustering label is calculated.

[0163] In step b3, when the confusion degree is greater than a preset threshold T2, the non-other clustering corpus before and after the adjustment is merged as the non-other clustering corpus of the current clustering result, and the other corpus is the other clustering corpus with a clustering label of other.

[0164] In a specific embodiment, the step of classifying and adjusting the clustering result includes three parts:

[0165] First, the first part is to change the label of the untrusted part of the non-other clustering corpus to the clustering corpus with a clustering label of other. Through the above steps, the clustering label and the predicted score value of all the clustering corpus are obtained. The clustering corpus with a predicted score value lower than a certain threshold 0.3 and inconsistent with the clustering label of non-other is exchanged to the clustering corpus with a clustering label of other. The label change combines the advantages of classification and clustering, and puts the untrusted part of the clustering corpus with a clustering label of non-other into other.

[0166] Then, it is judged whether the number of corpus with the cluster label of other exceeds 10% of the threshold of the overall training corpus. If yes, the new cluster result is executed with the above-mentioned clustering algorithm. The corpus in the cluster with the number of corpus exceeding a preset threshold N2 is marked as new cluster corpus, and the new cluster corpus is added to the training corpus of this time.

[0167] Finally, the new cluster corpus and the cluster corpus before classification adjustment are classified and merged. The process of classification and merging needs to calculate the confusion degree of the cluster corpus before and after classification adjustment. The formula for calculating the confusion degree is as follows:

[0168]

[0169] Ncatei, j represents the number of corpus with the actual intention of catei but misclassified into catej; Ncatei represents the actual number of corpus with the actual intention of catei; Ncatei represents the number of corpus predicted to be catei; and Ncatei represents the number of corpus actually predicted to be catei. catei,catej Ncatei, j represents the number of corpus with the actual intention of catei but misclassified into catej; Ncatei represents the actual number of corpus with the actual intention of catei; Ncatei represents the number of corpus predicted to be catei; and Ncatei represents the number of corpus actually predicted to be catei. catei Ncatei, j represents the number of corpus with the actual intention of catei but misclassified into catej; Ncatei represents the actual number of corpus with the actual intention of catei; Ncatei represents the number of corpus predicted to be catei; and Ncatei represents the number of corpus actually predicted to be catei. Ncatei, j represents the number of corpus with the actual intention of catei but misclassified into catej; Ncatei represents the actual number of corpus with the actual intention of catei; Ncatei represents the number of corpus predicted to be catei; and Ncatei represents the number of corpus actually predicted to be catei.

[0170] Step B4: taking the classification-adjusted cluster result as the current cluster result.

[0171] Obtaining the current cluster result and executing the following steps:

[0172] Dividing the cluster corpus in the cluster result into training corpus and prediction corpus according to the cluster label.

[0173] Training the model based on the training corpus to obtain a trained initial classification model.

[0174] Inputting the prediction corpus into the trained initial classification model for prediction to obtain a prediction score value.

[0175] Determining the precision recall F value of the cluster result based on the cluster label in the cluster result and the prediction score value.

[0176] In a specific embodiment, the PRF value of the cluster result does not reach a preset threshold, and the cluster result is unreasonable. Therefore, classification adjustment needs to be performed on the cluster label corresponding to the cluster result and the cluster corpus corresponding to the cluster label. After obtaining the classification-adjusted cluster result, the classification-adjusted cluster result is taken as the current cluster result. The steps of training and predicting the model based on the cluster label in the cluster result and the corresponding cluster corpus are repeatedly executed to determine the target intention recognition model.

[0177] Further, the clustering corpus in the clustering result is divided into training corpus and prediction corpus according to the clustering label, the classification result of the classification model is obtained through the training corpus, the prediction score value of the clustering result is obtained through the prediction corpus, the PRF value corresponding to the clustering result is determined based on the classification result and the prediction score value, and the target intent recognition model corresponding to the PRF value is determined.

[0178] In step B5, until the PRF value reaches a preset threshold value, the clustering result is reasonable, and the initial classification model is output as the target intent recognition model.

[0179] If the PRF value reaches the preset threshold value, the clustering result is reasonable, and the classification model trained by the above clustering result is output as the target intent recognition model.

[0180] In the embodiment, whether the classification model is reasonable is determined by judging the PRF value of the classification result of the classification model, whether the training data of the classification model is reasonable is further determined, and whether the clustering result containing the training data is reasonable is further determined. If it is reasonable, the classification model is directly output to obtain the target intent recognition model. If it is not reasonable, the clustering result needs to be classified and adjusted to obtain corresponding model training data, and then the target intent recognition model is obtained, so that the correctness of automatically creating the target intent recognition model is improved, and the efficiency of automatically creating the target intent recognition model is improved.

[0181] The application further provides a model construction device. Figure 9 The model construction device of the application comprises:

[0182] The acquisition module 10 is configured to acquire training corpus for constructing a model.

[0183] The clustering module 20 is configured to perform clustering processing on the training corpus based on a pre-trained clustering model to obtain a corresponding clustering result, wherein the clustering result comprises a clustering label and clustering corpus corresponding to the clustering label.

[0184] The determination module 30 is configured to perform model training and prediction based on the clustering label and the corresponding clustering corpus in the clustering result, and determine a target intent recognition model according to the model training and prediction result.

[0185] In addition, the application further provides a computer readable storage medium, preferably a computer readable storage medium, which stores a model construction program, and the model construction program is executed by a processor to implement the steps of the model construction method described above.

[0186] In the embodiments of the model construction device and medium of the application, all the technical features of the above-mentioned model construction method are included, and the description and explanation are basically the same as those of the above-mentioned model construction method, which will not be repeated here.

[0187] It should be noted that the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or system that includes the recited element.

[0188] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0189] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and the necessary general hardware platform, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) as described above, and includes a number of instructions for making a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present application.

[0190] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made by using the content of the specification and drawings of the present application, is also included in the patent protection scope of the present application.

Claims

1. A model construction method, characterized in that, The model construction method includes the following steps: Obtain the training corpus for building the model; Based on a pre-trained clustering model, the training corpus is clustered to obtain the corresponding clustering results, wherein the clustering results include clustering labels and the clustered corpus corresponding to the clustering labels; Based on the clustering labels and corresponding clustering corpora in the clustering results, the model is trained and predicted, and the target intent recognition model is determined according to the model training and prediction results. The steps of training and predicting the model based on the clustering labels and corresponding clustering corpus in the clustering results, and determining the target intent recognition model based on the model training and prediction results, include: The clustered data in the clustering results are divided into training data and prediction data according to the clustering labels. The model is trained based on the training corpus to obtain a trained initial classification model; The prediction corpus is input into the trained initial classification model for prediction, and the prediction score is obtained. Based on the cluster labels in the clustering results and the predicted scores, the Precise Recall (PRF) value of the clustering results is determined. Based on the PRF value, the corresponding target intent recognition model is determined; The step of determining the corresponding target intent recognition model based on the PRF value includes: Determine whether the PRF value reaches a preset threshold; If the PRF value reaches a preset threshold, the clustering result is reasonable, and the initial classification model is output as the target intent recognition model. If the PRF value does not reach the preset threshold, the clustering result is unreasonable. The clustering result is then adjusted to obtain the adjusted clustering result. Use the clustering result after classification adjustment as the current clustering result, and return to the execution steps: The clustered data in the clustering results are divided into training data and prediction data according to the clustering labels. The model is trained based on the training corpus to obtain a trained initial classification model; The prediction corpus is input into the trained initial classification model for prediction, and the prediction score is obtained. Based on the cluster labels in the clustering results and the predicted scores, the Precise Recall (PRF) value of the clustering results is determined. Until the PRF value reaches a preset threshold, the clustering result is considered reasonable, and the initial classification model is output as the target intent recognition model. The clustering results include corpora of other clusters labeled "other" and corpora of non-other clusters labeled "non-other". The step of classifying and adjusting the clustering results to obtain the adjusted clustering results includes: Adjust the other clustering corpus and the non-other clustering corpus to obtain the clustering corpus corresponding to the adjusted clustering labels; Calculate the confusion level of the clustered corpus corresponding to the adjusted clustering labels; When the confusion level is greater than the preset threshold T2, the non-other clustered data before and after the classification is merged and adjusted as the non-other clustered data of the current clustering result, and the other data are used as the other clustered data with the clustering label "other". The step of adjusting the "other" clustering corpus and the non-"other" clustering corpus to obtain the clustering corpus corresponding to the adjusted clustering labels includes: Obtain the predicted score of the non-other clustered corpus; If the predicted score of the non-other clustered corpus is lower than the preset threshold T1, then the clustering label of the non-other clustered corpus will be changed to other. When the number of other cluster corpora with the clustering label "other" exceeds a preset threshold N2, the intra-cluster corpora in the clustering results that exceed the preset threshold N2 are marked as new cluster corpora. We obtained the adjusted "other" clustered corpus and the adjusted non-"other" clustered corpus; The step of clustering the training corpus based on a pre-trained clustering model to obtain the corresponding clustering results includes: The training corpus is sequentially shuffled and segmented to obtain clustered sample corpus. Based on the hierarchical agglomerative clustering algorithm (HAC), the clustered sample corpus is clustered to obtain cluster labels and the corresponding clustered corpus. The steps of performing clustering processing on the clustering sample corpus based on the hierarchical agglomerative clustering algorithm (HAC) to obtain cluster labels and the corresponding clustered corpus include: The clustered sample corpus is classified, and the clustered sample corpus of the same type after classification is divided into a cluster to obtain the clustering labels corresponding to different types of clusters. Based on the clustering labels corresponding to the clusters, determine the intra-cluster corpus corresponding to different types of clustering labels; If the number of corpora within a cluster is greater than a preset threshold N1, then the corpora within a cluster are labeled with the cluster as the cluster label, and the corpora within the cluster corresponding to the cluster label are the corresponding cluster corpora. If the number of corpora within a cluster is not greater than a preset threshold N1, then the corpora within a cluster are labeled with "other", and the corpora within a cluster corresponding to the cluster label are the corresponding clustered corpora.

2. The model construction method as described in claim 1, characterized in that, The steps for obtaining the training corpus for building the model include: Obtain raw corpus from the business side; The original corpus is preprocessed to obtain training corpus for model construction; The preprocessing methods include removing stop words, using full-width characters, removing emoticons, removing greetings and meaningless questions, using standardized punctuation marks, and removing one or more of the following:

3. A model building apparatus, characterized in that, The model building apparatus includes: The acquisition module is used to acquire the training corpus for building the model; The clustering module is used to perform clustering processing on the training corpus based on a pre-trained clustering model to obtain the corresponding clustering results, wherein the clustering results include clustering labels and the clustered corpus corresponding to the clustering labels; The determination module is used to train and predict the model based on the clustering labels in the clustering results and the corresponding clustering corpus, and to determine the target intent recognition model based on the model training and prediction results. The step of training and predicting the model based on the clustering labels and corresponding clustering corpus in the clustering results, and determining the target intent recognition model based on the model training and prediction results, includes: The clustered data in the clustering results are divided into training data and prediction data according to the clustering labels. The model is trained based on the training corpus to obtain a trained initial classification model; The prediction corpus is input into the trained initial classification model for prediction, and the prediction score is obtained. Based on the cluster labels in the clustering results and the predicted scores, the Precise Recall (PRF) value of the clustering results is determined. Based on the PRF value, the corresponding target intent recognition model is determined; The step of determining the corresponding target intent recognition model based on the PRF value includes: Determine whether the PRF value reaches a preset threshold; If the PRF value reaches a preset threshold, the clustering result is reasonable, and the initial classification model is output as the target intent recognition model. If the PRF value does not reach the preset threshold, the clustering result is unreasonable. The clustering result is then adjusted to obtain the adjusted clustering result. Use the clustering result after classification adjustment as the current clustering result, and return to the execution steps: The clustered data in the clustering results are divided into training data and prediction data according to the clustering labels. The model is trained based on the training corpus to obtain a trained initial classification model; The prediction corpus is input into the trained initial classification model for prediction, and the prediction score is obtained. Based on the cluster labels in the clustering results and the predicted scores, the Precise Recall (PRF) value of the clustering results is determined. Until the PRF value reaches a preset threshold, the clustering result is considered reasonable, and the initial classification model is output as the target intent recognition model. The clustering results include corpora of other clusters labeled "other" and corpora of non-other clusters labeled "non-other". The process of classifying and adjusting the clustering results to obtain the adjusted clustering results includes: Adjust the other clustering corpus and the non-other clustering corpus to obtain the clustering corpus corresponding to the adjusted clustering labels; Calculate the confusion level of the clustered corpus corresponding to the adjusted clustering labels; When the confusion level is greater than the preset threshold T2, the non-other clustered data before and after the classification is merged and adjusted as the non-other clustered data of the current clustering result, and the other data are used as the other clustered data with the clustering label "other". The step of adjusting the "other" clustering corpus and the non-"other" clustering corpus to obtain the clustering corpus corresponding to the adjusted clustering labels includes: Obtain the predicted score of the non-other clustered corpus; If the predicted score of the non-other clustered corpus is lower than the preset threshold T1, then the clustering label of the non-other clustered corpus will be changed to other. When the number of other cluster corpora with the clustering label "other" exceeds a preset threshold N2, the intra-cluster corpora in the clustering results that exceed the preset threshold N2 are marked as new cluster corpora. We obtained the adjusted "other" clustered corpus and the adjusted non-"other" clustered corpus; The clustering process based on the pre-trained clustering model, which performs clustering on the training corpus to obtain the corresponding clustering results, includes: The training corpus is sequentially shuffled and segmented to obtain clustered sample corpus. Based on the hierarchical agglomerative clustering algorithm (HAC), the clustered sample corpus is clustered to obtain cluster labels and the corresponding clustered corpus. The hierarchical agglomerative clustering algorithm (HAC) performs clustering processing on the clustered sample corpus to obtain cluster labels and the corresponding clustered corpus, including: The clustered sample corpus is classified, and the clustered sample corpus of the same type after classification is divided into a cluster to obtain the clustering labels corresponding to different types of clusters. Based on the clustering labels corresponding to the clusters, determine the intra-cluster corpus corresponding to different types of clustering labels; If the number of corpora within a cluster is greater than a preset threshold N1, then the corpora within a cluster are labeled with the cluster as the cluster label, and the corpora within the cluster corresponding to the cluster label are the corresponding cluster corpora. If the number of corpora within a cluster is not greater than a preset threshold N1, then the corpora within a cluster are labeled with "other", and the corpora within a cluster corresponding to the cluster label are the corresponding clustered corpora.

4. A model building device, characterized in that, The model building device includes: a memory, a processor, and a model building program stored in the memory and executable on the processor, wherein the model building program, when executed by the processor, implements the steps of the model building method as described in any one of claims 1 to 2.

5. A medium, said medium being a computer-readable storage medium, characterized in that, The computer-readable storage medium stores a model building program, which, when executed by a processor, implements the steps of the model building method as described in any one of claims 1 to 2.

Images