Intelligent scene dialogue analysis method and system based on model recognition

Abstract

The invention discloses an intelligent scene dialogue analysis method and system based on model recognition, and relates to the technical field of scene dialogue analysis, and the method comprises the steps: collecting historical dialogue data on an intelligent scene dialogue platform, and obtaining historical dialogue standard data through text cleaning, word segmentation and part-of-speech tagging, data tagging and other processing; dividing data according to dialogue text length information, selecting a long and short-term memory network model, a gating loop unit model and a Transform model to perform a contrast experiment to construct an initial model, and selecting the model with the optimal performance as a basic model; and finally, after the real-time dialogue data of the user is preprocessed, a corresponding basic model is selected for analysis according to the dialogue length. The system comprises an acquisition module, a data processing module, a basic model construction module and an analysis module to realize the process of the method. The method and the system aim to improve the accuracy and the efficiency of intelligent scene dialogue analysis, and overcome the defects in the aspects of data processing and model application in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of scene dialogue analysis, and in particular to an intelligent scene dialogue analysis method and system based on model recognition. Background Art

[0002] With the development of artificial intelligence technology, intelligent scene dialogue applications are becoming more and more widespread, such as intelligent customer service, smart home control, etc. However, existing dialogue analysis methods are difficult to accurately understand complex and diverse dialogue scenarios. The intelligent scene dialogue analysis method and system based on model recognition can use advanced model recognition technology to accurately identify dialogue scenarios, understand semantics and intentions, and improve the efficiency and quality of dialogue interaction. It is of great significance in improving user experience and optimizing services.

[0003] The existing intelligent scene dialogue analysis methods and systems based on model recognition are difficult to fully handle the noise and error information in the dialogue data when faced with dialogue texts containing multiple languages. At the same time, they are unable to accurately segment words and mark parts of speech. Therefore, it is necessary to provide an intelligent scene dialogue analysis method and system based on model recognition to solve the above-mentioned problems. Summary of the invention

[0004] In order to solve the above technical problems, a method and system for intelligent scene dialogue analysis based on model recognition are provided. This technical solution solves the problem that the existing method and system for intelligent scene dialogue analysis based on model recognition proposed in the above background technology is difficult to fully process the noise and error information in the dialogue data when faced with dialogue texts containing multiple languages, and it is impossible to accurately segment words and mark parts of speech.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is: An intelligent scene dialogue analysis method based on model recognition, comprising: Collect historical conversation data on the intelligent scene conversation platform, perform text cleaning on the historical conversation data, and obtain historical conversation text information; Perform word segmentation and part-of-speech tagging on the historical dialogue text information to obtain the part-of-speech information of the historical dialogue text; According to the historical conversation text information and the part-of-speech information of the historical conversation text, the historical conversation data is annotated to obtain the historical conversation standard data; Obtaining the length information of each historical conversation text in the historical conversation standard data, and recording it as the historical conversation text length sub-information; Dividing the historical conversation standard data according to the historical conversation text length sub-information to obtain the historical conversation standard sub-data; Select the corresponding model to conduct a comparative experiment on the historical dialogue standard sub-data, complete the construction of the initial model, and obtain the accuracy of each initial model; According to the accuracy of each initial model, the initial model with the best performance is selected as the basic model to obtain the basic model of intelligent scene dialogue analysis; Acquire real-time user conversation data and perform data preprocessing on the real-time user conversation data to obtain conversation data to be analyzed. Finally, use the basic model of intelligent scenario conversation analysis to analyze the conversation data to be analyzed.

[0006] In an optional embodiment, the collecting of historical conversation data on the intelligent scene conversation platform, performing text cleaning on the historical conversation data, and obtaining historical conversation text information specifically includes: Obtain the type information of the smart scene dialogue platform to determine the corresponding interface. Use the SDK officially provided by the smart scene dialogue platform to collect public dialogue data and obtain historical dialogue data. Based on the intelligent scene dialogue platform, the preset scene category information corresponding to the historical dialogue data is obtained, and the historical dialogue data is preliminarily classified according to the preset scene category information to obtain the historical dialogue data of different preset scene categories; The historical conversation data of different preset scenario categories are stored in the same data set, and the text of the historical conversation data in the data set is cleaned in turn, including: S1.1. Use regular expressions to remove noise information from the corresponding conversation text in the historical conversation data; S1.2, according to the preset scene category information corresponding to the historical dialogue data, calling the corresponding corpus, and using the historical dialogue data and the corresponding corpus to train an error correction model corresponding to the preset scene category information, and then correcting the dialogue text in the historical dialogue data through the error correction model, and using the dialogue text in the historical dialogue data after error correction as the historical dialogue text information; Among them, the intelligent scene dialogue platform includes an online customer service platform, an intelligent voice assistant and a social media platform.

[0007] In an optional embodiment, the word segmentation and part-of-speech tagging of the historical conversation text information to obtain the part-of-speech information of the historical conversation text specifically includes: Obtaining language type information corresponding to the dialogue text in the historical dialogue data after error correction, and performing Chinese-English segmentation processing on the dialogue text in the historical dialogue data after error correction according to the language type information, and recording segmentation position information at the same time; Segment the Chinese and English texts in the historical dialogue text to obtain the historical dialogue Chinese text and the historical dialogue English text; Select the corresponding word segmentation tool to perform word segmentation on the Chinese text of historical dialogues and the English text of historical dialogues, including: S2.1. Based on the intelligent scene dialogue platform, the preset scene category information corresponding to the historical dialogue data is obtained again, thereby obtaining the preset scene field information; S2.2, according to the preset scene domain information, load the corresponding custom Chinese dictionary, and use the Jieba word segmentation tool to segment the historical conversation Chinese text, cut the Chinese text into individual Chinese words, and then store the Chinese words corresponding to each historical conversation Chinese text into the same list to obtain the first historical conversation text information; S2.2, load the corresponding custom English dictionary, use the word segmentation function in the NLTK library to perform word segmentation on the historical conversation English text, cut the English text into individual English words, and then store the English words corresponding to each historical conversation English text into the same list to obtain the second historical conversation text information; Part-of-speech tagging is performed on the first historical conversation text information and the second historical conversation text information, including: S3.1. Based on the first historical conversation text information, obtain all Chinese pinyins corresponding to the Chinese words in the first historical conversation text information, and obtain all homophonic Chinese words corresponding to the Chinese words based on the Chinese pinyins; S3.2, based on the second historical conversation text information, traverse each English word in the second historical conversation text information, and obtain all recombined English words corresponding to each English word; S3.3, according to the preset scene category information corresponding to the historical conversation data, the custom Chinese dictionary corresponding to the Chinese words and the custom English dictionary corresponding to the English text are loaded respectively, and then all homophonic Chinese words and reorganized English words are screened by using the preset scene category information, the custom Chinese dictionary and the custom English dictionary, and the homophonic Chinese words and reorganized English words contained in the custom Chinese dictionary and the custom English dictionary are extracted to obtain the first historical conversation standard text information and the second historical conversation standard text information; S3.4, using the Jieba word segmentation tool to perform part-of-speech tagging on the Chinese words in the first historical dialogue standard text information, using the part-of-speech tagger in the NLTK library to perform part-of-speech tagging on the English words in the second historical dialogue standard text information, and using the part-of-speech tagging results of the first historical dialogue standard text information and the second historical dialogue standard text information to record the part-of-speech information of the historical dialogue text; S3.5. By segmenting the position information, the first historical dialogue standard text information and the second historical dialogue standard text information after part-of-speech tagging are reorganized to update the historical dialogue text information.

[0008] In an optional embodiment, the data annotation of the historical conversation data is performed according to the historical conversation text information and the historical conversation text part-of-speech information to obtain the historical conversation standard data, specifically including: The updated historical conversation text information is sent to professional annotators for annotation to obtain manual part-of-speech annotation information of the historical conversation text; Compare the part-of-speech information of the historical conversation text with the part-of-speech manual annotation information of the historical conversation text, remove the part-of-speech information of the historical conversation text that does not match the manual annotation information of the historical conversation, and obtain the standard part-of-speech information of the historical conversation text; The historical conversation data is annotated using the standard part-of-speech information of the historical conversation text to obtain the standard data of the historical conversation.

[0009] In an optional embodiment, the length information of each historical conversation text in the historical conversation standard data is obtained, recorded as historical conversation text length sub-information, and specifically includes: Obtain the total number of Chinese characters and the total number of letters of English words in each historical conversation text in the historical conversation standard data; The sum of the total number of Chinese words and the total number of letters in English words in each historical dialogue text in the historical dialogue standard data is used as the length information of each historical dialogue text in the historical dialogue standard data to obtain the historical dialogue text length sub-information.

[0010] In an optional embodiment, the historical conversation standard data is divided according to the historical conversation text length sub-information to obtain the historical conversation standard sub-data, specifically including: Traverse all the historical conversation text length sub-information to obtain the median value of all historical conversation text lengths; Classify the historical conversation standard data whose historical conversation text length is greater than or equal to the median of all historical conversation text lengths as the first historical conversation standard data, and classify the historical conversation standard data whose historical conversation text length is less than the median of all historical conversation text lengths as the second historical conversation standard data; The first historical conversation standard data and the second historical conversation standard data are respectively stored in different data sets to obtain historical conversation standard sub-data.

[0011] In an optional embodiment, the selecting of the corresponding model to conduct a comparative experiment on the historical dialogue standard sub-data, completing the construction of the initial model at the same time, and obtaining the accuracy rate corresponding to each initial model specifically includes: Extracting historical conversation standard data corresponding to the first historical conversation standard data and the second historical conversation standard data from the historical conversation standard sub-data respectively, and dividing them into 70% training set, 15% validation set and 15% test set respectively, to obtain a first training set, a first validation set, a first test set, a second training set, a second validation set and a second test set; Obtain a long short-term memory network model, a gated recurrent unit model, and a Transformer model respectively, set initial hyperparameters for the long short-term memory network model, the gated recurrent unit model, and the Transformer model, and complete the construction of the initial model; The initial model is trained using the first training set and the second training set respectively, and the first validation set, the first test set, the second validation set and the second test set are used to train the initial model. , get the accuracy corresponding to each initial model, where TP is a true positive example, TN is a true negative example, FP is a false positive example, and FN is a false negative example.

[0012] In an optional embodiment, the step of selecting the initial model with the best performance as the basic model according to the accuracy rate corresponding to each initial model to obtain the basic model for intelligent scene dialogue analysis specifically includes: Traverse all initial models and take the initial model with the highest accuracy as the initial model with the best performance, that is, the basic model, so as to obtain the basic model of intelligent scene dialogue analysis; Among them, the intelligent scene dialogue analysis basic model includes a long dialogue analysis basic model and a short dialogue analysis basic model.

[0013] In an optional embodiment, the real-time user conversation data is obtained, and data preprocessing is performed on the real-time user conversation data to obtain conversation data to be analyzed, and finally the conversation data to be analyzed is analyzed using the intelligent scenario conversation analysis basic model, specifically including: Obtaining the length information of the conversation text to be analyzed in the conversation data to be analyzed, and obtaining the median value of the length of all historical conversation texts; If the length of the conversation text to be analyzed is greater than or equal to the median length of all historical conversation texts, the long conversation analysis basic model is used to analyze the conversation data to be analyzed; If the length of the conversation text to be analyzed is less than the median length of all historical conversation texts, the short conversation analysis basic model is used to analyze the conversation data to be analyzed.

[0014] Furthermore, an intelligent scene dialogue analysis system based on model recognition is proposed, which is used to implement any of the above analysis methods, including: A collection module, which is used to collect historical conversation data on the intelligent scene conversation platform; A data processing module, the data processing module is used to perform text cleaning on the historical conversation data to obtain historical conversation text information, perform word segmentation and part-of-speech tagging on the historical conversation text information to obtain the part-of-speech information of the historical conversation text, perform data tagging on the historical conversation data according to the historical conversation text information and the part-of-speech information of the historical conversation text to obtain historical conversation standard data, obtain the length information of each historical conversation text in the historical conversation standard data, record it as historical conversation text length sub-information, divide the historical conversation standard data according to the historical conversation text length sub-information, and obtain historical conversation standard sub-data; A basic model construction module, which is used to select a corresponding model to conduct a comparative experiment on the historical dialogue standard sub-data, complete the construction of the initial model, and obtain the accuracy rate corresponding to each initial model. According to the accuracy rate corresponding to each initial model, the initial model with the best performance is selected as the basic model to obtain the basic model of intelligent scene dialogue analysis; The analysis module is used to obtain the user's real-time conversation data, perform data preprocessing on the user's real-time conversation data, obtain the conversation data to be analyzed, and finally analyze the conversation data to be analyzed using the intelligent scene conversation analysis basic model.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This solution proposes an intelligent scene dialogue analysis method and system based on model recognition, which removes noise information of dialogue text in historical dialogue data through regular expressions, and then calls the corpus training error correction model according to preset scene category information to correct the dialogue text, thereby obtaining high-quality historical dialogue text information; This scheme proposes an intelligent scene dialogue analysis method and system based on model recognition. It performs Chinese and English segmentation according to language type information, loads a custom dictionary based on preset scene domain information, and uses the Jieba word segmentation tool and the word segmentation function and part-of-speech tagger in the NLTK library for processing. It also improves the accuracy of word segmentation and part-of-speech tagging by filtering homophones and reorganizing words, and extracts text features more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of an intelligent scene dialogue analysis method based on model recognition proposed by the present invention; Figure 2 A flowchart of the present invention for performing word segmentation on historical conversation text information; Figure 3 A flowchart of part-of-speech tagging of historical conversation text information in the present invention; Figure 4 This is a system framework diagram of an intelligent scene dialogue analysis system based on model recognition proposed in the present invention. DETAILED DESCRIPTION

[0017] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0018] Reference Figure 1 - Figure 4 As shown, a method for intelligent scene dialogue analysis based on model recognition includes: Collect historical conversation data on the intelligent scene conversation platform, perform text cleaning on the historical conversation data, and obtain historical conversation text information; Perform word segmentation and part-of-speech tagging on the historical dialogue text information to obtain the part-of-speech information of the historical dialogue text; According to the historical conversation text information and the part-of-speech information of the historical conversation text, the historical conversation data is annotated to obtain the historical conversation standard data; Obtaining the length information of each historical conversation text in the historical conversation standard data, and recording it as the historical conversation text length sub-information; Dividing the historical conversation standard data according to the historical conversation text length sub-information to obtain the historical conversation standard sub-data; Select the corresponding model to conduct a comparative experiment on the historical dialogue standard sub-data, complete the construction of the initial model, and obtain the accuracy of each initial model; According to the accuracy of each initial model, the initial model with the best performance is selected as the basic model to obtain the basic model of intelligent scene dialogue analysis; Acquire real-time user conversation data and perform data preprocessing on the real-time user conversation data to obtain conversation data to be analyzed. Finally, use the basic model of intelligent scenario conversation analysis to analyze the conversation data to be analyzed.

[0019] Furthermore, historical conversation data is collected on the intelligent scene conversation platform, and the text of the historical conversation data is cleaned to obtain historical conversation text information, including: Obtain the type information of the smart scene dialogue platform to determine the corresponding interface. Use the SDK officially provided by the smart scene dialogue platform to collect public dialogue data and obtain historical dialogue data. Based on the intelligent scene dialogue platform, the preset scene category information corresponding to the historical dialogue data is obtained, and the historical dialogue data is preliminarily classified according to the preset scene category information to obtain the historical dialogue data of different preset scene categories; The historical conversation data of different preset scenario categories are stored in the same data set, and the text of the historical conversation data in the data set is cleaned in turn, including: S1.1. Use regular expressions to remove noise information from the corresponding conversation text in the historical conversation data; S1.2, according to the preset scene category information corresponding to the historical dialogue data, calling the corresponding corpus, and using the historical dialogue data and the corresponding corpus to train an error correction model corresponding to the preset scene category information, and then correcting the dialogue text in the historical dialogue data through the error correction model, and using the dialogue text in the historical dialogue data after error correction as the historical dialogue text information; Among them, the intelligent scenario dialogue platform includes online customer service platform, intelligent voice assistant and social media platform.

[0020] Specifically, in this embodiment, first, the specific type of the intelligent scene dialogue platform must be determined, such as whether it is an online customer service platform (such as a customer service system used by enterprises to communicate with customers), an intelligent voice assistant (such as a voice interaction assistant on a mobile phone), or a social media platform (such as WeChat, Weibo, etc.). After clarifying the platform type, find the corresponding interface according to the type. Use the software development kit (SDK) officially provided by the intelligent scene dialogue platform to collect the public dialogue data on the platform through this interface, so that the historical dialogue data is obtained. Then, based on the historical dialogue data that has been obtained, the preset scene category information corresponding to these historical dialogue data is obtained from the intelligent scene dialogue platform. For example, in an online customer service platform, the preset scene categories may include product consultation, after-sales service, complaint suggestions, etc.; in a social media platform, there may be scene categories such as daily communication, topic discussion, and advertising. Then, based on these preset scene category information, the historical dialogue data is preliminarily classified, and the historical dialogue data belonging to the same preset scene category is classified into one category, thereby obtaining historical dialogue data of different preset scene categories. Afterwards, the historical dialogue data of these different preset scene categories are stored in the same data set. Next, perform text cleaning operations on the historical conversation data in this data set: S1.1: Use regular expressions to process the corresponding conversation texts in the historical conversation data. Regular expressions are a powerful text processing tool that can match and remove noise information in the conversation text according to specific rules, such as some special characters, garbled characters, irrelevant punctuation marks, etc., making the conversation text cleaner and neater.

[0021] S1.2: According to the preset scene category information corresponding to the historical dialogue data obtained previously, call the corresponding corpus. The corpus stores a large amount of language data related to the preset scene. Then, use the historical dialogue data and the corresponding corpus to train an error correction model, which is specifically for the preset scene category information. After the training is completed, use this error correction model to correct the dialogue text in the historical dialogue data, find and correct the errors in the dialogue text, such as typos, grammatical errors, etc. Finally, the dialogue text in the historical dialogue data that has been corrected is determined as the historical dialogue text information.

[0022] Furthermore, the historical dialogue text information is segmented and POS tagged to obtain the POS information of the historical dialogue text, including: Obtaining language type information corresponding to the dialogue text in the historical dialogue data after error correction, and performing Chinese-English segmentation processing on the dialogue text in the historical dialogue data after error correction according to the language type information, and recording segmentation position information at the same time; Segment the Chinese and English texts in the historical dialogue text to obtain the historical dialogue Chinese text and the historical dialogue English text; Select the corresponding word segmentation tool to perform word segmentation on the Chinese text of historical dialogues and the English text of historical dialogues, including: S2.1. Based on the intelligent scene dialogue platform, the preset scene category information corresponding to the historical dialogue data is obtained again, thereby obtaining the preset scene field information; S2.2, according to the preset scene domain information, load the corresponding custom Chinese dictionary, and use the Jieba word segmentation tool to segment the historical conversation Chinese text, cut the Chinese text into individual Chinese words, and then store the Chinese words corresponding to each historical conversation Chinese text into the same list to obtain the first historical conversation text information; S2.2, load the corresponding custom English dictionary, use the word segmentation function in the NLTK library to perform word segmentation on the historical conversation English text, cut the English text into individual English words, and then store the English words corresponding to each historical conversation English text into the same list to obtain the second historical conversation text information; Part-of-speech tagging is performed on the first historical conversation text information and the second historical conversation text information, including: S3.1. Based on the first historical conversation text information, obtain all Chinese pinyins corresponding to the Chinese words in the first historical conversation text information, and obtain all homophonic Chinese words corresponding to the Chinese words based on the Chinese pinyins; S3.2, based on the second historical conversation text information, traverse each English word in the second historical conversation text information, and obtain all recombined English words corresponding to each English word; S3.3, according to the preset scene category information corresponding to the historical conversation data, the custom Chinese dictionary corresponding to the Chinese words and the custom English dictionary corresponding to the English text are loaded respectively, and then all homophonic Chinese words and reorganized English words are screened by using the preset scene category information, the custom Chinese dictionary and the custom English dictionary, and the homophonic Chinese words and reorganized English words contained in the custom Chinese dictionary and the custom English dictionary are extracted to obtain the first historical conversation standard text information and the second historical conversation standard text information; S3.4, using the Jieba word segmentation tool to perform part-of-speech tagging on the Chinese words in the first historical dialogue standard text information, using the part-of-speech tagger in the NLTK library to perform part-of-speech tagging on the English words in the second historical dialogue standard text information, and using the part-of-speech tagging results of the first historical dialogue standard text information and the second historical dialogue standard text information to record the part-of-speech information of the historical dialogue text; S3.5. By segmenting the position information, the first historical dialogue standard text information and the second historical dialogue standard text information after part-of-speech tagging are reorganized to update the historical dialogue text information.

[0023] Specifically, in this embodiment, the language type information is obtained, and the language type of the dialogue text can be determined by simple character features. For example, if most of the characters in the text are letters within the ASCII code range, it is determined to be English; if it contains a large number of Chinese characters (determined by the Unicode range, such as \u4e00-\u9fa5), it is determined to be Chinese. For a mixed text of Chinese and English, the character distribution ratio can be further analyzed to determine the main language type. Some mature language detection libraries can also be used, such as the langdetect library in Python, which can easily detect the language type of the text. For the Chinese and English segmentation and recording of the position, traverse the dialogue text and distinguish between Chinese and English according to the Unicode range of the characters. When encountering Chinese characters, mark it as the Chinese part; when encountering English letters, mark it as the English part. At the same time, record the starting and ending positions of the segmentation.

[0024] It is understandable that in order to obtain the preset scene domain information in the word segmentation process, the preset scene category information corresponding to the historical dialogue data is obtained from the configuration file or database of the intelligent scene dialogue platform. For example, in the e-commerce customer service scenario, the preset scene categories may include product consultation, order inquiry, after-sales processing, etc. Based on these scene categories, the corresponding preset scene domain information is further determined, such as the type of product, the order process, etc. For Chinese word segmentation, you need to install the jieba library, use the jieba.load_userdict() method to load the custom Chinese dictionary, and then use the jieba.cut() method to segment the Chinese text of the historical dialogue. For English word segmentation, you need to install the NLTK library and use the nltk.tokenize.word_tokenize() method to segment the English text of the historical dialogue.

[0025] When tagging parts of speech, you can obtain homophones of Chinese characters and reorganized English words. You can use some open source pinyin libraries, such as the pypinyin library, to obtain the pinyin of Chinese characters, and then search for homophones by pinyin to obtain Chinese homophones. For reorganized English words, reorganized English words are obtained by traversing the permutations and combinations of English letters. For filtering standard text information, you need to load a custom Chinese dictionary and an English dictionary, traverse homophones of Chinese characters and reorganized English words, and determine whether they are in the corresponding dictionary. If they are in the dictionary, extract them to obtain the first historical dialogue standard text information and the second historical dialogue standard text information.

[0026] For Chinese part-of-speech tagging, the jieba.posseg module can be used to tag the Chinese words in the first historical dialogue standard text information. For English part-of-speech tagging, the nltk.pos_tag() method in the NLTK library is used to tag the English words in the second historical dialogue standard text information. Finally, according to the previously recorded segmentation position information, the first historical dialogue standard text information and the second historical dialogue standard text information after part-of-speech tagging are reorganized to update the historical dialogue text information.

[0027] Furthermore, based on the historical conversation text information and the part-of-speech information of the historical conversation text, the historical conversation data is annotated to obtain the historical conversation standard data, which specifically includes: The updated historical conversation text information is sent to professional annotators for annotation to obtain manual part-of-speech annotation information of the historical conversation text; Compare the part-of-speech information of the historical conversation text with the part-of-speech manual annotation information of the historical conversation text, remove the part-of-speech information of the historical conversation text that does not match the manual annotation information of the historical conversation, and obtain the standard part-of-speech information of the historical conversation text; The historical conversation data is annotated using the standard part-of-speech information of the historical conversation text to obtain the standard data of the historical conversation.

[0028] Specifically, the part-of-speech manual tagging information of historical dialogue texts is obtained: first, the updated historical dialogue text information obtained after a series of previous processing (including text cleaning, Chinese-English segmentation, word segmentation, part-of-speech tagging, and text information update) is sent to professional taggers in an appropriate way (such as using a special data tagging platform, sending emails, or using specific project management tools). These professional taggers have relevant language knowledge and tagging experience. They will tag each word in the updated historical dialogue text information according to professional standards and understanding of text semantics, thereby obtaining the part-of-speech manual tagging information of historical dialogue texts. For example, in a dialogue text about e-commerce consultation, the tagger will clearly point out that the part of speech of the word "commodity" is a noun, and the part of speech of the word "purchase" is a verb, etc. Obtain the standard part-of-speech information of historical dialogue texts: Then, the part-of-speech information of historical dialogue texts obtained by automatic processing of the program (that is, the part-of-speech information obtained after Chinese-English segmentation, word segmentation, part-of-speech tagging, etc.) is carefully compared with the part-of-speech manual tagging information of historical dialogue texts given by professional taggers. During the comparison process, the word-by-word part-of-speech tagging results of the two are checked to see if they are consistent. For those parts of speech information of historical dialogue text that do not match the manual tagging information of historical dialogues, that is, the parts where the program's automatic tagging results are different from the manual tagging results, they are removed from the original part-of-speech information of historical dialogue texts. After such screening and processing, the remaining part-of-speech information of historical dialogue texts is determined as the standard part-of-speech information of historical dialogue texts. For example, if the program automatically tags the word "fast" as a noun, and the manual tagging is an adjective, since the two do not match, the result of "noun" automatically marked by the program is removed, and the tagging result of "adjective" that matches the manual tagging is finally retained. Obtaining the standard data of historical dialogues: Finally, the obtained standard part-of-speech information of historical dialogue texts is used to perform comprehensive data tagging on the original historical dialogue data. This means that the standard part-of-speech information of historical dialogue texts is accurately applied to each section of historical dialogue data, and each word in the historical dialogue data is given an accurate, manually reviewed and confirmed part-of-speech tag. After such labeling operations, standard data of historical conversations are obtained. These data can be used for subsequent model training, analysis and other tasks, providing a high-quality data foundation for intelligent scene conversation analysis.

[0029] Furthermore, the length information of each historical conversation text in the historical conversation standard data is obtained and recorded as the historical conversation text length sub-information, which specifically includes: Obtain the total number of Chinese characters and the total number of letters of English words in each historical conversation text in the historical conversation standard data; The sum of the total number of Chinese words and the total number of letters in English words in each historical dialogue text in the historical dialogue standard data is used as the length information of each historical dialogue text in the historical dialogue standard data to obtain the historical dialogue text length sub-information.

[0030] Specifically, first, for each historical dialogue text in the historical dialogue standard data, the total number of Chinese words and the total number of letters in English words need to be counted. Count the total number of Chinese words: traverse the Chinese words in each historical dialogue text. Since the text has been segmented in the previous processing, it is possible to check whether the words in the text are Chinese (by judging the Unicode encoding range of the characters, for example, the characters between \u4e00 and \u9fa5 are Chinese characters). Each time a Chinese word is recognized, the counter is incremented by 1. After all the words in the historical dialogue text are traversed, the value recorded by the counter is the total number of Chinese words in the text. For example, a text "I like apples", in which "I", "like" and "apple" are all Chinese words, the counter starts from 0 and increments by 1 in sequence, and finally the total number of Chinese words is 3. Count the total number of letters in English words: traverse the English words in each historical dialogue text. For each English word, calculate the number of letters it contains. This can be achieved by obtaining the character length of the word (in programming, the length function of a string can return the number of characters in a word). The number of letters in each English word is accumulated into a sum variable. After traversing all the English words in the historical dialogue text, the value recorded in the sum variable is the total number of letters in the English words in this text. For example, a text "I like apples", where the length of "I" is 1, the length of "like" is 4, and the length of "apples" is 6, the total number of letters in the English word after accumulation is 1 + 4 + 6 = 11. Then, the total number of Chinese words and the total number of letters in the English word obtained in each historical dialogue text are added. The result of this addition is used as the length information of the historical dialogue text. The above addition operation is repeated for each historical dialogue text, so as to obtain the length information corresponding to each historical dialogue text. This length information is called the historical dialogue text length sub-information, and the historical dialogue standard data can be further processed and analyzed based on this sub-information, such as classification by length.

[0031] Furthermore, the historical conversation standard data is divided according to the historical conversation text length sub-information to obtain the historical conversation standard sub-data, which specifically includes: Traverse all the historical conversation text length sub-information to obtain the median value of all historical conversation text lengths; Classify the historical conversation standard data whose historical conversation text length is greater than or equal to the median of all historical conversation text lengths as the first historical conversation standard data, and classify the historical conversation standard data whose historical conversation text length is less than the median of all historical conversation text lengths as the second historical conversation standard data; The first historical conversation standard data and the second historical conversation standard data are respectively stored in different data sets to obtain historical conversation standard sub-data.

[0032] Specifically, get the median of all historical conversation text lengths: First, we already have all the historical conversation text length sub-information, which represents the length of each historical conversation text. We need to sort out these length data. Collect all the historical conversation text length sub-information into a list or set, which contains the text length values ​​corresponding to each historical conversation. Sort the length values ​​in this set from small to large. The purpose of sorting is to facilitate the subsequent determination of the median.

[0033] Next, determine the median value based on the number of sorted data. If the number of data is an odd number, the median is the value in the middle after sorting; if the number of data is an even number, the median is usually the average of the two middle values. Through this calculation, we get the median value of the length of all historical conversation texts.

[0034] Divide the standard data of historical conversations: prepare the standard data of historical conversations, which are the historical conversation data with annotated information obtained after a series of processing (such as text cleaning, part-of-speech tagging, etc.). Check the sub-information of the length of the historical conversation text corresponding to each section of the standard data of historical conversations in turn. Compare the length with the median we just calculated. If the text length of a certain section of historical conversation is greater than or equal to the median, then classify this section of the standard data of historical conversations into the first standard data of historical conversations. This part of data represents a relatively long historical conversation. If the text length of a certain section of historical conversations is less than the median, classify it into the second standard data of historical conversations. This part of data represents a relatively short historical conversation. Store the divided data: prepare two different data sets, for example, you can use data structures such as lists, dictionaries, or database tables to store data. Store the first standard data of historical conversations in one of the data sets, which is specifically used to store longer standard data of historical conversations. Store the second standard data of historical conversations in another data set, which is specifically used to store shorter standard data of historical conversations. After such storage operations, we obtain standard sub-data of historical conversations, which are stored in two different data sets respectively, so as to facilitate subsequent different processing and analysis of long and short conversation data, such as using different models for training.

[0035] Furthermore, the corresponding model is selected to conduct comparative experiments on the historical dialogue standard sub-data, and the construction of the initial model is completed at the same time, and the accuracy rate corresponding to each initial model is obtained, including: Extracting historical conversation standard data corresponding to the first historical conversation standard data and the second historical conversation standard data from the historical conversation standard sub-data respectively, and dividing them into 70% training set, 15% validation set and 15% test set respectively, to obtain a first training set, a first validation set, a first test set, a second training set, a second validation set and a second test set; Obtain a long short-term memory network model, a gated recurrent unit model, and a Transformer model respectively, set initial hyperparameters for the long short-term memory network model, the gated recurrent unit model, and the Transformer model, and complete the construction of the initial model; The initial model is trained using the first training set and the second training set respectively, and the first validation set, the first test set, the second validation set and the second test set are used to train the initial model. , get the accuracy corresponding to each initial model, where TP is a true positive example, TN is a true negative example, FP is a false positive example, and FN is a false negative example.

[0036] Specifically, the initial hyperparameters are set for the three models (LSTM, GRU, and Transformer) obtained. Hyperparameters are parameters that need to be manually set before model training, such as learning rate, number of hidden layers, number of neurons, number of iterations, etc. Different hyperparameter settings will affect the training process and performance of the model. By setting the initial hyperparameters reasonably, the initial model is constructed so that the model has the conditions to start training. True positive examples: the number of samples that the model predicts as positive examples (for example, predicting that the conversation belongs to a specific category) and the actual situation is indeed a positive example. For example, when judging whether a conversation is a customer complaint, the model predicts "it is a complaint", and the actual conversation is indeed a customer complaint. This is counted as a true positive example. True negative examples: the number of samples that the model predicts as negative examples (for example, predicting that the conversation does not belong to a specific category), and the actual situation is indeed a negative example. For example, the model predicts that a conversation is "not a complaint", and it is actually not a complaint. This is a true negative example. False positive examples: the number of samples that the model predicts as positive examples, but the actual situation is a negative example. For example, the model predicts that a conversation is "a complaint", but in fact, this conversation is not a complaint. This is a false positive example. False negative examples: The number of samples that the model predicts as negative examples, but are actually positive examples. For example, if the model predicts that a conversation is "not a complaint", but it is actually a customer complaint, this is a false negative example.

[0037] Furthermore, the initial model with the best performance is selected as the basic model according to the accuracy corresponding to each initial model, and the basic model of intelligent scene dialogue analysis is obtained, which specifically includes: Traverse all initial models and take the initial model with the highest accuracy as the initial model with the best performance, that is, the basic model, so as to obtain the basic model of intelligent scene dialogue analysis; Among them, the basic model of intelligent scene dialogue analysis includes the basic model of long dialogue analysis and the basic model of short dialogue analysis.

[0038] Furthermore, the real-time user conversation data is obtained and preprocessed to obtain the conversation data to be analyzed. Finally, the conversation data to be analyzed is analyzed using the intelligent scenario conversation analysis basic model, which specifically includes: Obtaining the length information of the conversation text to be analyzed in the conversation data to be analyzed, and obtaining the median value of the length of all historical conversation texts; If the length of the conversation text to be analyzed is greater than or equal to the median length of all historical conversation texts, the long conversation analysis basic model is used to analyze the conversation data to be analyzed; If the length of the conversation text to be analyzed is less than the median length of all historical conversation texts, the short conversation analysis basic model is used to analyze the conversation data to be analyzed.

[0039] Specifically, to determine the relationship between the length of the conversation text to be analyzed and the median, it is necessary to compare the length of the conversation text to be analyzed just obtained with the median of the length of all historical conversation texts. Use the long conversation analysis basic model for analysis: If the length of the conversation text to be analyzed is greater than or equal to the median of the length of all historical conversation texts, it means that this conversation to be analyzed is relatively long, and the long conversation analysis basic model is selected at this time. Input the conversation data to be analyzed into the long conversation analysis basic model, and the model will analyze the conversation data to be analyzed based on the patterns and rules learned from previous training. The analysis process may include identifying the topic, emotional tendency, intention, etc. of the conversation, and finally output the analysis results. Use the short conversation analysis basic model for analysis: If the length of the conversation text to be analyzed is less than the median of the length of all historical conversation texts, it means that this conversation to be analyzed is relatively short, and the short conversation analysis basic model is selected at this time. Input the conversation data to be analyzed into the short conversation analysis basic model, and the model will analyze the conversation data to be analyzed based on its own training results, which may also involve analysis of topics, emotions, intentions, etc., and output the corresponding analysis results.

[0040] Furthermore, an intelligent scene dialogue analysis system based on model recognition is proposed, which is used to implement any of the above analysis methods, including: A collection module, which is used to collect historical conversation data on the intelligent scene conversation platform; The data processing module is used to perform text cleaning on the historical conversation data to obtain historical conversation text information, perform word segmentation and part-of-speech tagging on the historical conversation text information to obtain part-of-speech information of the historical conversation text, perform data tagging on the historical conversation data according to the historical conversation text information and the part-of-speech information of the historical conversation text to obtain historical conversation standard data, obtain the length information of each historical conversation text in the historical conversation standard data, record it as historical conversation text length sub-information, divide the historical conversation standard data according to the historical conversation text length sub-information, and obtain historical conversation standard sub-data; The basic model construction module is used to select the corresponding model to conduct comparative experiments on the historical dialogue standard sub-data, complete the construction of the initial model, and obtain the accuracy rate corresponding to each initial model. According to the accuracy rate corresponding to each initial model, the initial model with the best performance is selected as the basic model to obtain the basic model of intelligent scene dialogue analysis; The analysis module is used to obtain the user's real-time conversation data, and pre-process the user's real-time conversation data to obtain the conversation data to be analyzed. Finally, the intelligent scenario conversation analysis basic model is used to analyze the conversation data to be analyzed.

[0041] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. An intelligent scene dialogue analysis method based on model recognition, characterized in that: include: Collect historical conversation data on the intelligent scene conversation platform, perform text cleaning on the historical conversation data, and obtain historical conversation text information; Perform word segmentation and part-of-speech tagging on the historical dialogue text information to obtain the part-of-speech information of the historical dialogue text; According to the historical conversation text information and the part-of-speech information of the historical conversation text, the historical conversation data is annotated to obtain the historical conversation standard data; Obtaining the length information of each historical conversation text in the historical conversation standard data, and recording it as the historical conversation text length sub-information; Dividing the historical conversation standard data according to the historical conversation text length sub-information to obtain the historical conversation standard sub-data; Select the corresponding model to conduct a comparative experiment on the historical dialogue standard sub-data, complete the construction of the initial model, and obtain the accuracy of each initial model; According to the accuracy of each initial model, the initial model with the best performance is selected as the basic model to obtain the basic model of intelligent scene dialogue analysis; Acquire real-time user conversation data and perform data preprocessing on the real-time user conversation data to obtain conversation data to be analyzed. Finally, use the basic model of intelligent scenario conversation analysis to analyze the conversation data to be analyzed.

2. According to the method of intelligent scene dialogue analysis based on model recognition according to claim 1, it is characterized in that: The collecting of historical conversation data on the intelligent scene conversation platform and text cleaning of the historical conversation data to obtain historical conversation text information specifically includes: Obtain the type information of the smart scene dialogue platform to determine the corresponding interface. Use the SDK officially provided by the smart scene dialogue platform to collect public dialogue data and obtain historical dialogue data. Based on the intelligent scene dialogue platform, the preset scene category information corresponding to the historical dialogue data is obtained, and the historical dialogue data is preliminarily classified according to the preset scene category information to obtain the historical dialogue data of different preset scene categories; The historical conversation data of different preset scenario categories are stored in the same data set, and the text of the historical conversation data in the data set is cleaned in turn, including: S1.

1. Use regular expressions to remove noise information from the corresponding conversation text in the historical conversation data; S1.2, according to the preset scene category information corresponding to the historical dialogue data, calling the corresponding corpus, and using the historical dialogue data and the corresponding corpus to train an error correction model corresponding to the preset scene category information, and then correcting the dialogue text in the historical dialogue data through the error correction model, and using the dialogue text in the historical dialogue data after error correction as the historical dialogue text information; Among them, the intelligent scene dialogue platform includes an online customer service platform, an intelligent voice assistant and a social media platform.

3. The method for intelligent scene dialogue analysis based on model recognition according to claim 1, characterized in that: The word segmentation and part-of-speech tagging of the historical conversation text information to obtain the part-of-speech information of the historical conversation text specifically includes: Obtaining language type information corresponding to the dialogue text in the historical dialogue data after error correction, and performing Chinese-English segmentation processing on the dialogue text in the historical dialogue data after error correction according to the language type information, and recording segmentation position information at the same time; Segment the Chinese and English texts in the historical dialogue text to obtain the historical dialogue Chinese text and the historical dialogue English text; Select the corresponding word segmentation tool to perform word segmentation on the Chinese text of historical dialogues and the English text of historical dialogues, including: S2.

1. Based on the intelligent scene dialogue platform, the preset scene category information corresponding to the historical dialogue data is obtained again, thereby obtaining the preset scene field information; S2.2, according to the preset scene domain information, load the corresponding custom Chinese dictionary, and use the Jieba word segmentation tool to segment the historical conversation Chinese text, cut the Chinese text into individual Chinese words, and then store the Chinese words corresponding to each historical conversation Chinese text into the same list to obtain the first historical conversation text information; S2.2, load the corresponding custom English dictionary, use the word segmentation function in the NLTK library to perform word segmentation on the historical conversation English text, cut the English text into individual English words, and then store the English words corresponding to each historical conversation English text into the same list to obtain the second historical conversation text information; Part-of-speech tagging is performed on the first historical conversation text information and the second historical conversation text information, including: S3.

1. Based on the first historical conversation text information, obtain all Chinese pinyins corresponding to the Chinese words in the first historical conversation text information, and obtain all homophonic Chinese words corresponding to the Chinese words based on the Chinese pinyins; S3.2, based on the second historical conversation text information, traverse each English word in the second historical conversation text information, and obtain all recombined English words corresponding to each English word; S3.3, according to the preset scene category information corresponding to the historical conversation data, the custom Chinese dictionary corresponding to the Chinese words and the custom English dictionary corresponding to the English text are loaded respectively, and then all homophonic Chinese words and reorganized English words are screened by using the preset scene category information, the custom Chinese dictionary and the custom English dictionary, and the homophonic Chinese words and reorganized English words contained in the custom Chinese dictionary and the custom English dictionary are extracted to obtain the first historical conversation standard text information and the second historical conversation standard text information; S3.4, using the Jieba word segmentation tool to perform part-of-speech tagging on the Chinese words in the first historical dialogue standard text information, using the part-of-speech tagger in the NLTK library to perform part-of-speech tagging on the English words in the second historical dialogue standard text information, and using the part-of-speech tagging results of the first historical dialogue standard text information and the second historical dialogue standard text information to record the part-of-speech information of the historical dialogue text; S3.

5. By segmenting the position information, the first historical dialogue standard text information and the second historical dialogue standard text information after part-of-speech tagging are reorganized to update the historical dialogue text information.

4. The method for intelligent scene dialogue analysis based on model recognition according to claim 3 is characterized in that: The historical conversation data is annotated according to the historical conversation text information and the historical conversation text part-of-speech information to obtain the historical conversation standard data, specifically including: The updated historical conversation text information is sent to professional annotators for annotation, and the part-of-speech manual annotation information of the historical conversation text is obtained; Compare the part-of-speech information of the historical conversation text with the part-of-speech manual annotation information of the historical conversation text, remove the part-of-speech information of the historical conversation text that does not match the manual annotation information of the historical conversation, and obtain the standard part-of-speech information of the historical conversation text; The historical conversation data is annotated using the standard part-of-speech information of the historical conversation text to obtain the standard data of the historical conversation.

5. The method for intelligent scene dialogue analysis based on model recognition according to claim 1, characterized in that: The length information of each historical conversation text in the historical conversation standard data is obtained, which is recorded as the historical conversation text length sub-information, and specifically includes: Obtain the total number of Chinese characters and the total number of letters of English words in each historical conversation text in the historical conversation standard data; The sum of the total number of Chinese words and the total number of letters in English words in each historical dialogue text in the historical dialogue standard data is used as the length information of each historical dialogue text in the historical dialogue standard data to obtain the historical dialogue text length sub-information.

6. The method for intelligent scene dialogue analysis based on model recognition according to claim 1, characterized in that: The historical conversation standard data is divided according to the historical conversation text length sub-information to obtain the historical conversation standard sub-data, which specifically includes: Traverse all the historical conversation text length sub-information to obtain the median value of all historical conversation text lengths; Classify the historical conversation standard data whose historical conversation text length is greater than or equal to the median of all historical conversation text lengths as the first historical conversation standard data, and classify the historical conversation standard data whose historical conversation text length is less than the median of all historical conversation text lengths as the second historical conversation standard data; The first historical conversation standard data and the second historical conversation standard data are respectively stored in different data sets to obtain historical conversation standard sub-data.

7. The method for intelligent scene dialogue analysis based on model recognition according to claim 1, characterized in that: The selection of the corresponding model conducts a comparative experiment on the historical dialogue standard sub-data, completes the construction of the initial model, and obtains the accuracy rate corresponding to each initial model, specifically including: Extracting historical conversation standard data corresponding to the first historical conversation standard data and the second historical conversation standard data from the historical conversation standard sub-data respectively, and dividing them into 70% training set, 15% validation set and 15% test set respectively, to obtain a first training set, a first validation set, a first test set, a second training set, a second validation set and a second test set; Obtain a long short-term memory network model, a gated recurrent unit model, and a Transformer model respectively, set initial hyperparameters for the long short-term memory network model, the gated recurrent unit model, and the Transformer model, and complete the construction of the initial model; The initial model is trained using the first training set and the second training set respectively, and the first validation set, the first test set, the second validation set and the second test set are used to train the initial model. , get the accuracy corresponding to each initial model, where TP is a true positive example, TN is a true negative example, FP is a false positive example, and FN is a false negative example.

8. The method for intelligent scene dialogue analysis based on model recognition according to claim 1, characterized in that: The method of selecting the best initial model as the basic model according to the accuracy rate corresponding to each initial model to obtain the basic model for intelligent scene dialogue analysis specifically includes: Traverse all initial models and take the initial model with the highest accuracy as the initial model with the best performance, that is, the basic model, so as to obtain the basic model of intelligent scene dialogue analysis; Among them, the intelligent scene dialogue analysis basic model includes a long dialogue analysis basic model and a short dialogue analysis basic model.

9. The method for intelligent scene dialogue analysis based on model recognition according to claim 8, characterized in that: The method of obtaining the user's real-time conversation data and preprocessing the user's real-time conversation data to obtain the conversation data to be analyzed, and finally analyzing the conversation data to be analyzed using the intelligent scene conversation analysis basic model, specifically includes: Obtaining the length information of the conversation text to be analyzed in the conversation data to be analyzed, and obtaining the median value of the length of all historical conversation texts; If the length of the conversation text to be analyzed is greater than or equal to the median length of all historical conversation texts, the long conversation analysis basic model is used to analyze the conversation data to be analyzed; If the length of the conversation text to be analyzed is less than the median length of all historical conversation texts, the short conversation analysis basic model is used to analyze the conversation data to be analyzed.

10. An intelligent scene dialogue analysis system based on model recognition, used to implement the analysis method according to any one of claims 1 to 9, characterized in that: include: A collection module, which is used to collect historical conversation data on the intelligent scene conversation platform; A data processing module, the data processing module is used to perform text cleaning on the historical conversation data to obtain historical conversation text information, perform word segmentation and part-of-speech tagging on the historical conversation text information to obtain the part-of-speech information of the historical conversation text, perform data tagging on the historical conversation data according to the historical conversation text information and the part-of-speech information of the historical conversation text to obtain historical conversation standard data, obtain the length information of each historical conversation text in the historical conversation standard data, record it as historical conversation text length sub-information, divide the historical conversation standard data according to the historical conversation text length sub-information, and obtain historical conversation standard sub-data; A basic model construction module, which is used to select a corresponding model to conduct a comparative experiment on the historical dialogue standard sub-data, complete the construction of the initial model, and obtain the accuracy rate corresponding to each initial model. According to the accuracy rate corresponding to each initial model, the initial model with the best performance is selected as the basic model to obtain the basic model of intelligent scene dialogue analysis; The analysis module is used to obtain the user's real-time conversation data, perform data preprocessing on the user's real-time conversation data, obtain the conversation data to be analyzed, and finally analyze the conversation data to be analyzed using the intelligent scene conversation analysis basic model.

Images