A method, device and medium for generating a multi-round dialogue model based on a knowledge base
By combining facial features and dialect model for multiple rounds of dialogue processing, the problem that the existing technology cannot perform dialect translation and private dialogue is solved, and a natural, personalized and intelligent user interaction experience is achieved, and the system's intelligence and user satisfaction are improved. Spend.
Patent Information
- Application Number
- CN202411194665.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2044-08-29
AI Technical Summary
The existing multi-round dialogue models cannot translate dialect based on the preset regional facial feature model and dialect model, and cannot conduct independent and private facial feature dialogue, which affects the use effect.
By receiving portrait pictures and voice signals from the user, facial feature extraction and speech recognition are performed, combined with regional facial feature models and dialect models, dialect conversion is carried out, standard language data is generated, and dialogue content is dynamically adjusted according to user facial features to ensure privacy protection and the continuity of dialogue.
It significantly improves the naturalness and personalized experience of user interaction, realizes intelligent conversion between dialects and Mandarin, enhances the intimacy and understanding of communication, ensures the continuity and privacy protection of dialogue, and improves the intelligence of the system and user satisfaction. Spend.
Smart Images
Figure CN118708703B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and medium for generating a multi-round dialogue model based on a knowledge base. Background Art
[0002] In today's era of information explosion, the way people interact with machines is undergoing unprecedented changes. From the initial simple command execution to today's intelligent systems capable of complex conversations, technological advances are gradually narrowing the distance between people and machines. Among them, the multi-round dialogue model, as a shining pearl in the field of artificial intelligence, is gradually penetrating into all aspects of our daily lives, from voice assistants in smart homes to intelligent customer service for enterprise-level customer service, all of which demonstrate its huge potential and broad application prospects.
[0003] At present, a Chinese invention patent with application number CN202311453002.1 discloses a multi-round dialogue training method for a large dialogue model, which belongs to the field of artificial intelligence. This invention amplifies the training data through data enhancement technology to generate diversified dialogue materials to improve the generalization ability of the model and avoid overfitting problems; introduces a dialogue history screening mechanism to focus on retaining historical information related to the current dialogue and remove irrelevant or redundant historical records to improve dialogue generation efficiency; uses a pre-trained model to introduce domain expertise into the training of the dialogue model to enhance the model's knowledge question-answering ability and scene adaptability. Although the relevant technology improves the multi-round dialogue ability of the large model by designing targeted data processing strategies and training processes, it is impossible to translate the acquired language signals into dialects according to the preset regional facial feature model and dialect model, and it is impossible to conduct independent and private conversations based on facial features, which affects the use effect. Summary of the invention
[0004] The technical problem solved by the present invention is that although the related technology improves the multi-round conversation capability of the large model by designing targeted data processing strategies and training processes, it is unable to perform dialect translation of the acquired language signals according to the preset regional facial feature model and dialect model, and is unable to conduct independent and private conversations based on facial features, which affects the use effect.
[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: In the first aspect, a method for generating a multi-round dialogue model based on a knowledge base comprises the following steps:
[0006] Step S1, receiving a portrait image and a voice signal from a user terminal, preprocessing the portrait image and locating feature points, extracting facial features based on the located feature points, acquiring current facial feature data, performing text recognition on the voice signal and generating original language data.
[0007] Step S2, comparing the current facial feature data with a preset regional facial feature model to generate a regional comparison result.
[0008] Step S3, extracting keywords from the original language data and calibrating them, generating keyword data and substituting them into the original language data to generate standard language data.
[0009] Step S4, extracting acoustic features and time-frequency features of the speech signal and outputting them as original feature data.
[0010] Step S5, inputting the regional comparison result into a preset dialect model, wherein the dialect model includes several dialect types, wherein the dialect type includes a Mandarin-dialect correspondence table, a region, and acoustic features and time-frequency features corresponding to the dialect type, searching for a dialect type corresponding to the region with the same regional comparison result, inputting the original feature data and the keyword data into the dialect model, matching the dialect type with the same acoustic features and time-frequency features among the found dialect types, and searching for a Mandarin-dialect correspondence table containing the keyword data according to the matched dialect type.
[0011] Step S6, according to the found Mandarin-dialect correspondence table, replace the keywords in the keyword data with dialects, establish a correspondence between keywords and dialects, and output dialect word data.
[0012] Step S7, replace the keyword data in the standard language data with dialect word data, output the replaced standard language data as dialect answer data, input the dialect answer data to the user end, temporarily store the conversation content as a temporary conversation signal, and when the user end receives and obtains facial feature data different from the current facial feature data, delete the temporary conversation signal and the current facial feature data.
[0013] Preferably, the step S1 includes the following sub-steps:
[0014] Step S101, receiving a portrait image from a user, graying, normalizing and removing noise from the portrait image, performing facial area recognition on the processed portrait image, locating feature points using a pre-trained feature point positioning model after recognition, calculating the distance and angle between feature points, generating a geometric feature vector, and outputting it as current facial feature data.
[0015] Step S102, receiving a voice signal from a user terminal, performing noise reduction, normalization and frame division processing on the voice signal, performing text recognition on the processed voice signal using a voice recognition model and generating original language data.
[0016] Preferably, step S2 includes the following sub-steps:
[0017] Step S201, inputting current facial feature data into a regional facial feature model, wherein the regional facial feature model is trained by a plurality of regional facial feature data.
[0018] Step S202, calculating the mean and standard deviation of the corresponding geometric feature vectors in the regional facial features and the current facial features and calculating the Euclidean distance, and outputting the regions whose Euclidean distances are all less than the mean plus one standard deviation as the regional comparison results.
[0019] Preferably, step S3 includes the following sub-steps:
[0020] Step S301, extracting keywords from the original language data, calculating the edit distance between the keywords and each word in the preset dictionary, searching for the word with the smallest edit distance and replacing the keywords, and outputting the replaced keywords as keyword data.
[0021] Step S302: Substitute the keyword data into the original language data to generate standard language data.
[0022] Preferably, step S4 includes the following sub-steps:
[0023] Step S401, windowing each frame of speech signal after frame processing, performing fast Fourier transform on the windowed speech signal to convert it into a frequency domain signal, passing the frequency domain signal through a group of Mel filters, taking the logarithm of the energy output by the Mel filter group to generate a Mel spectrum, performing discrete cosine transform on the Mel spectrum, obtaining Mel frequency cepstrum coefficients and outputting them as acoustic features.
[0024] Step S402: output the Mel spectrum of step S401 as a time-frequency feature.
[0025] Step S403: output the acoustic features and time-frequency features as original feature data.
[0026] Preferably, step S5 includes the following sub-steps:
[0027] Step S501: input the regional comparison result into a preset dialect model to search for the dialect type corresponding to the region with the same regional comparison result.
[0028] Step S502: According to the found dialect type, the original feature data is compared with the acoustic features and time-frequency features corresponding to the dialect type, and the dialect type with the same acoustic features and time-frequency features is matched.
[0029] Step S503: input the keyword data into the matched dialect type, and search for a Mandarin-dialect correspondence table containing the keyword data.
[0030] Preferably, step S6 includes the following sub-steps:
[0031] Step S601, according to the found Mandarin-dialect correspondence table, find the dialect corresponding to the keyword data and replace it.
[0032] Step S602, establishing a correspondence between keywords and dialects and outputting them as dialect word data, and storing the dialect word data.
[0033] Preferably, the step S7 includes the following sub-steps:
[0034] Step S701, replace the keyword data with dialect data, and substitute into the standard language data, output the replaced standard language data as dialect answer data, input the dialect answer data to the user end, and store the dialogue under the current facial feature data as a temporary dialogue signal.
[0035] Step S702, when the user terminal receives and obtains facial feature data different from the current facial feature data, the temporary dialogue signal is deleted and steps S1 to S7 are repeated.
[0036] In a second aspect, an electronic device is provided, characterized in that it includes a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in any one of the above-described methods for generating a multi-round dialogue model based on a knowledge base are executed.
[0037] In a third aspect, a storage medium stores a computer program thereon, characterized in that when the computer program is executed by a processor, the steps in any one of the above methods for generating a multi-round dialogue model based on a knowledge base are executed.
[0038] The beneficial effects of the present invention are as follows: the integration of facial recognition and dialect processing significantly improves the naturalness and personalized experience of user interaction, automatically adjusts the conversation language according to the user's regional characteristics, realizes intelligent conversion between dialect and Mandarin, enhances the intimacy and understanding of communication, and at the same time, dynamically updates user information and manages conversation history, ensuring the continuity and privacy protection of the conversation, and improving the intelligence of the system and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A flowchart of the steps of a method for generating a multi-round dialogue model based on a knowledge base is provided for one embodiment of the present invention. DETAILED DESCRIPTION
[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.
[0041] Embodiment 1, refer to Figure 1 , provides a method for generating a multi-round dialogue model based on a knowledge base, comprising the following steps:
[0042] Step S1, receiving a portrait image and a voice signal from a user terminal, preprocessing the portrait image and locating feature points, extracting facial features based on the located feature points, acquiring current facial feature data, performing text recognition on the voice signal and generating original language data.
[0043] Through preprocessing and feature point positioning, the image quality is improved and key facial parts are identified, laying the foundation for subsequent facial feature extraction, obtaining the facial feature data of the current user, providing important information for subsequent regional comparison, and converting the voice signal into a processable text form, that is, the original language data, in preparation for subsequent language processing.
[0044] Step S2, comparing the current facial feature data with a preset regional facial feature model to generate a regional comparison result.
[0045] By comparing the current facial feature data with the preset regional facial feature model, the region to which the user may belong is determined, providing approximate regional information for subsequent dialect recognition.
[0046] Step S3, extracting keywords from the original language data and calibrating them, generating keyword data and substituting them into the original language data to generate standard language data.
[0047] Extract key information from the original language data and make necessary calibration to ensure the accuracy of keywords, provide an accurate vocabulary basis for dialect conversion, and reintegrate the calibrated keywords into the original language data to form more standardized and accurate standard language data.
[0048] Step S4, extracting acoustic features and time-frequency features of the speech signal and outputting them as original feature data.
[0049] Analyze the acoustic and time-frequency features of speech signals, which are crucial for subsequent dialect recognition and conversion.
[0050] Step S5, inputting the regional comparison result into a preset dialect model, wherein the dialect model includes several dialect types, wherein the dialect type includes a Mandarin-dialect correspondence table, a region, and acoustic features and time-frequency features corresponding to the dialect type, searching for a dialect type corresponding to the region with the same regional comparison result, inputting the original feature data and the keyword data into the dialect model, matching the dialect type with the same acoustic features and time-frequency features among the found dialect types, and searching for a Mandarin-dialect correspondence table containing the keyword data according to the matched dialect type.
[0051] Using the regional comparison results and the original feature data of the voice signal, the dialect model is searched for the dialect type that matches the user's region and voice features. Based on the matching dialect type, the Mandarin-dialect correspondence table containing the keyword data is further searched to prepare for the dialect conversion of the keyword.
[0052] Step S6, according to the found Mandarin-dialect correspondence table, replace the keywords in the keyword data with dialects, establish a correspondence between keywords and dialects, and output dialect word data.
[0053] According to the Mandarin-dialect correspondence table, the keywords in the keyword data are replaced with corresponding dialect words, a correspondence between the keywords and the dialect words is established, and the keyword data after dialect conversion, namely the dialect word data, is output.
[0054] Step S7, replace the keyword data in the standard language data with dialect word data, output the replaced standard language data as dialect answer data, input the dialect answer data to the user end, temporarily store the conversation content as a temporary conversation signal, and when the user end receives and obtains facial feature data different from the current facial feature data, delete the temporary conversation signal and the current facial feature data.
[0055] Replace the keyword data in the standard language data with dialect word data to generate the final dialect answer data, so that the conversation content is more in line with the language habits of the user's area. Send the dialect answer data to the user end and temporarily store the conversation content for subsequent processing. When the user's facial features change, automatically clean up the old conversation records to maintain the timeliness and security of the system data.
[0056] Step S1 includes the following sub-steps:
[0057] Step S101, receiving a portrait image from a user, graying, normalizing and removing noise from the portrait image, performing facial area recognition on the processed portrait image, locating feature points using a pre-trained feature point positioning model after recognition, calculating the distance and angle between feature points, generating a geometric feature vector, and outputting it as current facial feature data.
[0058] Step S101 is the starting point of data input, ensuring that the user's portrait image can be obtained.
[0059] Converting color portrait images into grayscale images simplifies the complexity of image processing while retaining sufficient facial feature information to facilitate subsequent processing, adjust the size, brightness and contrast parameters of portrait images, remove noise points or noise areas in portrait images, improve image quality, and reduce interference and errors in subsequent feature point positioning.
[0060] Accurately locate the position of the face in the image, provide a precise area range for subsequent feature point positioning, and use the pre-trained feature point positioning model to automatically identify and locate key feature points in the facial area. These feature points are crucial for subsequent facial feature extraction.
[0061] By analyzing the distance and angle between feature points, vectors that can describe facial geometric features are generated. These geometric feature vectors are an important part of subsequent facial feature data.
[0062] The calculated geometric feature vector is output as the current facial feature data for use in subsequent steps.
[0063] Step S102, receiving a voice signal from a user terminal, performing noise reduction, normalization and frame division processing on the voice signal, performing text recognition on the processed voice signal using a voice recognition model and generating original language data.
[0064] Step S102 is also the starting point of data input, ensuring that the user's voice signal can be acquired.
[0065] Remove background noise from speech signals, improve the signal-to-noise ratio of speech signals, provide clear speech data for subsequent text recognition, adjust the volume and frequency of speech signals, improve the accuracy of recognition models, and divide continuous speech signals into multiple short time frames for more detailed processing and analysis.
[0066] Apply the speech recognition model to the processed speech signal to convert the speech signal into readable text form, that is, raw language data. Its effect directly affects the accuracy of subsequent language processing and dialect conversion. The output of the speech recognition model is output as raw language data for use in subsequent steps.
[0067] Step S2 includes the following sub-steps:
[0068] Step S201, inputting current facial feature data into a regional facial feature model, wherein the regional facial feature model is trained by a plurality of regional facial feature data.
[0069] The current facial feature data obtained after preprocessing and feature point positioning is input into the pre-trained regional facial feature model. This model is trained based on the facial feature data of people in multiple different regions and can recognize and distinguish the differences in facial features in different regions. By inputting the current facial feature data, the regional facial feature model begins to compare and match it.
[0070] Step S202, calculating the mean and standard deviation of the corresponding geometric feature vectors in the regional facial features and the current facial features and calculating the Euclidean distance, and outputting the regions whose Euclidean distances are all less than the mean plus one standard deviation as the regional comparison results.
[0071] For each geometric feature vector in the facial feature dataset of a region, the mean and standard deviation are calculated, reflecting the typical value and variation range of the facial features of the people in the region. The Euclidean distance is calculated between the geometric feature vector in the current facial feature data and each geometric feature vector in the regional facial feature dataset. The Euclidean distance is a common method for measuring the distance between two points in a multidimensional space. It is used here to evaluate the degree of difference between the current facial features and the facial features of each region.
[0072] Based on the calculated Euclidean distance, those regions whose distances are less than the mean plus one standard deviation are screened out as the regional comparison results. The purpose is to find those regions that are closest or most similar to the current facial features, thereby providing regional information for subsequent dialect recognition and conversion, and ensuring the accuracy and reliability of the regional comparison results.
[0073] Step S3 includes the following sub-steps:
[0074] Step S301, extracting keywords from the original language data, calculating the edit distance between the keywords and each word in the preset dictionary, searching for the word with the smallest edit distance and replacing the keywords, and outputting the replaced keywords as keyword data.
[0075] Accurately extract keywords from the original language data. These keywords are the core information points in sentences and phrases. By calculating the edit distance between the keywords and each word in the preset dictionary, that is, the minimum number of single-character editing operations required to transform from one string to another, evaluate the similarity between the keywords and the words in the dictionary, find the dictionary word with the smallest edit distance to the keyword, and replace the original keyword with the word. This helps to correct possible spelling errors, abbreviation misunderstandings and standardization of dialect vocabulary. The replaced keywords are output as keyword data, which is more standardized and accurate, providing a reliable vocabulary basis for subsequent language processing.
[0076] Step S302: Substitute the keyword data into the original language data to generate standard language data.
[0077] The keyword data after edit distance comparison and replacement is reintegrated into the original language data to ensure the accuracy and consistency of key information. By substituting the keyword data, more standardized, easy-to-understand and process standard language data is generated, which not only improves the overall quality of the language data, but also provides better input data for subsequent dialect conversion and natural language processing.
[0078] Step S4 includes the following sub-steps:
[0079] Step S401, windowing each frame of speech signal after frame processing, performing fast Fourier transform on the windowed speech signal to convert it into a frequency domain signal, passing the frequency domain signal through a group of Mel filters, taking the logarithm of the energy output by the Mel filter group to generate a Mel spectrum, performing discrete cosine transform on the Mel spectrum, obtaining Mel frequency cepstrum coefficients and outputting them as acoustic features.
[0080] Frame processing divides the continuous speech signal into a series of short-time frames, each frame is 20-30 milliseconds, and the frame shift is 10-15 milliseconds, which is convenient for subsequent processing. Windowing applies the Hamming window function to each frame to reduce the discontinuity of the frame boundary, which helps to maintain the continuity of the signal in the frequency domain analysis and improve the analysis accuracy. The fast Fourier transform converts the windowed time domain signal into a frequency domain signal to analyze the spectral characteristics of the signal. The obtained frequency domain signal is passed through a set of Mel filters based on the auditory characteristics of the human ear through the Mel filter group. The Mel filter group simulates the difference in sensitivity of the human ear to sounds of different frequencies, so that the low-frequency part has a high resolution and the high-frequency part has a low resolution, which helps to extract features that are more in line with the characteristics of human auditory perception. After passing through the Mel filter group, the output energy of each filter is taken logarithm, which helps to compress the dynamic range so that signals of different intensities can contribute more fairly in subsequent processing. The output of the Mel filter after taking the logarithm is used as the Mel spectrum, and then it is converted into Mel frequency cepstrum coefficients through discrete cosine transform, which effectively represents the spectral envelope of the speech, while reducing the data dimension and improving computational efficiency.
[0081] Step S402: output the Mel spectrum of step S401 as a time-frequency feature.
[0082] The Mel spectrum generated in step S401 is directly used as the time-frequency feature. Although the Mel spectrum itself is extracted from the frequency domain signal, it is used as a feature that combines time domain and frequency domain information in step S402, which is very useful for capturing the spectral characteristics of the speech signal that change over time.
[0083] Step S403: output the acoustic features and time-frequency features as original feature data.
[0084] The acoustic features obtained in step S401 and the time-frequency features directly obtained in step S402 are combined and output as original feature data. By combining these two features, the acoustic characteristics of the speech signal can be more comprehensively described, thereby improving the processing effect.
[0085] Step S5 includes the following sub-steps:
[0086] Step S501: input the regional comparison result into a preset dialect model to search for the dialect type corresponding to the region with the same regional comparison result.
[0087] The regional comparison results are used as input and imported into the preset dialect model. The dialect model includes several dialect types, including a Mandarin-dialect correspondence table, regions, and acoustic features and time-frequency features corresponding to the dialect types. By finding regions that match the regional comparison results, the dialect type that the user may use can be determined, and the starting point of the dialect conversion can be determined.
[0088] Step S502: According to the found dialect type, the original feature data is compared with the acoustic features and time-frequency features corresponding to the dialect type, and the dialect type with the same acoustic features and time-frequency features is matched.
[0089] After determining the dialect type that the user may use, the original feature data is further compared with the acoustic features and time-frequency features corresponding to the dialect type to verify whether the acoustic characteristics of the speech signal match the expected dialect type. If the match is successful, it means that the speech signal does belong to that dialect type, thereby enhancing the accuracy of dialect recognition and helping to reduce conversion errors caused by dialect misjudgment.
[0090] Step S503: input the keyword data into the matched dialect type, and search for a Mandarin-dialect correspondence table containing the keyword data.
[0091] After determining the dialect type that matches the speech signal, the keyword data is input into the matched dialect type to look up the Mandarin-dialect correspondence table containing these keywords. The Mandarin-dialect correspondence table is a data structure containing Mandarin vocabulary and its corresponding dialect vocabulary. By looking up the correspondence table, the expression of the keyword in a specific dialect can be found to prepare for subsequent dialect conversion.
[0092] Step S6 includes the following sub-steps:
[0093] Step S601, according to the found Mandarin-dialect correspondence table, find the dialect corresponding to the keyword data and replace it.
[0094] According to the Mandarin-dialect correspondence table found in step S5, each keyword in the keyword data is replaced one by one with the corresponding dialect vocabulary, thereby realizing the dialectization of the keywords. This allows the keywords originally expressed in Mandarin to be converted into expressions that conform to the dialect habits of the user's area. The replaced keyword data is closer to the user's language environment, which helps to improve the user experience and the naturalness of the conversation.
[0095] Step S602, establishing a correspondence between keywords and dialects and outputting them as dialect word data, and storing the dialect word data.
[0096] After completing the dialect replacement of keywords, we further establish a clear relationship between these keywords and their corresponding dialect vocabulary, and output this relationship together with the replaced dialect vocabulary as dialect word data. This not only ensures the accurate correspondence between the keywords and their dialect expressions, but also provides convenience for subsequent processing. The output dialect word data is an important component of the subsequent generation of dialect answer data, which ensures that the answer content conforms to the user's dialect habits in terms of vocabulary usage.
[0097] Step S7 includes the following sub-steps:
[0098] Step S701, replace the keyword data with dialect data, and substitute into the standard language data, output the replaced standard language data as dialect answer data, input the dialect answer data to the user end, and store the dialogue under the current facial feature data as a temporary dialogue signal.
[0099] According to the correspondence between keywords and dialect vocabulary established in the previous step, the keywords in the standard language data are replaced with the corresponding dialect vocabulary, so that the output language data is closer to the language habits of the user's area, enhancing the user experience. The replaced keywords are reintegrated into the standard language data to form dialect answer data, which not only retains the structure and semantics of the original language, but also incorporates dialect elements to make the answer more natural and localized. The processed dialect answer data is sent to the user end so that users can see the answers expressed in the language they are familiar with, which helps to improve user satisfaction and interaction efficiency. The conversation content under the current facial feature data is stored as a temporary conversation signal, which helps to update or delete old conversation records according to changes in the user's facial features in subsequent processing, thereby maintaining the timeliness and accuracy of system data.
[0100] Step S702, when the user terminal receives and obtains facial feature data different from the current facial feature data, the temporary dialogue signal is deleted and steps S1 to S7 are repeated.
[0101] Through real-time feedback from the user side or the facial recognition mechanism within the system, it is detected whether the user's facial feature data has changed. When it is detected that the user's facial feature data is different from the currently stored facial feature data, it is considered that the user may have changed or is no longer the previous user. Therefore, the previously stored temporary conversation signal is deleted, which helps to protect user privacy and avoid confusion between conversation records between different users. The entire processing flow starts again from receiving the portrait image and voice signal from the user side, ensuring that the system can perform accurate voice processing and dialect conversion for new users, always keeping the interaction with the user smooth and efficient, while also reflecting flexibility and scalability.
[0102] By preprocessing portrait images and locating feature points, the user's facial feature data is extracted, providing important reference information for subsequent regional comparison and dialect recognition. The extracted facial feature data is compared with the preset regional facial feature model to determine the region to which the user may belong. The dialect type that matches the user's region and voice characteristics is searched in the dialect model to ensure the accuracy and pertinence of the dialect conversion. The voice signal is recognized by text to generate raw language data, and keywords are extracted for calibration. The acoustic features and time-frequency features of the voice signal are obtained to provide the necessary voice information for subsequent dialect conversion. According to the matched dialect type, and keyword data, look up the corresponding dialect vocabulary in the Mandarin-dialect correspondence table, replace the keyword with the dialect vocabulary, output the replaced standard language data as dialect answer data, and send it to the user end, so that the output language data is closer to the language habits of the user's area, improving user satisfaction, and storing the conversation content under the current facial feature data as a temporary conversation signal for subsequent processing. When it is detected that the user's facial feature data has changed, the old temporary conversation signal is deleted and the entire processing flow is re-executed to ensure that the conversation record can be updated in real time according to the change of the user's status, so as to maintain the timeliness, security and accuracy of the data.
[0103] In the second embodiment, the present application provides an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in any one of the above methods are executed. Through the above technical solution, the processor and the memory are interconnected and communicate with each other through a communication bus and / or other forms of connection mechanisms (not marked). The memory stores a computer program executable by the processor. When the electronic device is running, the processor executes the computer program to execute the method in any optional implementation of the above embodiment to achieve the following functions: receiving a portrait image and a voice signal from the user end, preprocessing the portrait image and locating feature points, extracting facial features based on the located feature points, performing text recognition on the voice signal, comparing the current facial feature data with a preset regional facial feature model, generating a regional comparison result, and extracting keywords from the original language data. And calibrate, generate keyword data, generate standard language data, extract acoustic features and time-frequency features of speech signals, output as original feature data, find the dialect type corresponding to the region with the same regional comparison result, input the original feature data and keyword data into the dialect model, match the dialect type with the same acoustic features and time-frequency features in the found dialect types, replace the keywords in the keyword data with dialects, establish the corresponding relationship between keywords and dialects, output dialect word data, output the replaced standard language data as dialect answer data, input the dialect answer data to the user end, temporarily store the conversation content as a temporary conversation signal, delete the temporary conversation signal and the current facial feature data.
[0104] In a third embodiment, the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in any one of the above methods are executed. Through the above technical solution, when the computer program is executed by the processor, the method in any optional implementation of the above embodiment is executed to achieve the following functions: receiving a portrait image and a voice signal of the user terminal, preprocessing the portrait image and locating feature points, extracting facial features according to the located feature points, performing text recognition on the voice signal, comparing the current facial feature data with a preset regional facial feature model, generating a regional comparison result, extracting keywords from the original language data and calibrating them, generating keyword data, generating standard language data, extracting acoustic features and time-frequency features of the voice signal, and outputting them as original feature data, searching for a dialect type corresponding to a region with the same regional comparison result, inputting the original feature data and keyword data into the dialect model, matching the dialect type with the same acoustic features and time-frequency features in the found dialect type, replacing the keywords in the keyword data with dialects, establishing a corresponding relationship between keywords and dialects, outputting dialect word data, outputting the replaced standard language data as dialect answer data, inputting the dialect answer data into the user terminal, temporarily storing the dialogue content as a temporary dialogue signal, and deleting the temporary dialogue signal and the current facial feature data.
[0105] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. Among them, the storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0106] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A method for generating a multi-round dialogue model based on a knowledge base, characterized in that: The steps include: Step S1, receiving a portrait image and a voice signal from a user terminal, preprocessing the portrait image and locating feature points, extracting facial features based on the located feature points, obtaining current facial feature data, performing text recognition on the voice signal and generating original language data; Step S2, comparing the current facial feature data with a preset regional facial feature model to generate a regional comparison result; Step S3, extracting keywords from the original language data and calibrating them, generating keyword data and substituting them into the original language data to generate standard language data; Step S4, extracting acoustic features and time-frequency features of the speech signal and outputting them as original feature data; Step S5, inputting the regional comparison result into a preset dialect model, wherein the dialect model includes several dialect types, wherein the dialect type includes a Mandarin-dialect correspondence table, a region, and acoustic features and time-frequency features corresponding to the dialect type, searching for a dialect type corresponding to a region having the same regional comparison result, inputting the original feature data and the keyword data into the dialect model, matching the dialect type having the same acoustic features and time-frequency features among the found dialect types, and searching for a Mandarin-dialect correspondence table containing keyword data according to the matched dialect type; Step S6, according to the found Mandarin-dialect correspondence table, replace the keywords in the keyword data with the dialect, establish the correspondence between the keywords and the dialect, and output the dialect word data; Step S7, replace the keyword data in the standard language data with dialect word data, output the replaced standard language data as dialect answer data, input the dialect answer data to the user end, temporarily store the conversation content as a temporary conversation signal, and when the user end receives and obtains facial feature data different from the current facial feature data, delete the temporary conversation signal and the current facial feature data.
2. The method for generating a multi-round dialogue model based on a knowledge base as claimed in claim 1, characterized in that: The step S1 includes the following sub-steps: Step S101, receiving a portrait image from a user terminal, graying, normalizing and removing noise from the portrait image, performing facial region recognition on the processed portrait image, locating feature points using a pre-trained feature point positioning model after recognition, calculating the distance and angle between feature points, generating a geometric feature vector, and outputting it as current facial feature data; Step S102, receiving a voice signal from a user terminal, performing noise reduction, normalization and frame division processing on the voice signal, performing text recognition on the processed voice signal using a voice recognition model and generating original language data.
3. The method for generating a multi-round dialogue model based on a knowledge base as claimed in claim 2, characterized in that: The step S2 includes the following sub-steps: Step S201, inputting current facial feature data into a regional facial feature model, wherein the regional facial feature model is trained by a plurality of regional facial feature data; Step S202, calculating the mean and standard deviation of the corresponding geometric feature vectors in the regional facial features and the current facial features and calculating the Euclidean distance, and outputting the regions whose Euclidean distances are all less than the mean plus one standard deviation as the regional comparison results.
4. The method for generating a multi-round dialogue model based on a knowledge base as claimed in claim 3, characterized in that: The step S3 includes the following sub-steps: Step S301, extracting keywords from the original language data, calculating the edit distance between the keywords and each word in the preset dictionary, searching for the word with the smallest edit distance and replacing the keywords, and outputting the replaced keywords as keyword data; Step S302: Substitute the keyword data into the original language data to generate standard language data.
5. The method for generating a multi-round dialogue model based on a knowledge base as claimed in claim 4, characterized in that: The step S4 includes the following sub-steps: Step S401, windowing each frame of speech signal after the frame processing, performing fast Fourier transform on the windowed speech signal to convert it into a frequency domain signal, passing the frequency domain signal through a group of Mel filters, taking the logarithm of the energy output by the Mel filter group to generate a Mel spectrum, performing discrete cosine transform on the Mel spectrum to obtain Mel frequency cepstrum coefficients and output them as acoustic features; Step S402, outputting the Mel spectrum of step S401 as time-frequency features; Step S403: output the acoustic features and time-frequency features as original feature data.
6. The method for generating a multi-round dialogue model based on a knowledge base as claimed in claim 5, characterized in that: The step S5 includes the following sub-steps: Step S501, inputting the regional comparison result into a preset dialect model, and searching for the dialect type corresponding to the region with the same regional comparison result; Step S502: according to the found dialect type, the original feature data is compared with the acoustic features and time-frequency features corresponding to the dialect type, and the dialect type with the same acoustic features and time-frequency features is matched; Step S503: input the keyword data into the matched dialect type, and search for a Mandarin-dialect correspondence table containing the keyword data.
7. The method for generating a multi-round dialogue model based on a knowledge base as claimed in claim 6, characterized in that: The step S6 comprises the following sub-steps: Step S601, according to the found Mandarin-dialect correspondence table, find the dialect corresponding to the keyword data and replace it; Step S602, establishing a correspondence between keywords and dialects and outputting them as dialect word data, and storing the dialect word data.
8. The method for generating a multi-round dialogue model based on a knowledge base as claimed in claim 7, characterized in that: The step S7 includes the following sub-steps: Step S701, replacing the keyword data with the dialect data and substituting into the standard language data, outputting the replaced standard language data as dialect answer data, inputting the dialect answer data to the user end, and storing the dialogue under the current facial feature data as a temporary dialogue signal; Step S702, when the user terminal receives and obtains facial feature data different from the current facial feature data, the temporary dialogue signal is deleted and steps S1 to S7 are repeated.
9. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method for generating a multi-round dialogue model based on a knowledge base as described in any one of claims 1 to 8 are executed.
10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method for generating a multi-round dialogue model based on a knowledge base as described in any one of claims 1 to 8 are executed.