Data comprehensive processing method and system

Abstract

The invention discloses a data comprehensive processing method and system, and relates to the text speech processing technology, and the method comprises the steps: obtaining the basic text information of a target person in advance, and building an RAG-based database according to the basic text information; acquiring voice data of a user, and recognizing text information in the voice data; retrieving in the database according to the recognized text information to obtain associated context information; generating a corresponding prompt word based on the context information and the text information; generating an output text by utilizing a large language model according to the generated cue word; and performing voice synthesis on the output text, and performing voice broadcast according to the synthesized voice. According to the invention, the appearance, tone and knowledge background of a human are simulated through an AI means, and a digital human system with a voice interaction function is constructed.

Description

Technical Field

[0001] This application relates to the technical field of text-to-speech processing, and particularly to a data comprehensive processing method and system. Background Art

[0002] Existing chatbots or personalized digital lives are all aimed at interacting with human users passively, that is, with the purpose of "answering every question", and it is difficult to achieve the function of actively guiding human users. Especially in some special application scenarios, it is difficult to have a positive guiding effect on human users. Although the method of AI synthesized video can synthesize videos with specified voice content of specified persons according to actual needs, it does not have the ability of real-time interaction.

[0003] A personalized digital life with the ability of active guidance should possess functions such as voice interaction, visual understanding, intelligent dialogue, and dialogue strategies. Currently, no publicly disclosed solutions meet the above requirements. Summary of the Invention

[0004] Embodiments of this application provide a data comprehensive processing method and system, which simulate a person's appearance, voice, and knowledge background through AI means to construct a digital human system with voice interaction function.

[0005] Embodiments of this application provide a data comprehensive processing method, including:

[0006] Pre-acquire the basic text information of the target person, and establish a RAG-based database according to the basic text information;

[0007] Acquire the voice data of the user, and recognize the text information in the voice data;

[0008] Retrieve in the database according to the recognized text information to obtain associated context information;

[0009] Generate corresponding prompt words based on the context information and the text information;

[0010] Generate output text using a large language model according to the generated prompt words;

[0011] Perform speech synthesis on the output text, and perform voice broadcast according to the synthesized voice.

[0012] Optionally, establishing a RAG-based database according to the basic text information further includes:

[0013] Extract the words with special semantics from the basic text information of the target person;

[0014] Supplement or delete the words with special semantics to form a hot word list;

[0015] Identifying text information in speech data includes:

[0016] Transcribing the speech data into text and identifying the text information according to the hot word list.

[0017] Optionally, according to the identified text information, retrieving in the database to obtain associated context information includes:

[0018] Embedding the identified text information through an encoder to form a text vector, where the encoder used is the same as the encoder for processing the basic text information in the database;

[0019] Comparing the formed text vector based on the vector index in the database;

[0020] Taking the top K results with the highest similarity to determine the corresponding K pieces of associated context information according to the top K results.

[0021] Optionally, when obtaining the user's video intercom data, it further includes:

[0022] Performing face detection based on the video intercom data;

[0023] Extracting face features according to the detected face;

[0024] Comparing the face features with the pre-registered face features and taking the result with the highest comparison similarity as the face recognition result;

[0025] And,

[0026] Performing emotion recognition based on the video intercom data;

[0027] Generating corresponding prompt words based on the face recognition result, emotion recognition result, the context information, and the text information.

[0028] Optionally, generating corresponding prompt words based on the context information and the text information includes:

[0029] Determining the event content included in the text information;

[0030] According to the pre-set prompt word template, filling the identified event content into the prompt word template, and,

[0031] When including the face recognition result and emotion recognition result, filling the face recognition result and emotion recognition result in the corresponding positions to generate prompt words.

[0032] Optionally, performing speech synthesis on the output text and performing speech broadcast according to the synthesized speech includes:

[0033] Pre-annotate multiple segments of voice materials of the target person, and use the annotated multiple segments of voice materials to train a voice broadcast model;

[0034] After training, input the text and use the voice broadcast model to broadcast it to determine the similarity between the synthesized voice and the actual voice;

[0035] Adjust the training parameters of the voice broadcast model according to the determined similarity until the similarity of the synthesized voice of the trained voice broadcast model meets the requirements.

[0036] Optionally, performing voice synthesis on the output text and performing voice broadcast according to the synthesized voice further includes: generating matching lip movements for the voice broadcast using lip synchronization.

[0037] An embodiment of the present application also provides a data comprehensive processing system, including a processor and a memory, where a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the data comprehensive processing method as described above are implemented.

[0038] An embodiment of the present application constructs a digital human system with voice interaction function by simulating a person's appearance, voice and knowledge background through AI means.

[0039] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the specific embodiments of the present application are specifically described below. Description of the Drawings

[0040] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0041] Figure 1 It is a schematic diagram of the overall process of the data comprehensive processing method according to the embodiment of the present application;

[0042] Figure 2 It is a flow chart of determining associated context information based on RAG for the data comprehensive processing method according to the embodiment of the present application;

[0043] Figure 3 It is a schematic diagram of the technical framework of the data comprehensive processing method according to the embodiment of the present application;

[0044] Figure 4 It is a flow chart of constructing a hot word list for the data comprehensive processing method according to the embodiment of the present application;

[0045] Figure 5 This is the training process of the voice broadcast model for the data comprehensive processing method of the embodiments of the present application. Specific implementation manners

[0046] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0047] The embodiments of the present application provide a data comprehensive processing method, as Figure 1 shown, including the following steps:

[0048] In step S101, the basic text information of the target person is obtained in advance, and a RAG-based database is established according to the basic text information. In the embodiments of the present application, the target person is taken as an example for illustration. The text data including the resume, file, chat record, diary, etc. of the target person are collected in advance. Secondly, the named entity recognition (NER) method is used to process the above text data, and the words with special semantic meanings such as person names, place names, and organization names are extracted.

[0049] The RAG technology provides the large language model with the information retrieved from certain data sources and corrects the generated answers based on this. It enables the large language model to be more accurate in answering questions in a certain sub-field according to the information in the database. The information in the database can include two categories: one is the experience of the target user, and the other is expert knowledge. The introduction of the first category of information is to avoid the problem that when the user mentions certain experiences during the conversation, the large language model gives incorrect answers due to lack of understanding of these experiences, resulting in a decrease in the sense of reality of the interaction. For example, if the user is talking about the college period of the target person, the RAG technology can retrieve the relevant information of the target person during this period, including the name of the college, the name of the major, the names of classmates, etc. When the large language model generates the output text, if these detailed information are added, the output text can be closer to the experience of the person.

[0050] As Figure 2As shown, the engineering implementation of RAG can adopt open-source frameworks such as LangChain or LlamaIndex. Using the basic RAG solution, it includes a corpus database with a large amount of text. The construction of this database is achieved by collecting and organizing the materials of target personnel and the words of professionals. The materials of target personnel include their resumes, files, chat records, diaries, etc., and can also be sorted biographies of similar figures. The text data in the database is first segmented into chunks, and then these chunks are embedded into vector encoders, and indexes are established for all these vectors.

[0051] In step S102, the voice data of the user is obtained, and the text information in the voice data is recognized. As Figure 3 shown, it can be specifically implemented through a speech recognition module. Automatic Speech Recognition (ASR) automatically transcribes human oral speech into text by computer. After decades of development, the recognition accuracy of speech recognition technology has been continuously improved in various languages. In specific examples, technologies such as whisper, MassivelyMultilingual Speech, PaddleSpeech, and FunASR can be used. In some examples, the FunASR model can be selected as an example to build the entire system process, and other speech recognition models or algorithms can also be applicable.

[0052] In step S103, according to the recognized text information, a search is performed in the said database to obtain associated context information. The search can be specifically implemented by vectorizing the text.

[0053] In step S104, based on the said context information and the said text information, corresponding prompt words are generated. The large language model generates the final output text, and one of the key factors controlling the content generation of the large language model is the prompt it receives. The prompt words are generated by combining the said context information and the said text information.

[0054] In step S105, according to the generated prompt words, a large language model is used to generate output text. In specific examples, for example, ChatGPT, Gemini, LLaMA, Claude, Tongyi Qianwen, Doubao, and DeepSeek can be used. The large language model not only continuously improves in performance, but also the openness of various models is increasing day by day, providing convenience for the application of the large language model in different scenarios. The input and output of the large language model are relatively simple, and its internal structure is "black box" - like. Therefore, in the embodiments of the present application, it is not modified and is directly used. The large language model can be deployed locally or called through the Internet in the form of an API interface. Generally, the larger the number of parameters of the model, the better its effect. In the present application, the large language model only needs to generate text for daily conversations and perform simple logical reasoning. Therefore, common models can also meet the usage requirements.

[0055] In step S106, the output text is synthesized into speech, and the synthesized speech is used for voice broadcast.

[0056] The embodiments of the present application simulate a person's appearance, voice, and knowledge background through AI means, and construct a digital human system with voice interaction functions.

[0057] In some embodiments, establishing a RAG - based database according to the basic text information further includes:

[0058] According to the basic text information of the target person, extract the words with special semantics therein. For example, extract the words with special semantic meanings such as personal names, place names, and organization names, which can be realized through third - party libraries such as open - source Jieba or SnowNLP.

[0059] As Figure 4 shown, supplement or delete the words with special semantics to form a hot - word list. For example, manually screen and modify the obtained candidate words, remove the general words, and supplement the associated words not appearing in the text data to obtain the final hot - word list.

[0060] Identifying the text information in the voice data includes:

[0061] Transcribe the voice data into text, and identify the text information according to the hot - word list.

[0062] In some embodiments, according to the identified text information, retrieve in the database to obtain associated context information includes:

[0063] Embed the identified text information through an encoder to form a text vector, where the encoder used is the same as the encoder for processing the basic text information in the database.

[0064] Compare the formed text vector based on the vector index in the said database;

[0065] For the top K results with the highest similarity, determine the corresponding K associated context information according to the top K results.

[0066] In some embodiments, when the video intercom data of the user is obtained, it further includes:

[0067] Perform face detection based on the said video intercom data; face recognition is used to confirm the relationship between the user and the target person. This relationship can be added to the prompt words as one of the bases for the large language model to generate the output text, so as to enable the digital human to speak in different ways when facing users with different relationships.

[0068] Extract face features according to the detected face.

[0069] Compare the said face features with the pre-registered face features, and take the result with the highest comparison similarity as the face recognition result.

[0070] The face recognition algorithm is relatively mature. In the scenario of this application, the posture of the user's face will not change significantly, and there will be no external obstructions such as glasses or masks. Existing open-source algorithms, such as facenet, InsightFace, OpenFace, deepface, etc., can meet the requirements. The face recognition process includes that the input picture is the image at the current moment captured by the camera. If the computing resources are tight, it can be set to process one frame every t seconds. The input picture first undergoes face detection to judge whether there is a face in the picture, and locate and normalize key points such as eyes, nose, and corners of the mouth. Secondly, feature extraction is performed on the detected face area, usually by using a neural network to calculate, and each face area obtains a feature vector. Finally, the calculated feature vector is compared with the registered face vectors in the face database, and the most similar one is selected as the result output.

[0071] Perform emotion recognition based on the said video intercom data. The embodiments of this application further focus on the emotional changes of the user. There are various methods for using a computer to recognize the human condition state, including those based on face images, based on voices, based on texts, etc. This application adopts the method based on face images. Compared with other means, the method based on face images has better accuracy. In specific applications, the quality of the face image is relatively high, and the real-time performance is good, which can accurately reflect the emotional state of the user at the current moment.

[0072] The facial emotion recognition algorithm is similar to the facial recognition algorithm. Most of the open-source algorithms mentioned above have the function of emotion recognition, and the emotions that can be distinguished basically include happiness, sadness, surprise, anger, disgust, and neutrality, etc. The principle will not be elaborated here.

[0073] Generate corresponding prompt words based on the facial recognition result, the emotion recognition result, the context information, and the text information.

[0074] In some embodiments, generating corresponding prompt words based on the context information and the text information includes:

[0075] Determine the event content included in the text information;

[0076] According to the pre-set prompt word template, fill the recognized event content into the prompt word template.

[0077] In a specific example, the generation of prompt words needs to comprehensively consider the input text, the context related to the input text, the identity information of the user, and the current emotional state of the user. Among them, the input text is the most core content, which contains the main content of the current round of conversation. When the user has a voice input, the input text is the content of the voice, which may be the question or the content told by the user to the digital human. When the user has no voice input for a long time, this part is replaced by the pre-designed psychological intervention words. From the perspective of the performance of the digital human, it is that the digital human takes the initiative to communicate with the user. The context is the known information most relevant to the input text, including the experience details of the target person and the knowledge provided by experts. The identity information will affect details such as the address, word choice, and tone in the output text. Similarly, the emotional state will also affect details such as the word choice, tone, and sentence pattern in the output text.

[0078] In the case of including the facial recognition result and the emotion recognition result, fill the facial recognition result and the emotion recognition result in the corresponding positions to generate prompt words.

[0079] For example, I am having a conversation with my XXX (such as father, mother, etc.). He / She looks XXX (emotional state, such as sad, surprised, peaceful, etc.) now. He / She says: "XXX" (input text content, or the whole sentence is replaced by the specified prompt statement for retrieving the expert words in the database). What should I say to him / her? Directly give the answer content with reference to the following background knowledge. XXX (background knowledge, that is, the context retrieved from the database through RAG).

[0080] In some embodiments, performing speech synthesis on the output text and performing speech broadcast according to the synthesized speech includes:

[0081] Pre-label multiple voice clips of the target person, and use the labeled voice clips to train the voice broadcast model. Figure 5 As shown, the naturalness of the speech synthesized by speech synthesis technology is getting better and better, and it is easier to train a model for a certain person. The embodiment of the present application uses existing methods to replicate the voice of the target person, for example, the open source project GPT-SoVITS can be used.

[0082] The preparation of sound materials refers to the collection of target personnel's voice audio. These materials can be daily conversations, readings or other voices. For example, they can be audio materials such as sending voice messages, recording videos, and holding online meetings in daily life. When preparing materials, try to select high-quality audio to avoid background noise and noise to improve the training effect. In the data preprocessing stage, the prepared voice materials are converted into 16kHz, 16-bit mono wav format, and each audio segment is annotated with text to ensure that the audio content matches the text completely. The annotation process can use the ASR tool that comes with the GPT-SoVITS project to first batch recognize the audio, and then manually proofread it to reduce the workload.

[0083] After training, the input text is broadcasted using the speech broadcast model to determine the similarity between the synthesized speech and the actual speech; the preprocessed audio and text annotations are combined into a training data set, the data is organized in the format required by the model, and the model training is started after the training parameters are set.

[0084] Adjust the training parameters of the speech broadcast model according to the determined similarity until the synthesized speech similarity of the trained speech broadcast model meets the requirements. After the training is completed, you can try to input some text for synthesis to evaluate whether the similarity between the synthesized speech and the real person meets the requirements. If not, you can train the model again, add more training data, and adjust parameters such as the learning rate. Repeat the process from training to evaluation until the requirements are met.

[0085] In some embodiments, performing speech synthesis on the output text and performing speech broadcast according to the synthesized speech further comprises: using lip synchronization to generate matching lip movements for the speech broadcast.

[0086] Lip-sync is a technology that achieves perfect synchronization between speech and image by generating lip movements that match the speech audio. The purpose of the lip-sync module is to synchronize the audio output of speech synthesis with the video image of the digital human to make the whole coordinated. The video can be an existing video of the target user or a video generated by the digital human construction technology.

[0087] Specifically, an open-source lip-sync model can be adopted, such as LatentSync and Easy-Wav2Lip, etc. The details are not elaborated here.

[0088] Compared with existing approximate solutions, the method of the present application can communicate with users in real time, rather than simply generating a video with fixed content. This method can make users feel that they are really talking to the target person, such as a relative, thus making up for regrets. In the method of the present application, except for the preparation stage which requires the participation of professionals, no real person is required to act during the use process, thus greatly reducing the demand for the number of professionals and enabling the method to be widely promoted.

[0089] The embodiment of the present application also proposes a data comprehensive processing system, including a processor and a memory. A computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the data comprehensive processing method as described above are implemented.

[0090] It should be noted that in each embodiment of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.

[0091] The serial numbers of the embodiments of the present application above are only for description and do not represent the superiority or inferiority of the embodiments.

[0092] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0093] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative rather than restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims. All of these fall within the protection scope of the present application.

Claims

1. A data comprehensive processing method, characterized in that, Including: Pre-acquire the basic text information of the target person, and establish a RAG-based database according to the basic text information; Obtain the voice data of the user, and recognize the text information in the voice data; Retrieve in the database according to the recognized text information to obtain associated context information; Generate corresponding prompt words based on the context information and the text information; Generate an output text using a large language model according to the generated prompt words; Perform speech synthesis on the output text and perform voice broadcast according to the synthesized voice.

2. The data comprehensive processing method according to claim 1, characterized in that, Establishing a RAG-based database according to the basic text information further includes: Extract the words with special semantics from the basic text information of the target person; Supplement or delete the words with special semantics to form a hot word list; Recognizing the text information in the voice data includes: Transcribe the voice data into text, and recognize the text information according to the hot word list.

3. The data comprehensive processing method according to claim 2, wherein, Retrieve in the database according to the recognized text information to obtain associated context information includes: Embed the recognized text information through an encoder to form a text vector, where the encoder used is the same as the encoder that processes the basic text information in the database; Compare the formed text vector based on the vector index in the database; Take the top K results with the highest similarity, and determine the corresponding K associated context information according to the top K results.

4. The data comprehensive processing method according to claim 2, characterized in that In the case of obtaining the video intercom data of the user, it further includes: Perform face detection based on the video intercom data; Extract face features according to the detected face; Compare the face features with the pre-registered face features, and take the result with the highest comparison similarity as the face recognition result; And, Perform emotion recognition based on the video intercom data; Generate corresponding prompt words based on the face recognition result, emotion recognition result, the context information and the text information.

5. The data comprehensive processing method according to claim 4, wherein Generating corresponding prompt words based on the context information and the text information includes: Determine the event content included in the text information; According to the pre-set prompt word template, fill the recognized event content into the prompt word template, and, In the case of including the face recognition result and emotion recognition result, fill the face recognition result and emotion recognition result in the corresponding positions to generate prompt words.

6. The data comprehensive processing method according to claim 1, characterized in that Performing speech synthesis on the output text and performing voice broadcast according to the synthesized voice includes: Pre-annotate multiple segments of voice materials of the target person, and train a voice broadcast model using the annotated multiple segments of voice materials; After training, input text and use the voice broadcast model to perform broadcast to determine the similarity between the synthesized voice and the actual voice; Adjust the training parameters of the voice broadcast model according to the determined similarity until the synthesized voice similarity of the trained voice broadcast model meets the requirements.

7. The data comprehensive processing method according to claim 6, wherein Performing speech synthesis on the output text and performing voice broadcast according to the synthesized voice further includes: generating matching lip movements for the voice broadcast using lip synchronization.

8. A data comprehensive processing system, characterized in that, It includes a processor and a memory, and a computer program is stored on the memory. When the computer program is executed by the processor, the steps of the data comprehensive processing method according to any one of claims 1 to 7 are implemented.

Images