Knowledge question and answer method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of knowledge questioning and answering, and provides a knowledge questioning and answering method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-answered question of a first user; determining a target knowledge material corresponding to the to-be-answered question based on the knowledge base; determining a target answer based on the target knowledge material; based on the target answer, generating a target virtual human video taking the second user as a virtual human image; the audio in the target virtual human video is obtained by performing voice synthesis on the target answer by using the voiceprint of the second user; the knowledge base is constructed based on at least one of audio, text and documents of the second user. Personalized speech synthesis and a knowledge question-answering system are unified, the problem of experience splitting caused by separation of the two systems in the prior art is avoided, it is ensured that the overall process from question-answering to speech output of a user is smooth and seamless, and the use experience of the user is improved; and the knowledge base of the second user is flexibly utilized to deeply adapt the personality characteristics of the user, so that the use experience of the user is improved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge question answering technology, and in particular to a knowledge question answering method, device, electronic device and storage medium. Background Art

[0002] At present, technologies such as speech synthesis, knowledge retrieval, and Large Language Model (LLM) have been widely used in the field of intelligent interaction.

[0003] However, existing voice interaction and knowledge question-answering systems are mostly implemented independently, and most of them only support generalized response modes and lack deep adaptation to user personality characteristics.

[0004] With the growing demand for personalization in applications such as personal electronic assistants, smart education, and knowledge payment, this limitation has become increasingly prominent, making it difficult for existing technologies to meet users' needs for accurate and natural experience in interactions. Summary of the invention

[0005] The present invention provides a knowledge question and answer method, device, electronic device and storage medium to solve the problem that voice interaction and knowledge question and answer systems in the prior art are mostly implemented independently and mostly only support universal response modes, lack deep adaptation to user personality characteristics, and reduce the user's usage experience.

[0006] The present invention provides a knowledge question answering method, comprising the following steps: Get the first user's unanswered questions; Determine the target knowledge material corresponding to the question to be answered based on the knowledge base; Based on the target knowledge material, determining a target answer to the question to be answered; Based on the target answer, generating a target virtual human video with the second user as the virtual human image; The audio in the target virtual human video is obtained by speech synthesis of the target answer based on the voiceprint of the second user; and the knowledge base is constructed based on at least one of the audio, text and document of the second user.

[0007] According to a knowledge question answering method provided by the present invention, determining a target answer to the question to be answered based on the target knowledge material includes: Based on the target knowledge material, construct a question prompt text; Based on the knowledge question answering model, the question prompt text is applied to generate the target answer.

[0008] According to a knowledge question answering method provided by the present invention, the step of constructing a question prompt text based on the target knowledge material includes: The question prompt text is constructed based on at least two of the target knowledge material, the answer style, the field to which it belongs, the emotional information of the first user, and the question to be answered.

[0009] According to a knowledge question answering method provided by the present invention, generating a target virtual human video with a second user as a virtual human image based on the target answer includes: Based on the virtual human image of the second user and the target answer, generating the target virtual human video; The virtual human image is constructed by collecting the image and voiceprint of the second user.

[0010] According to a knowledge question-answering method provided by the present invention, the target virtual human video is generated based on the virtual human image of the second user and the target answer, and then the method further includes: Determining the broadcasting duration of the target answer; When the broadcast duration is greater than a preset threshold, emotion recognition is performed on the target answer to obtain an emotion recognition result; Based on the emotion recognition result, the target virtual human video is optimized.

[0011] According to a knowledge question answering method provided by the present invention, the step of optimizing the target virtual human video based on the emotion recognition result includes: Based on the emotion recognition result, the synthesized audio in the target virtual human video and the expression and / or action of the virtual human image are optimized.

[0012] According to a knowledge question answering method provided by the present invention, the step of determining the target knowledge material corresponding to the question to be answered based on the knowledge base includes: Determine candidate knowledge materials corresponding to the question to be answered based on the knowledge base; The candidate knowledge material is screened to obtain the target knowledge material.

[0013] The present invention also provides a knowledge question-answering device, comprising the following modules: An acquisition unit, used for acquiring a question to be answered by the first user; A first determining unit, configured to determine, based on a knowledge base, a target knowledge material corresponding to the question to be answered; A second determining unit, configured to determine a target answer to the question to be answered based on the target knowledge material; A generating unit, configured to generate a target virtual human video with the second user as the virtual human image based on the target answer; The audio in the target virtual human video is obtained by speech synthesis of the target answer based on the voiceprint of the second user; and the knowledge base is constructed based on at least one of the audio, text and document of the second user.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described knowledge question-and-answer methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the knowledge question-answering methods described above.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned knowledge question and answer methods.

[0017] The knowledge question and answer method, device, electronic device and storage medium provided by the present invention obtain the unanswered question of the first user, determine the target knowledge material corresponding to the unanswered question based on the knowledge base, and then determine the target answer to the unanswered question based on the target knowledge material. Finally, based on the target answer, a target virtual human video with the second user as the virtual human image is generated, wherein the knowledge base is constructed based on at least one of the audio, text and document of the second user, and the audio in the target virtual human video is obtained by speech synthesis of the target answer based on the voiceprint of the second user. On the one hand, personalized speech synthesis is unified with the knowledge question and answer system to avoid the experience fragmentation problem caused by the separation of the two in the prior art, ensure that the overall process from question and answer to voice output is smooth and seamless, and improve the user experience; on the other hand, the knowledge base of the second user is flexibly used in knowledge retrieval and answer generation, so as to deeply adapt the personality characteristics of the first user and further improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the knowledge question and answer method provided by the present invention.

[0020] Figure 2 It is a flowchart of step 130 in the knowledge question and answer method provided by the present invention.

[0021] Figure 3 It is a schematic diagram of the process of optimizing the target virtual human video in the knowledge question-answering method provided by the present invention.

[0022] Figure 4 It is a flowchart of step 120 in the knowledge question and answer method provided by the present invention.

[0023] Figure 5 This is the second flow chart of the knowledge question and answer method provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the knowledge question and answer device provided by the present invention.

[0025] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0028] Currently, personalized speech synthesis and knowledge question-answering systems have been widely used in the field of intelligent interaction. However, in existing solutions, the two are usually separated and no effective fusion mechanism has been formed.

[0029] On the one hand, with the support of speech synthesis technology, some voice interaction systems can generate voice output with a specific tone by sampling the voiceprint of user voice data, but the output content of such systems is mostly based on text broadcasts, lacking the ability to deeply adapt to and update the user's personalized knowledge in real time, and it is difficult to generate personalized answers that highly meet the needs of specific users. On the other hand, question-answering systems based on knowledge retrieval usually generate content by building an external knowledge base system combined with a large language model. Such systems can quickly extract information from the knowledge base and provide accurate content feedback, but they usually ignore the personalized needs in the interaction process. The answer style and content are still pre-designed and universal, and cannot reflect the expression habits and tone of a specific user.

[0030] In summary, the existing technical solutions mainly remain at generating personalized voices through standard speech synthesis models or multi-timbre templates, but it is difficult to achieve deep personalization in the joint use of knowledge questions and answers and speech synthesis. Especially in personalized interactive scenarios, the existing solutions cannot effectively simulate the user's thinking habits, language characteristics and unique knowledge, and lack all-round personalization from content to timbre.

[0031] Based on the above problems, the present invention provides a knowledge question answering method. Figure 1 is one of the flow charts of the knowledge question answering method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0032] Step 110: Obtain the unanswered questions of the first user.

[0033] Specifically, the unanswered questions of the first user may be obtained. Here, the first user refers to a system user, that is, an external user, and the first user may be defined as a Class C user.

[0034] Among them, the questions to be answered refer to the questions that need to be answered in the future. The questions to be answered can be in voice form or in text form. The questions in voice form can come from the user's direct oral statement or can be obtained through a sound pickup device. The sound pickup device here can be a smart phone, a tablet computer, or a smart appliance, such as a stereo, a television, and an air conditioner. After the sound pickup device obtains the voice data through the microphone array, it can also amplify and reduce the noise of the voice data.

[0035] The text-based questions may be directly input by the user, or may be obtained by transcribing the collected audio, or may be obtained by collecting images through image collection devices such as scanners, mobile phones, cameras, tablets, and performing OCR (Optical Character Recognition) on the images. This embodiment of the present invention does not specifically limit this.

[0036] Step 120, determining the target knowledge material corresponding to the question to be answered based on a knowledge base; the knowledge base is constructed based on at least one of the audio, text and document of the second user.

[0037] Specifically, after obtaining the unanswered question of the first user, the target knowledge material corresponding to the unanswered question can be determined based on the knowledge base. Here, the target knowledge material corresponding to the unanswered question is the material recommended by the knowledge base and used to determine the target answer.

[0038] The knowledge base is constructed based on at least one of the audio, text and document of the second user. That is, the knowledge base can be constructed based on the audio and text of the second user, or based on the audio and document of the second user, etc. Among them, the second user is an entity user, and the second user can be defined as a P-type user.

[0039] It is understandable that P-type users are responsible for knowledge production, while C-type users are responsible for knowledge consumption.

[0040] Specifically, the second user's audio, text, document and other knowledge materials can be collected, and the knowledge materials can be cleaned and organized through the capabilities of large language models such as audio transcription and document parsing, and the knowledge can be vectorized and stored to obtain a knowledge base.

[0041] Among them, the vectorized storage of knowledge is equivalent to knowledge construction. Text data can be converted into vector representation by using natural language processing technology, such as Word2Vec, and Transformer-based models such as BERT (Bidirectional Encoder Representations from Transformers) model, etc., which is not specifically limited in the embodiment of the present invention.

[0042] Correspondingly, determining the target knowledge material corresponding to the question to be answered based on the knowledge base is a knowledge retrieval step, that is, utilizing the capability of vector knowledge fuzzy retrieval to obtain the target knowledge material of the second user that is closest to the question to be answered by the first user, as prior knowledge for the subsequent question-and-answer session.

[0043] For example, the question to be answered can be converted into a query vector, and the similarity between the query vector and each vector in the knowledge base can be calculated. The similarity between the query vector and each vector in the knowledge base can be calculated using cosine similarity, Euclidean distance, etc., which is not specifically limited in the embodiments of the present invention.

[0044] It can be understood that these methods are able to quantify the differences between vectors, thereby achieving the effect of fuzzy retrieval.

[0045] Step 130: Determine a target answer to the question to be answered based on the target knowledge material.

[0046] Specifically, after obtaining the target knowledge material, the target answer to the question to be answered can be determined. Here, the target answer is the knowledge answer to the question to be answered that is finally obtained.

[0047] For example, the target knowledge material can be used as question prompt text, and the question prompt text can be used to determine the target answer to the question to be answered. The target knowledge material, answer style and field to which it belongs can also be used as question prompt text, and the question prompt text can be used to determine the target answer to the question to be answered. The embodiments of the present invention do not make specific limitations on this.

[0048] Step 140, generating a target virtual human video with the second user as the virtual human image based on the target answer; The audio in the target virtual human video is obtained by voice synthesis of the target answer based on the voiceprint of the second user.

[0049] Specifically, after obtaining the target answer, a target virtual human video with the second user as the virtual human image may be generated based on the target answer.

[0050] The audio in the target virtual human video is obtained by synthesizing the target answer with the voiceprint of the second user. The speech synthesis of the target answer with the voiceprint of the second user can be implemented by a system with a large language model deployed. The large language model can be a pre-trained language model such as iFlytek Spark, XLNet (extreme Multi-label Learning Network), ROBERTa (Robustly Optimized BERT approach), etc., which is not specifically limited in the embodiment of the present invention.

[0051] Here, the virtual human image of the second user is constructed by collecting the image and voiceprint of the second user. It is understandable that when collecting the image of the second user, a virtual human image can be constructed using a picture virtual human or super-anthropomorphic technology, or a face detection algorithm can be used to locate the face area of ​​the second user, and then a posture estimation algorithm can be used to locate the key parts of the second user's body, such as the head, torso and limbs. For capturing facial expressions, a facial feature point detection algorithm can be used to locate the key feature points of the second user's face, and the expression information can be inferred accordingly. Finally, based on the captured posture and expression information, a virtual human image can be constructed, and the embodiments of the present invention do not specifically limit this.

[0052] According to the target's answer, appropriate actions and expressions can be selected from the pre-made animation library and applied to the avatar, or the avatar's expressions and actions can be directly generated based on the image avatar or super-anthropomorphic technology. In addition, if the target answer is voice content, lip synchronization processing is required to ensure that the avatar's mouth shape matches the voice content.

[0053] After animation synthesis and lip synchronization processing, a target virtual human video in which a virtual human image reads the target answer can be generated.

[0054] It can be understood that the knowledge base is constructed based on at least one of the second user's audio, text and documents, the audio in the target virtual human video is obtained by speech synthesis of the target answer based on the second user's voiceprint, the target answer is determined based on the knowledge base, and then based on the target answer, a target virtual human video with the second user as the virtual human image is generated, thereby effectively simulating the user's thinking habits, language characteristics and unique knowledge, and realizing all-round personalization from content to timbre.

[0055] The method provided by the embodiment of the present invention obtains the unanswered question of the first user, determines the target knowledge material corresponding to the unanswered question based on the knowledge base, and then determines the target answer to the unanswered question based on the target knowledge material. Finally, based on the target answer, a target virtual human video with the second user as the virtual human image is generated, wherein the knowledge base is constructed based on at least one of the audio, text and document of the second user, and the audio in the target virtual human video is obtained by speech synthesis of the target answer based on the voiceprint of the second user. On the one hand, personalized speech synthesis is unified with the knowledge question and answer system to avoid the experience fragmentation problem caused by the separation of the two in the prior art, ensuring that the overall process from question and answer to voice output is smooth and seamless, thereby improving the user experience; on the other hand, the knowledge base of the second user is flexibly utilized in knowledge retrieval and answer generation, thereby deeply adapting the personality characteristics of the first user to further improve the user experience.

[0056] Based on the above embodiments, Figure 2 is a flow chart of step 130 in the knowledge question answering method provided by the present invention, such as Figure 2 As shown, step 130 includes: Step 131, constructing a question prompt text based on the target knowledge material; Step 132, based on the knowledge question answering model, apply the question prompt text to generate the target answer.

[0057] Specifically, the question prompt text may be constructed based on the target knowledge material, or based on at least two of the target knowledge material, answer style, field, emotional information of the first user, and the question to be answered.

[0058] Here, the answer style may include a formal and rigorous style, a friendly and natural style, a concise and clear style, a dialectical and analytical style, etc., and the embodiment of the present invention does not make any specific limitation to this.

[0059] The fields may include finance, agriculture, and computer, etc., which are not specifically limited in the embodiments of the present invention.

[0060] The question to be answered, i.e., the original question asked by the first user, may be "What are the basic structures and functions of the financial market?" or "What water-saving technologies are there for agricultural irrigation?", etc., and this embodiment of the present invention does not specifically limit this.

[0061] It should be noted that the question prompt text includes the emotional information of the first user, so as to more comprehensively understand the needs and context of the first user, and then generate a target answer that is more in line with the personalized characteristics of the first user, further improving the user experience.

[0062] In summary, the question to be answered raised by the first user is semantically packaged and formatted, and a question prompt text (prompt) is generated that contains target knowledge material (prior knowledge), answer style, field, emotional information of the first user, and the question to be answered, so as to achieve emotional optimization processing of the target answer and ensure the consistency and fluency of the emotion of the final feedback.

[0063] Here, the knowledge question answering model may be a large language model, and the large language model may be a pre-trained model that has the ability to apply question prompt text and generate target answers.

[0064] It can be understood that the question prompt text is equivalent to a prompt, that is, prior knowledge. Based on the knowledge question-answering model, the question prompt text is applied to generate a target answer, thereby ensuring that a more accurate and targeted target answer can be generated based on the question prompt text, that is, ensuring the accuracy and effectiveness of the subsequent target virtual human video with the second user as the virtual human image generated based on the target answer.

[0065] It should be noted that if the target knowledge material is not returned after the knowledge retrieval, it means that the second user does not have an acceptable relevant statement for the first user's question to be answered, and it is necessary to splice an indicator of no prior knowledge in the question prompt text; if multiple knowledge materials are returned after the knowledge retrieval, it is necessary to perform knowledge screening on the multiple knowledge materials to ensure the accuracy of the prior knowledge match.

[0066] The method provided by the embodiment of the present invention constructs a question prompt text based on the target knowledge material, and then generates a target answer by applying the question prompt text based on the knowledge question and answer model. In this process, the question prompt text is equivalent to a prompt, and the question prompt text is applied based on the knowledge question and answer model to generate a target answer, thereby ensuring that a more accurate and targeted target answer can be generated based on the question prompt text, that is, ensuring the accuracy and effectiveness of the subsequent generation of a target virtual human video with the second user as the virtual human image based on the target answer.

[0067] Based on the above embodiment, step 140 includes: Step 141, generating the target virtual human video based on the virtual human image of the second user and the target answer; The virtual human image is constructed by collecting the image and voiceprint of the second user.

[0068] Specifically, the virtual image of the second user is constructed by collecting the image and voiceprint of the second user. Here, when collecting the image of the second user, a video of the second user can be collected. The video of the second user can be a pre-shot and stored video or a real-time collected video stream, which is not specifically limited in the embodiment of the present invention.

[0069] Furthermore, an image frame sequence can be acquired from the video of the second user, the image frame sequence comprising multiple frames of images, each frame of image being derived from the video of the second user, and the multiple frames of image being arranged in chronological order in the video of the second user, thereby forming an image frame sequence.

[0070] It should be noted that when the video of the second user is collected, it is usually collected in a uniform order based on the total number of frames of the video of the second user, so that the time intervals between each frame image are equal. Alternatively, each frame image in the video of the second user can be directly combined in time sequence to form an image frame sequence.

[0071] Furthermore, after the image frame sequence is acquired, a preprocessing operation may be performed on the image frame sequence, and the preprocessing operation includes but is not limited to normalization processing, sharpening processing, denoising processing, etc.

[0072] Therefore, a virtual human image may be constructed for the image frame sequence of the second user using a picture virtual human, and a virtual human image may also be constructed for the video of the second user using a super-anthropomorphic technology.

[0073] When collecting the voiceprint of the second user, the audio of the second user can be collected. Here, the audio of the second user can be obtained through a sound pickup device. The sound pickup device can be a smart phone, a tablet computer, or a smart appliance, such as a stereo, a television, and an air conditioner. After the sound pickup device obtains the audio of the second user through a microphone array, the audio of the second user can also be amplified and noise reduced. The embodiment of the present invention does not specifically limit this.

[0074] Finally, according to the content of the target answer, appropriate actions and expressions can be selected from the pre-made animation library and applied to the virtual human image, or the virtual human image's expressions and actions can be directly generated based on the picture virtual human or super-anthropomorphic technology. In addition, if the target answer is voice content, lip synchronization processing is required to ensure that the lip shape of the virtual human image matches the voice content. After animation synthesis and lip synchronization processing, a target virtual human video can be generated in which the virtual human image broadcasts the target answer.

[0075] It can be understood that by modeling the multimodal features of the second user, such as audio features and video features, through the personalized virtual image acquisition module, the foundation is laid for further realizing a higher level of personalized interaction, meeting the future upgrade needs of virtual human technology, and achieving high scalability of dynamic personalized adaptation.

[0076] The method provided by the embodiment of the present invention generates a target virtual human video based on the virtual human image of the second user and the target answer, wherein the virtual human image is constructed by collecting the image and voiceprint of the second user, thereby realizing personalized adaptation of the voiceprint, and then realizing the personalization of the voiceprint and subsequent personalized adaptation of the virtual human image, thereby improving the user experience.

[0077] Based on the above embodiments, Figure 3 is a flow chart of optimizing the target virtual human video in the knowledge question answering method provided by the present invention, such as Figure 3 As shown, step 141, and then further includes: Step 1411, determining the broadcasting duration of the target answer; Step 1412, when the broadcast duration is greater than a preset threshold, emotion recognition is performed on the target answer to obtain an emotion recognition result; Step 1413: Optimize the target virtual human video based on the emotion recognition result.

[0078] Specifically, considering that the broadcasting duration of the target answer meets a certain length, an emotional feedback loop can be constructed.

[0079] First, the broadcasting duration of the target answer can be determined, and when the broadcasting duration is greater than a preset threshold, emotion recognition is performed on the target answer to obtain an emotion recognition result.

[0080] Here, emotion recognition of the target answer may be performed using a large language model, or a support vector machine (SVM), decision tree, random forest, KNN (k-nearest neighbor), naive Bayes, etc., which is not specifically limited in the embodiment of the present invention.

[0081] After obtaining the emotion recognition result, the target virtual human video can be optimized based on the emotion recognition result. For example, based on the emotion recognition result, the audio of the target virtual human video, the expression and action of the virtual human image can be optimized, and personalized learning resource recommendation and homework difficulty adjustment can also be optimized, which is not specifically limited in the embodiment of the present invention.

[0082] The method provided by the embodiment of the present invention determines the broadcasting duration of the target answer, and when the broadcasting duration is greater than a preset threshold, performs emotion recognition on the target answer to obtain an emotion recognition result. Finally, based on the emotion recognition result, the target virtual human video is optimized, emotion recognition analysis is added, and the emotion recognition result is used as an intervention factor for the synthesized target virtual human video to optimize the generation effect of the target virtual human video.

[0083] Based on the above embodiment, step 1413 includes: Based on the emotion recognition result, the synthesized audio in the target virtual human video and the expression and / or action of the virtual human image are optimized.

[0084] Specifically, the synthesized audio in the target virtual human video and the expression and / or action of the virtual human image can be optimized based on the emotion recognition result. That is, based on the emotion recognition result, the synthesized audio in the target virtual human video and the expression of the virtual human image can be optimized, the synthesized audio in the target virtual human video and the action of the virtual human image can be optimized, and the synthesized audio in the target virtual human video and the expression and action of the virtual human image can also be optimized, which is not specifically limited in the embodiment of the present invention.

[0085] For example, based on the emotion recognition results, the emotion labels (such as happiness, sadness, anger, etc.) are mapped to the pitch features of the synthesized audio in the corresponding virtual human video. That is, based on the mapping results, the pitch, volume, speech speed and other parameters of the synthesized audio are adjusted to match the expression of specific emotions.

[0086] For example, happy emotions can correspond to higher pitch, faster speaking speed and louder volume; sad emotions can correspond to lower pitch, slower speaking speed and lower volume.

[0087] For another example, the emotion recognition results can also be applied to the optimization of the super-anthropomorphic image. Through the emotion recognition results, the expression and movement of the super-anthropomorphic image can be adjusted to make it closer to human emotional expression. For example, when the emotion recognition result is joy, the super-anthropomorphic image can show a smile and cheerful movements; when the emotion recognition result is sadness, it can show a sad and heavy expression and movement.

[0088] Based on the above embodiments, Figure 4is a flow chart of step 120 in the knowledge question answering method provided by the present invention, such as Figure 4 As shown, step 120 includes: Step 121, determining candidate knowledge materials corresponding to the question to be answered based on the knowledge base; Step 122: perform knowledge screening on the candidate knowledge materials to obtain the target knowledge materials.

[0089] Specifically, considering that there may be multiple knowledge materials corresponding to the question to be answered based on the knowledge base, candidate knowledge materials corresponding to the question to be answered can be determined based on the knowledge base, wherein the candidate knowledge materials refer to knowledge materials that may be used as target knowledge materials in the future.

[0090] After obtaining the candidate knowledge materials, the candidate knowledge materials can be screened to obtain the target knowledge materials. The knowledge screening can be to calculate the similarity between the two vectors corresponding to the multiple candidate knowledge materials and the question to be answered, and sort the similarity calculation results from small to large, and take the candidate knowledge material with the lowest ranking as the target knowledge material.

[0091] The method provided by the embodiment of the present invention determines the candidate knowledge materials corresponding to the question to be answered based on the knowledge base, and then performs knowledge screening on the candidate knowledge materials to obtain the target knowledge materials, thereby performing knowledge screening on the knowledge materials to ensure the accuracy and reliability of the subsequent generation of the target answer based on the target knowledge materials, that is, to ensure the accuracy of the prior knowledge matching.

[0092] In related technologies, speech synthesis and knowledge question-answering systems are mostly implemented independently, that is, they are separate system architectures and lack a fusion mechanism; there is a lack of effective use of users' personal knowledge bases, and most systems cannot flexibly call users' exclusive knowledge bases or historical data, thus lacking support for users' personal knowledge in knowledge retrieval and answer generation; there is a lack of real-time performance, which is mainly manifested in the lack of real-time detection and switching of users' expression styles.

[0093] Based on any of the above embodiments, Figure 5 This is the second flow chart of the knowledge question answering method provided by the present invention. Figure 5 As shown in the figure, the main body of the method is divided into three major modules, including personalized virtual image acquisition module, knowledge base question and answer module, and personalized interaction module. Among them, the personalized virtual image acquisition module and the knowledge base question and answer module are system front-end application modules, which are mainly used for user information collection, virtual image reproduction, and knowledge vector storage; the personalized interaction module includes the request twisting of user questions and the processing of output content form.

[0094] The stakeholders of this method include the main body users and system users. It is necessary to virtualize the personal image of the main body users in advance, involving the first two modules of the three main modules in the method: the system user is an external user who asks questions and conducts question-answering behavior with the personalized interaction module to obtain relevant knowledge and answers returned by the main body users. The two types of users are now distinguished by symbols, and the user main body is recorded as a P-type user and the system user is recorded as a C-type user.

[0095] The above-mentioned personalized virtual image collection module mainly includes the following steps: user voiceprint collection, uploading the recorded audio uploaded by P-type users to the voiceprint recognition engine, collecting user voiceprint features, and returning the unique voiceprint identifier of P-type users for storage; user image collection, collecting photos or personal videos uploaded by P-type users, and constructing the virtual image export channel of the P-type user (super-anthropomorphic image construction module) through picture virtual people or super-anthropomorphic technology for use in subsequent personalized interaction modules.

[0096] The knowledge base question and answer module mainly includes the following two sub-links: the core knowledge question and answer module and the retrieval-based knowledge base. As the core module of this method, it is necessary to package the original questions (questions to be answered) of category C users before question and answer, and construct a prompt containing multiple information combinations such as question prompt text (prior knowledge), answer style, field, original question, etc.; the prior knowledge is provided by the retrieval-based knowledge base.

[0097] Among them, the retrieval-type knowledge base includes two steps: knowledge construction and knowledge retrieval. The main steps of knowledge construction are: pre-collecting knowledge materials such as audio recordings and personal documents of P-type users, cleaning and organizing knowledge materials through AI capabilities such as document parsing and transcription, and vectorizing and storing knowledge; the main steps of knowledge retrieval are: inputting user questions of C-type users into the upper-level service, performing content retrieval on user questions of C-type users, and using the fuzzy retrieval capability of vector knowledge to obtain target knowledge materials of P-type users with similar question content based on the personal knowledge base of P-type users as prior knowledge for subsequent question-and-answer sessions.

[0098] Among them, the upper-layer service is a service at a higher level in the service architecture. It relies on the basic functions and data provided by the underlying service and implements more specific business logic or functions on this basis. It can be understood that the upper-layer service is responsible for receiving and processing user input, such as user questions, user input, etc., and converting it into a form that can be understood and processed by the underlying service. After receiving the user's question, the upper-layer service processes it according to the business logic, such as converting the user's question into a format, and performing knowledge retrieval in the P-type user personal knowledge base based on the format-converted user question to obtain the target knowledge material, etc. The upper-layer service calls the functions and data provided by the underlying service when necessary to implement more complex business logic or functions, for example, based on the super-anthropomorphic image and the target answer output by the large language model, the target virtual human video is generated. The upper-layer service returns the target virtual human video to the user to meet the user's knowledge question and answer needs.

[0099] The core knowledge question and answer mainly includes the following details: if no prior knowledge content is returned after knowledge retrieval based on the user's unanswered questions, that is, no knowledge material is returned, it means that the P-type user has not responded to the content of the C-type user's question, and there is a relevant expression that can be adopted, so it is necessary to mark the absence of prior knowledge in the prompt splicing; if multiple knowledge materials are returned after knowledge retrieval based on the user's unanswered questions, it is necessary to increase the screening of knowledge materials to ensure the accuracy of prior knowledge matching. The above returned assembled prompt is used as a parameter, and a question and answer request is made based on the large language model to obtain the returned target answer as the input content of the above personalized interaction module.

[0100] Furthermore, after obtaining the response to the question asked by Category C users through the knowledge quiz, the target answer is processed through the personalized virtual image. The main steps include: first, the target answer is sent as input to the broadcast module, and the target answer is broadcast in combination with the unique voiceprint identifier collected by the user and the super-anthropomorphic image; secondly, when the broadcast content of the target answer accumulates to a certain length, an emotional feedback loop is constructed, that is, emotional analysis of all the previous content is performed to predict the emotion of the subsequent broadcast content, which is used as a fine-tuning parameter for personalized interaction to optimize the synthesized audio tone, super-anthropomorphic image expressions and movements, etc.

[0101] It can be understood that the personalized personal knowledge question and answer interactive system in the embodiment of the present invention, with the help of knowledge retrieval, speech synthesis and large language model question and answer technologies, deeply learns the user's expression habits, thinking mode and voice characteristics to provide a more accurate and natural interactive question and answer experience.

[0102] The method provided by the embodiment of the present invention deeply integrates personalized speech synthesis with the knowledge question and answer system, and realizes seamless connection of information collection, knowledge management, and speech output through modular design. This integrated design avoids the experience of speech synthesis and knowledge question and answer being separated in the prior art, making the process from question and answer to speech feedback smoother and more natural, and can detect and switch the user's expression style in real time, further improving the user's experience.

[0103] The knowledge question and answer device provided by the present invention is described below. The knowledge question and answer device described below and the knowledge question and answer method described above can be referenced to each other.

[0104] Based on any of the above embodiments, the present invention provides a knowledge question answering device, Figure 6 is a schematic diagram of the structure of the knowledge question answering device provided by the present invention, such as Figure 6 As shown, the device comprises: An acquisition unit 610 is used to acquire a question to be answered by the first user; A first determining unit 620 is used to determine the target knowledge material corresponding to the question to be answered based on the knowledge base; A second determining unit 630 is used to determine a target answer to the question to be answered based on the target knowledge material; A generating unit 640 is configured to generate a target virtual human video with the second user as the virtual human image based on the target answer; The audio in the target virtual human video is obtained by speech synthesis of the target answer based on the voiceprint of the second user; and the knowledge base is constructed based on at least one of the audio, text and document of the second user.

[0105] The device provided by the embodiment of the present invention obtains the unanswered question of the first user, determines the target knowledge material corresponding to the unanswered question based on the knowledge base, and then determines the target answer to the unanswered question based on the target knowledge material. Finally, based on the target answer, a target virtual human video with the second user as the virtual human image is generated, wherein the knowledge base is constructed based on at least one of the audio, text and document of the second user, and the audio in the target virtual human video is obtained by speech synthesis of the target answer based on the voiceprint of the second user. On the one hand, personalized speech synthesis is unified with the knowledge question and answer system to avoid the experience fragmentation problem caused by the separation of the two in the prior art, ensure that the overall process from question and answer to voice output is smooth and seamless, and improve the user experience; on the other hand, the knowledge base of the second user is flexibly utilized in knowledge retrieval and answer generation, so as to deeply adapt the personality characteristics of the first user and further improve the user experience.

[0106] Based on any of the foregoing embodiments, the second determining unit 630 specifically includes: A question prompt text construction unit is used to construct a question prompt text based on the target knowledge material; A target answer generation unit is used to generate the target answer based on the knowledge question answering model and applying the question prompt text.

[0107] Based on any of the above embodiments, the constructing question prompt text unit is specifically used for: The question prompt text is constructed based on at least two of the target knowledge material, the answer style, the field to which it belongs, the emotional information of the first user, and the question to be answered.

[0108] Based on any of the above embodiments, the generating unit 640 is specifically configured to: Based on the virtual human image of the second user and the target answer, generating the target virtual human video; The virtual human image is constructed by collecting the image and voiceprint of the second user.

[0109] Based on any of the above embodiments, it further includes an emotion recognition unit, and the emotion recognition unit is specifically used to: A broadcast duration determination unit, used to determine the broadcast duration of the target answer; A recognition unit, configured to perform emotion recognition on the target answer when the broadcast duration is greater than a preset threshold, and obtain an emotion recognition result; An optimization unit is used to optimize the target virtual human video based on the emotion recognition result.

[0110] Based on any of the above embodiments, the optimization unit is specifically used to: Based on the emotion recognition result, the synthesized audio in the target virtual human video and the expression and / or action of the virtual human image are optimized.

[0111] Based on any of the foregoing embodiments, the first determining unit 620 is specifically configured to: Determine candidate knowledge materials corresponding to the question to be answered based on the knowledge base; The candidate knowledge material is screened to obtain the target knowledge material.

[0112] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the knowledge question-answering method, which includes: obtaining the question to be answered of the first user; determining the target knowledge material corresponding to the question to be answered based on the knowledge base; determining the target answer to the question to be answered based on the target knowledge material; generating a target virtual human video with the second user as the virtual human image based on the target answer; the audio in the target virtual human video is obtained by synthesizing the target answer by voice using the voiceprint of the second user; the knowledge base is constructed based on at least one of the audio, text and document of the second user.

[0113] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0114] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the knowledge question and answer method provided by the above methods, which includes: obtaining a question to be answered by a first user; determining a target knowledge material corresponding to the question to be answered based on a knowledge base; determining a target answer to the question to be answered based on the target knowledge material; generating a target virtual human video with a second user as a virtual human image based on the target answer; the audio in the target virtual human video is obtained by speech synthesis of the target answer based on the voiceprint of the second user; the knowledge base is constructed based on at least one of the audio, text and document of the second user.

[0115] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the knowledge question and answer method provided by the above-mentioned methods, the method comprising: obtaining a question to be answered of a first user; determining a target knowledge material corresponding to the question to be answered based on a knowledge base; determining a target answer to the question to be answered based on the target knowledge material; generating a target virtual human video with a second user as a virtual human image based on the target answer; the audio in the target virtual human video is obtained by speech synthesis of the target answer using the voiceprint of the second user; the knowledge base is constructed based on at least one of the audio, text and document of the second user.

[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge question answering method, characterized in that: include: Get the first user's unanswered questions; Determine the target knowledge material corresponding to the question to be answered based on the knowledge base; Based on the target knowledge material, determining a target answer to the question to be answered; Based on the target answer, generating a target virtual human video with the second user as the virtual human image; The audio in the target virtual human video is obtained by speech synthesis of the target answer based on the voiceprint of the second user; and the knowledge base is constructed based on at least one of the audio, text and document of the second user.

2. The knowledge question answering method according to claim 1, characterized in that: Determining a target answer to the question to be answered based on the target knowledge material includes: Based on the target knowledge material, construct a question prompt text; Based on the knowledge question answering model, the question prompt text is applied to generate the target answer.

3. The knowledge question answering method according to claim 2, characterized in that: The step of constructing a question prompt text based on the target knowledge material includes: The question prompt text is constructed based on at least two of the target knowledge material, the answer style, the field to which it belongs, the emotional information of the first user, and the question to be answered.

4. The knowledge question answering method according to claim 1, characterized in that: The step of generating a target virtual human video with the second user as the virtual human image based on the target answer includes: Based on the virtual human image of the second user and the target answer, generating the target virtual human video; The virtual human image is constructed by collecting the image and voiceprint of the second user.

5. The knowledge question answering method according to claim 4, characterized in that: The method further comprises generating the target virtual human video based on the virtual human image of the second user and the target answer, and then: Determining the broadcasting duration of the target answer; When the broadcast duration is greater than a preset threshold, emotion recognition is performed on the target answer to obtain an emotion recognition result; Based on the emotion recognition result, the target virtual human video is optimized.

6. The knowledge question answering method according to claim 5, characterized in that: The step of optimizing the target virtual human video based on the emotion recognition result includes: Based on the emotion recognition result, the synthesized audio in the target virtual human video and the expression and / or action of the virtual human image are optimized.

7. The knowledge question answering method according to any one of claims 1 to 6, characterized in that: The determining, based on the knowledge base, the target knowledge material corresponding to the question to be answered includes: Determine candidate knowledge materials corresponding to the question to be answered based on the knowledge base; The candidate knowledge material is screened to obtain the target knowledge material.

8. A knowledge question-answering device, characterized in that: include: An acquisition unit, used for acquiring a question to be answered by the first user; A first determining unit, configured to determine, based on a knowledge base, a target knowledge material corresponding to the question to be answered; A second determining unit, configured to determine a target answer to the question to be answered based on the target knowledge material; A generating unit, configured to generate a target virtual human video with the second user as the virtual human image based on the target answer; The audio in the target virtual human video is obtained by speech synthesis of the target answer based on the voiceprint of the second user; and the knowledge base is constructed based on at least one of the audio, text and document of the second user.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the knowledge question and answer method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the knowledge question and answer method as described in any one of claims 1 to 7 is implemented.

Images