Interactive virtual expert image generation method and system

Abstract

The invention provides an interactive virtual expert image generation method and system. The method comprises the following steps: acquiring voice data of a user, preset character image data and preset voice data; determining input text data according to the user voice data; inputting the input text data into a question and answer model for processing, and determining virtual expert image response text data; determining synthetic voice of the virtual expert image according to preset voice data and response text data; inputting the preset character image data into a preset rendering model, and determining a virtual expert image model; determining face and limb animations of the virtual expert image according to the synthesized voice and a virtual expert image model; and synthesizing the face and limb animation of the virtual expert image with the synthesized voice, and determining an interactive virtual expert image animation. According to the invention, natural, smooth and vivid user interaction is realized. Interrogation service, medication reminding and health management are provided for patients, the burden of doctors is relieved, and the medical efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for generating an interactive virtual expert image. Background Art

[0002] Thanks to the rapid development of technologies such as artificial intelligence, virtual reality, augmented reality, and machine learning, digital human technology has made significant progress in the past few decades. The "digital human + medical" model came into being, accurately matching the urgent needs of the market and showing great development potential. From the perspective of technical implementation, the deep integration of medical knowledge graphs and digital human technology can not only provide patients with convenient consultation services, medication reminders, and health management support, but also help doctors efficiently obtain key medical information, significantly improving the efficiency of diagnosis and treatment.

[0003] Faced with challenges such as offline queues, cumbersome payment, large number of manual inquiries, tight human resources in small and medium-sized hospitals, and difficulty in obtaining expert appointments in large hospitals in the traditional medical system, these pain points not only increase the workload of medical staff, but may also delay the best time for diagnosis and treatment for patients. The introduction of digital human technology has appropriately filled this gap and brought significant results in reducing costs and increasing efficiency for medical institutions.

[0004] Although the current question-answering model of medical knowledge graph has begun to take shape, its plain text output still limits its further expansion of application scope to a certain extent. At the same time, existing digital life technologies generally rely on complex artificial animation design and manual modeling processes, which is not only tedious and time-consuming, but also consumes a lot of human resources and time costs. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a method and system for generating an interactive virtual expert image.

[0006] According to one aspect of the present invention, there is provided a method for generating an interactive virtual expert image, comprising:

[0007] Acquire the user's voice data, preset character image data and preset voice data;

[0008] Determining input text data according to the user voice data;

[0009] Inputting the input text data into the question-answering model for processing to determine the virtual expert image answer text data;

[0010] Determining the synthesized voice of the virtual expert image according to the preset voice data and the response text data;

[0011] Inputting the preset character image data into a preset rendering model to determine a virtual expert image model;

[0012] Determining facial and body animation of the virtual expert image according to the synthesized speech and the virtual expert image model;

[0013] The face and body animation of the virtual expert image are synthesized with the synthesized speech to determine an interactive virtual expert image animation.

[0014] Preferably, the voice data of the user is obtained through a recording device.

[0015] The preset character image data is a character image collected from the Internet or in reality, and is used to synthesize a virtual expert image;

[0016] The preset voice data is voice data collected from the network or in real life, and is used to synthesize the voice of the virtual expert image. Preferably, the step of determining the input text data based on the user voice data includes:

[0017] T n =FunASR(S)

[0018] Among them, T n represents the input text data, S represents the user voice data, and FunASR represents the process of processing the voice data.

[0019] Preferably, the step of inputting the input text data into the question-answering model for processing to determine the virtual expert image answering text data comprises:

[0020] R n (EXT,RE)=Chat(T n , [EXT, KEY], N)

[0021] Among them, R n represents the response text data of the virtual expert image, Chat represents the process of the question-answering model responding to the input text data in the nth round of interaction, EXT represents the disease name data extracted from the input text, KEY represents the data related to the disease name extracted from the input text, RE represents the data related to the disease name obtained from the question-answering model, N represents the number of rounds of dialogue, and [EXT, KEY] represents the request body in the response process of the question-answering model.

[0022] Preferably, the question-answering model is a model based on a knowledge graph, which can mark related diseases from the disease description text, and based on this, obtain the cause of the disease, mark the location where the corresponding disease occurs, find the symptoms of the disease, check the corresponding physiological indicators through the symptoms, determine the corresponding disease, and after determining the disease, determine the required drugs for the disease based on the treatment method.

[0023] Preferably, determining the synthesized voice of the virtual expert image according to the preset voice data and the response text data includes:

[0024] The preset voice data is input into the pre-trained voice cloning model to determine the voice characteristics of the digital human.

[0025] F A =f(A)

[0026] Among them, F A represents the voice feature of the virtual expert image, A represents the preset voice data, and f represents the voice feature extraction operation performed using the pre-trained voice cloning model;

[0027] S=Voice(F A ,R n )

[0028] Wherein, S represents the synthesized speech of the virtual expert image, Voice represents the synthesized speech operation performed by the speech cloning model, and R n The virtual expert image's response text data is represented.

[0029] Preferably, the step of inputting the preset character image data into a preset rendering model to determine the virtual expert image model includes:

[0030] Y=Human(Figure)

[0031] Among them, Y represents the virtual expert image model, Figure represents the preset character image data, and Human represents the rendering operation using the preset rendering model. The rendering models include the HumanNeRF model and the HumanGaussian model, which can respectively obtain a three-dimensional character image by rendering text or pictures.

[0032] Preferably, determining the face and body animation of the virtual expert image based on the synthesized speech and the virtual expert image model includes:

[0033] A=Pan(S,Y)

[0034] Wherein, A represents the synthesized animation of the virtual expert image, Pan represents the animation generation operation using the action generation model, and S represents the synthesized speech of the virtual expert image.

[0035] Preferably, synthesizing the face and body animation of the virtual expert image with the synthesized speech to determine the interactive virtual expert image animation includes:

[0036] D=A+S

[0037] Wherein, D represents the interactive virtual expert image animation, and + represents the synthesis processing operation.

[0038] According to a second aspect of the present invention, there is provided an interactive virtual expert image generation system, comprising:

[0039] Data acquisition module: acquires user voice data, preset character image data and preset voice data, and determines input text data according to the user voice data;

[0040] Question and answer module: input the input text data into the question and answer model for processing, and determine the virtual expert image answer text data;

[0041] A speech synthesis module: determining the synthesized speech of the virtual expert image according to the preset speech data and the response text data;

[0042] Rendering module: inputting the preset character image data into a preset rendering model to determine the virtual expert image model;

[0043] Animation determination module: determining the face and body animation of the virtual expert image according to the synthesized speech and the virtual expert image model;

[0044] Synthesis module: synthesizes the face and body animation of the virtual expert image with the synthesized speech to determine the interactive virtual expert image animation.

[0045] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0046] The embodiment of the present invention introduces a method for generating an interactive virtual expert image, which integrates preset voice data, character image data and user voice data, and successfully constructs a virtual expert image with audio and animation effects with the help of a pre-trained question-answering model and an action generation model. This innovative method not only realizes the natural, smooth and vivid interaction between users and virtual experts, but also breaks through the limitation of traditional technology that relies solely on text output, greatly broadening the application scenarios. By introducing a virtual expert system and integrating animation and voice display methods, this method effectively improves the patient's understanding ability and memory effect, and significantly optimizes the patient's user experience.

[0047] The embodiment of the present invention introduces an interactive virtual expert image generation method, which can provide patients with convenient consultation services, reduce the workload of doctors, improve the overall efficiency of medical services, thereby meeting the diversified needs of the market and effectively filling the blank areas in the current medical system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0049] Figure 1 It is a flow chart of a method for generating an interactive virtual expert image in one embodiment of the present invention.

[0050] Figure 2 It is a flow chart of a method for determining interactive virtual expert image response text data in a preferred embodiment of the present invention.

[0051] Figure 3 It is a structural diagram of a question-answering model based on a knowledge graph in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0053] like Figure 1 As shown, in one embodiment of the present invention, a method for generating an interactive virtual expert image is provided, and the specific process is as follows:

[0054] S11, obtaining the user's voice data, preset character image data and preset voice data.

[0055] S12, determining input text data according to the user voice data acquired in S11.

[0056] S13, inputting the input text data of S12 into the question-answering model for processing, and determining the virtual expert image response text data.

[0057] S14, determining the synthesized voice of the virtual expert image according to the voice data and response text data preset in S11.

[0058] S15, inputting the character image data preset in S11 into a preset rendering model to determine a virtual expert image model.

[0059] S16, determining the face and body animation of the virtual expert image according to the synthesized speech of S14 and the virtual expert image model of S15.

[0060] S17, synthesizing the face and body animation of the virtual expert image in S16 with the synthesized speech in S14 to determine the interactive virtual expert image animation.

[0061] The above-mentioned embodiment, through preset voice data, preset character image data and the user's voice, adopts a pre-trained question-answering model and a pre-trained action generation model to synthesize a virtual expert image animation with audio, and effectively realizes the interaction between the user and the virtual expert image, and generates a realistic virtual expert image through a pre-trained rendering model and presents it to the user.

[0062] In a preferred embodiment of the present invention, S11 is implemented to obtain the user's voice data, preset character image data and preset voice data.

[0063] The preset character image data can be a character image collected online or in real life, the preset voice data can be a voice data collected online or in real life, the preset voice data is used to synthesize the voice of the virtual expert image, and the preset image data is used to synthesize the virtual expert image. The user's voice data uses real voice data, which is generally obtained through a recording device.

[0064] In a preferred embodiment of the present invention, S12 is implemented to determine the input text data according to the user voice data, specifically:

[0065] T n =FunASR(S)

[0066] Among them, T n represents input text data, S represents user voice data, and FunASR represents the operation of processing voice data.

[0067] In some specific embodiments, the speech recognition model may adopt a FunAsr model, and the FunAsr model may be used to convert the user's speech data into text data and input it into the question-answering model.

[0068] The above embodiment converts the user's voice data into text data, which is beneficial to the subsequent processing of the question-answering model.

[0069] In a preferred embodiment of the present invention, S13 is implemented to input the input text data into the question-answering model for processing, and determine the virtual expert image answer text data, which may include S131 to S133, such as Figure 2 shown.

[0070] S131, input text data into a preset question-answering model to determine the disease name and corresponding keyword data in the input text data, wherein the text data includes input text data converted from user voice data.

[0071] S132, determining disease data related to the question-answering model based on the disease name and corresponding keyword data in the input text data.

[0072] S133, determining the response text data of the virtual expert image according to the relevant disease data in the question-answering model.

[0073] In a preferred embodiment, in S133, inputting the input text data into the question-answering model for processing and determining the virtual expert image answering text data may include:

[0074] R n (EXT,RE)=Chat(T n , [EXT, KEY], N)

[0075] Among them, R n Represents the response text data of the virtual expert image, Chat represents the process of the question-answering model responding to the input text data in the nth round of interaction, EXT represents the disease name data extracted from the input text, KEY represents the data related to the disease name extracted from the input text, RE represents the data related to the disease name obtained from the question-answering model, N represents the number of rounds of dialogue, and [EXT, KEY] represents the request body in the question-answering model's response process.

[0076] In the above-mentioned embodiment of the present invention, the question-answering model can be used to effectively realize the interaction between the user and the virtual expert image.

[0077] In a preferred embodiment, the question-answering model can use a preset knowledge graph model. Figure 3As shown in the figure, related diseases are marked from a large number of disease description texts, and the causes of the diseases are further found based on this. At the same time, the parts where the corresponding diseases occur are marked, and the symptoms of the diseases are found. The corresponding physiological indicators are checked through the symptoms, and the corresponding diseases are further determined. After the diseases are determined, the drugs needed for the diseases are determined according to the treatment methods. The corresponding disease knowledge graph is obtained by marking the above entities and the corresponding entity relationships.

[0078] The steps for establishing a method based on a knowledge graph may include:

[0079] DB=Clean(Data)

[0080] Among them, DB represents the knowledge graph, Data represents the existing medical disease data, and Clean represents the knowledge graph construction process, which specifically includes data cleaning, knowledge extraction and knowledge fusion of Data to obtain key knowledge triples and construct a knowledge graph.

[0081] The above-mentioned embodiment adopts a preset knowledge graph model to determine the text data of the virtual expert image response, which can accurately obtain the information required by the user and improve the efficiency of the virtual expert image response.

[0082] In a preferred embodiment of the present invention, the step S14 of determining the synthesized speech of the virtual expert image according to the preset speech data and the response text data may include:

[0083] F A =f(A)

[0084] S=Voice(FA,R n )

[0085] Among them, F A represents the voice characteristics of the virtual expert image, A represents the preset voice data, f represents the voice feature extraction operation using the pre-trained voice cloning model, S represents the synthesized voice of the virtual expert image, Voice represents the synthesized voice operation using the voice cloning model, and R n Indicates the response text data of the virtual expert image.

[0086] In the above preferred embodiment, a pre-trained voice cloning model is used to imitate the timbre characteristics of a specified real person to improve the user experience quality.

[0087] In order to better obtain the voice data of the virtual expert image, in some specific embodiments, the preset voice cloning model can adopt the SV2TTS model to clone the voice characteristics of a real person to obtain the voice data of the virtual expert image.

[0088] Specifically, the following steps may be included:

[0089] First, the preset voice data is input into the SV2TTS voice cloning model to extract the sound features of the preset voice data.

[0090] Secondly, the response text data is input into the SV2TTS voice cloning model to obtain the voice data of the virtual expert image as the audio of this virtual expert image.

[0091] The above embodiment adopts the SV2TTS voice cloning model to determine the synthesized voice of the virtual expert image, which can effectively imitate the sound characteristics of the preset voice data and enhance the user's interactive interest.

[0092] In a preferred embodiment of the present invention, S15 is implemented to input the preset character image data into the preset rendering model to determine the virtual expert image model, including:

[0093] Y=Human(Figure)

[0094] Among them, Y represents the virtual expert image model, Figure represents the preset character image data, and Human represents the rendering operation using the preset rendering model.

[0095] In the above embodiments of the present invention, a realistic virtual expert image can be generated and presented to the user through a pre-trained rendering model.

[0096] In some specific embodiments, depending on whether the preset character image is a photo or text, the preset rendering model may adopt a HumanNeRF model or a HumanGaussian model, wherein HumanNeRF renders a picture into a three-dimensional character image, and HumanGaussian directly generates a three-dimensional character image through text.

[0097] The above embodiments use the HumanNeRF model or the HumanGaussian model to generate a virtual expert image, which has the characteristic of realistic image.

[0098] In order to obtain a more vivid virtual expert image, in a preferred embodiment of the present invention, S16 is implemented to synthesize the speech and the virtual expert image model to determine the face and body animation of the virtual expert image, including:

[0099] A=Pan(S,Y)

[0100] Among them, A represents the synthetic animation of the virtual expert image, and Pan represents the animation generation operation using the action generation model.

[0101] In some specific embodiments, the action generation model may use a PantoMatrix model to infer the facial and body movements of the virtual expert image.

[0102] In the above embodiment, the PantoMatrix model is used to generate the face and body movements of the virtual expert image, making the virtual expert image more vivid and improving the user's interactive experience.

[0103] In a preferred embodiment of the present invention, S17 is implemented to synthesize the generated face and body animation with the synthesized speech to determine the interactive virtual expert image animation, including:

[0104] D=A+S

[0105] Wherein, D represents the interactive virtual expert image animation, and + represents the synthesis processing operation.

[0106] In some specific embodiments, the interactive virtual expert image animation and the synthesized speech may be synthesized using the ffmpeg program, and audio may be added to the virtual expert image animation in a manner of embedding the audio into a corresponding position of the virtual expert image animation.

[0107] The above-mentioned embodiment adds audio to the audioless virtual expert image animation to synthesize a complete interactive virtual expert image animation, which can intelligently interact with the user in terms of vision, hearing and language.

[0108] Of course, in other embodiments, each of steps S11 to S17 can be adopted and used alone, or a combination of multiple steps can be used.

[0109] Based on the same technical concept, in another embodiment of the present invention, an interactive virtual expert image generation system is also provided, comprising:

[0110] Data acquisition module: acquires the user's voice data, preset character image data and preset voice data, and determines the input text data according to the user's voice data;

[0111] Question and answer module: inputs the input text data into the question and answer model for processing, and determines the virtual expert image answer text data;

[0112] Speech synthesis module: determines the synthesized speech of the virtual expert image according to the preset speech data and the response text data;

[0113] Rendering module: input the preset character image data into the preset rendering model to determine the virtual expert image model;

[0114] Animation determination module: determines the face and body animation of the virtual expert image based on the synthesized speech and the virtual expert image model;

[0115] Synthesis module: synthesizes the face and body animation of the virtual expert image with the synthesized speech to determine the interactive virtual expert image animation.

[0116] It should be noted that the above method and system may correspond to each other. Since the principles of the method and system are similar, the implementation process of the system and the method may refer to each other and will not be described in detail here.

[0117] The steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, that is, the embodiments in the system can be understood as preferred examples for implementing the method, which will not be described in detail here.

[0118] Those skilled in the art know that, in addition to implementing the system and its various devices / modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices / modules provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices / modules provided by the present invention can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component, and the devices for implementing various functions can also be considered as both software modules for implementing the method and structures within the hardware component.

[0119] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A method for generating an interactive virtual expert image, characterized in that: include: Acquire the user's voice data, preset character image data and preset voice data; Determining input text data according to the user voice data; Inputting the input text data into the question-answering model for processing to determine the virtual expert image answer text data; Determining the synthesized voice of the virtual expert image according to the preset voice data and the response text data; Inputting the preset character image data into a preset rendering model to determine a virtual expert image model; Determining facial and body animation of the virtual expert image according to the synthesized speech and the virtual expert image model; The face and body animation of the virtual expert image are synthesized with the synthesized speech to determine an interactive virtual expert image animation.

2. The method for generating an interactive virtual expert image according to claim 1, characterized in that: Obtaining voice data of the user through a recording device; The preset character image data is a character image collected from the Internet or in reality, and is used to synthesize a virtual expert image; The preset voice data is voice data collected from the network or in reality, and is used to synthesize the voice of the virtual expert image.

3. The method for generating an interactive virtual expert image according to claim 1, characterized in that: The step of determining input text data according to the user voice data comprises: T n =Funs(S) Among them, T n represents the input text data, S represents the user voice data, and funs represents the process of processing the voice data.

4. The method for generating an interactive virtual expert image according to claim 1, characterized in that: The step of inputting the input text data into the question-answering model for processing and determining the virtual expert image answering text data comprises: R n (EXT,RE)=Chat(T n ,[EXT,KEY],N) Among them, R n represents the answering text data of the virtual expert image, Chat represents the process of the question-answering model answering the input text data in the nth round of interaction, T n Represents the input text data, EXT represents the disease name data extracted from the input text, KEY represents the data related to the disease name extracted from the input text, RE represents the data related to the disease name obtained from the question-answering model, N represents the number of rounds of dialogue, and [EXT, KEY] represents the request body in the question-answering model's response process.

5. The method for generating an interactive virtual expert image according to claim 4, characterized in that: The question-answering model is a model based on a knowledge graph, which can mark related diseases from the disease description text, and based on this, obtain the cause of the disease, mark the location where the corresponding disease occurs, find the symptoms of the disease, check the corresponding physiological indicators through the symptoms, determine the corresponding disease, and after determining the disease, determine the required drugs for the disease based on the treatment method.

6. The method for generating an interactive virtual expert image according to claim 1, characterized in that: Determining the synthesized voice of the virtual expert image according to the preset voice data and the response text data includes: The preset voice data is input into the pre-trained voice cloning model to determine the voice characteristics of the virtual expert, F A =f(A) Among them, F A represents the voice feature of the virtual expert image, A represents the preset voice data, and f represents the voice feature extraction operation performed using the pre-trained voice cloning model; S=Voice(F A ,R n ) Wherein, S represents the synthesized speech of the virtual expert image, Voice represents the synthesized speech operation performed by the speech cloning model, and R n The virtual expert image's response text data is represented.

7. The method for generating an interactive virtual expert image according to claim 1, characterized in that: The step of inputting the preset character image data into a preset rendering model to determine the virtual expert image model includes: Y=Human(Figure) Among them, Y represents the virtual expert image model, Figure represents the preset character image data, and Human represents the rendering operation using the preset rendering model. The rendering models include the HumanNeRF model and the HumanGaussian model, which can respectively obtain a three-dimensional character image by rendering text or pictures.

8. The method for generating an interactive virtual expert image according to claim 1, characterized in that: Determining the face and body animation of the virtual expert image according to the synthesized speech and the virtual expert image model includes: A=Pan(S,Y) Wherein, A represents the synthesized animation of the virtual expert image, Pan represents the animation generation operation using the action generation model, S represents the synthesized speech of the virtual expert image, and Y represents the virtual expert image model.

9. The method for generating an interactive virtual expert image according to claim 1, characterized in that: The step of synthesizing the face and body animation of the virtual expert image with the synthesized speech to determine the interactive virtual expert image animation includes: Wherein, D represents the interactive virtual expert image animation, represents a synthesis processing operation, and S represents the synthesized speech of the virtual expert image.

10. An interactive virtual expert image generation system, characterized in that: include: Data acquisition module: acquires user voice data, preset character image data and preset voice data, and determines input text data according to the user voice data; Question and answer module: input the input text data into the question and answer model for processing, and determine the virtual expert image answer text data; A speech synthesis module: determining the synthesized speech of the virtual expert image according to the preset speech data and the response text data; Rendering module: inputting the preset character image data into a preset rendering model to determine the virtual expert image model; Animation determination module: determining the face and body animation of the virtual expert image according to the synthesized speech and the virtual expert image model; Synthesis module: synthesizes the face and body animation of the virtual expert image with the synthesized speech to determine the interactive virtual expert image animation.

Images