Method for generating virtual character and interacting with user

Abstract

The invention provides a method for generating a virtual character and interacting with a user, and solves the problems of insufficient personification degree and limited interaction capability of the virtual character in the prior art. The method comprises the steps of obtaining information of a virtual character to be generated, generating a cue word Prompt used for generating a virtual character image, generating a character image through the cue word, generating timbre, fixed verbal skill voice and video based on the virtual character image, and optimizing the timbre, the fixed verbal skill voice and the video; according to the virtual character and the large language model, interaction between the user and the virtual character is achieved, the anthropomorphic degree of the virtual character is improved, and the interaction ability of the virtual character is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for generating a virtual character and interacting with a user. Background Art

[0002] As digital transformation accelerates, virtual characters, as the core vehicle for human-computer interaction, are showing tremendous potential for application in multiple fields. In healthcare, virtual caregivers can provide 24-hour psychological counseling; in education, virtual teachers can enable personalized instruction; and in commercial services, digital employees are gradually taking on roles such as customer service and shopping guides. However, existing virtual character systems face significant bottlenecks in practical application:

[0003] 1. Insufficient anthropomorphism

[0004] Most virtual images still have problems such as stiff expressions and mechanical dialogues. User surveys show that 67% of respondents believe that existing virtual characters "lack real emotional expression."

[0005] 2. Limited ability to interact continuously

[0006] During long-term interactions, virtual characters often have problems such as memory confusion and incorrect responses, resulting in 40% of users giving up after using the app for one week. Summary of the Invention

[0007] The present invention provides a method for generating a virtual character and interacting with a user, so as to solve the problems in the prior art of insufficient anthropomorphism of the virtual character and limited interaction capability.

[0008] In a first aspect, the present invention provides a method for generating a virtual character, which specifically comprises the following steps:

[0009] Step S1: Obtain information of the virtual character to be generated;

[0010] Step S2: constructing a prompt word for outputting a large model result based on the information of the virtual character;

[0011] Step S3: inputting the prompt word Prompt into the Wenshengtu model, and generating an image of the virtual character through the Wenshengtu model;

[0012] Step S4: selecting a timbre style based on the information of the virtual character, and generating the timbre of the virtual character through a TTS (text-to-speech) tool;

[0013] Step S5: Generate a voice text that the virtual character needs to output based on the information of the virtual character, input the voice text into a TTS tool, and generate a voice corresponding to the voice text through the TTS tool;

[0014] Step S6: input the image and the voice of the virtual character into a lip synchronization model, and generate a video of the virtual character speaking according to the lip synchronization model.

[0015] Preferably, in step S1, the information of the virtual character includes character information such as gender, age, hairstyle, clothing, body proportions, skin color, character settings, and timbre style of the virtual character to be generated.

[0016] Preferably, in step S4, the TTS tools include Microsoft Azure, Google TTS and a custom voice generation tool.

[0017] Preferably, in step S4, after the timbre of the virtual character is generated, the timbre is further slightly adjusted to ensure that the timbre of the virtual character is natural, recognizable, and consistent with the image of the virtual character.

[0018] Preferably, in step S5, the voice text includes texts such as self-introduction, conversation opening remarks, fixed answer templates, and random texts of different lengths.

[0019] Preferably, in step S5, after the speech is generated, the speech is also adjusted, including information such as pitch, speaking speed, and emotional expression, to ensure that the speech meets the expected effect.

[0020] Preferably, when it is necessary to generate a video in which the virtual character does not speak, the image of the virtual character is input into an image-generated video model, and the video in which the virtual character does not speak is generated by the image-generated video model.

[0021] More preferably, the video of the virtual character when not speaking includes natural body swaying, blinking, hair fluttering, facial expressions, etc., to simulate the performance of listening to others speak.

[0022] A second aspect of the present invention provides a virtual character generation device, which specifically includes the following modules:

[0023] A virtual character information acquisition module is used to obtain information about the virtual character to be generated;

[0024] A prompt word construction module, used to construct a prompt word Prompt for outputting a large model result based on the information of the virtual character;

[0025] A virtual character image generation module, configured to input the prompt word Prompt into the Wenshengtu macro model and generate an image of the virtual character using the Wenshengtu macro model;

[0026] A virtual character timbre generation module is used to select a timbre style based on the information of the virtual character and generate the timbre of the virtual character through a TTS (text-to-speech) tool;

[0027] A virtual character speech generation module is used to generate the speech text that the virtual character needs to output based on the information of the virtual character, input the speech text into a TTS tool, and generate speech corresponding to the speech text through the TTS tool;

[0028] The virtual character video generation module is used to input the image and voice of the virtual character into a lip synchronization model, and generate a video of the virtual character speaking according to the lip synchronization model.

[0029] Preferably, in the virtual character information acquisition module, the virtual character information includes the gender, age, hairstyle, clothing, body proportions, skin color, character settings, voice style and other character information of the virtual character to be generated.

[0030] Preferably, in the virtual character timbre generation module, the TTS tools include Microsoft Azure, Google TTS and a custom sound generation tool.

[0031] Preferably, in the virtual character timbre generating module, after the timbre of the virtual character is generated, the timbre is also slightly adjusted to ensure that the timbre of the virtual character is natural, recognizable, and consistent with the image of the virtual character.

[0032] Preferably, in the virtual character voice generation module, the voice text includes texts such as self-introduction, dialogue opening remarks, fixed answer templates, and random texts of different lengths.

[0033] Preferably, in the virtual character voice generation module, after the voice is generated, the voice is also adjusted, including information such as pitch, speaking speed, and emotional expression, to ensure that the voice meets the expected effect.

[0034] Preferably, when it is necessary to generate a video in which the virtual character does not speak, the image of the virtual character is input into an image-generated video model, and the video in which the virtual character does not speak is generated by the image-generated video model.

[0035] More preferably, the video of the virtual character when not speaking includes natural body swaying, blinking, hair fluttering, facial expressions, etc., to simulate the performance of listening to others speak.

[0036] A third aspect of the present invention provides a method for interacting with a virtual character, which is applied to the virtual character described in the first or second aspect of the present invention, and the method specifically comprises the following steps:

[0037] Step S7: When no keyword for waking up the virtual character is detected, the virtual character displays a silent video; when a keyword for waking up the virtual character is detected, the virtual character answers using a fixed answer template;

[0038] Step S8: After the virtual character is awakened, the user's voice is monitored, the user's voice is converted into text, and the text is input into the large language model to generate a text answer;

[0039] Step S9: Generate and output audio based on the timbre of the virtual character and video based on the image of the virtual character according to the text of the answer.

[0040] Preferably, in step S9, before outputting the audio based on the timbre of the virtual character and the video based on the image of the virtual character, the audio and video are optimized to ensure synchronization and smooth and natural output of the audio and video.

[0041] Compared with the prior art, the present invention has the following obvious outstanding substantial features and significant advantages:

[0042] The present invention provides a method for generating a virtual character and interacting with a user, addressing the existing issues of insufficient anthropomorphism and limited interactive capabilities of virtual characters. By acquiring information about the virtual character to be generated, generating a prompt for generating an image of the virtual character, generating an image of the character using the prompt, generating and optimizing the timbre, fixed-speech voice, and video based on the virtual character image, and enabling user interaction with the virtual character based on the virtual character and a large language model, the present invention not only improves the virtual character's anthropomorphism but also enhances its interactive capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0044] Figure 1 This is a flow chart of a method for generating a virtual character and interacting with a user according to a preferred embodiment of the present invention.

[0045] Figure 2 It is a structural diagram of a virtual character generation device according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention provides a method for generating a virtual character and interacting with a user. To make the objectives, technical solutions, and effects of the present invention more clear and explicit, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention.

[0047] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0048] Example 1:

[0049] like Figure 1 As shown, the method for generating a virtual character and interacting with a user described in this embodiment specifically includes the following steps:

[0050] Step S1, obtaining information of the virtual character to be generated; for example, the gender, age, hairstyle, clothing, body proportions, skin color, character settings, voice style and other character information of the virtual character to be generated.

[0051] Step S2: Based on the information of the virtual character, construct a prompt word for outputting the large model result, such as "Please help me generate a male doctor who is about 25 years old, of yellow race, and has black hair." For character information without set requirements, a random selection can be made. For clothing that is not set, the final generated virtual character image shows that the clothing worn includes multiple types and is not unique. If necessary, the clothing can be pre-set.

[0052] Step S3: Input the prompt word Prompt into the Wenshengtu macromodel, and generate an image of the virtual character using the Wenshengtu macromodel. In this embodiment, the Wenshengtu macromodel uses Stable Diffusion or Midjunery. After selecting the Wenshengtu macromodel to generate the virtual character image, if you are not satisfied with the result, you can adjust the prompt word Prompt and the parameters of the Wenshengtu macromodel and regenerate the virtual character image until you are satisfied. In addition, if you have high requirements for the virtual character image, you can manually optimize the generated character image to enrich the details of the virtual character image and adjust any unsatisfactory aspects, such as adjusting the image clarity, image size, image format, background color, etc. Generating a virtual image using the Wenshengtu macromodel is not a selection process similar to creating a game character.

[0053] Step S4: Based on the avatar's information, a timbre style is selected and a TTS (text-to-speech) tool is used to generate the avatar's timbre. TTS tools include Microsoft Azure, Google TTS, and custom voice generation tools. In this embodiment, Google TTS is used to generate the avatar's timbre. After the avatar's timbre is generated, subtle adjustments are made to ensure that the avatar's timbre is natural, recognizable, and consistent with the avatar's image.

[0054] Step S5: Based on the avatar's information, generate the desired voice text for the avatar. Input the voice text into a text-to-speech (TTS) tool, which then generates the corresponding speech. The voice text includes text such as a self-introduction, a conversation starter, fixed answer templates, and random text of varying lengths. After the speech is generated, adjustments are made to the voice, including pitch, speaking rate, and emotional expression, to ensure it meets the desired effect. To enhance the fluency of subsequent interactions between the avatar and the user, the timbre can be adjusted during the timbre generation process by adjusting relevant parameters, including timbre proportion, speaking rate, volume, pitch, and emotion. Once a satisfactory timbre is achieved for a particular scenario, the parameters can be saved for future use. This primarily applies to two scenarios: pre-generated fixed speech audio, such as self-introductions and greetings; and real-time generation of spoken audio based on the chat content during a user chat.

[0055] Step S6: Input the image and the voice of the virtual character into a lip-sync model, and generate a video of the virtual character speaking based on the lip-sync model. When it is necessary to generate a video of the virtual character without speaking, input the image of the virtual character into a graph-generated video model, and generate a video of the virtual character when not speaking through the graph-generated video model. The video of the virtual character when not speaking includes natural body swaying, blinking, hair fluttering, facial expressions, etc., which are used to simulate the performance of listening to others speak. If you are not satisfied with the video of the virtual character speaking or not speaking, you can choose to regenerate the relevant video through the graph-generated model, or you can choose to manually optimize the video to enrich the video plot, adjust the video clarity, adjust the video format, adjust the background color in the video, etc., and edit the video appropriately to ensure smooth and natural transitions during continuous playback.

[0056] Step S7: When no keyword for waking up the virtual character is detected, the virtual character displays a silent video; when a keyword for waking up the virtual character is detected, the virtual character answers using a fixed answer template.

[0057] Step S8: After the virtual character is awakened, the user's voice output is monitored, the user's voice output is converted into text, and the text is input into a large language model to generate a text of the answer.

[0058] Step S9: Generate and output audio based on the timbre of the virtual character and video based on the image of the virtual character based on the text of the answer. Before outputting the audio based on the timbre of the virtual character and the video based on the image of the virtual character, optimize the audio and video to ensure synchronization and smooth and natural output.

[0059] Example 2:

[0060] like Figure 2 As shown, the virtual character generation device described in this embodiment specifically includes the following modules: a virtual character information acquisition module, a prompt word construction module, a virtual character image generation module, a virtual character timbre generation module, a virtual character voice generation module and a virtual character video generation module.

[0061] The virtual character information acquisition module is used to obtain information about the virtual character to be generated; wherein the virtual character information includes character information such as gender, age, hairstyle, clothing, body proportions, skin color, character settings, and voice style of the virtual character to be generated.

[0062] A prompt word construction module, used to construct a prompt word Prompt for outputting a large model result based on the information of the virtual character;

[0063] A virtual character image generation module, configured to input the prompt word Prompt into the Wenshengtu macro model and generate an image of the virtual character using the Wenshengtu macro model;

[0064] The avatar voice generation module is used to select a voice style based on the avatar's information and generate the avatar's voice using a TTS (text-to-speech) tool. These TTS tools include Microsoft Azure, Google TTS, and custom voice generation tools. Once the avatar's voice is generated, subtle adjustments are made to ensure it's natural, recognizable, and consistent with the avatar's image.

[0065] The avatar speech generation module is used to generate the speech text that the avatar needs to output based on the avatar's information. This speech text is then input into a text-to-speech (TTS) tool, which then generates the speech corresponding to the speech text. The speech text includes text such as self-introductions, conversation starters, fixed answer templates, and random text of varying lengths. After the speech is generated, adjustments are made to the speech, including pitch, speaking speed, and emotional expression, to ensure that it meets the desired effect.

[0066] The avatar video generation module is configured to input the avatar's image and speech into a lip-sync model, and generate a video of the avatar speaking based on the lip-sync model. To generate a video of the avatar without speaking, the avatar's image is input into an image-based video model, and a silent video of the avatar is generated using the image-based video model. The silent video of the avatar includes natural body swaying, blinking, hair movement, and facial expressions, simulating the experience of listening to someone speak.

[0067] While the specific embodiments of the present invention have been described in detail above, these are merely exemplary and the present invention is not limited thereto. For those skilled in the art, any equivalent modifications and substitutions to the present invention are also within the scope of the present invention. Therefore, any equivalent changes and modifications made without departing from the spirit and scope of the present invention are intended to be encompassed within the scope of the present invention.

Claims

1. A method for generating a virtual character, characterized in that: The specific steps include: Step S1: Obtain information of the virtual character to be generated; Step S2: constructing a prompt word for outputting a large model result based on the information of the virtual character; Step S3: inputting the prompt word Prompt into the Wenshengtu model, and generating an image of the virtual character through the Wenshengtu model; Step S4: selecting a timbre style based on the information of the virtual character, and generating the timbre of the virtual character through a TTS tool; Step S5: Generate a voice text that the virtual character needs to output based on the information of the virtual character, input the voice text into a TTS tool, and generate a voice corresponding to the voice text through the TTS tool; Step S6: input the image and the voice of the virtual character into a lip synchronization model, and generate a video of the virtual character speaking according to the lip synchronization model.

2. A method for generating a virtual character according to claim 1, characterized in that: In step S1, the information of the virtual character includes the gender, age, hairstyle, clothing, body proportions, skin color, character settings, and timbre style information of the virtual character to be generated.

3. A method for generating a virtual character according to claim 1, characterized in that: In step S4, the TTS tools include Microsoft Azure, Google TTS, and a custom voice generation tool.

4. A method for generating a virtual character according to claim 1, characterized in that: In step S4, after the timbre of the virtual character is generated, the timbre is finely adjusted to ensure that the timbre of the virtual character is natural, recognizable, and consistent with the image of the virtual character.

5. The method for generating a virtual character according to claim 1, wherein: In step S5, the voice text includes a self-introduction, a conversation starter, a fixed answer template, and random texts of different lengths.

6. A method for generating a virtual character according to claim 1, characterized in that: In step S5, after the speech is generated, the speech is adjusted, including the pitch, speaking speed, and emotional expression information, to ensure that the speech meets the expected effect.

7. A method for generating a virtual character according to claim 1, characterized in that: When it is necessary to generate a video of the virtual character without speaking, the image of the virtual character is input into the image-generated video model, and the video of the virtual character when not speaking is generated by the image-generated video model; the video of the virtual character when not speaking includes the natural swaying of the body, blinking, fluttering of hair, and facial expressions, which are used to simulate the performance of listening to others speak.

8. A virtual character generation device, characterized in that: Specifically, it includes the following modules: A virtual character information acquisition module is used to obtain information about the virtual character to be generated; A prompt word construction module, used to construct a prompt word Prompt for outputting a large model result based on the information of the virtual character; A virtual character image generation module, configured to input the prompt word Prompt into the Wenshengtu macro model and generate an image of the virtual character using the Wenshengtu macro model; A virtual character timbre generation module is used to select a timbre style based on the information of the virtual character and generate the timbre of the virtual character through a TTS tool; A virtual character speech generation module is used to generate the speech text that the virtual character needs to output based on the information of the virtual character, input the speech text into a TTS tool, and generate speech corresponding to the speech text through the TTS tool; The virtual character video generation module is used to input the image and voice of the virtual character into a lip synchronization model, and generate a video of the virtual character speaking according to the lip synchronization model.

9. The method for interaction between a virtual character and a user according to claim 1, wherein: The specific steps include: Step S7: When no keyword for waking up the virtual character is detected, the virtual character displays a silent video; when a keyword for waking up the virtual character is detected, the virtual character answers using a fixed answer template; Step S8: After the virtual character is awakened, the user's voice is monitored, the user's voice is converted into text, and the text is input into the large language model to generate a text answer; Step S9: Generate and output audio based on the timbre of the virtual character and video based on the image of the virtual character according to the text of the answer.

10. The method for interaction between a virtual character and a user according to claim 9, characterized in that: In step S9, before outputting the audio based on the timbre of the virtual character and the video based on the image of the virtual character, the audio and video are optimized to ensure that the audio and video are synchronized and output smoothly and naturally.

Images