A virtual human control method, device, equipment and storage medium

Abstract

The application discloses a virtual human control method and device, equipment and a storage medium, wherein the method comprises: performing real-time speech recognition on a real-time collected audio stream based on a preset time period to obtain text information corresponding to a target audio stream in the preset time period; converting the text information into a synthetic audio with a time length equal to that of the preset time period; and driving the lip change of the virtual human based on the synthetic audio.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and includes, but is not limited to, a virtual human control method, apparatus, device, and storage medium. Background Technology

[0002] When using voice to drive virtual human lip movements in real time, the virtual human's lips will tremble due to the influence of surrounding noise (noise, other voices, etc.). The shortcomings of existing audio denoising solutions are: they are limited by the performance of audio denoising algorithms, cannot completely identify and remove background noise, and cannot separate the voices of other speakers, thus failing to completely solve the problem of virtual human lip trembling. Summary of the Invention

[0003] In view of this, embodiments of this application provide a virtual human control method, apparatus, device, and storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a virtual human control method, the method comprising:

[0006] Real-time speech recognition is performed on the real-time acquired audio stream based on a preset time period to obtain the text information corresponding to the target audio stream within the preset time period;

[0007] The text information is converted into a synthesized audio with a duration equal to the preset time period;

[0008] The synthesized audio drives the virtual human's lip movements.

[0009] Secondly, embodiments of this application provide a virtual human control device, the device comprising:

[0010] The recognition module is used to perform real-time speech recognition on the real-time acquired audio stream based on a preset time period to obtain the text information corresponding to the target audio stream within the preset time period.

[0011] The conversion module is used to convert the text information into synthesized audio of the same duration as the preset time period;

[0012] A driving module is used to drive the lip movements of the virtual human based on the synthesized audio.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement the above-described method.

[0014] Fourthly, embodiments of this application provide a storage medium storing executable instructions for implementing the above-described method when executed by a processor.

[0015] In this embodiment, real-time speech recognition is first performed on the real-time acquired audio stream based on a preset time period to obtain text information corresponding to the target audio stream within the preset time period. Then, the text information is converted into synthesized audio of the same duration as the preset time period. Finally, the synthesized audio is used to drive the virtual human's lip movements. In this way, using pure synthesized audio to drive the virtual human's lip movements effectively solves the problem of lip tremors in virtual humans, and the synthesized audio can achieve a playback duration equal to that of the synchronously acquired target audio stream, reflecting the speed at which the virtual human speaks like a real person. Attached Figure Description

[0016] Figure 1A A schematic diagram illustrating the implementation process of a virtual human control method provided in this application embodiment;

[0017] Figure 1B A schematic diagram of a virtual human provided in an embodiment of this application;

[0018] Figure 2 A schematic diagram illustrating the implementation process of a method for converting and synthesizing audio provided in an embodiment of this application;

[0019] Figure 3 A schematic diagram illustrating the implementation process of a method for driving a virtual human, provided in an embodiment of this application;

[0020] Figure 4 A schematic diagram illustrating the implementation process of a virtual mouth lip stabilization method provided in this application embodiment;

[0021] Figure 5 This is a schematic diagram of the composition structure of a control device provided in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of the embodiments will be further described in detail below with reference to the accompanying drawings. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0024] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0025] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0027] This application provides a virtual human control method, such as... Figure 1A As shown, the method includes:

[0028] Step S110: Perform real-time speech recognition on the real-time acquired audio stream based on a preset time period to obtain the text information corresponding to the target audio stream within the preset time period;

[0029] During implementation, microphone components can be used to capture real-time audio streams from real people.

[0030] Here, users can set a preset time period according to their actual needs. The shorter the preset time period, the better the real-time performance of the virtual human's updates. For example, the preset time period can be set to 10 milliseconds or 50 milliseconds, with 10 milliseconds providing better real-time performance than 50 milliseconds.

[0031] During implementation, at least one target audio stream can be collected in real time according to a preset time period and in chronological order; real-time speech recognition can be performed on the target audio stream to obtain the text information corresponding to the target audio stream within the preset time period. Here, the algorithm for speech recognition to text is not limited.

[0032] In some embodiments, if the target audio stream collected within a preset time period cannot be recognized as text due to the slow speaking speed of a real person, multiple target audio streams collected within preset time periods can be accumulated for speech recognition to obtain text information.

[0033] Step S120: Convert the text information into synthesized audio of the same duration as the preset time period;

[0034] Here, text information can be converted into audio information, resulting in pure speech.

[0035] During implementation, the duration of the synthesized audio can be determined based on the duration of a preset time period, that is, the duration of the synthesized audio obtained by converting text information is equal to the preset time.

[0036] Step S130: Drive the virtual human's lip changes based on the synthesized audio.

[0037] Because the synthesized audio is the same length as the preset time, the synthesized audio can achieve a playback duration equal to that of the synchronously acquired target audio stream, reflecting the speed at which the virtual human speaks.

[0038] Figure 1B A schematic diagram of a virtual human is provided for an embodiment of this application, such as... Figure 1B As shown, the diagram includes a virtual human 11 located in the center of the screen.

[0039] During implementation, synthesized speech can be used to drive, for example... Figure 1B The changes in the lips of the virtual human 11 shown.

[0040] In some embodiments, synthesized audio is used solely to drive changes in the virtual human's lips and facial expressions. For example, when the virtual human is giving a speech, the voice provided to the virtual human is a real person's voice produced in real time, and the virtual human's lips can adapt to the changes in the real person's audio in real time.

[0041] In some embodiments, the synthesized audio is also used to generate adapted speech for the virtual human, meaning the virtual human can speak based on the synthesized audio. For example, when the virtual human is giving a speech, the voice provided to the virtual human is the voice generated by the synthesized audio, and the virtual human's lips can change in accordance with the synthesized audio.

[0042] The basic principle of "voice-driven lip-syncing" technology is to generate corresponding sound signals based on the input text information, and then simulate human oral cavity movements based on these sound signals to produce speech signals corresponding to the input text. By simulating the movements of the human oral cavity through computer simulation, it produces mouth shapes and / or sounds corresponding to the input text. Compared with traditional speech synthesis technology, it has higher accuracy and naturalness.

[0043] In this embodiment, real-time speech recognition is first performed on the real-time acquired audio stream based on a preset time period to obtain text information corresponding to the target audio stream within the preset time period. Then, the text information is converted into synthesized audio of the same duration as the preset time period. Finally, the synthesized audio is used to drive the virtual human's lip movements. In this way, using pure synthesized audio to drive the virtual human's lip movements effectively solves the problem of lip tremors in virtual humans, and the synthesized audio can achieve a playback duration equal to that of the synchronously acquired target audio stream, reflecting the speed at which the virtual human speaks like a real person.

[0044] In some embodiments, the above step S120 "converting the text information into synthesized audio having the same duration as the preset time period", as Figure 2 shown, includes the following steps:

[0045] Step S210: determining the text information as valid information;

[0046] Here, the audio stream includes at least non-human speech sounds such as noise and sounds emitted by objects, and valid speech sounds of real humans.

[0047] During implementation, the text information obtained by recognizing the valid speech of a real person in the audio stream is valid text information.

[0048] Step S220: determining the speech duration corresponding to each character in the text information;

[0049] Here, the text information may include one character, two characters, or more than two characters.

[0050] During implementation, the speech duration corresponding to each character can be determined by analyzing the speech duration of the speech corresponding to each character in the audio stream. For example, the speech duration corresponding to each of the characters "I", "am", "Xiao", "Wang" in the collected audio stream "I am Xiao Wang" can be determined.

[0051] Step S230: converting each character in the text information into synthesized audio with the same duration as the corresponding speech duration based on the speech duration corresponding to each character.

[0052] In the process of converting text information into synthesized audio, each character in the text information can be converted into synthesized audio with the same duration as the corresponding speech duration based on the speech duration corresponding to each character. For example, when the text information is "I am Xiao Wang", the speech duration of each of "I", "am", "Xiao", "Wang" in the synthesized audio is respectively equal to the speech duration of each of "I", "am", "Xiao", "Wang" in the collected audio stream "I am Xiao Wang".

[0053] In the embodiments of the present application, firstly, the text information is determined as valid information; then the speech duration corresponding to each character in the text information is determined; finally, based on the speech duration corresponding to each character, each character in the text information is converted into synthesized audio with the same duration as the corresponding speech duration. In this way, since the duration of each speech in the synthesized audio is one-to-one correspondingly equal to the duration of each speech in the collected audio stream, the effect that the synthesized audio can more reasonably reflect the speech rate of a real person can be achieved on the basis of effectively eliminating the lip jitter of the virtual human.

[0054] In some embodiments, the above step S130 "driving the lip movement of the virtual human based on the synthesized audio" can be implemented through the following process:

[0055] The synthesized audio drives the virtual human to produce sound, and drives the lips of the virtual human to change based on the speech duration of the text corresponding to each speech in the synthesized audio.

[0056] Here, the synthesized audio can not only drive the lip movement of the virtual human, but also drive the virtual human to produce sound, that is, the virtual human can emit synthesized sound different from that of a real person.

[0057] During implementation, the synthesized audio can drive the virtual human to produce sound according to different durations corresponding to each character.

[0058] Driving the lip movement of the virtual human based on the speech duration of the text corresponding to each speech in the synthesized audio can realize the synchronization between the lip movement of the virtual human and the speech duration of each character, making the lip movement of the virtual human more reasonable.

[0059] Here, since the synthesized audio drives both the lip movement and the sound production of the virtual human, the synchronization between the lip movement and the sound production of the virtual human can be achieved more accurately.

[0060] In the embodiments of the present application, using the synthesized audio to drive the virtual human can achieve the effect that the synthesized audio can more reasonably reflect the speaking speed of a real person, and make the lip movement of the virtual human more reasonable.

[0061] In some embodiments, the above step S120 "converting the text information into synthesized audio having the same duration as the preset time period", as Figure 3 described, further includes the following step:

[0062] Step S310: determining the speech frequency corresponding to each character in the text information;

[0063] Here, the speech frequency can reflect the intonation level of a real person's speech.

[0064] During implementation, the speech frequency corresponding to each character can be determined by parsing the speech frequency of the speech corresponding to each character in the audio stream. For example, the speech frequency corresponding to each of the characters "I", "am", "Xiao", "Wang" in the collected audio stream "I am Xiao Wang" can be determined.

[0065] Correspondingly, the above step S230 "converting each said character in said text information into synthesized audio with the same duration as the corresponding speech duration based on the speech duration corresponding to each said character" can be implemented through the following process:

[0066] Step S320: Based on the speech duration and speech frequency corresponding to each of the characters, convert each of the characters in the text information into a synthesized audio with the same duration as the corresponding speech duration and adapted to the speech frequency.

[0067] In the process of converting text information into synthesized audio, each character in the text information can be converted into synthesized audio with the same duration and frequency as its corresponding voice, based on the voice duration and frequency of each character. For example, in the case of the text information "I am Xiao Wang", the voice duration and frequency of each of the characters "I", "is", "Xiao", and "Wang" in the synthesized audio are equal to the voice duration of each of the characters "I", "is", "Xiao", and "Wang" in the acquired audio stream "I am Xiao Wang", and are also compatible with the voice frequencies.

[0068] Correspondingly, step S130 above, "driving the lip changes of the virtual human based on the synthesized audio," can be achieved through the following process:

[0069] Step S330: Based on the synthesized audio, drive the virtual human to speak at a frequency that matches the speech frequency, and simultaneously drive the virtual human's lips to change based on the speech duration of each text corresponding to the speech in the synthesized audio.

[0070] In this embodiment, the speech frequency corresponding to each character in the text information is first determined. Then, based on the speech duration and speech frequency corresponding to each character, each character in the text information is converted into a synthesized audio with the same duration as the corresponding speech and adapted to the speech frequency. Finally, the frequency of the virtual human's speech is driven to adapt to the speech frequency based on the synthesized audio, and the virtual human's lips are driven to change according to the speech duration of each character in the synthesized audio. In this way, while effectively removing lip tremors in the virtual human, the synthesized audio can more reasonably reflect the speaking speed and intonation of a real person, and the changes in the virtual human's lips can be made more realistic.

[0071] In some embodiments, step S120 above, "converting the text information into synthesized audio of the same duration as the preset time period," includes the following steps:

[0072] Step 121: Determine that the text information is invalid;

[0073] Here, because the audio stream includes at least noise and sounds from objects that are not human speech, these sounds cannot be interpreted, meaning the text information recognized from the audio stream is invalid. For example, during a real person's speech, there may be sounds of objects hitting each other, outdoor vehicles, or a child crying; the information recognized from these sounds cannot be interpreted and is therefore invalid.

[0074] Step 122: Set the synthesized audio to blank audio;

[0075] Here, you can preset blank audio, which is audio that does not contain any sound.

[0076] During implementation, if the text information is determined to be invalid, the synthesized audio can be set to blank audio. That is, if it is determined that there is no valid real human voice in the collected audio stream, the synthesized audio can be set to blank audio.

[0077] Correspondingly, step S130 above, "driving the lip changes of the virtual human based on the synthesized audio," can be achieved through the following process:

[0078] Step 123: During the preset time period, the virtual human is driven not to speak based on the blank audio, and the lips of the virtual human are set to a closed state.

[0079] Here, since blank audio can represent the absence of real human speech during the preset time period, the virtual human can be driven to remain silent based on blank audio, and the virtual human's lips can be set to a closed state.

[0080] In this embodiment, the text information is first determined to be invalid; then the synthesized audio is set to blank audio; finally, within the preset time period, the virtual human is driven to remain silent based on the blank audio, while simultaneously setting the virtual human's lips to a closed state. This allows the virtual human to remain silent and have its lips closed when no valid real person is speaking, making the virtual human's state more realistic and effectively preventing lip tremors.

[0081] This application provides a method for virtual lip stabilization, such as... Figure 4 As shown, the method includes the following steps:

[0082] Step S410: The local microphone performs real-time audio stream acquisition according to a preset time period to obtain audio blocks;

[0083] Here, the real-time audio stream refers to real-time human speech. Users can set a preset time period according to their actual needs. The shorter the preset time period, the better the real-time performance of the virtual human's updates. For example, the preset time period can be set to 10 milliseconds or 50 milliseconds, with 10 milliseconds providing better real-time performance than 50 milliseconds.

[0084] A microphone can be a standalone device or an integrated device within an electronic device.

[0085] Here, the duration of the audio block is equal to the duration of the preset time period.

[0086] During implementation, at least one segment of the target audio stream can be collected in real time based on a preset time period and in chronological order.

[0087] Step S420: Perform real-time speech recognition on the audio block;

[0088] During implementation, speech recognition is performed on the audio block to obtain the corresponding text information. No restrictions are placed on the speech recognition method used.

[0089] In some embodiments, if the target audio stream collected within a preset time period cannot be recognized as text due to the slow speaking speed of a real person, multiple audio blocks (target audio streams) collected within the preset time period can be accumulated for speech recognition to obtain text information.

[0090] Step S430: Determine whether the current audio block can be recognized as text;

[0091] If the current audio block can be recognized as text, proceed to step S440;

[0092] If it is determined that the current audio block cannot be recognized as text, proceed to step S470.

[0093] Step S440: Send the audio block to the voice optimization module;

[0094] Here, the speech optimization module is used to convert text into synthesized audio (blocks) based on audio blocks. Text information can be converted into audio information, thus obtaining clean speech.

[0095] Step S450: The voice optimization module generates a synthesized audio block with a duration equal to the preset time period based on the recognized text and the audio block, and sends it to the virtual human driving system.

[0096] Here, the virtual human driving system is used to drive the virtual human to speak and drive virtual lip movements.

[0097] Figure 1B A schematic diagram of a virtual human is provided for an embodiment of this application, such as... Figure 1B As shown, the diagram includes a virtual human 11 located in the center of the screen.

[0098] During implementation, the virtual human driving system can utilize synthesized speech to drive, such as... Figure 1B The changes in the lips of the virtual human 11 shown.

[0099] In some embodiments, the virtual human driving system can utilize synthesized audio solely to drive the virtual human's lip and facial expressions. For example, when the virtual human is giving a speech, the voice provided to the virtual human is a real person's voice emitted in real time, and the virtual human's lips can adapt to the real person's audio changes in real time.

[0100] In some embodiments, the virtual human driving system can also utilize synthesized audio to generate adapted speech for the virtual human, meaning the virtual human can speak based on the synthesized audio. For example, when the virtual human is giving a speech, the voice provided to the virtual human is a voice generated from synthesized audio, and the virtual human's lips can adapt to the synthesized audio changes.

[0101] Thus, the improved text-to-speech (TTS) algorithm can produce clean speech, significantly reduce jitter, and because the duration of the synthesized audio block is equal to the preset time period, it can keep the lip movements synchronized with real human speech.

[0102] Step S460: The virtual human driving system drives the virtual human's lip movements according to the received synthesized audio blocks;

[0103] During implementation, pure synthesized audio was used to drive the virtual human's lip movements, resulting in no trembling of the virtual human's lips.

[0104] Step S470: Send a blank audio block of the same length as the preset time period to the virtual human driving system;

[0105] Here, because the audio stream includes at least noise and sounds from objects that are not human speech, these sounds cannot be interpreted, meaning the text information recognized from the audio stream is invalid. For example, during a real person's speech, there may be sounds of objects hitting each other, outdoor vehicles, or a child crying; the information recognized from these sounds cannot be interpreted and is therefore invalid.

[0106] Here, you can preset blank audio, which is audio that does not contain any sound.

[0107] During implementation, if the text information is determined to be invalid, the synthesized audio can be set to blank audio. That is, if it is determined that there is no valid real human voice in the collected audio stream, the synthesized audio can be set to blank audio.

[0108] Step S480: The virtual human driving system drives the virtual human's lip changes based on the received noise-free blank audio block.

[0109] Here, since blank audio represents the absence of effective human speech within the preset time period, the virtual human driving system can drive the virtual human to remain silent based on noise-free blank audio and set the virtual human's lips to a closed state. This makes the virtual human's lip state more realistic and effectively avoids lip tremors.

[0110] During implementation, since blank audio can represent the absence of effective human speech within the preset time period, the virtual human can be driven to remain silent based on the blank audio, and the virtual human's lips can be set to a closed state, with no trembling of the virtual human's lips.

[0111] In this embodiment, a method combining speech recognition and text-to-speech is used. When text is detected in the audio, the audio is used to drive the virtual human; otherwise, the virtual human's lips are not driven. In this way, regardless of the type of background noise, if it cannot be recognized as text, the virtual human's lip movements will not be driven, resulting in a significant anti-shake effect. When multiple people speak simultaneously, only a single character can be recognized, without affecting the virtual human's lip movement, and lip tremors are effectively eliminated.

[0112] Based on the foregoing embodiments, this application provides a virtual human control device, which includes various modules, each module including sub-modules, which can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor unit (MPU), digital signal processor (DSP), or field programmable gate array (FPGA), etc.

[0113] Figure 5 This is a schematic diagram of the composition structure of the virtual human control device provided in the embodiments of this application, as shown below. Figure 5 As shown, the device 500 includes:

[0114] The recognition module 510 is used to perform real-time speech recognition on the real-time acquired audio stream based on a preset time period to obtain the text information corresponding to the target audio stream within the preset time period.

[0115] The conversion module 520 is used to convert the text information into synthesized audio of the same duration as the preset time period;

[0116] The driving module 530 is used to drive the lip changes of the virtual human based on the synthesized audio.

[0117] In some embodiments, the conversion module 520 includes a first determining submodule, a second determining submodule, and a conversion submodule, wherein the first determining submodule is used to determine that the text information is valid information; the second determining submodule is used to determine the speech duration corresponding to each character in the text information; and the conversion submodule is used to convert each character in the text information into a synthesized audio with a duration equal to the corresponding speech duration based on the speech duration corresponding to each character.

[0118] In some embodiments, the driving module 530 is further configured to drive the virtual human to speak based on the synthesized audio, and to drive the virtual human's lips to change based on the speech duration of each speech character in the synthesized audio.

[0119] In some embodiments, the conversion module 520 further includes a third determining submodule, configured to determine the speech frequency corresponding to each character in the text information; correspondingly, the conversion submodule is further configured to convert each character in the text information into a synthesized audio with the same duration as the corresponding speech duration and adapted to the speech frequency based on the speech duration and speech frequency corresponding to each character.

[0120] In some embodiments, the driving module 530 is further configured to drive the frequency of the virtual human's speech to be adapted to the speech frequency based on the synthesized audio, and simultaneously drive the virtual human's lips to change based on the speech duration of each speech character in the synthesized audio.

[0121] In some embodiments, the conversion module 520 includes a fourth determining submodule and a setting submodule, wherein the fourth determining submodule is used to determine that the text information is invalid information; and the setting submodule is used to set the synthesized audio to blank audio.

[0122] In some embodiments, the driving module 540 is further configured to drive the virtual human to remain silent based on the blank audio during the preset time period, while simultaneously setting the virtual human's lips to a closed state.

[0123] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0124] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause electronic devices (such as mobile phones, tablets, laptops, desktop computers, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0125] Correspondingly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the control method provided in the above embodiments.

[0126] Correspondingly, embodiments of this application provide an electronic device, Figure 6 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the hardware entity of the device 600 includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the program, it implements the steps in the control method provided in the above embodiments.

[0127] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) of the processor 602 and various modules in the electronic device 600, and can be implemented by flash memory or random access memory (RAM).

[0128] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0129] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0130] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0132] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0134] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0135] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0136] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0137] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0138] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0139] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A virtual human control method, the method comprising: Real-time speech recognition is performed on the real-time acquired audio stream based on a preset time period to obtain the text information corresponding to the target audio stream within the preset time period; The text information is converted into a synthesized audio with a duration equal to the preset time period; The virtual human's lip movements are driven by the synthesized audio. The step of converting the text information into synthesized audio of the same duration as the preset time period includes: The text information is determined to be valid; the speech duration and frequency corresponding to each character in the text information are determined; the speech frequency is used to characterize the pitch of a real person speaking; Based on the speech duration and speech frequency corresponding to each of the characters, each of the characters in the text information is converted into a synthesized audio with the same duration as the corresponding speech duration and adapted to the speech frequency. When the target audio stream cannot be interpreted, the text information is determined to be invalid. Set the synthesized audio to blank audio; The step of driving the virtual human's lip changes based on the synthesized audio includes: When the text information is valid, the frequency of the virtual human's speech is adapted to the speech frequency based on the synthesized audio, and the virtual human's lips are driven to change according to the speech duration of each text corresponding to the speech in the synthesized audio.

2. The method of claim 1, wherein driving the lip movements of the virtual human based on the synthesized audio comprises: During the preset time period, the virtual human remains silent based on the blank audio, while the virtual human's lips are set to a closed state.

3. A virtual human control device, the device comprising: The recognition module is used to perform real-time speech recognition on the real-time acquired audio stream based on a preset time period to obtain the text information corresponding to the target audio stream within the preset time period. The conversion module is used to convert the text information into synthesized audio of the same duration as the preset time period; A driving module is used to drive changes in the virtual human's lips based on the synthesized audio; The conversion module includes: The first determining submodule is used to determine that the text information is valid information; The second determining submodule is used to determine the speech duration corresponding to each character in the text information; The conversion submodule is used to convert each of the text information into a synthesized audio with the same duration as the corresponding speech duration, based on the speech duration corresponding to each of the texts. The fourth determination submodule is used to determine that the text information is invalid when the target audio stream cannot be interpreted; The settings submodule is used to set the synthesized audio to blank audio.

4. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the program to implement the steps of the method of any one of claims 1 or 2.

5. A storage medium storing executable instructions for causing a processor to execute, thereby implementing the steps of the method of any one of claims 1 or 2.

Images