Generation method and generation equipment of digital human interaction video

Abstract

The embodiment of the invention provides a generation method and generation equipment of a digital human interaction video. In the method, after a generation device obtains a target voice, a basic video and a facial image set can be obtained from a pre-constructed material library, key phoneme switching information corresponding to the target voice can be obtained by mapping each original phoneme in the target voice into one of a plurality of key phonemes, and in addition, the key phoneme switching information corresponding to the target voice can be obtained. Based on the facial image set and the key phoneme switching information, a target facial image sequence matched with the phoneme change condition in the target voice can be generated, and then the target facial image sequence is attached to the facial region of the digital human in the basic video to obtain a target video.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and more particularly to a method and device for generating digital human interactive videos. Background Art

[0002] With the rapid development of technologies such as artificial intelligence, digital human technology has become one of the most cutting-edge scientific and technological products in the digital economy era. Digital humans can simulate real-life expressions, movements, etc. to interact with users in multiple fields such as entertainment, education, and customer service. In order to achieve the interactive effect between digital humans and users, the terminal device needs to render each frame of the digital human interactive video in real time so that the rendered digital human's body movements and lip shapes are consistent with the voice content currently expressed. The rendering process of the above-mentioned interactive video needs to occupy or consume more processing resources, especially graphics processing unit (GPU) resources, which makes the computational cost of real-time interaction of digital humans high. In addition, the rendering process of the above-mentioned interactive video needs to rely on GPU resources, which also makes digital human interactive products unable to be widely used in various terminal devices, such as mobile phones.

[0003] Therefore, how to reduce the processing resources occupied or consumed by digital humans during real-time interaction is a technical problem that needs to be solved urgently.

[0004] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the invention

[0005] The present specification provides a method and device for generating a digital human interactive video, which can not only ensure that the facial changes of the digital human during the interaction are natural and the facial lip shape matches its output content, but also reduce the processing resources consumed when the digital human performs real-time interaction.

[0006] In the first aspect, the present specification provides a method for generating a digital human interactive video. The method includes: obtaining a target voice, wherein the target voice represents the information that the digital human needs to express through voice during interaction; obtaining a basic video and a facial image set from a pre-built material library, wherein the basic video represents the limb movement information of the digital human, and the facial image set represents the facial changes involved when the digital human switches pronunciation between different key phonemes among multiple key phonemes; by mapping each original phoneme in the target voice to one of the multiple key phonemes, obtaining the key phoneme switching information corresponding to the target voice; based on the facial image set and the key phoneme switching information, generating a target facial image sequence that matches the phoneme changes in the target voice; and fitting the target facial image sequence to the facial area of ​​the digital human in the basic video to obtain the target video.

[0007] In some embodiments, the multiple key phonemes include silent phonemes, and M non-silent phonemes in the phoneme table that meet target conditions, the target conditions including: the amplitude of the change between the pronunciation mouth shape and the closed mouth shape of the phoneme is greater than a preset threshold, the silent phoneme corresponds to the closed mouth shape, and M is an integer greater than 1.

[0008] In some embodiments, the facial image set includes: a plurality of facial images involved when the digital human switches pronunciation from the silent phonemes to the M non-silent phonemes respectively, and a plurality of facial images involved when the digital human switches pronunciation from each non-silent phoneme to other M-1 non-silent phonemes respectively.

[0009] In some embodiments, at least some of the face images in the face image collection are integrated into a sprite image.

[0010] In some embodiments, the facial image set is obtained in the following manner: obtaining a real-life video and a basic image, the real-life video showing the facial changes of a real person when the pronunciation of different key phonemes among the multiple key phonemes is switched, and the basic image showing the image of the digital person; migrating the facial changes shown in the real-life video to the basic image to obtain a synthetic video, the synthetic video showing the facial changes of the digital person when the pronunciation of different key phonemes among the multiple key phonemes is switched; and extracting key frames from the synthetic video to obtain the facial image set.

[0011] In some embodiments, migrating the facial changes shown in the live-action video to the base image to obtain the composite video includes: using a face reenactment model to migrate the facial changes shown in the live-action video to the base image to obtain the composite video.

[0012] In some embodiments, for any i-th original phoneme in the target speech, the i-th original phoneme is mapped to the key phoneme among the multiple key phonemes that has the greatest similarity with the pronunciation mouth shape of the i-th original phoneme, where i is an integer greater than or equal to 1; the key phoneme switching information includes: a key phoneme sequence obtained by mapping the original phonemes in the target speech, and a timestamp of each key phoneme in the key phoneme sequence.

[0013] In some embodiments, the key phoneme switching information corresponding to the target speech is obtained by mapping each original phoneme in the target speech to one of the multiple key phonemes, including: determining the timestamp corresponding to each original phoneme in the target speech; mapping each original phoneme in the target speech to a key phoneme among the multiple key phonemes that has the greatest similarity with the pronunciation mouth shape of the original phoneme; and generating the key phoneme sequence based on the key phonemes mapped to the original phonemes in the order of the original phonemes in the target speech, and determining the timestamp of each key phoneme in the key phoneme sequence based on the timestamp corresponding to each original phoneme.

[0014] In some embodiments, determining the timestamp corresponding to each original phoneme in the target speech includes: performing speech recognition on the target speech to obtain multiple text units and timestamps corresponding to the multiple text units; and mapping the multiple text units to multiple original phonemes, and obtaining a timestamp corresponding to each original phoneme in the multiple original phonemes based on the timestamps corresponding to the multiple text units.

[0015] In some embodiments, mapping the multiple text units to multiple original phonemes includes: obtaining a target language adopted by the target speech; and mapping the multiple text units to the multiple original phonemes based on a mapping relationship corresponding to the target language, wherein the mapping relationship represents a mapping relationship between the text units in the target language and the original phonemes.

[0016] In some embodiments, the generating of a target facial image sequence matching the phoneme changes in the target speech based on the facial image set and the key phoneme switching information comprises the following steps: for a first phoneme and a second phoneme in the key phoneme sequence, the first phoneme and the second phoneme being any two adjacent key phonemes in the key phoneme sequence: determining frame index information involved in a process of switching from the first phoneme to the second phoneme based on a timestamp of the first phoneme and a timestamp of the second phoneme, obtaining a standard facial image sequence corresponding to a pronunciation switching process from the first phoneme to the second phoneme from the facial image set, and generating an updated facial image sequence corresponding to a pronunciation switching process from the first phoneme to the second phoneme based on the frame index information and the standard facial image sequence; the step further comprises combining updated facial image sequences corresponding to pronunciation switching processes of different phonemes according to a switching order between phonemes in the key phoneme sequence to obtain the target facial image sequence.

[0017] In some embodiments, generating an updated facial image sequence corresponding to the pronunciation switching process from the first phoneme to the second phoneme based on the frame index information and the standard facial image sequence includes any one of the following: if the number of frames represented by the frame index information is equal to the number of facial images in the standard facial image sequence, then using the standard facial image sequence as the updated facial image sequence; if the number of frames represented by the frame index information is greater than the number of facial images in the standard facial image sequence, then interpolating the standard facial image sequence based on the frame index information to obtain the updated facial image sequence; or if the number of frames represented by the frame index information is less than the number of facial images in the standard facial image sequence, then sampling the standard facial image sequence based on the frame index information to obtain the updated facial image sequence.

[0018] In some embodiments, the number of facial images in the target facial image sequence is equal to the total duration of the target speech multiplied by the frame rate of the base video, and the step of fitting the target facial image sequence to the mouth area of ​​the digital human in the base video to obtain the target video includes: extracting a video clip from the base video, the number of image frames in the video clip being equal to the number of facial images in the target facial image sequence; and fitting the target facial image sequence to the facial area of ​​the digital human in the video clip frame by frame to obtain the target video.

[0019] In some embodiments, the step of fitting the target facial image sequence to the facial region of the digital human in the video clip frame by frame includes fitting the mouth region in the target facial image sequence to the mouth region of the digital human in the video clip frame by frame; or fitting the facial region in the target facial image sequence to the facial region of the digital human in the video clip frame by frame.

[0020] In some embodiments, for any natural number i, the i-th frame facial image in the target facial image sequence is fitted to the facial region of the digital human in the i-th frame limb image of the video clip, including: based on the interocular distance in the i-th frame facial image and the interocular distance in the i-th frame limb image, scaling the i-th frame facial image so that the face in the scaled i-th frame facial image is adapted to the face in the i-th frame limb image; based on the facial key point information corresponding to the i-th frame facial image, capturing the facial region from the scaled i-th frame facial image to obtain a facial sub-image; based on the facial key point information corresponding to the i-th frame facial image and the facial key point information corresponding to the i-th frame limb image, determining the facial region of the digital human in the i-th frame limb image; and fitting the facial sub-image to the facial region of the digital human.

[0021] In some embodiments, the step of fitting the facial sub-image to the facial region of the digital human comprises: performing Poisson fusion on the facial sub-image and the facial region of the digital human to achieve the step of fitting the facial sub-image to the facial region of the digital human.

[0022] In some embodiments, the method further includes synchronously playing the target voice and the target video to present the interactive effect of the digital human.

[0023] In some embodiments, the method further includes: merging the target voice and the target video to generate target audio and video; and playing the target audio and video to present the interactive effect of the digital human, or sending the target audio and video to a terminal device to present the interactive effect of the digital human through the terminal device.

[0024] In a second aspect, the present specification also provides a device for generating a digital human interactive video. The generating device includes at least one storage medium and at least one processor. The at least one storage medium stores at least one instruction set for generating a digital human interactive video. The at least one processor is in communication connection with the at least one storage medium. When the generating device is running, the at least one processor reads the at least one instruction and executes any one of the methods described in the first aspect according to the instructions of the at least one instruction set.

[0025] It can be seen from the above technical solutions that the method for generating interactive videos of digital humans provided in this specification can pre-build a library of materials including basic videos and facial image sets. When it is necessary to generate interactive videos, each original phoneme in the target speech is mapped to multiple key phonemes, and then the facial image set is used to generate a target facial image sequence that adapts to the changes in the phonemes in the target speech, and then the target facial image sequence is fitted to the facial area of ​​the digital human in the basic video to obtain an interactive video. The above solution can ensure that the facial changes, especially the changes in the mouth shape, of the digital human in the interactive process match the output voice content, thereby ensuring the fidelity of the digital human and the user's experience. On this basis, the above solution improves the efficiency of generating interactive videos of digital humans by mapping numerous original phonemes to a few key phonemes, reduces the processing resources consumed by digital humans during real-time interaction, and improves and reduces the computational cost of real-time interaction of digital humans.

[0026] The methods and other functions of the digital human interactive video generation device provided in this specification will be partially listed in the following description. The creative aspects of the digital human interactive video generation method and the generation device provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1A A schematic diagram of an application scenario of digital human interactive video generation provided according to an embodiment of this specification is shown;

[0029] Figure 1B A schematic diagram of a scene in which a target user is interacting with a digital human according to some embodiments of this specification is shown;

[0030] Figure 2 A schematic diagram of the structure of a generating device provided according to an embodiment of this specification is shown;

[0031] Figure 3 A flowchart of an interactive video generation method provided according to an embodiment of this specification is shown;

[0032] Figure 4 The present invention shows a process of constructing a material library and a content composition according to some embodiments of the present specification;

[0033] Figure 5shows a portion of images in a basic video provided according to some embodiments of the present specification;

[0034] Fig. 6A A partial facial image in a facial image collection provided according to some embodiments of the present specification is shown;

[0035] Figure 6B A partial facial image in a facial image collection provided according to some embodiments of the present specification is shown;

[0036] Figure 7 A flowchart of obtaining key phoneme switching information provided in some embodiments of this specification is shown; and

[0037] Figure 8 Some image frames in a target video provided according to some embodiments of the present specification are shown. DETAILED DESCRIPTION

[0038] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0039] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0040] In view of the following description, these and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of the combination and manufacture of the parts can be significantly improved. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0041] The flowcharts used in this specification illustrate the operations implemented by a system according to some embodiments in this specification. It should be clearly understood that the operations in the flowchart may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0042] It should be noted that the user data obtained in this specification is authorized by the user and does not involve user privacy.

[0043] For the convenience of description, the terms involved in this specification are first explained as follows.

[0044] A Digital Human, also known as a virtual avatar or virtual digital human, refers to a product that exists in the non-physical world, is created and used by computer means, and has multiple human characteristics (appearance characteristics, human performance capabilities, interaction capabilities, etc.).

[0045] The International Phonetic Alphabet (IPA) is a set of symbols used to represent the pronunciations of various languages in the world. It is developed by the International Phonetic Association and aims to provide linguists with a standardized pronunciation representation method that is not restricted by a specific language. The International Phonetic Alphabet includes vowels, consonants, other symbols (such as tones, vowel lengths, stresses, etc.), and some additional symbols.

[0046] A phoneme is the smallest unit divided from the perspective of speech and is the smallest sound segment that makes up the pronunciation of a word. A phoneme is the smallest speech unit in a language that can distinguish meanings. For example, in English, the word "bat" (bat) includes three phonemes, / b / , and / t / ; the word "pat" (pat) includes three phonemes, / p / , and / t / . The words "bat" and "pat" can be distinguished by phonemes. In Chinese, "猫" (cat) includes three phonemes, / m / , / a / , and / o / ; "麻" (hemp) includes three phonemes, / m / , / a / , and . The words "猫" and "麻" can be distinguished by phonemes. The IPA is a symbol system used to record phonemes.

[0047] Poisson Image Editing is an image processing technology that realizes seamless fusion between a source image and a target image by solving the Poisson equation. The core idea of this method is to fuse the source image into the target image while retaining the gradient information of the source image, so that the fused image presents a natural and continuous effect visually.

[0048] Before describing the specific embodiments of this specification, the application scenarios to which the technical solutions provided in this specification are applicable are first introduced.

[0049] Digital humans can replace manual services or separate voice services, respond to user interaction needs in real time, and interact with users in real time to provide visual services. Among them, interaction needs can include services that users want to obtain. For example, interaction needs can be questions or requests raised by users. For example, in the e-commerce customer service industry, user interaction needs can include answering questions about product details, size selection, payment methods or delivery issues. For another example, in the bank customer service industry, user interaction needs can include providing operations such as account registration, account query, account loss or transfer. For another example, in the cultural and tourism industry, user interaction needs can include providing travel routes, travel time, catering and accommodation suggestions. Digital human interaction videos can be videos of digital humans providing interactive services to users. For example, after a user raises questions about product details, the user's questions can be answered in real time by playing digital human interaction videos, so that the user can experience face-to-face voice communication. For another example, digital human interaction videos can provide users with operations such as how to register an account, and assist users in completing operations during the interaction with users. For another example, digital human interaction videos can provide users with real-time suggestions on travel routes. Due to different interaction requirements raised by users, the feedback or output content of the digital human when interacting with users is also different. Different output content will cause the digital human to present different lip shapes. Lip shapes synchronized with the content will enhance the realism of the digital human, so that users can get the experience of face-to-face voice communication with real service personnel.

[0050] A digital human can be a two-dimensional (2-Dimensional, 2D) digital human or a three-dimensional (3-Dimensional, 3D) digital human. 3D digital humans are usually created and presented using professional three-dimensional modeling, animation and rendering technologies, as well as more powerful computing resources. 3D digital humans usually require manual real-time driving to give the digital human an image, expression and movement. Compared with 3D digital humans, 2D digital humans do not require or require less complex three-dimensional graphics processing capabilities, and have better compatibility on different platforms or devices. 2D digital humans take up less computing resources and storage space during operation, and are suitable for running on low-performance devices. For example, running on a mobile phone. For the sake of ease of narration, this application mainly introduces digital humans as 2D digital humans as an example.

[0051] In the prior art, when generating digital humans, such as 2D digital human interactive videos, a large amount of video data of people speaking is usually used to train a neural network model to learn how to change the lip shape in the picture or video to the lip shape in the input voice. After the user's real-time interaction needs are raised, the device determines the content and corresponding image that the digital human needs to output or feedback, and then uses the neural network model to render the images with different lip shapes frame by frame in real time to obtain the lip shape details, thereby obtaining the corresponding digital human video. However, there are often high requirements for devices that use model reasoning, such as the need to use GPU resources for reasoning. This makes the cost of video generation and the cost of real-time interaction of digital humans very high, and the requirements for device performance are very high.

[0052] Therefore, this specification provides a method for generating interactive videos of digital humans, which can pre-build a library including basic videos and facial image sets. When it is necessary to generate interactive videos, each original phoneme in the target speech is mapped to multiple key phonemes, and then the facial image set is used to generate a target facial image sequence that adapts to the changes in the phonemes in the target speech, and then the target facial image sequence is fitted to the facial area of ​​the digital human in the basic video to obtain an interactive video. The above scheme can ensure that the facial changes, especially the changes in the mouth shape, of the digital human in the interactive process match the output speech content, thereby ensuring the fidelity of the digital human and the user's experience. On this basis, the above scheme improves the generation efficiency of digital human interactive videos by mapping numerous original phonemes to a few key phonemes, reduces the processing resources consumed by digital humans during real-time interaction, and improves and reduces the computing cost of real-time interaction of digital humans.

[0053] The digital human interactive video generation method and generation device provided in this specification can be applied to various service scenarios using digital humans. For example, in the above-mentioned customer service industry, the method can generate customer service digital human interactive videos that provide services to consumers. For another example, in the education industry, the method can generate teacher digital human interactive videos that provide online courses and training materials to students. For another example, in the media industry, the method can generate anchor digital human interactive videos that provide viewers with a rich viewing experience. For another example, in the cultural and travel industry, the method can generate tour guide digital human interactive videos that provide tourists with a vivid travel experience.

[0054] The digital human interactive video generation method of this specification can also be applied to any digital human video generation scenario, and will not be described in detail here.

[0055] Figure 1A The following is a schematic diagram of an application scenario of digital human interactive video generation according to an embodiment of the present specification. The application scenario may be the above-mentioned customer service industry, education industry, media industry, cultural travel industry, etc. Figure 1AAs shown, the scenario 100 may include a target user 110 , a terminal device 120 , a server 130 , a network 140 , and a database 150 .

[0056] The target user 110 may be a user who has an interactive demand, thereby triggering the generation of a digital human interactive video (hereinafter referred to as the interactive video). The terminal device 120 may be a device that responds to the interactive demand of the target user 110 and interacts with the target user 110 by playing the interactive video. The interactive video may include a digital human 160. The digital human 160 may output feedback content to the target user 110. In some embodiments, the digital human 160 may also perform body movements corresponding to the feedback content. Those skilled in the art will understand that the digital human 160 is a virtual character displayed on the display screen of the terminal device 120. Figure 1A For the sake of convenience in the example, the digital human 160 is not displayed on the display screen of the terminal device 120.

[0057] In some embodiments, the interactive video generation method can be executed on the terminal device 120. At this time, the terminal device 120 can store the data or instructions of the interactive video generation method described in this specification, and can execute or be used to execute the data or instructions. The terminal device 120 may include a hardware device with data information processing functions and the necessary programs required to drive the hardware device to work. In some embodiments, the terminal device 120 may include a mobile device, such as a mobile phone. The performance of the mobile phone is relatively low, and it is usually impossible to generate a digital human video through the aforementioned, existing model reasoning. In some embodiments, the terminal device 120 may include a tablet computer, a laptop computer, a built-in device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, etc., or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, etc., or any combination thereof. In some embodiments, the virtual reality device or the augmented reality device may include a virtual reality helmet, virtual reality glasses, a virtual reality patch, an augmented reality helmet, augmented reality glasses, an augmented reality patch, or the like, or any combination thereof. For example, virtual reality devices or augmented reality devices may include Google Glass, head mounted displays, VR, etc. In some embodiments, built-in devices in a motor vehicle may include an onboard computer, an onboard television, etc.

[0058] In some embodiments, the terminal device 120 may also provide the target user 110 with interactive capabilities so that the target user 110 can input its interactive needs to the terminal device 120. Then the terminal device 120 generates an interactive video based on the interactive needs. For example, the terminal device 120 may be installed with one or more application programs (APPs). APP can provide the target user 110 with the ability and interface to interact with the outside world through the network 140. The target user 110 can input interactive needs through the interface of the APP. APP includes but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, mailbox terminal devices, social platform software, etc. A target APP may be installed on the terminal device 120. The target APP can provide interactive capabilities to the target user 110. Thus, the target user 110 can trigger the generation of an interactive video through the target APP. The target APP can execute the interactive video generation method described in this specification in response to the trigger. The interactive video generation method will be described in detail in the following content. For another example, the terminal device 120 may be equipped with input devices including but not limited to a keyboard, a mouse, a camera, a microphone, or a touch screen. For example, the target user 110 may input his / her interaction requirements to the terminal device 120 through a keyboard. For another example, the target user 110 may voice the interaction requirements, the microphone may obtain the voice information of the target user 110, and analyze the user's interaction requirements based on the voice information. For another example, the target user 110 may make a specific action, the camera may obtain the body information of the target user 110, and analyze the user's interaction requirements based on the body information.

[0059] In some embodiments, the terminal device 120 may be used as both a device for generating an interactive video and a device for outputting an interactive video. For example, the terminal device 120 may include, but is not limited to, a display device. The terminal device 120 may output the interactive video through the display device. For another example, the terminal device 120 may include both a display device and a speaker to output the generated interactive video. For example, the terminal device 120 may output the interactive video through the display device and output the audio corresponding to the interactive video through the speaker.

[0060] Figure 1B A schematic diagram of a digital human interaction scenario provided according to some embodiments of the present specification is shown. Figure 1B The scenario shown may be an inquiry scenario, taking the example that the terminal device 120 is a mobile phone used by the target user 110. Figure 1BAs shown, the target user 110 uses the voice input button at the bottom of the mobile phone to express the need to check the weather to the digital human 160 in the self-service machine. The terminal device 120 can generate and output an interactive video to the target user 110 by executing the interactive video generation method of this specification. In the interactive video, the digital human can feedback or reply to the user's living bill, thereby interacting with the target user 110 in real time. For example, the interaction between the target user 110 and the digital human 160 includes the following two rounds of interaction:

[0061] The target user 110 first asks the digital human 160 a question (U1): "What is the weather like today?"

[0062] In the generated interactive video, Digital Human 160 replied (D1): <The highest temperature in your area is 20℃, the lowest temperature is 12℃, the wind direction is northeasterly, the wind force is level 3, the relative humidity is 36%, the ultraviolet intensity is very strong, and the air quality index is 36, which is excellent. >.

[0063] The target user 110 asks the digital human 160 again in voice (U2): <What should I wear to go out comfortably today? >.

[0064] In the generated interactive video, Digital Human 160 replied (D2): <Today's weather is relatively cool, so you can choose a long-sleeved T-shirt or a thin long-sleeved shirt for the top, jeans or casual pants for the bottom, and either sports shoes or casual shoes. If the weather is cooler, you can choose shoes that cover the ankles. >

[0065] Continue to refer to Figure 1A , the server 130 may be a server that provides various services to the terminal device 120. For example, the server 130 may provide support to the target APP deployed on the terminal device 120. In some embodiments, the interactive video generation method described in this specification may be executed on the server 130. At this time, the server 130 may store data or instructions for executing the interactive video generation method described in this specification, and may execute or be used to execute the data or instructions. The server 130 may include a hardware device with a data information processing function and a necessary program for driving the hardware device to work. In this case, after the terminal device 120 receives the interactive demand of the target user 110, it may send the interactive demand to the server 130. The server 130 generates an interactive video by executing the interactive video generation method provided in this specification, and sends the interactive video to the terminal device 120. Thus, the terminal device 120 plays the interactive video to achieve real-time interaction with the target user 110.

[0066] The network 140 can facilitate the exchange of information and / or data. The network 140 can be any type of wired or wireless network, or a combination thereof. For example, the network 140 can include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan / urban network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, or a similar network. The network 140 may include one or more network access points. For example, the network 140 may include a wired or wireless network access point, such as a base station and / or an Internet exchange point. Through the access point, one or more components of the terminal device 120, the server 130, and the database 150 can be connected to the network 140 to exchange data and / or information, wherein the target user 110 can be connected to the network 140 via the terminal device 120. Figure 1A As shown, the server 130, the terminal device 120, and the database 150 may be connected to the network 140, and may transmit information and / or data to each other via the network 140. For example, the terminal device 120 may obtain services from the server 130 via the network 140.

[0067] The database 150 may be used to store the data and / or instructions required by the terminal device 120 or the server 130 in the process of executing the interactive video generation method described in this specification. The server 130 and the terminal device 120 may have access to the database 150. The database 150 may be directly connected to the server 130 and the terminal device 120. Among them, the database 150 may be a part of the server 130. The database 150 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM) or the like, or any combination thereof. Exemplary mass storage may include non-transitory storage mediums such as disks, optical disks, and solid-state drives. Example removable storage may include flash drives, floppy disks, optical disks, memory cards, zip disks, tapes, and the like. Typical volatile read-write memory may include random access memory (RAM). Example RAM may include dynamic RAM (DRAM), dual date rate synchronous dynamic RAM (DDR-SDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero capacitance RAM (Z-RAM). Exemplary ROMs may include mask ROM (MROM), programmable ROM (PROM), virtual programmable ROM (PEROM), electronic programmable ROM (EEPROM), compact disk (CD-ROM) and digital versatile disk ROM, etc. Database 150 can be implemented on a cloud platform. Just by way of example, a cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, a cloud-to-cloud cloud or the like, or any combination thereof. The storage medium may include a remote independent storage medium, such as the above-mentioned disk, hard disk and solid-state drive, etc. The server 130 / terminal device 120 can access or update data stored in a remote independent storage medium through a mobile device with access rights. The storage medium may include a local storage medium. For example, a data storage hard disk of a local server or the IOT device itself, etc.

[0068] It should be understood that Figure 1A and Figure 1B The number of terminal devices 120, servers 130 or networks 140 in the embodiment is only for illustration. Any number of terminal devices 120, servers 130 and networks 140 may be provided as required.

[0069] It should be noted that the method for generating digital human interactive video provided in this specification can be completely executed on the terminal device 120, can be completely executed on the server 130, or can be partially executed on the terminal device 120 and partially executed on the server 130.

[0070] Figure 2The structure diagram of a digital human interactive video generation device provided according to an embodiment of the present specification is shown. The digital human interactive video generation device 200 (hereinafter referred to as the generation device 200) can be used as Figure 1A and / or Figure 1B The terminal device 120 in the embodiment can also be used as Figure 1A The server 130 in.

[0071] like Figure 2 As shown, the generating device 200 includes at least one storage medium 230 and at least one processor 220. In some embodiments, the generating device 200 may further include a communication port 250 and an internal communication bus 210. Meanwhile, the generating device 200 may further include an I / O component 260.

[0072] Internal communication bus 210 may connect various system components, including storage media 230 and processor 220. I / O component 260 supports input / output between generating device 200 and other components.

[0073] The storage medium 230 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. The storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set is a computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the method provided in this specification.

[0074] The communication port 250 is used for data communication between the generating device 200 and the outside world. For example, the generating device 200 can be connected to the network through the communication port 250. At least one processor 220 is connected to at least one storage medium 230 through the internal communication bus 210. At least one processor 220 is used to execute the at least one instruction set. When the generating device 200 is running, the at least one processor 220 reads the at least one instruction set and executes the interactive video generating method provided in this specification according to the instructions of the at least one instruction set.

[0075] The processor 220 can execute all the steps included in the interactive video generation method. The processor 220 can be in the form of one or more processors. The processor 220 can issue execution instructions. The processor 220 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0076] For illustration purposes only, only one processor 220 is shown in the generating device 200 in the drawings of this specification. However, it should be noted that the generating device 200 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor as described in this specification, or may be performed jointly by multiple processors. For example, if the processor 220 of the generating device 200 in this specification performs step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 220 (for example, the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0077] Figure 3 A flow chart of an interactive video generation method provided according to an embodiment of the present specification is shown. As previously shown, the server 130 and / or the terminal device 120 may execute the interactive video generation method S300 (hereinafter referred to as method S300). Method S300 may be executed entirely on the terminal device 120, or entirely on the server 130, or partially on the terminal device 120 and partially on the server 130. For example, the processor 220 in the terminal device 120 may be communicatively connected to the storage medium 230. The processor 220 may read an instruction set stored in its local storage medium and / or database, and then execute method S300 according to the provisions of the instruction set. The following description will be given by taking the execution of method S300 by the processor 220 as an example. Figure 3 As shown, method S300 may include:

[0078] S310: Obtain target speech. The target speech represents the information that the digital human 160 needs to express through speech during interaction.

[0079] For example, the target voice may be feedback information or reply information made by the digital human 160 based on the interaction requirements proposed by the target user 110. Figure 1B D1 in the scene: <The highest temperature in your area is 20℃, the lowest temperature is 12℃, the wind direction is northeast wind, wind force 3, relative humidity is 36%, ultraviolet intensity is very strong, and the air quality index is 36, which is excellent. >. D2: <Today's weather is relatively cool. You can choose a long-sleeved T-shirt or a thin long-sleeved shirt for the top, jeans or casual pants for the bottom, and sneakers or casual shoes are both fine. If the weather is cool, you can choose shoes that cover the ankles. >. The target voice may include some or all of the information that needs to be expressed through voice during the multiple interactions between the digital human 160 and the target user 110. For example, the target voice includes D1, D2, and the information in the subsequent interactions between the digital human 160 and the target user 110.

[0080] In some embodiments, the target voice may be information pre-stored in the database 150 or the storage medium 230. After the target user 110 triggers the keyword, the processor 220 may obtain the information corresponding to the keyword from the pre-stored information in real time. For example, the target user 110 triggers the keyword <weather> in U1. The digital human 160 may reply with the information D1 corresponding to the keyword. The target user 110 simultaneously triggers the keywords <wear>, <go out>, and <comfortable> in U2. The digital human 160 may reply with the information D2 corresponding to the keyword. Among them, the processor 220 may directly obtain the target voice. The processor 220 may also first obtain text information and then convert the text information into the target voice.

[0081] In some embodiments, the target voice may also be an artificial intelligence (AI) model that analyzes the user's interaction needs in real time and outputs intelligent, high-quality feedback information. The AI ​​model may directly output the target voice. The AI ​​model may also output text information first and then convert the text information into the target voice.

[0082] In some embodiments, the target voice may be real-time feedback from a real person, and information that needs to be outputted in the form of a digital human 160. For example, in the media industry, a digital human anchor may be controlled by a real person. After the target user 110 raises an interactive demand, the real person may give corresponding feedback. The processor 220 may obtain the voice feedback given by the real person in real time and use it as the target voice.

[0083] S330: Obtain a basic video and a set of facial images from a pre-built material library, wherein the basic video represents the body movement information of the digital human, and the set of facial images represents the facial changes involved when the digital human switches pronunciation between different key phonemes among a plurality of key phonemes.

[0084] The material library may be a collection of resources used to generate interactive videos. The material library may exist before the processor 220 executes method S300. The material library may be pre-constructed by the processor 220 or pre-constructed by the server 130. For ease of description, the construction of the material library by the processor 220 is taken as an example for introduction. The material package may be pre-stored in the storage medium 230 or stored in the database 150. When generating an interactive video, the processor 220 may obtain the pre-constructed material library in real time. The processor 220 may obtain all the materials in the material library, or may obtain part of the materials in the material library according to the generation requirements of the interactive video.

[0085] Figure 4 The figure shows the composition process and content composition of a material library provided according to some embodiments of this specification. Figure 4 As shown, the material library L may include basic videos and a collection of facial images.

[0086] The basic video may represent the limb movement information of the digital human 160. The limb movement information of the digital human 160 may include information such as the limb position, movement state or movement trajectory of the digital human 160. The action presented by the digital human 160 may be changed by changing the limb movement information of the digital human 160. The basic video may include multiple frames of images when the digital human 160 presents different actions. For example, the digital human 160 may present one or more actions including making a finger gesture of the number 1, touching the abdomen with both hands, touching the abdomen with one hand and raising the other hand, hanging both hands, cheering, clapping with both hands, denying and rejecting, spreading both hands, thinking, giving a thumbs up with both hands, listening closely, making a gesture of an upward trend chart, waving and saying hello, etc.

[0087] In some embodiments, the basic video can be obtained through machine learning training based on real-life photos or videos. In some embodiments, the digital human can be customized by inputting custom parameters through existing digital human generation tools to obtain the basic video.

[0088] It is worth noting that the digital human 160 in the basic video includes a human face. The facial features of the digital human 160 in the basic video may be fixed parameters, that is, the face of the digital human 160 may always remain consistent without change. For example, the digital human 160 may always keep its mouth closed when performing different actions. The human face included in the basic video may be the same as the human face in the face image collection to ensure the harmony of the face of the digital human 160 obtained by subsequent splicing.

[0089] Figure 5 Partial images in a basic video provided according to some embodiments of the present specification are shown. Figure 5 Three frames of images in the basic video are shown, Frame1, Frame2 and Frame3.

[0090] In Frame 1, the head of the digital human 160 slightly sways to the right. In Frame 2, the head of the digital human 160 remains in the middle. In Frame 3, the head of the digital human 160 slightly sways to the left. The action in Frame 2 can be a transition action between Frame 1 and Frame 3. In the three frames, the digital human 160 always keeps its mouth closed.

[0091] It is worth mentioning that Figure 5 The three frames of images may be three consecutive frames of images in the basic video, or may be three discontinuous frames of images.

[0092] In some embodiments, in addition to obtaining the basic video, facial key points corresponding to the basic video can also be obtained through the generation tool or machine learning. The facial key points corresponding to the basic video can be points that characterize the face or head position of the digital human 160. For example, the key points can include but are not limited to the midpoint of the top of the head, the top of the ear, the tip of the chin, and other feature points. The rotation angle, tilt angle, etc. of the digital human's head can be calculated through the key points, thereby determining the head position of the digital human 160.

[0093] like Figure 5 As shown, the head of the digital human 160 in Frame2 remains in the middle. In Frame3, the head of the digital human 160 tilts to the left. The different positions of the head of the digital human 160 in Frame2 and Frame3 can be determined by key points. By determining the head position of the digital human 160 in the basic video, an accurate splicing point can be determined for subsequent face splicing. The specific determination method will be introduced later. The key point corresponding to the basic video can be the coordinate information of the key point. The coordinate information of the key point can be the relative coordinate of the key point relative to the preset point. For example, the preset point is the left vertex of the image frame.

[0094] In some embodiments, the facial key points in the material library L may also include lip key points corresponding to the basic video. The mouth key points corresponding to the basic video may be points that characterize the shape and position of the mouth of the digital human 160. The lip key points may include but are not limited to the corners of the mouth, the philtrum, the lip beads, the outer sides of the upper and lower lips, the inner sides of the upper and lower lips, etc. The position of the mouth of the digital human 160 or the area where the lip shape is located may be calculated through the mouth key points. The lip key point information corresponding to the basic video may be the coordinate information of the lip key points.

[0095] In some embodiments, the key points corresponding to the basic video may be obtained by performing key point detection on the basic video after the basic video is obtained.

[0096] In some embodiments, the material library L may also include the interocular distance corresponding to the basic video. The interocular distance corresponding to the basic video may be the distance between the centers of the two eyes of the digital human 160. The interocular distance is related to the size and proportion of the head (face) and is an important parameter for characterizing the size of the head (face). The interocular distance corresponding to the basic video may include data of the interocular distance of the digital human 160 in some or all frame images in the basic video.

[0097] In some embodiments, the interocular distance corresponding to the basic video and the lip key points corresponding to the basic video can be obtained after data processing based on the basic video and the facial key points corresponding to the basic video.

[0098] Continue to see Figure 4 The material library L may also include a set of facial images. The set of facial images may represent facial changes involved when the digital human 160 switches pronunciation between different key phonemes in the plurality of key phonemes.

[0099] The multiple key phonemes may include silent phonemes and M non-silent phonemes in the phoneme table that meet the target condition, where M is an integer greater than 1. The target condition may include: the change range between the pronunciation mouth shape and the closed mouth shape of the phoneme is greater than a preset threshold. The silent phoneme represents no pronunciation. The mouth shape corresponding to the silent phoneme may be a closed mouth shape.

[0100] The preset threshold can be obtained based on manual experience or calculated based on an artificial intelligence model, and this specification does not limit this.

[0101] The phoneme table includes the phonemes shown in the International Phonetic Alphabet IPA as an example. Different phonemes can correspond to different pronunciation mouth shapes. When some phonemes are pronounced, the corresponding pronunciation mouth shape changes more than the closed mouth shape. For example, the pronunciation of vowels such as [u] and [u] is different from that of [u]. For example, when pronouncing the tip of the tongue, such as [t] and [d], the mouth shape is that the tip of the tongue touches the back of the lower teeth. When pronouncing, the mouth shape protrudes forward. For some phonemes, the corresponding mouth shape during pronunciation has a relatively small change compared to the closed - mouth shape. For example, when pronouncing consonants such as [m] and [n], the sound is mainly emitted through the nose, and the positions of the tongue and lips are relatively fixed, so the change in the mouth shape is relatively small. The digital human 160 can present these mouth shapes with relatively small changes. The digital human 160 can also not present these mouth shapes. Since the mouth shape changes are small, the digital human 160 does not present these mouth - shape changes, which can save processing resources and does not affect the viewing experience of the target user 110.

[0102] In some embodiments, the M phonemes in the preset phoneme set may include 6 vowels and 2 consonants { / b / , / f / }. The pronunciation references for the above - mentioned phonemes are {yi, ai, a, wu, ou, ao, bu, fu}. These 8 phonemes are 8 phonemes that are highly correlated with mouth - shape changes and have a large discrimination degree according to the standards of the IPA International Phonetic Alphabet when people with different languages speak. The mouth shapes corresponding to these 8 phonemes and the transitional changes between them can cover the vowel - consonant representations related to human mouth - shape changes.

[0103] The set of facial images can include multiple facial images involved when the digital human 160 switches from a silent phoneme to each of the M non - silent phonemes during pronunciation. For example, the set of facial images includes multiple facial images from not making a sound to pronouncing the phoneme / i / . For another example, the set of facial images includes multiple facial images from not pronouncing to pronouncing the phoneme For another example, the set of facial images includes multiple facial images from pronouncing the phoneme to not pronouncing. For another example, the set of facial images includes multiple facial images from pronouncing the phoneme / f / to not pronouncing.

[0104] The set of facial images can also include multiple facial images involved when the digital human 160 switches from each non - silent phoneme to the other M - 1 non - silent phonemes during pronunciation. For example, the set of facial images includes multiple facial images from pronouncing the phoneme to pronouncing the phoneme / i / . For another example, the set of facial images includes multiple facial images from pronouncing the phoneme to pronouncing the phoneme / i / .

[0105] Fig. 6A shows some of the facial images in the set of facial images provided according to some embodiments of this specification. Figure 6B shows some of the facial images in the set of facial images provided according to some embodiments of this specification.

[0106] Fig. 6AThe facial images of the digital human 160 when switching from silent phonemes to four of the eight phonemes are shown. (I): / i / , the pronunciation mouth shape is that the corners of the lips are spread to both sides. (II): / u / , the lips protrude forward and close together to form a small circle. (III): Lips spread as far apart as possible, mouth open wide, with about two fingers' width between upper and lower teeth. (IV): The lips are slightly rounded, and the tongue is lowered and retracted. In some embodiments, for each switching process, the facial image set may include 8 facial images in the above pronunciation switching process ( Fig. 6A Only 5 figures are shown).

[0107] Figure 6B The facial images of the digital human 160 when the pronunciation of the phoneme / i / is switched to four of the other seven phonemes are shown. (I): The phoneme / i / is switched to the phoneme / u / . (II): The phoneme / i / is switched to the phoneme / u / . (III): Phoneme / i / switches to phoneme (IV): Switching from phoneme / i / to phoneme / b / . In some embodiments, the mouth of the digital human 160 is already open when switching between phonemes, so switching to the mouth shape corresponding to another non-silent phoneme is faster than switching to the mouth shape corresponding to the closed mouth. Therefore, for each switching process, the facial image set may include 4 facial images in the above pronunciation switching process.

[0108] Since the mouth shape of the digital human 160 changes most obviously when it pronounces different phonemes, and has a relatively high impact on the user's viewing experience, the facial changes involved in the face image collection mainly include the above-mentioned mouth shape changes. In some embodiments, in addition to the mouth shape changes, the real person may also cause other parts of the face to change when the pronunciation switches. When the face is changed, the eyebrows may be raised, etc. Therefore, the facial changes involved in the face image collection include the entire face, including changes in all parts of the face, such as eyes, eyebrows, cheeks, etc.

[0109] Continue to see Figure 4 , the following combination Figure 4 The generation process of the face image collection is described.

[0110] (1) Obtain real-life video and basic images.

[0111] The live video may show the facial changes of the live person when the live person switches the pronunciation of different key phonemes among the multiple key phonemes. The basic image may show the image of the digital person 160. The basic image may be a static image. The image of the digital person 160 includes but is not limited to the appearance and clothing of the digital person 160. The image of the digital person 160 may be designed or selected according to the corporate image or user needs.

[0112] (2) The facial changes shown in the real-life video are transferred to the basic image to obtain a composite video. The composite video can show the facial changes of the digital human 160 when the digital human 160 switches the pronunciation of different key phonemes among the multiple key phonemes.

[0113] For example, the facial changes of a pronunciation video recorded by a professional broadcasting model are re-interpreted on the basic image to obtain a synthetic video with consistent facial changes. For example, the lip shape changes of the broadcasting model during the pronunciation switching process can be reproduced on the digital human 160 on the basic image or re-interpreted by the digital human 160.

[0114] In some embodiments, the migrating the facial changes shown in the live-action video to the base image to obtain a synthetic video may include the following steps: using a face reenactment model to migrate the facial changes shown in the live-action video to the base image to obtain a synthetic video.

[0115] The Face Reenactment Model is a deep learning algorithm for generating or modifying facial images and video content. The Face Reenactment Model can reconstruct or synthesize digital facial representations based on input features (such as base images or live-action videos). The model relies on generative adversarial networks (GANs), convolutional neural networks (CNNs), and other advanced computer vision techniques to achieve high-fidelity real-time or offline facial animation, emotional expression conversion, and facial attribute editing.

[0116] In some embodiments, the face re-enactment model can be obtained through training. The training data of the face re-enactment model can include sample real-life videos and sample basic images. The training goal of the face re-enactment model can include constraining the difference between the synthetic video output by it and the pre-labeled or generated synthetic video to be within a first preset difference threshold. The first preset difference threshold can be determined based on experience or based on a model training process.

[0117] (3) Extract key frames from the synthetic video to obtain a set of facial images.

[0118] The key frame may be a facial image in the synthesized video that has a high ability to represent the change in lip shape. The lip shape change between key frames is relatively large. In some embodiments, the key frame may be uniformly selected from the synthesized video. For example, the digital human 160 in the synthesized video switches from the phoneme / i / to the phoneme / u / in a total of 5 frames, and the processor 220 may select the 1st frame, the 3rd frame, and the 5th frame as the corresponding facial images.

[0119] By using the face reenactment model to obtain a facial image set, it can be ensured that the facial expression of the digital human 160 is very natural, smooth and real when the digital human 160 is subsequently generated, thereby providing a good viewing experience for the target user 110.

[0120] In some embodiments, at least part of the face images in the face image set can be integrated into a sprite image. For example, the sprite image can include M regions. The M regions can correspond to the M key phonemes one by one. Each region includes multiple face images involved when the key phoneme is switched to a silent phoneme for pronunciation, and multiple face images involved when the key phoneme is switched to other M-1 non-silent phonemes for pronunciation.

[0121] The multiple facial images corresponding to each phoneme may be located in the region corresponding to the key phoneme. When selecting the multiple facial images corresponding to the key phoneme, the processor 220 may locate the region corresponding to the key phoneme from the sprite map, and then obtain all or part of the facial images in the corresponding region.

[0122] By integrating the face image collection into a sprite image, the number of requests when loading the screen can be significantly reduced, thereby achieving faster screen loading speed, improving efficiency, and enhancing the user experience of the target user.

[0123] In some embodiments, the material library L may further include the distance between the eyes in the face image. In some embodiments, the material library L may further include the key points of the face in the face image. In some embodiments, due to the existence of the sprite image, the distance between the eyes and the key points of the face may be the distance between the eyes corresponding to the sprite image and the key points of the face corresponding to the sprite image, respectively. Figure 4 As shown, the material library L may include the inter-eye distance corresponding to the Sprite image and the facial key points corresponding to the Sprite image.

[0124] In some embodiments, the facial landmarks in the facial map may include lip landmarks in the facial map.

[0125] In some embodiments, the pre-built material library L may be updated regularly or irregularly to adapt to different market or user needs.

[0126] By pre-building a library of materials, the time for creating resources in the video production process can be reduced, and the amount of calculation can be greatly reduced while ensuring image quality. When the subsequent interactive video is generated, the processor 220 only needs to read the pre-processed image, thereby greatly reducing the resource consumption during real-time interaction.

[0127] Continue to refer to Figure 3 , the method S300 may further include:

[0128] S350: Obtain key phoneme switching information corresponding to the target speech by mapping each original phoneme in the target speech to one of the multiple key phonemes.

[0129] The original phoneme may be a phoneme originally existing in the target speech.

[0130] For any i-th original phoneme in the target speech, the i-th original phoneme can be mapped to the key phoneme with the greatest similarity to the pronunciation mouth shape of the i-th original phoneme among the multiple key phonemes. The key phoneme switching information can include a key phoneme sequence obtained by mapping the original phonemes in the target speech, and a timestamp of each key phoneme in the key phoneme sequence.

[0131] The original phonemes and the corresponding key phonemes have similar pronunciation mouth shapes. By mapping a large number of original phonemes to a small number of key phonemes, it can not only ensure that the facial mouth shape of the digital human 160 matches its output content during the interaction process, but also reduce the processing resources consumed by the digital human 160 during real-time interaction.

[0132] Figure 7 FIG. 2 shows a flowchart of obtaining key phoneme switching information according to some embodiments of the present specification. Figure 7 As shown, step S350 may include:

[0133] S351: Determine the timestamp corresponding to each original phoneme in the target speech.

[0134] Timestamps can be used to identify the time when a particular event occurs. Here, timestamps can be used to record or identify the pronunciation time of key phonemes. Timestamps can be a moment or a time range. For example, the timestamp is moment T. For another example, the timestamp is the time range from moment T1 to moment T2, recorded as [T1, T2]. For another example, the timestamp is a time range with a duration of 10ms, 100ms, 1s, 5s, 10s, etc.

[0135] Specifically, step S351 may include:

[0136] S351 - 1 : Perform speech recognition on the target speech to obtain a plurality of text units and timestamps corresponding to the plurality of text units.

[0137] The text unit can be a letter, a word, a sentence, etc. Taking Chinese as an example, the target speech can be a sentence, and the text unit can be a Chinese character. For example: the target speech is . By performing speech recognition on the target speech, four text units [, <Love>, <China>, <Country>] and four Chinese characters can be obtained. By performing speech recognition on the target speech, timestamps corresponding to multiple text units can also be obtained. For example, corresponds to timestamp I, <Love> corresponds to timestamp II, <China> corresponds to timestamp III, and <Country> corresponds to timestamp IV. As mentioned above, the timestamp can be a moment or a time range. In daily conversations, the average speaking time of each word is about 1.7 seconds to 2 seconds. When the timestamp is a time range, the speaking time of each text unit can be between 1.5 seconds and 2.5 seconds, so as to increase the realism of the digital human 160's facial changes, especially the lip shape changes, and enhance the viewing experience of the target user 110.

[0138] There may be segments in the interactive video that do not require the digital human 160 to output voice information. For example, the target voice is Figure 1B D2 in the figure. The digital human 160 made a semantic pause between <Today's weather is relatively cool> and <You can choose a long-sleeved T-shirt or a thin long-sleeved shirt as your top>. That is, after saying <Today's weather is relatively cool>, it took a while before saying <You can choose a long-sleeved T-shirt or a thin long-sleeved shirt as your top> again. The semantic pause can be reflected in the facial lip shape, which can correspond to the digital human 160's lip shape not changing. For example, the digital human 160 maintains a closed mouth shape. Therefore, in some embodiments, after obtaining the timestamp corresponding to the text unit, the processor 220 can also process the timestamp as a silent segment, fill in the segments without pronunciation, so that the entire timestamp sequence of the target voice is continuous, so as to subsequently generate a harmonious interactive video.

[0139] S351 - 3 : Map the multiple text units to multiple original phonemes, and obtain a timestamp corresponding to each original phoneme in the multiple original phonemes based on the timestamps corresponding to the multiple text units.

[0140] A text unit can be mapped to one original phoneme or multiple original phonemes, and multiple text units can be mapped to multiple original phonemes.

[0141] In some embodiments, the multiple text units may be mapped to the multiple original phonemes in the following manner: obtaining a target language adopted by the target speech; and mapping the multiple text units to the multiple original phonemes based on a mapping relationship corresponding to the target language.

[0142] Among them, the mapping relationship represents the mapping relationship between the text unit and the original phoneme in the target language. The mapping relationship between the text unit and the original phoneme is different depending on the target language used by the target speech. For example, if the target language is Chinese, the mapping relationship can be pinyin. The mapping relationship between the text unit and the original phoneme can be provided by, but not limited to, the pypinyin library. For another example, if the target language is English, the mapping relationship is phonetic symbols. The mapping relationship between the text unit and the original phoneme can be provided by, but not limited to, the cmudict library. The target language can be other languages. The mapping relationship between the text unit and the original phoneme of other languages ​​can be obtained through, but not limited to, the library officially provided by IPA.

[0143] For example, the target speech may be the Chinese word "I love China". The original phonemes in the target speech are represented by Chinese pinyin including {I: Love: [ / a / , / I / ]; Chinese: country: For example, the target language could be English The original phonemes in the target language are represented using phonetic symbols and include

[0144] After the processor 220 obtains the timestamps corresponding to multiple text units, it can obtain the timestamps corresponding to multiple original phonemes. For example, if the text units in the target language are [, <love>, <China>], and the original phonemes of the text units are Timestamp I, Timestamp II, Timestamp III, Timestamp IV. Then / w / corresponds to Timestamp I1, corresponds to Timestamp I2; / a / corresponds to Timestamp II1, / I / corresponds to Timestamp II2; corresponds to Timestamp III1, corresponds to Timestamp III2, corresponds to Timestamp III3; / g / corresponds to Timestamp IV1, / u / corresponds to Timestamp IV2, corresponds to Timestamp IV3.

[0145] Continuing to refer to Figure 7 , step S350 may further include:

[0146] S353: Map each original phoneme in the target voice to one key phoneme in the multiple key phonemes that has the greatest similarity in pronunciation mouth shape with this original phoneme.

[0147] For example, the original phonemes include { / p / , / m / }. The key phoneme with the greatest similarity to these two original phonemes among the 8 key phonemes is / b / . These three original phonemes are all bilabial consonants, and the two lips need to be closed to form an obstruction during pronunciation. That is to say, the pronunciation mouth shapes of these three are relatively similar. Therefore, the original phonemes { / p / , / m / } can be mapped to the key phoneme / b / . Another example is that a phoneme that belongs to both the original phonemes and the key phonemes is mapped to itself during mapping. For example, the original phoneme includes / u / . Then its mapped phoneme is still itself / u / .

[0148] In some embodiments, the similarity between pronunciation mouth shapes can be obtained by comparing the similarity between the pronunciation mouth shape and the closed - mouth shape. For example, the pronunciation mouth shape A has a similarity a with the closed - mouth shape, and the pronunciation mouth shape B has a similarity b with the closed - mouth shape. The similarity between the pronunciation mouth shape A and the pronunciation mouth shape B can be a / b. In some implementations, the similarity between the pronunciation mouth shape and the closed - mouth shape can be evaluated by measuring the position change (displacement magnitude) of the lips during pronunciation relative to when the mouth is closed.

[0149] S355: According to the order of the original phonemes in the target voice, generate the key phoneme sequence based on the key phonemes after mapping of the original phonemes, and determine the timestamps of the key phonemes in the key phoneme sequence based on the timestamps corresponding to the original phonemes.

[0150] The order of the original phonemes in the target speech may be the time order of the appearance / sounding of the original phonemes in the target speech. The key phoneme sequence may represent the order of appearance / sounding of the key phonemes. The timestamps corresponding to the mapped key phonemes may be the same as the timestamps of the corresponding original phonemes.

[0151] For example, the original phonemes in <How are you> may include {you: [ / n / , / i / ]; good: [ / h / , / a / , / u / ]; are you: [ / m / , / a / ]}. The key phoneme sequence generated by the key phonemes after the original phonemes are mapped may be { / i / , / i / , / f / , / a / , / u / , / b / , / a / }. The timestamps corresponding to the original phonemes { / n / , / i / } are I1 and I2 respectively. Then the timestamps corresponding to the first two { / i / , / i / } of the key phonemes obtained by mapping are also I1 and I2.

[0152] Continue to see Figure 3 , step S300 may further include:

[0153] S370: Based on the facial image set and the key phoneme switching information, generate a target facial image sequence that matches the phoneme changes in the target speech.

[0154] The key phoneme switching information may include a key phoneme sequence obtained by mapping each original phoneme in the target speech, and a timestamp of each key phoneme in the key phoneme sequence.

[0155] In some embodiments, the target facial image sequence may be generated by sequentially performing the following steps:

[0156] (1) Perform the following steps for the first phoneme and the second phoneme in the key phoneme sequence, wherein the first phoneme and the second phoneme are any two adjacent key phonemes in the key phoneme sequence:

[0157] (i) Based on the time stamp of the first phoneme and the time stamp of the second phoneme, frame index information involved in a process of switching from the first phoneme to the second phoneme is determined.

[0158] Among them, the frame index information may refer to the frame in which the pronunciation of the key phoneme appears. The frame index information involved in the process of switching from the first phoneme to the second phoneme may include all frames in the entire pronunciation switching process from the first phoneme to the second phoneme. For example, the first phoneme appears or sounds in the 10th frame, and the second phoneme appears or sounds in the 20th frame. Then the 10th frame to the 20th frame are the frame index information for switching from the first phoneme to the second phoneme.

[0159] In some embodiments, the frame index information corresponding to the phoneme switching process can be determined based on the total duration of the target speech, the target frame rate, and the timestamps corresponding to the first phoneme and the second phoneme. For example, the total duration of the target speech is 20 seconds, and the target frame rate is 30FPS, then the total number of frames of the corresponding interactive video is 600 frames. The timestamp of the first phoneme is 1s, and the timestamp of the second phoneme is 1.5s. The first phoneme appears or sounds in the 30th frame. The second phoneme appears or sounds in the 45th frame. The frame index information involved in the switching of the two phonemes is the 30th frame to the 45th frame.

[0160] (ii) Obtaining a standard facial image sequence corresponding to the pronunciation switching process from the first phoneme to the second phoneme from the facial image set.

[0161] As mentioned above, for each key phoneme, the facial image set includes multiple facial images involved when the silent phoneme is switched to the key phoneme for pronunciation, and multiple facial images involved when the key phoneme is switched to other M-1 non-silent phonemes for pronunciation.

[0162] In some embodiments, the facial image set can be integrated in one or more sprite images. Taking integration in a sprite image as an example, the sprite image can include M+1 areas, and the M+1 areas correspond to M key phonemes and silent phonemes one by one. The area corresponding to the silent phoneme includes multiple facial images involved when the silent phoneme switches to the M key phonemes for pronunciation. The area corresponding to each key phoneme includes multiple facial images involved when the key phoneme switches to other M-1 key phonemes for pronunciation. The sprite image can also only include M areas, and the M areas correspond to the M key phonemes one by one. Each area includes multiple facial images involved when the key phoneme switches to other M-1 key phonemes for pronunciation, and also includes multiple facial images involved when the silent phoneme switches to the phoneme for pronunciation.

[0163] Therefore, in some embodiments, a standard facial image sequence corresponding to the pronunciation switching process from the first phoneme to the second phoneme can be obtained from the facial image collection in the following manner: a target area corresponding to the first phoneme is located from the Sprite image, and then multiple facial images involved in the switching from the first phoneme to the second phoneme are determined from the target area, and a standard facial image sequence is generated based on the multiple facial images.

[0164] (iii) generating an updated facial image sequence corresponding to a pronunciation switching process from the first phoneme to the second phoneme based on the frame index information and the standard facial image sequence.

[0165] The frame index information can synchronize the actual speech rate of the target voice or key phoneme with the frames in the video. The standard facial image sequence is the facial image of the key phoneme in the material library. The number of frames represented by the frame index information may be the same as or different from the number of lip shape images in the standard lip shape image sequence.

[0166] In some embodiments, if the number of frames represented by the frame index information is equal to the number of facial images in the standard facial image sequence, the standard facial image sequence is used as the updated facial image sequence. For example, the frame index information from the first phoneme to the second phoneme is the 5th frame to the 9th frame, and the number of facial images in the standard lip shape image sequence corresponding to the first phoneme to the second phoneme is also 4. Then these 4 facial images can be directly used as the updated facial image sequence.

[0167] In some embodiments, if the number of frames represented by the frame index information is greater than the number of facial images in the standard facial image sequence, the standard facial image sequence is interpolated based on the frame index information to obtain the updated facial image sequence. The number of frames being greater than the number of facial images indicates that the standard facial image sequence cannot provide enough frames for the video. In some embodiments, the processor 220 can generate new facial images by interpolation to make up the number of facial images to the number of frames. In some embodiments, the processor 220 can also extract more key frames from the synthesized video to make up the number of facial images to the number of frames.

[0168] In some embodiments, if the number of frames represented by the frame index information is less than the number of facial images in the standard facial image sequence, the standard facial image sequence is sampled based on the frame index information to obtain the updated facial image sequence. The number of frames being less than the number of facial images indicates that the lip shape images provided by the standard facial image sequence exceed those required by the video. The processor 220 can select some facial images by sampling to reduce the number of facial images to the number of frames.

[0169] The updated facial image sequence is obtained through the above steps, ensuring that each key phoneme in the interactive video can be synchronized with the correct frame, thereby creating a smooth audio-visual experience for the target user 110.

[0170] (2) According to the switching order between the phonemes in the key phoneme sequence, the updated facial image sequences corresponding to the pronunciation switching processes of different phonemes are combined to obtain the target facial image sequence.

[0171] The updated facial image sequences corresponding to the pronunciation switching processes of different phonemes are combined to correspond to the switching order of the phonemes. For example, the key phoneme sequence includes the sequence of phoneme A switching to phoneme B, phoneme B switching to phoneme E, and phoneme E switching to phoneme A. The target facial image sequence includes the updated facial image sequence of phoneme A switching to phoneme B, the updated facial image sequence of phoneme B switching to phoneme E, and the updated facial image sequence of phoneme E switching to phoneme A.

[0172] Continue to see Figure 3 , step S300 further includes S390: fitting the target facial image sequence to the facial region of the digital human 160 in the basic video to obtain a target video.

[0173] The face of the digital human 160 in the base video may be in a silent state. For example, the mouth of the digital human face region may be in a closed state. The processor 220 uses the face image in the target face image sequence to replace the face region of the digital human 160 in the base video, and a target video with facial changes, especially lip shape changes and body movements, may be obtained.

[0174] In some embodiments, the target video may be obtained by the following steps: extracting a video clip from the base video; and fitting the target facial image sequence to the facial region of the digital human 160 in the video clip frame by frame to obtain the target video.

[0175] The number of image frames in the video clip is equal to the number of facial images in the target facial image sequence. That is, the duration of the video clip is equal to the duration of the target speech. In this way, the facial regions in the video clip can be replaced, and the facial regions in the video clip are all from the same face, thereby ensuring the coherence and harmony of the image of the digital human 160.

[0176] In some embodiments, for any natural number i, fitting the i-th frame face image in the target face image sequence to the face region of the digital human 160 in the i-th frame limb image of the video clip may include the following steps:

[0177] (1) Based on the inter-eye distance in the i-th frame of the facial image and the inter-eye distance in the i-th frame of the limb image, the i-th frame of the facial image is scaled so that the face in the scaled i-th frame of the facial image is matched with the face in the i-th frame of the limb image.

[0178] In some embodiments, the eye spacing in the limb image may be the eye spacing corresponding to the aforementioned basic video. The eye spacing in the face image may be the eye spacing corresponding to the aforementioned sprite image. As mentioned above, the eye spacing is related to the size and proportion of the head (face), and is an important parameter for characterizing the size of the head (face). The face size in the face image may be different from the face size in the limb image, and direct fitting may result in an incoordination between the face and the limbs.

[0179] The processor 220 can calculate the size of each face based on the inter-eye distance in the facial image and the inter-eye distance in the limb image, thereby scaling the facial image so that the face in the facial image fits the face / face in the limb image. For example, the processor 220 can calculate the ratio of the inter-eye distance in the facial image to the inter-eye distance in the limb image. If the ratios are consistent, the processor 220 does not need to scale the facial image. If the ratio is greater than 1, the processor 220 can scale the facial image until the ratio of the inter-eye distance between the two is 1. If the ratio is less than 1, the processor 220 can enlarge the facial image until the ratio of the inter-eye distance between the two is 1.

[0180] By adjusting the size of the facial image, the mouth shape in the facial image is adapted to the face in the limb image, thereby avoiding the phenomenon of the digital human 160 being unnatural or uncoordinated due to the inconsistency of the sizes of the two combined materials.

[0181] (2) Based on the facial key point information corresponding to the i-th frame facial image, a facial region is captured from the scaled i-th frame facial image to obtain a facial sub-image.

[0182] The facial key points corresponding to the facial image may be the facial key points corresponding to the aforementioned sprite image. The facial key point information may be the coordinate information of the key points. The facial key point information may also be the displacement of the key points relative to the fixed point. Based on the facial key point information, the processor 220 may determine the precise position and range of the facial region in the facial image, thereby extracting a facial sub-image containing only the facial region from the entire head in the facial image.

[0183] (3) Based on the facial key point information corresponding to the i-th frame facial image and the facial key point information corresponding to the i-th frame limb image, the facial region of the digital human 160 is determined in the i-th frame limb image.

[0184] In some embodiments, the processor 220 may process the facial key point information corresponding to the limb map based on the facial key point information corresponding to the face map. The processor 220 may add the displacement of the key point in the face map relative to the fixed point to the facial key point corresponding to the limb map, thereby determining the facial area corresponding to the size in the face map in the limb map, thereby ensuring the accuracy of subsequent fitting.

[0185] (4) Fitting the facial sub-image to the facial area of ​​the digital human 160.

[0186] In some embodiments, fitting the facial sub-image to the facial region of the digital human 160 may include the following steps: performing Poisson fusion on the facial sub-image and the facial region of the digital human 160 to fit the facial sub-image to the facial region of the digital human 160. Using Poisson fusion, a seamless transition region can be created between the facial sub-image and the facial region, so that the synthesized image looks natural, thereby making the image of the digital human 160 more harmonious and natural, and providing a better experience for the target user 110.

[0187] In some embodiments, the facial region fitting based on key points can be achieved through the neural network model. For example, by inputting the facial image and the limb image into the fitting model, the facial region in the facial image can be fitted to the facial region in the limb image.

[0188] Figure 8 Some image frames in a target video provided according to some embodiments of the present specification are shown. Figure 8 3 shows body images with facial sub-images attached to the facial regions of three frames, Frame 1', Frame 2', and Frame 3'. The digital human 160 has both body movements and facial changes, especially lip shape changes.

[0189] As mentioned above, since the lip shape changes in the face of the digital human 160 are most obvious when pronouncing different phonemes, and have a relatively high impact on the user's viewing experience, the facial changes involved in the facial image set mainly include the above-mentioned lip shape changes. Therefore, in some embodiments, the step of fitting the target facial image sequence to the facial region of the digital human 160 in the video clip frame by frame may include: fitting the mouth region in the target facial image sequence to the mouth region of the digital human 160 in the video clip frame by frame.

[0190] The processor 220 can capture the mouth area from the scaled face image based on the mouth key point information in the face key points to obtain a mouth sub-image, and then determine the mouth area of ​​the digital human 160 in the corresponding frame of the limb image based on the mouth key point information corresponding to the face image and the mouth key point information corresponding to the limb image, and then fit the mouth sub-image to the mouth area of ​​the digital human 160.

[0191] As mentioned above, in some embodiments, in addition to the changes in lip shape, the real person may also cause changes in other parts of the face when switching pronunciation. Or the facial expression of the digital human 160 in the basic video / video clip is not rich or the facial expression of the digital human 160 in the target facial image sequence is richer, and the overall expression is related to the pronunciation. In this case, the target facial image sequence is fitted frame by frame to the facial area of ​​the digital human 160 in the video clip, which may include: the facial area in the target facial image sequence is fitted frame by frame to the facial area of ​​the digital human 160 in the video clip. The facial area may be the entire facial area.

[0192] In some embodiments, when method S300 is executed by the terminal device 120, method S300 may further include the following steps: synchronously playing the target voice and the target video to present the interactive effect of the digital human. When the target voice and the target video are synchronously played, the changes in the lip shape of the digital human 160 in the target video match the changes in the factors in the target voice. For example, when the target voice is played to the point where the digital human 160 outputs the word "country", the lip shape of the digital human 160 is roughly circular and protrudes forward. Synchronously playing the target voice and the target video can ensure that the lip shape of the digital human 160 in the target voice and the target video is completely corresponding in time, thereby enhancing the realism and immersion of the interaction both auditorily and visually.

[0193] In some embodiments, after generating the target video, method S300 may further include the following steps: merging the target voice and the target video to generate the target audio and video. When method S300 is executed by the terminal device 120, after generating the target audio and video, the terminal device 120 may play the target audio and video to present the interactive effect of the digital human 160. When method S300 is executed by the server, after generating the target audio and video, the server may send the target audio and video to the terminal device 120 to present the interactive effect of the digital human 160 through the terminal device 120.

[0194] In summary, the digital human interactive video generation method S300 provided in this specification can pre-build a material library including a basic video and a facial image set. When it is necessary to generate an interactive video, each original phoneme in the target voice is mapped to multiple key phonemes, and then the facial image set is used to generate a target facial image sequence that is suitable for the phoneme change situation, and then the target facial image sequence is fitted to the facial area of ​​the digital human 160 in the basic video to obtain an interactive video. The above scheme can ensure that the facial changes of the digital human 160 during the interaction process, especially the changes in the lip shape, match the voice content output by it, thereby ensuring the fidelity of the digital human 160 and the user's experience. On this basis, the above scheme maps a large number of original phonemes to a small number of key phonemes, and generates a target video by using the pre-built material library through the fitting technology. It is not necessary to perform real-time rendering of each frame image in the video, thereby improving the generation efficiency of the digital human interactive video and reducing the processing resources consumed by the digital human 160 when performing real-time interaction. On the one hand, it can reduce the computational cost of generating interactive videos, and on the other hand, it enables the digital human interactive product to be applied to various terminal devices, thereby improving the wide application.

[0195] On the other hand, the present specification provides a non-transitory storage medium storing at least one set of executable instructions for interactive video generation. When the executable instructions are executed by the processor, the executable instructions instruct the processor to implement the steps of the interactive video generation method S300 described in the present specification. In some possible implementations, various aspects of the present specification can also be implemented in the form of a program product, which includes a program code. When the program product is run on the generating device 200, the program code is used to enable the generating device 200 to perform the steps of the interactive video generation method S300 described in the present specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on the generating device 200. However, the program product of the present specification is not limited to this. In the present specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with the instruction execution system. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the generation device 200, partially on the generation device 200, as a stand-alone software package, partially on the generation device 200 and partially on a remote computing device, or entirely on a remote computing device.

[0196] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0197] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0198] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "embodiments" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, particular features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0199] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0200] Each patent, patent application, publication of patent applications, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein, except to the extent that it is inconsistent or conflicting with this document or that has a limiting effect on the broadest scope of the claims, may be incorporated herein by reference and used for all purposes now or hereafter associated with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall prevail.

[0201] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A method for generating a digital human interactive video, comprising: Obtaining a target voice, wherein the target voice represents information that the digital human needs to express through voice during interaction; Obtaining a basic video and a facial image set from a pre-built material library, wherein the basic video represents the body movement information of the digital human, and the facial image set represents the facial changes involved when the digital human switches pronunciation between different key phonemes among a plurality of key phonemes; By mapping each original phoneme in the target speech to one of the plurality of key phonemes, key phoneme switching information corresponding to the target speech is obtained; Based on the facial image set and the key phoneme switching information, generating a target facial image sequence that matches the phoneme changes in the target speech; and The target facial image sequence is fitted to the facial region of the digital human in the base video to obtain a target video.

2. The method according to claim 1, wherein: The multiple key phonemes include silent phonemes and M non-silent phonemes in the phoneme table that meet target conditions, wherein the target conditions include: the change amplitude between the pronunciation mouth shape and the closed mouth shape of the phoneme is greater than a preset threshold, the silent phoneme corresponds to the closed mouth shape, and M is an integer greater than 1.

3. The method according to claim 2, wherein: The face image collection includes: A plurality of facial images involved when the digital human switches pronunciation from the silent phonemes to the M non-silent phonemes, and The digital human includes multiple facial images involved when switching pronunciation from each non-silent phoneme to other M-1 non-silent phonemes.

4. The method according to claim 3, wherein: At least some of the face images in the face image set are integrated into a sprite image.

5. The method according to claim 2, wherein: The face image set is obtained in the following manner: Obtaining a real-person video and a basic image, wherein the real-person video shows facial changes of the real person when the real person switches pronunciation of different key phonemes among the multiple key phonemes, and the basic image shows the image of the digital person; Migrating the facial changes shown in the real-person video to the basic image to obtain a synthetic video, wherein the synthetic video shows the facial changes of the digital human when the digital human switches the pronunciation of different key phonemes among the multiple key phonemes; as well as The face image set is obtained by extracting key frames from the synthetic video.

6. The method according to claim 5, wherein: The step of transferring the facial changes shown in the live video to the base image to obtain a synthesized video includes: A face reenactment model is used to transfer the facial changes shown in the real-life video to the basic image to obtain the synthetic video.

7. The method according to claim 1, wherein: For any i-th original phoneme in the target speech, the i-th original phoneme is mapped to a key phoneme among the multiple key phonemes having the greatest similarity to the pronunciation mouth shape of the i-th original phoneme, where i is an integer greater than or equal to 1; The key phoneme switching information includes: a key phoneme sequence obtained by mapping each original phoneme in the target speech, and a timestamp of each key phoneme in the key phoneme sequence.

8. The method according to claim 7, wherein: The step of mapping each original phoneme in the target speech to one of the plurality of key phonemes to obtain key phoneme switching information corresponding to the target speech includes: Determine the timestamp corresponding to each original phoneme in the target speech; Mapping each original phoneme in the target speech to a key phoneme among the plurality of key phonemes that has the greatest similarity to the pronunciation mouth shape of the original phoneme; and The key phoneme sequence is generated based on the key phonemes mapped from the original phonemes according to the order of the original phonemes in the target speech, and the timestamp of each key phoneme in the key phoneme sequence is determined based on the timestamp corresponding to each original phoneme.

9. The method according to claim 8, wherein: The determining of the timestamp corresponding to each original phoneme in the target speech includes: Performing speech recognition on the target speech to obtain a plurality of text units and timestamps corresponding to the plurality of text units; and The multiple text units are mapped to multiple original phonemes, and the timestamp corresponding to each original phoneme in the multiple original phonemes is obtained based on the timestamps corresponding to the multiple text units.

10. The method according to claim 9, wherein: Mapping the plurality of text units to a plurality of original phonemes comprises: Obtaining a target language used by the target speech; and Based on a mapping relationship corresponding to the target language, the multiple text units are mapped to the multiple original phonemes, wherein the mapping relationship represents a mapping relationship between the text units and the original phonemes in the target language.

11. The method according to claim 7, wherein: The step of generating a target facial image sequence matching the phoneme changes in the target speech based on the facial image set and the key phoneme switching information comprises: For the first phoneme and the second phoneme in the key phoneme sequence, the first phoneme and the second phoneme are any two adjacent key phonemes in the key phoneme sequence: determining frame index information involved in a process of switching from the first phoneme to the second phoneme based on the timestamp of the first phoneme and the timestamp of the second phoneme, obtaining a standard facial image sequence corresponding to the pronunciation switching process from the first phoneme to the second phoneme from the facial image set, and generating an updated facial image sequence corresponding to a pronunciation switching process from the first phoneme to the second phoneme based on the frame index information and the standard facial image sequence; and According to the switching order between the phonemes in the key phoneme sequence, the updated facial image sequences corresponding to the pronunciation switching processes of different phonemes are combined to obtain the target facial image sequence.

12. The method according to claim 11, wherein: The step of generating an updated facial image sequence corresponding to the pronunciation switching process from the first phoneme to the second phoneme based on the frame index information and the standard facial image sequence includes any one of the following: If the number of frames represented by the frame index information is equal to the number of facial images in the standard facial image sequence, then using the standard facial image sequence as the updated facial image sequence; If the number of frames represented by the frame index information is greater than the number of facial images in the standard facial image sequence, interpolating the standard facial image sequence based on the frame index information to obtain the updated facial image sequence; or If the number of frames represented by the frame index information is less than the number of facial images in the standard facial image sequence, the standard facial image sequence is sampled based on the frame index information to obtain the updated facial image sequence.

13. The method according to claim 1, wherein: The number of facial images in the target facial image sequence is equal to the total duration of the target speech multiplied by the frame rate of the basic video, and the target video is obtained by fitting the target facial image sequence to the facial region of the digital human in the basic video, including: Cutting out a video segment from the base video, wherein the number of image frames in the video segment is equal to the number of facial images in the target facial image sequence; and The target facial image sequence is fitted frame by frame to the facial region of the digital human in the video clip to obtain the target video.

14. The method according to claim 13, wherein: The step of fitting the target facial image sequence to the facial region of the digital human in the video clip frame by frame includes: Fitting the mouth region in the target facial image sequence to the mouth region of the digital human in the video clip frame by frame; or The facial region in the target facial image sequence is fitted frame by frame to the facial region of the digital human in the video clip.

15. The method according to claim 13, wherein: For any natural number i, the i-th frame face image in the target face image sequence is fitted to the face region of the digital human in the i-th frame limb image of the video clip, comprising: Based on the inter-eye distance in the i-th frame of the facial image and the inter-eye distance in the i-th frame of the limb image, scaling the i-th frame of the facial image so that the face in the scaled i-th frame of the facial image is adapted to the face in the i-th frame of the limb image; Based on the facial key point information corresponding to the i-th frame facial image, a facial region is intercepted from the scaled i-th frame facial image to obtain a facial sub-image; Based on the facial key point information corresponding to the i-th frame of the facial image and the facial key point information corresponding to the i-th frame of the limb image, determining the facial region of the digital human in the i-th frame of the limb image; and The facial sub-image is fitted to the facial region of the digital human.

16. The method according to claim 15, wherein: The step of fitting the facial sub-image to the facial region of the digital human comprises: The facial sub-image is Poisson-fused with the facial region of the digital human, so as to fit the facial sub-image to the facial region of the digital human.

17. The method according to claim 1, wherein: The method further comprises: The target voice and the target video are played synchronously to present the interactive effect of the digital human.

18. The method according to claim 1, wherein: The method further comprises: Merging the target voice and the target video to generate target audio and video; and The target audio and video are played to present the interactive effect of the digital human, or the target audio and video are sent to a terminal device to present the interactive effect of the digital human through the terminal device.

19. A device for generating a digital human interactive video, comprising: At least one storage medium storing at least one instruction set for generating a digital human interaction video; as well as at least one processor, in communication with the at least one storage medium, Wherein, when the generating device is running, the at least one processor reads the at least one instruction and executes the method according to any one of claims 1-18 according to the instructions of the at least one instruction set.

Images