Generation method and generation equipment of digital human interaction video

Abstract

The embodiment of the invention provides a digital human interaction video generation method and interaction equipment. According to the method, after a generation device obtains a target voice, a basic video and a mouth shape image set can be obtained from a pre-constructed material library, a target mouth shape image sequence matched with the phoneme change condition in the target voice is generated based on the target voice and the mouth shape image set, and then the target mouth shape image sequence is obtained. And fitting the target mouth shape image sequence to the mouth area of the digital human in the basic video, thereby obtaining a target video.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for generating a digital human interactive video and a generated video. Background Art

[0002] With the rapid development of technologies such as artificial intelligence, digital human technology has become one of the most cutting-edge scientific and technological products in the digital economy era. Digital humans can simulate real-life expressions, movements, etc. to interact with users in multiple fields such as entertainment, education, and customer service. In order to achieve the interactive effect between digital humans and users, the terminal device needs to render each frame of the digital human interactive video in real time so that the rendered digital human's body movements and lip shapes are consistent with the voice content currently expressed. The rendering process of the above-mentioned interactive video needs to occupy or consume more processing resources, especially graphics processing unit (GPU) resources, which makes the computational cost of real-time interaction of digital humans high. In addition, the rendering process of the above-mentioned interactive video needs to rely on GPU resources, which also makes digital human interactive products unable to be widely used in various terminal devices.

[0003] Therefore, how to reduce the processing resources occupied or consumed by digital humans during real-time interaction is a technical problem that needs to be solved urgently.

[0004] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the invention

[0005] The present specification provides a method and device for generating a digital human interactive video, which can not only ensure that the lip shape changes during the digital human interaction process match its output content, but also reduce the processing resources consumed when the digital human performs real-time interaction.

[0006] In the first aspect, the present specification provides a method for generating a digital human interactive video. The method comprises: obtaining a target voice, wherein the target voice represents the information that the digital human needs to express through voice during interaction; obtaining a basic video and a lip-shaped graph set from a pre-built material library, wherein the basic video represents the limb movement information of the digital human, and the lip-shaped graph set represents the lip shape change information involved in the pronunciation of different phonemes in a preset phoneme set; based on the lip-shaped graph set, generating a target lip-shaped graph sequence that matches the phoneme change in the target voice; and fitting the target lip-shaped graph sequence to the mouth area of ​​the digital human in the basic video to obtain a target video.

[0007] In some embodiments, the preset phoneme set includes M phonemes in the phoneme table that meet the target condition, the target condition includes that the change range between the pronunciation mouth shape and the closed mouth shape of the phoneme is greater than a preset threshold, and M is an integer greater than 1. The mouth shape graph set includes a plurality of mouth shape graphs corresponding to each of the M phonemes, and the plurality of mouth shape graphs corresponding to each phoneme represent the mouth shape change information of the phoneme during the pronunciation process.

[0008] In some embodiments, generating a target lip shape graph sequence that matches the phoneme changes in the target speech based on the lip shape graph set includes the following steps: determining at least one key phoneme in the target speech and a timestamp corresponding to the at least one key phoneme, wherein the key phoneme belongs to the preset phoneme set; and generating the target lip shape graph sequence based on the at least one key phoneme, the timestamp corresponding to the at least one key phoneme, and the lip shape graph set.

[0009] In some embodiments, the determining of at least one key phoneme in the target speech and the timestamp corresponding to the at least one key phoneme includes the following steps: performing speech recognition on the target speech to obtain multiple text units and the timestamps corresponding to the multiple text units; mapping the multiple text units to multiple phonemes, and obtaining the timestamps corresponding to the multiple phonemes based on the timestamps corresponding to the multiple text units; locating the at least one key phoneme from the multiple phonemes, and obtaining the timestamp corresponding to the at least one key phoneme.

[0010] In some embodiments, mapping the multiple text units to multiple phonemes includes the following steps: obtaining a target language adopted by the target speech; and mapping the multiple text units to the multiple phonemes based on a mapping relationship corresponding to the target language, wherein the mapping relationship represents a mapping relationship between text units and phonemes in the target language.

[0011] In some embodiments, the target lip shape graph sequence is generated based on the at least one key phoneme, the timestamp corresponding to the at least one key phoneme, and the lip shape graph set, and for each key phoneme, the following steps are included: based on the total duration of the target speech, the target frame rate, and the timestamp corresponding to the key phoneme, the frame index information corresponding to the key phoneme is determined; the standard lip shape graph sequence corresponding to the key phoneme is obtained from the lip shape graph set; based on the frame index information and the standard lip shape graph sequence, an updated lip shape graph sequence corresponding to the key phoneme is generated; and the updated lip shape graph sequences corresponding to each of the at least one key phoneme are spliced ​​to obtain the target lip shape graph sequence.

[0012] In some embodiments, the lip shape diagram set is integrated into a Sprite diagram, the Sprite diagram includes M areas, the M areas correspond one-to-one to the M phonemes, wherein the multiple lip shape diagrams corresponding to each phoneme are located in the area corresponding to the phoneme; obtaining the standard lip shape diagram sequence corresponding to the key phoneme from the lip shape diagram set includes: locating the target area corresponding to the key phoneme from the Sprite diagram, and generating the standard lip shape diagram sequence based on the multiple lip shape diagrams in the target area.

[0013] In some embodiments, generating the updated lip shape diagram sequence corresponding to the key phonemes based on the frame index information and the standard lip shape diagram sequence includes any one of the following: if the number of frames represented by the frame index information is equal to the number of lip shape diagrams in the standard lip shape diagram sequence, then using the standard lip shape diagram sequence as the updated lip shape diagram sequence; if the number of frames represented by the frame index information is greater than the number of lip shape diagrams in the standard lip shape diagram sequence, then interpolating the standard lip shape diagram sequence based on the frame index information to obtain the updated lip shape diagram sequence; or if the number of frames represented by the frame index information is less than the number of lip shape diagrams in the standard lip shape diagram sequence, then sampling the standard lip shape diagram sequence based on the frame index information to obtain the updated lip shape diagram sequence.

[0014] In some embodiments, the number of lip shape graphs in the target lip shape graph sequence is equal to the total duration of the target speech multiplied by the frame rate of the base video, and the step of fitting the target lip shape graph sequence to the mouth area of ​​the digital human in the base video to obtain the target video comprises the following steps: extracting a video clip from the base video, the number of image frames in the video clip being equal to the number of lip shape graphs in the target lip shape graph sequence; and fitting the target lip shape graph sequence to the mouth area of ​​the digital human in the video clip frame by frame to obtain the target video.

[0015] In some embodiments, for any natural number i, fitting the i-th frame of the lip shape map in the target lip shape map sequence to the mouth area of ​​the digital human in the i-th frame of the limb map of the video clip includes the following steps: scaling the i-th frame of the lip shape map based on the inter-eye distance in the i-th frame of the lip shape map and the inter-eye distance in the i-th frame of the limb map so that the mouth shape in the scaled i-th frame of the lip shape map is adapted to the face in the i-th frame of the limb map; based on the mouth key point information corresponding to the i-th frame of the lip shape map, capturing the mouth area from the scaled i-th frame of the lip shape map to obtain a mouth sub-map; based on the mouth key point information corresponding to the i-th frame of the lip shape map and the mouth key point information corresponding to the i-th frame of the limb map, determining the mouth area of ​​the digital human in the i-th frame of the limb map; and fitting the mouth sub-map to the mouth area of ​​the digital human.

[0016] In some embodiments, the step of fitting the mouth sub-image to the mouth region of the digital human comprises the following steps: performing Poisson fusion on the mouth sub-image and the mouth region of the digital human to fit the mouth sub-image to the mouth region of the digital human.

[0017] In some embodiments, for any natural number i, multiple lip shape graphs corresponding to the ith phoneme in the lip shape graph set are obtained in the following manner: obtaining three-dimensional image parameters corresponding to the digital human; obtaining initial facial parameters and pronunciation facial parameters corresponding to the ith phoneme, wherein the initial facial parameters are facial parameters of the character in a closed mouth state, and the pronunciation facial parameters are facial parameters of the character when pronouncing the ith phoneme; interpolating based on the initial facial parameters and the pronunciation facial parameters to obtain a facial parameter sequence of the ith phoneme during the pronunciation process; and rendering the three-dimensional image parameters and the facial parameter sequence through a rendering tool to obtain multiple lip shape graphs corresponding to the ith phoneme.

[0018] In some embodiments, the basic video is obtained by: obtaining the three-dimensional image parameters corresponding to the digital human; obtaining the limb movement parameters corresponding to the character when performing preset actions; and rendering the three-dimensional image parameters and the limb movement parameters through a rendering tool to obtain the basic video.

[0019] In some embodiments, the method further includes the following steps: synchronously playing the target voice and the target video to present the interactive effect of the digital human.

[0020] In some embodiments, the method further includes the following steps: merging the target voice and the target video to generate target audio and video; and playing the target audio and video to present the interactive effect of the digital human, or sending the target audio and video to a terminal device to present the interactive effect of the digital human through the terminal device.

[0021] In a second aspect, the present specification also provides a device for generating a digital human interactive video. The generating device includes at least one storage medium and at least one processor. The at least one storage medium stores at least one instruction set for generating a digital human interactive video. The at least one processor is in communication connection with the at least one storage medium. When the generating device is running, the at least one processor reads the at least one instruction and executes any one of the methods described in the first aspect according to the instructions of the at least one instruction set.

[0022] As can be seen from the above technical solutions, the digital human interactive video generation method provided in this specification can pre-construct a material library including a basic video and a lip shape map set. When it is necessary to generate an interactive video, the generation device can obtain the basic video and the lip shape map set from the material library, and use the lip shape map set to generate a target lip shape map sequence that adapts to the phoneme changes of the target voice, and then fit the target lip shape map sequence to the basic video to obtain the interactive video. The above scheme avoids the real-time image rendering work during the interaction process, so it can reduce the resource consumption in the interactive video generation process. In this way, the digital human interactive product does not need to use or use less GPU resources, or only uses relatively low CPU resources to achieve the generation of digital human interactive videos. On the one hand, it can reduce the computational cost of generating interactive videos, and on the other hand, the digital human interactive product can be applied to various terminal devices to improve the wide application. In addition, the interactive video generation method can ensure that the lip shape changes of the digital human match the voice content output by fitting the target lip shape map sequence to the mouth area of ​​the digital human in the basic video, so as to provide users with a good experience and immersion.

[0023] The methods and other functions of the digital human interactive video generation device provided in this specification will be partially listed in the following description. The creative aspects of the digital human interactive video generation method and the generation device provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1A A schematic diagram of an application scenario of digital human interactive video generation provided according to an embodiment of this specification is shown;

[0026] Figure 1B A schematic diagram of a scene in which a target user is interacting with a digital human according to some embodiments of this specification is shown;

[0027] Figure 2 A schematic diagram of the structure of a generating device provided according to an embodiment of this specification is shown;

[0028] Figure 3 A flowchart of an interactive video generation method provided according to an embodiment of this specification is shown;

[0029] Figure 4 The present invention shows a process of constructing a material library and a content composition according to some embodiments of the present specification;

[0030] Figure 5 shows a portion of images in a basic video provided according to some embodiments of the present specification;

[0031] Figure 6 Some lip shape diagrams in a lip shape diagram set provided according to some embodiments of the present specification are shown;

[0032] Figure 7 A flowchart of generating a target lip shape sequence according to some embodiments of the present disclosure is shown;

[0033] Figure 8 A flowchart of obtaining a target video according to some embodiments of this specification is shown; and

[0034] Fig. 9 Some image frames in a target video provided according to some embodiments of the present specification are shown. DETAILED DESCRIPTION

[0035] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0036] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0037] In view of the following description, these and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of the combination and manufacture of the parts can be significantly improved. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0038] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations in the flowchart may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0039] It should be noted that the user data obtained in this specification is authorized by the user and does not involve user privacy.

[0040] For the convenience of description, the terms involved in this specification are first explained as follows.

[0041] A Digital Human, also known as a virtual avatar or virtual digital human, refers to a product that exists in the non-physical world, is created and used by computer means, and has multiple human characteristics (appearance characteristics, human performance capabilities, interaction capabilities, etc.).

[0042] Rendering is the process of converting 3D models, scenes, or animations in a computer into 2D images or videos. This process involves calculating the interaction of light with the surface of objects, including effects such as shadows, reflections, and refractions, to generate realistic images. In the generation of Digital Human videos, rendering technology is particularly important because it determines the realism and visual quality of the Digital Human image.

[0043] The International Phonetic Alphabet (IPA) is a set of symbols used to represent the pronunciations of various languages in the world. It is developed by the International Phonetic Association and aims to provide linguists with a standardized method of pronunciation representation that is not restricted by a specific language. The International Phonetic Alphabet includes vowels, consonants, other symbols (such as tones, lengths, stresses, etc.), and some additional symbols.

[0044] A phoneme is the smallest unit divided from the perspective of speech and is the smallest sound segment that constitutes the pronunciation of a word. A phoneme is the smallest speech unit in a language that can distinguish meanings. For example, in English, the word "bat" (bat) includes three phonemes, / b / , and / t / ; the word "pat" (pat) includes three phonemes, / p / , and / t / . The words "bat" and "pat" can be distinguished by phonemes. In Chinese, "猫" (cat) includes three phonemes, / m / , / a / , and / o / ; "麻" (hemp) includes three phonemes, / m / , / a / , and The words "猫" and "麻" can be distinguished by phonemes. The IPA is a symbol system used to record phonemes.

[0045] Poisson Image Editing is an image processing technique that achieves seamless fusion between source and target images by solving the Poisson equation. The core idea of ​​this method is to fuse the source image into the target image while retaining the gradient information of the source image, so that the fused image appears natural and continuous visually.

[0046] Before describing the specific embodiments of this specification, the application scenarios to which the technical solutions provided in this specification are applicable are first introduced.

[0047] Digital humans can replace manual services or separate voice services, respond to user interaction needs in real time, and interact with users in real time to provide visual services. Among them, interaction needs can include services that users want to obtain. For example, interaction needs can be questions or requests raised by users. For example, in the e-commerce customer service industry, user interaction needs can include answering questions about product details, size selection, payment methods or delivery issues. For another example, in the bank customer service industry, user interaction needs can include providing operations such as account registration, account query, account loss or transfer. For another example, in the cultural and tourism industry, user interaction needs can include providing travel routes, travel time, catering and accommodation suggestions. Digital human interaction videos can be videos of digital humans providing interactive services to users. For example, after a user raises questions about product details, the user's questions can be answered in real time by playing digital human interaction videos, so that the user can experience face-to-face voice communication. For another example, digital human interaction videos can provide users with operations such as how to register an account, and assist users in completing operations during the interaction with users. For another example, digital human interaction videos can provide users with real-time suggestions on travel routes. Due to different interaction requirements raised by users, the feedback or output content of the digital human when interacting with users is also different. Different output content will cause the digital human to present different lip shapes. Lip shapes synchronized with the content will enhance the realism of the digital human, so that users can get the experience of face-to-face voice communication with real service personnel.

[0048] In the prior art, when generating a digital human interactive video, the content and corresponding images that the digital human needs to output or feedback can be determined based on the interactive needs raised by the user in real time, and then the graphics GPU is used to render the images with different lip shapes frame by frame in real time to obtain the corresponding digital human video. Although the above real-time rendering method can also obtain a digital human interactive video, real-time rendering will occupy a large amount of GPU computing resources and has extremely high performance requirements for the GPU, making the cost of video generation and the cost of real-time interaction of digital humans very high.

[0049] Therefore, this specification provides a method for generating interactive videos of digital humans, which can pre-build a library of materials including basic videos and lip graph sets. When it is necessary to generate interactive videos, the lip graph set is used to generate a target lip graph sequence that adapts to the changes in the phonemes in the target speech, and then the target lip graph sequence is fitted to the mouth area of ​​the digital human in the basic video to obtain an interactive video. The above scheme can ensure that the lip shape changes during the digital human interaction match the voice content it outputs, thereby ensuring the fidelity of the digital human and the user's experience. On this basis, the above scheme avoids real-time rendering of each frame of the interactive video, so it can reduce the processing resources consumed by the digital human when performing real-time interaction, thereby reducing the computational cost of the digital human's real-time interaction.

[0050] The digital human interactive video generation method and generation device provided in this specification can be applied to various service scenarios using digital humans. For example, in the above-mentioned customer service industry, the method can generate customer service digital human interactive videos that provide services to consumers. For another example, in the education industry, the method can generate teacher digital human interactive videos that provide online courses and training materials to students. For another example, in the media industry, the method can generate anchor digital human interactive videos that provide viewers with a rich viewing experience. For another example, in the cultural and travel industry, the method can generate tour guide digital human interactive videos that provide tourists with a vivid travel experience.

[0051] The digital human interactive video generation method of this specification can also be applied to any digital human video generation scenario, and will not be described in detail here.

[0052] Figure 1A The following is a schematic diagram of an application scenario of digital human interactive video generation according to an embodiment of the present specification. The application scenario may be the above-mentioned customer service industry, education industry, media industry, cultural travel industry, etc. Figure 1A As shown, scenario 001 may include a target user 100 , a terminal device 200 , a server 300 , a network 400 , and a database 500 .

[0053] The target user 100 may be a user who has an interactive demand, thereby triggering the generation of a digital human interactive video (hereinafter referred to as the interactive video). The terminal device 200 may be a device that responds to the interactive demand of the target user 100 and interacts with the target user 100 by playing an interactive video. The interactive video may include a digital human 201. The digital human 201 may output feedback content to the target user 100. In some embodiments, the digital human 201 may also perform body movements corresponding to the feedback content. The appearance and clothing of the digital human 201 may be designed or selected according to the corporate image or user needs. Those skilled in the art will understand that the digital human 201 is a virtual character displayed on the display screen of the terminal device 200. The attached Figure 1A For the sake of convenience in the example, the digital human 201 is not displayed on the display screen of the terminal device 200.

[0054] In some embodiments, the interactive video generation method can be executed on the terminal device 200. At this time, the terminal device 200 can store the data or instructions of the interactive video generation method described in this specification, and can execute or be used to execute the data or instructions. The terminal device 200 may include a hardware device with data information processing functions and the necessary programs required to drive the hardware device to work. In some embodiments, the terminal device 200 may include a mobile device, a tablet computer, a laptop computer, a built-in device of a motor vehicle, or similar content, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or a similar device, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, etc., or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, etc., or any combination thereof. In some embodiments, the virtual reality device or the augmented reality device may include a virtual reality helmet, virtual reality glasses, a virtual reality patch, an augmented reality helmet, augmented reality glasses, an augmented reality patch, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, a head-mounted display, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an on-board computer, an on-board TV, etc.

[0055] In some embodiments, the terminal device 200 may also provide the target user 100 with interactive capabilities so that the target user 100 can input its interactive needs to the terminal device 200. Then the terminal device 200 generates an interactive video based on the interactive needs. For example, the terminal device 200 may be installed with one or more application programs (APPs). APP can provide the target user 100 with the ability and interface to interact with the outside world through the network 400. The target user 100 can input interactive needs through the interface of the APP. APP includes but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, mailbox terminal devices, social platform software, etc. A target APP may be installed on the terminal device 200. The target APP can provide interactive capabilities to the target user 100. Thus, the target user 100 can trigger the generation of an interactive video through the target APP. The target APP can execute the interactive video generation method described in this specification in response to the trigger. The interactive video generation method will be described in detail in the following content. For another example, the terminal device 200 may be equipped with input devices including but not limited to a keyboard, a mouse, a camera, a microphone, or a touch screen. For example, the target user 100 may input his / her interaction needs to the terminal device 200 through a keyboard. For another example, the target user 100 may voice the interaction needs, the microphone may obtain the voice information of the target user 100, and analyze the user's interaction needs based on the voice information. For another example, the target user 100 may make a specific action, the camera may obtain the body information of the target user 100, and analyze the user's interaction needs based on the body information.

[0056] In some embodiments, the terminal device 200 can be used as both a device for generating interactive videos and a device for outputting interactive videos. For example, the terminal device 200 may include, but is not limited to, a display device. The terminal device 200 can output interactive videos through the display device. For another example, the terminal device 200 can include both a display device and a speaker to output the generated interactive video. For example, the terminal device 200 can output the interactive video through the display device and output the audio corresponding to the interactive video through the speaker.

[0057] Figure 1B A schematic diagram of a digital human interaction scenario provided according to some embodiments of the present specification is shown. Figure 1B The scenario shown may be a bank scenario, and the terminal device 200 may be a bank's self-service machine. Figure 1BAs shown, the target user 100 expresses the need to open a new bank account to the digital human 201 in the self-service machine through voice. The terminal device 200 can generate and output an interactive video to the target user 100 by executing the interactive video generation method of this specification. In the interactive video, the digital human gives feedback or answers on how to open a new bank account, thereby interacting with the target user 100 in real time. For example, the interaction between the target user 100 and the digital human 201 includes the following two rounds of interaction:

[0058] The target user 100 first asks the digital human 201 a question (U1) in voice: <How to open a new account? >.

[0059] In the generated interactive video, Digital Human 201 replied (D1): <Hello, to open a new bank account, you need to bring your ID card to the counter to find a staff member. After the identity authentication is successful, you can open a new bank account. >.

[0060] The target user 100 asks the digital human 201 another question (U2): "I don't have my ID card with me today. Can I apply for it?"

[0061] In the generated interactive video, Digital Human 201 answered (D2): <Electronic ID card is also OK, can you provide the electronic ID card to the staff? >

[0062] Continue to refer to Figure 1A , the server 300 may be a server that provides various services to the terminal device 200. For example, the server 300 may provide support to the target APP deployed on the terminal device 200. The interactive video generation method described in this specification may be executed on the server 300. At this time, the server 300 may store data or instructions for executing the interactive video generation method described in this specification, and may execute or be used to execute the data or instructions. The server 300 may include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work. In this case, after the terminal device 200 receives the interactive demand of the target user 100, the interactive demand may be sent to the server 300. The server 300 generates an interactive video by executing the interactive video generation method provided in this specification, and sends the interactive video to the terminal device 200. Thus, the terminal device 200 plays the interactive video to achieve real-time interaction with the target user 100.

[0063] The network 400 can facilitate the exchange of information and / or data. The network 400 can be any type of wired or wireless network, or a combination thereof. For example, the network 400 can include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan / urban network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, or a similar network. The network 400 may include one or more network access points. For example, the network 400 may include a wired or wireless network access point, such as a base station and / or an Internet exchange point. Through the access point, one or more components of the terminal device 200, the server 300, and the database 500 can be connected to the network 400 to exchange data and / or information, wherein the target user 100 can connect to the network 400 via the terminal device 200. Figure 1A As shown, the server 300, the terminal device 200, and the database 500 may be connected to the network 400, and may transmit information and / or data to each other through the network 400. For example, the terminal device 200 may obtain services from the server 300 through the network 400.

[0064] The database 500 can be used to store the data and / or instructions required by the terminal device 200 or the server 300 in the process of executing the interactive video generation method described in this specification. The server 300 and the terminal device 200 may have access to the database 500. The database 500 may be directly connected to the server 300 and the terminal device 200. Among them, the database 500 may be a part of the server 200. The database 500 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM) or the like, or any combination thereof. Exemplary mass storage may include non-transitory storage mediums such as disks, optical disks, and solid-state drives. Example removable storage may include flash drives, floppy disks, optical disks, memory cards, zip disks, tapes, etc. Typical volatile read-write memory may include random access memory (RAM). Example RAM may include dynamic RAM (DRAM), dual date rate synchronous dynamic RAM (DDR-SDRAM), static RAM (SRAM), thyristor RAM (T-RAM) and zero capacitance RAM (Z-RAM), etc. Exemplary ROMs may include mask ROM (MROM), programmable ROM (PROM), virtual programmable ROM (PEROM), electronic programmable ROM (EEPROM), compact disk (CD-ROM) and digital versatile disk ROM, etc. Database 500 can be implemented on a cloud platform. Just by way of example, a cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, a cloud-to-cloud cloud or the like, or any combination thereof. The storage medium may include a remote independent storage medium, such as the above-mentioned disk, hard disk and solid-state hard disk, etc. The server 300 / terminal device 200 can access or update data stored in a remote independent storage medium through a mobile device with access rights. The storage medium may include a local storage medium. For example, a data storage hard disk of a local server or the IOT device itself, etc.

[0065] It should be understood that Figure 1A and Figure 1B The number of terminal devices 200, servers 300 or networks 400 in the embodiment is only for illustration. Any number of terminal devices 200, servers 300 and networks 400 may be provided as required.

[0066] It should be noted that the method for generating digital human interactive video provided in this specification can be completely executed on the terminal device 200, can be completely executed on the server 300, or can be partially executed on the terminal device 200 and partially executed on the server 300.

[0067] Figure 2The structure diagram of a digital human interactive video generation device provided according to an embodiment of the present specification is shown. The digital human interactive video generation device 600 (hereinafter referred to as the generation device 600) can be used as Figure 1A and / or Figure 1B The terminal device 200 in the embodiment can also be used as Figure 1A Server 300 in.

[0068] like Figure 2 As shown, the generating device 600 includes at least one storage medium 630 and at least one processor 620. In some embodiments, the generating device 600 may further include a communication port 650 and an internal communication bus 610. Meanwhile, the generating device 600 may further include an I / O component 660.

[0069] Internal communication bus 610 may connect various system components, including storage media 630 and processor 620. I / O component 660 supports input / output between generating device 600 and other components.

[0070] The storage medium 630 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 632, a read-only storage medium (ROM) 634, or a random access storage medium (RAM) 636. The storage medium 630 also includes at least one instruction set stored in the data storage device. The instruction set is a computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the method provided in this specification.

[0071] The communication port 650 is used for data communication between the generating device 600 and the outside world. For example, the generating device 600 can be connected to the network through the communication port 650. At least one processor 620 is connected to at least one storage medium 630 through the internal communication bus 610. At least one processor 620 is used to execute the at least one instruction set. When the generating device 600 is running, the at least one processor 620 reads the at least one instruction set and executes the interactive video generating method provided in this specification according to the instructions of the at least one instruction set.

[0072] The processor 620 can execute all the steps included in the interactive video generation method. The processor 620 can be in the form of one or more processors. The processor 620 can issue execution instructions. The processor 620 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0073] For illustration purposes only, only one processor 620 is shown in the generating device 600 in the drawings of this specification. However, it should be noted that the generating device 600 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor as described in this specification, or may be performed jointly by multiple processors. For example, if the processor 620 of the generating device 600 in this specification performs step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 620 (e.g., the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0074] Figure 3 A flow chart of an interactive video generation method provided according to an embodiment of the present specification is shown. As shown above, the server 300 and / or the terminal device 200 can execute the interactive video generation method S300 (hereinafter referred to as method S300). Method S300 can be executed entirely on the terminal device 200, or entirely on the server 300, or partially on the terminal device 200 and partially on the server 300. For example, the processor 620 in the terminal device 200 can be communicatively connected to the storage medium 630. The processor 620 can read an instruction set stored in its local storage medium and / or database, and then execute method S300 according to the provisions of the instruction set. The following is an introduction to the execution of method S300 by the processor 620 as an example. Figure 3 As shown, method S300 may include:

[0075] S310: Obtain target speech.

[0076] The target speech may represent the information that the digital human 201 needs to express through speech during interaction. For example, the target speech may be the feedback information or reply information made by the digital human 201 based on the interaction requirements proposed by the target user 100. Figure 1B D1 in the scenario: <Hello, to open a new bank account, you need to bring your ID card to the counter to find a staff member. After the identity authentication is successful, you can open a new bank account>. D2: <Electronic ID card is also OK. Can you provide the electronic ID card to the staff member?>. The target voice may include part or all of the information that needs to be expressed through voice during the more than 100 interactions between the digital human 201 and the target user. For example, the target voice includes information from D1, D2, and the subsequent interactions between the digital human 201 and the target user 100.

[0077] In some embodiments, the target voice may be information pre-stored in the database 500 or the storage medium 630. After the target user 100 triggers the keyword, the processor 620 may obtain the information corresponding to the keyword from the pre-stored information in real time. For example, the target user 100 triggers the keyword <new account> in U1. The digital human 201 may reply with the information D1 corresponding to the keyword. The target user 100 simultaneously triggers the keywords <no> and <ID card> in U2. The digital human 201 may reply with the information D2 corresponding to the keyword. Among them, the processor 620 may directly obtain the target voice. The processor 620 may also first obtain the text information and then convert the text information into the target voice.

[0078] In some embodiments, the target voice may also be an artificial intelligence (AI) model that analyzes the user's interaction needs in real time and outputs intelligent, high-quality feedback information. The AI ​​model may directly output the target voice. The AI ​​model may also output text information first and then convert the text information into the target voice.

[0079] In some embodiments, the target voice may be real-time feedback from a real person, information that needs to be outputted in the form of a digital human 201. For example, in the media industry, the anchor of a digital human image may be controlled by a real person. After the target user 100 raises an interactive demand, the real person may give corresponding feedback. The processor 620 may obtain the voice feedback given by the real person in real time and use it as the target voice.

[0080] S320: Obtain a basic video and lip-sync graph collection from a pre-built library.

[0081] The material library may be a collection of resources used to generate interactive videos. The material library may exist before the processor 620 executes method S300. The material library may be pre-built by the processor 620 or pre-built by the server 300. For ease of description, the construction of the material library by the processor 620 is taken as an example for introduction. The material package may be pre-stored in the storage medium 630 or stored in the database 500. When generating an interactive video, the processor 620 may obtain the pre-built material library in real time. The processor 620 may obtain all the materials in the material library or obtain part of the materials in the material library according to the generation requirements of the interactive video.

[0082] Figure 4 The figure shows the composition process and content composition of a material library provided according to some embodiments of this specification. Figure 4 As shown, the material library L may include basic videos and a collection of lip shape graphs.

[0083] The basic video may represent the body movement information of the digital human 201. The body movement information of the digital human 201 may include information such as the body position, movement state or movement trajectory of the digital human 201. The action presented by the digital human 201 may be changed by changing the body movement information of the digital human 201. The basic video may include multiple frames of images when the digital human 201 presents different actions. For example, the digital human 201 may present one or more actions of making a gesture of the number 1 with fingers, stroking the abdomen with both hands, stroking the abdomen with one hand and raising the other hand, lowering both hands, cheering, clapping with both hands, holding fists with both hands, making a heart shape with both hands, denying and rejecting, folding hands, thinking, giving a thumbs up with both hands, listening closely, making a gesture of an upward trend chart, waving and saying hello, etc.

[0084] In some embodiments, Figure 4 As shown, the basic video can be obtained as follows:

[0085] P410: Obtain the corresponding limb movement parameters when the character performs the preset action.

[0086] Among them, the character can be a motion capture actor. The motion capture actor can wear special clothing with sensors and perform various actions and performances, which are captured and converted into actions of the digital model. The preset action can be the action to be presented by the digital human 201 in the interactive video. When the motion capture actor performs the preset action, the corresponding body movement parameters can be used to generate the action of the digital human 201 or as a reference for generating the action of the digital human 201. The preset action can be one or more of the actions listed above. For example, the preset action can include stroking the abdomen with both hands, and stroking the abdomen with the left hand and raising the right hand. The preset action can also include transition actions between the actions listed above. For example, when the digital human 201 changes from stroking the abdomen with both hands to stroking the abdomen with the left hand and raising the right hand, the preset action can also include actions when the right hand is raised at different angles. The transition action can make the body movement of the digital human 201 more coherent, making the viewing experience of the target user 100 smoother.

[0087] P420: Obtain the three-dimensional image parameters corresponding to the digital human.

[0088] The three-dimensional image parameters corresponding to the digital human 201 may include parameters defining the external image of the digital human 201. For example, the three-dimensional image parameters may include parameters such as the color, glossiness, and transparency of the digital human 201's skin. For another example, the three-dimensional image parameters may include parameters such as the style, material, and texture of the digital human's 201 clothing. The three-dimensional image parameters corresponding to the digital human 201 may also define parameters of the surrounding environment of the digital human 201. For example, the three-dimensional image parameters may include parameters of the plants around the digital human 201. For another example, the three-dimensional image parameters may include parameters of the building behind the digital human 201. These three-dimensional image parameters may make the digital human 201 more natural and realistic in various application scenarios.

[0089] P430: Rendering the three-dimensional image parameters and the limb movement parameters through a rendering tool to obtain the basic video.

[0090] The processor 620 can convert the information contained in the three-dimensional image parameters and the limb movement parameters into visual content through a rendering tool. A series of static images (frames) can be obtained through the rendering tool. The rendering tool can also play the above images at a preset frame rate to obtain the basic video. The rendering tool can be Unreal Engine, Unity, Lumion, etc.

[0091] It is worth noting that the digital human 201 in the basic video includes a human face. However, since no face-related parameters are input during rendering, the facial features parameters of the digital human 201 in the basic video can be fixed parameters, that is, the face of the digital human 201 can always remain consistent without change. For example, when the digital human 201 performs different actions, the mouth shape can always be kept closed. The human face included in the basic video can be the same as the human face in the lip shape map set to ensure the facial harmony of the digital human 201 obtained by subsequent splicing.

[0092] Figure 5 Partial images in a basic video provided according to some embodiments of the present specification are shown. Figure 5 Three frames of images in the basic video are shown, Frame 1, Frame 2 and Frame 3. The actions of the digital human 201 and the clothes worn by the digital human 201 in the images can be obtained by rendering the above parameters using a rendering tool.

[0093] In Frame 1, the digital human 201 holds his abdomen with both hands and keeps his head in the middle. This action may be the action of the digital human 201 when listening to the target user 100. In Frame 3, the digital human 201 holds his abdomen with his left hand and raises his right hand, and his head tilts slightly to the right with his hand. This action may be the action of the digital human 201 when providing feedback to the target user 100. In Frame 2, the digital human 201 also holds his abdomen with his left hand and raises his right hand, but the angle of the right hand is smaller than that in Frame 3, and the head remains in the middle. The action in Frame 2 may be a transition action between Frame 1 and Frame 3. In the three frames, the digital human 201 always keeps his mouth closed.

[0094] It is worth mentioning that Figure 5 The three frames of images may be three consecutive frames of images in the basic video, or may be three discontinuous frames of images.

[0095] In some embodiments, after rendering the limb movement parameters and the three-dimensional image parameters by the rendering tool, in addition to obtaining the basic video, the key points corresponding to the basic video can also be obtained. The key points corresponding to the basic video can be points that characterize the face or head position of the digital human 201. For example, the key points can include but are not limited to the midpoint of the top of the head, the top of the ear, the tip of the chin, and other feature points. The rotation angle, tilt angle, etc. of the digital human head 201 can be calculated through the key points, so as to determine the head position of the digital human 201.

[0096] like Figure 5As shown, the head of the digital human 201 in Frame2 remains in the middle. In Frame3, the head of the digital human 201 tilts to the right. The different positions of the head of the digital human 201 in Frame2 and Frame3 can be determined by key points. By determining the head position of the digital human 201 in the basic video, an accurate splicing point can be determined for the subsequent splicing interface. The specific determination method will be introduced later. The key point corresponding to the basic video can be the coordinate information of the key point. The coordinate information of the key point can be the relative coordinate of the key point relative to the preset point. For example, the preset point is the left vertex of the image frame.

[0097] In some embodiments, the key points corresponding to the basic video may be obtained by performing key point detection on the basic video after the basic video is obtained.

[0098] In some embodiments, the material library L may also include the interocular distance corresponding to the basic video. The interocular distance corresponding to the basic video may be the distance between the centers of the two eyes of the digital human 201. The interocular distance is related to the size and proportion of the head (face) and is an important parameter for characterizing the size of the head (face). The interocular distance corresponding to the basic video may include data on the interocular distance of the digital human 201 in some or all frame images in the basic video.

[0099] In some embodiments, the material library L may further include lip key points corresponding to the basic video. The mouth key points corresponding to the basic video may be points that characterize the shape and position of the mouth of the digital human 201. The lip key points may include but are not limited to the corners of the mouth, the philtrum, the lip beads, the outer sides of the upper and lower lips, the inner sides of the upper and lower lips, etc. The position of the mouth of the digital human 201 or the area where the lip shape is located may be calculated through the mouth key points. The lip key point information corresponding to the basic video may be the coordinate information of the lip key points.

[0100] In some embodiments, the interocular distance corresponding to the basic video and the lip key points corresponding to the basic video may be obtained after data processing based on the basic video and the key points corresponding to the basic video.

[0101] Continue to see Figure 4 , the material library L may also include a mouth shape graph set. The mouth shape graph set may represent the mouth shape change information involved in the pronunciation process of different phonemes in the preset phoneme set. Among them, the preset phoneme set may include M phonemes in the phoneme table that meet the target condition, where M is an integer greater than 1. The target condition may include: the change range between the pronunciation mouth shape and the closed mouth shape of the phoneme is greater than a preset threshold.

[0102] The phoneme table may be the phonemes shown in the International Phonetic Alphabet IPA. Different phonemes may correspond to different pronunciation mouth shapes. When some phonemes are pronounced, the corresponding pronunciation mouth shape changes more significantly than the closed mouth shape. For example, the pronunciation of vowels such as [u] and [u] is different from that of [u]. For example, when pronouncing the tip of the tongue, such as [t] and [d], the mouth shape is that the tip of the tongue touches the back of the lower teeth. When speaking, the mouth shape is protruding forward. When pronouncing some phonemes, the corresponding pronunciation mouth shape does not change much compared to the closed mouth shape. For example, when pronouncing consonants such as [m] and [n], the sound is mainly produced through the nose, and the positions of the tongue and lips are relatively fixed, so the pronunciation mouth shape does not change much. Digital human 201 can present these mouth shapes with little change. Digital human 201 may also not present these mouth shapes. Since the mouth shape does not change much, digital human 201 does not present these changes in mouth shape, thereby saving processing resources and not affecting the viewing experience of target user 100.

[0103] In some embodiments, the M phonemes in the preset phoneme set may include 6 vowels. and 2 consonants { / b / , / f / }. These 8 phonemes are based on the IPA International Phonetic Alphabet standard, and are highly correlated with the changes in mouth shape when people of different languages ​​speak, and have a large degree of distinction. The mouth shapes corresponding to these 8 phonemes and their changes and transitions can cover the vowel and consonant representations related to human and mouth shape changes.

[0104] The mouth shape graph set may include multiple mouth shape graphs corresponding to each of the M phonemes, and the multiple mouth shape graphs corresponding to each phoneme represent the mouth shape change information of the phoneme during the pronunciation process. The mouth shape will gradually change to the final mouth shape during the pronunciation process. Therefore, in some embodiments, each phoneme may include an initial mouth shape graph during pronunciation, a final mouth shape graph (pronunciation mouth shape), and a transition mouth shape graph between the two in the mouth shape set. The initial mouth shape may, for example, be a closed mouth shape that does not make any sound. The final mouth shape may, for example, be any one or more of the 8 mouth shapes corresponding to the above-mentioned 8 phonemes. The transition mouth shape graph may be obtained by an interpolation algorithm based on the initial mouth shape graph and the final mouth shape graph. The interpolation algorithm may include, but is not limited to, linear interpolation, Hermite polynomial interpolation, Lagrange polynomial interpolation, and the like.

[0105] The preset threshold can be obtained based on manual experience or calculated based on an artificial intelligence model, and this specification does not limit this.

[0106] By setting the transition lip shape, the lip shape of the subsequently generated digital human 201 can change more smoothly, the digital human 201 can be more realistic, and the viewing experience of the target user 100 can be better.

[0107] In some embodiments, see Figure 4 , for any natural number i, the multiple lip shape graphs corresponding to the i-th phoneme in the lip shape graph set can be obtained in the following way:

[0108] P420: Obtain the three-dimensional image parameters corresponding to the digital human.

[0109] P450: Obtaining initial facial parameters and pronunciation facial parameters corresponding to the i-th phoneme.

[0110] The initial facial parameters may be facial parameters of the character in a closed mouth state, and the pronunciation facial parameters may be facial parameters of the character when pronouncing the i-th phoneme. Facial parameters may be parameters that characterize the facial state of the character. For example, facial parameters may include parameters that affect the shape of the lips or mouth. Facial parameters may include but are not limited to the degree of outward convergence of the corners of the mouth, the degree of inward convergence of the corners of the mouth, the degree of jaw opening, etc. The character has different mouth shapes when the mouth is closed and when pronouncing different phonemes, and the facial parameters are also different. The mouth shape state of the character can be determined by the facial parameters.

[0111] The character may be, for example, a real person, such as a face capture actor. By using a high-precision camera to capture the reflective positioning points on the actor's face, the facial parameters of the face capture actor in a closed-mouth state and when pronouncing the i-th phoneme are obtained. The character may also be, for example, a face model. By manually pronouncing the phoneme in accordance with the standard pronunciation, the facial parameters of the face model in a closed-mouth state and when pronouncing the i-th phoneme are obtained.

[0112] P460: Perform interpolation based on the initial facial parameters and the pronunciation facial parameters to obtain a facial parameter sequence of the i-th phoneme during the pronunciation process.

[0113] As mentioned above, the transitional mouth shape diagram between the initial mouth shape diagram and the final mouth shape (pronunciation mouth shape) diagram can be obtained by the interpolation algorithm. The processor 620 can obtain the transitional facial parameters by using the interpolation algorithm between the initial facial parameters and the pronunciation facial parameters, so as to obtain the complete facial parameter sequence of the i-th phoneme during the pronunciation process, so as to prepare for the subsequent generation of the transitional mouth shape diagram.

[0114] P470: Rendering the three-dimensional image parameters and the facial parameter sequence by means of a rendering tool to obtain a plurality of lip shape diagrams corresponding to the i-th phoneme.

[0115] For the specific method, please refer to the content of rendering the three-dimensional image parameters and body movement parameters by the rendering tool mentioned above. The multiple mouth shape graphs corresponding to the phonemes can come from the same mouth, so as to ensure the coherence and harmony of the image of the digital human 201 in the subsequent interactive video generation.

[0116] In some embodiments, the lip shape map can also be obtained by rendering and then clustering images based on real-life facial capture data.

[0117] Figure 6Some lip shape graphs in a lip shape graph collection provided according to some embodiments of the present specification are shown. Figure 6 The following are mouth shape diagrams showing the pronunciation of 4 of the 8 phonemes. (I): The final mouth shape is that the lips protrude forward. (II): / i / , the final mouth shape is that the corners of the lips spread out to the sides. (III): / a / , the final mouth shape is that the root of the tongue is lowered and the lips spread out to the sides. (IV): / ε / . The final mouth shape is that the lips spread out slightly to the sides. Figure 6 It also includes the transitional lip shape from closed mouth to the last 4 lip shapes.

[0118] As an example, in the vowel In the process, the mouth gradually opens from closed mouth, and the mouth shape gradually changes from closed mouth shape to lips protruding forward. There may be multiple transitional mouth shape images obtained by interpolation algorithm between the closed mouth shape image and the mouth shape image of mouth shape (I). Figure 6 Only two transition lip shape images are shown. In the actual material library L, each phoneme may correspond to more than two transition lip shape images. For example, each phoneme may correspond to a closed mouth lip shape image, six transition lip shape images and a final lip shape image.

[0119] In some embodiments, continue to refer to Figure 4 After rendering the three-dimensional image parameters and facial parameter sequence by the rendering tool, in addition to obtaining the mouth shape diagram set, the key points corresponding to the mouth shape diagram can also be obtained. The key points corresponding to the mouth shape diagram are points that characterize the facial state of the digital human 201, especially the mouth state. The key points corresponding to the mouth shape diagram correspond to the aforementioned facial coefficients. As mentioned above, the key points corresponding to the mouth shape diagram may include but are not limited to the corners of the mouth, the philtrum, the lip beads, the outer sides of the upper and lower lips, the inner sides of the upper and lower lips, etc. The position of the mouth of the digital human 201 or the area where the mouth shape is located can be calculated through the mouth key points. The mouth shape key point information corresponding to the mouth shape diagram can be the coordinate information of the mouth shape key points.

[0120] In some embodiments, Figure 4 As shown, the mouth shape diagram set is integrated into a sprite diagram. The sprite diagram may include M regions. The M regions may correspond to the M phonemes one by one. Among them, the multiple mouth shape diagrams corresponding to each phoneme may be located in the region corresponding to the phoneme. When selecting the multiple mouth shape diagrams corresponding to the phoneme, the processor 620 may locate the region corresponding to the phoneme from the sprite diagram, and then obtain all or part of the mouth shape diagrams in the corresponding region.

[0121] By integrating the lip shape collection into a sprite, the number of requests when loading the screen can be significantly reduced, thereby achieving faster screen loading speed, improving efficiency, and enhancing the user experience of the target user by 100%.

[0122] In some embodiments, the material library L may further include the eye spacing in the mouth shape diagram. In some embodiments, the material library L may further include the mouth shape key points in the mouth shape diagram. In some embodiments, due to the existence of a sprite diagram, the eye spacing and the mouth shape key points may be the eye spacing corresponding to the sprite diagram and the mouth shape key points corresponding to the sprite diagram, respectively. Figure 4 As shown, the material library L may include the inter-eye distance corresponding to the Sprite image and the lip shape key points corresponding to the Sprite image.

[0123] In some embodiments, the pre-built material library L may be updated regularly or irregularly to adapt to different market or user needs.

[0124] By pre-building a library of materials, the time for creating resources in the video production process can be reduced, and the amount of calculation can be greatly reduced while ensuring image quality. When the subsequent interactive video is generated, the processor 620 only needs to read the pre-processed image, thereby greatly reducing the resource consumption during real-time interaction.

[0125] Continue to refer to Figure 3 , the method S300 may further include:

[0126] S330: Based on the lip shape graph set, generate a target lip shape graph sequence that matches the phoneme changes in the target speech.

[0127] The lip-shaped graph set may include lip-shaped images corresponding to the phonemes with the largest lip-shaped change amplitude among all the phonemes. The phonemes in the target speech can be obtained based on the target speech. The number of lip-shaped graphs in the target lip-shaped graph sequence can be equal to the total duration of the target speech multiplied by the frame rate of the basic video. The frame rate is the number of frames displayed per second in the interactive video (Frames Per Second, FPS). The frame rate of the basic video may include but is not limited to 24FPS, 25FPS, 30FPS or 60FPS, etc. The processor 620 determines the number of lip-shaped graphs in the target lip-shaped graph sequence according to the total duration of the target speech and the frame rate of the basic video. For example, if the total duration of the target speech is 20 seconds and the frame rate is 25FPS, the number of lip-shaped graphs in the target lip-shaped graph sequence is 500. For another example, if the total duration of the target speech is 10 seconds and the frame rate is 30FPS, the number of lip-shaped graphs in the target lip-shaped graph sequence is 300.

[0128] By using a target lip shape sequence that matches the phoneme changes in the target speech, the content fed back by the digital human 201 to the target user 100 can correspond to the lip shape of the digital human 201, thereby making the digital human 201 more realistic and providing the target user 100 with a better experience.

[0129] Figure 7 FIG. 4 shows a flow chart of generating a target lip shape sequence according to some embodiments of the present invention. Figure 7 As shown, step S330 may include:

[0130] S331: Determine at least one key phoneme in the target speech and a timestamp corresponding to the at least one key phoneme.

[0131] Among them, the key phoneme belongs to a preset phoneme set. The change range between the key phoneme's pronunciation mouth shape and the closed mouth shape is greater than a preset threshold. The timestamp can be used to identify the time of occurrence of a specific event. Here, the timestamp can be used to record or identify the pronunciation time of the key phoneme. The timestamp can be a moment or a time range. For example, the timestamp is moment T. For another example, the timestamp is the time range from moment T1 to moment T2, recorded as [T1, T2]. For another example, the timestamp is a time range with a duration of 10ms, 100ms, 1s, 5s, 10s, etc.

[0132] Specifically, step S331 may include:

[0133] S331 - 1 : Perform speech recognition on the target speech to obtain a plurality of text units and timestamps corresponding to the plurality of text units.

[0134] The text unit can be a letter, a word, a sentence, etc. Taking Chinese as an example, the target speech can be a sentence, and the text unit can be a Chinese character. For example, the target speech is . By performing speech recognition on the target speech, four text units [, <Love>, <China>, <Country>] and four Chinese characters can be obtained. By performing speech recognition on the target speech, timestamps corresponding to multiple text units can also be obtained. For example, corresponds to timestamp I, <Love> corresponds to timestamp II, <China> corresponds to timestamp III, and <Country> corresponds to timestamp IV. As mentioned above, the timestamp can be a moment or a time range. In daily conversations, people's average pronunciation time for each word is about 1.7 seconds to 2 seconds. When the timestamp is a time range, the pronunciation time of each text unit can be between 1.5 seconds and 2.5 seconds, so as to increase the realism of the lip shape changes of the digital human 201 and enhance the viewing experience of the target user 100.

[0135] There may be segments in the interactive video that do not require the digital human 201 to output voice information. For example, the target voice is Figure 1BD2 <Electronic ID card is also acceptable, can you provide the electronic ID card to the staff? >. The digital human 201 made a semantic pause between <Electronic ID card is also acceptable> and <Can you provide the electronic ID card to the staff>. That is, after saying <Electronic ID card is also acceptable>, it took a while before saying <Can you provide the electronic ID card to the staff> again. The semantic pause can be reflected in the lip shape, which can correspond to the digital human 201's lip shape not changing. For example, the digital human 201 maintains a closed mouth lip shape. Therefore, in some embodiments, after obtaining the timestamp corresponding to the text unit, the processor 620 can also process the timestamp as a silent segment, fill in the segments without pronunciation, so that the entire timestamp sequence of the target voice is continuous, so as to generate a harmonious interactive video later.

[0136] S331 - 2: Map the multiple text units to multiple phonemes, and obtain timestamps corresponding to the multiple phonemes based on the timestamps corresponding to the multiple text units.

[0137] A text unit can be mapped to one phoneme or multiple phonemes, and multiple text units can be mapped to multiple phonemes.

[0138] In some embodiments, the multiple text units may be mapped to the multiple phonemes in the following manner: obtaining a target language adopted by the target speech; and mapping the multiple text units to the multiple phonemes based on a mapping relationship corresponding to the target language.

[0139] Among them, the mapping relationship represents the mapping relationship between the text unit and the phoneme in the target language. The mapping relationship between the text unit and the phoneme is different depending on the target language used by the target speech. For example, if the target language is Chinese, the mapping relationship can be pinyin. The mapping relationship between the text unit and the phoneme can be provided by, but not limited to, the pypinyin library. For another example, if the target language is English, the mapping relationship is phonetic symbols. The mapping relationship between the text unit and the phoneme can be provided by, but not limited to, the cmudict library. The target language can be other languages. The mapping relationship between the text unit and the phoneme of other languages ​​can be obtained through, but not limited to, the library officially provided by IPA.

[0140] For example, the target speech may be the Chinese word "I love China". The phonemes in the target speech are represented by Chinese pinyin including {I: like: middle: country: For example, the target language could be English 。The phonemes in the target language are represented using phonetic symbols and include

[0141] After obtaining the timestamps corresponding to multiple text units, the processor 620 can obtain the timestamps corresponding to multiple phonemes. For example, if the text units of the target language are [, <Love>, <China>], and the phonemes of the text units are Timestamp I, Timestamp II, Timestamp III, Timestamp IV. Then / w / corresponds to Timestamp I1, corresponds to Timestamp I2; / a / corresponds to Timestamp II1, corresponds to Timestamp II2]; corresponds to Timestamp III1, corresponds to Timestamp III2, corresponds to Timestamp III3; / g / corresponds to Timestamp IV1, / u / corresponds to Timestamp IV2, corresponds to Timestamp IV3.

[0142] S331-3: Locate the at least one key phoneme from the multiple phonemes and obtain the timestamps corresponding to the at least one key phoneme.

[0143] As mentioned above, the key phoneme belongs to a preset phoneme set. The change range between the pronunciation mouth shape of the key phoneme and the closed mouth shape is greater than a preset threshold. The phonemes belonging to the preset phoneme set among the multiple phonemes are located as key phonemes. The key phoneme can include one or more. For example, the key phonemes of can include The processor 620 can obtain the timestamps corresponding to the key phonemes based on the timestamps of the multiple phonemes. In daily conversations, the average vocalization time of a single vowel phoneme is about 100 milliseconds to 300 milliseconds, and the vocalization time of compound vowels and some consonants is about 200 - 400 milliseconds.

[0144] In some embodiments, when extracting the key phonemes, the processor 620 can also perform phoneme merging and adjustment to optimize the representation of the speech, making it more in line with the pronunciation rules of natural language, so that the overall visual effect of the subsequent generated interactive video is more natural. For example, the target speech is <Kimi is a Moonshot AI>. For the convenience of description, all the phonemes in the target speech are merged and adjusted. The processing of the key factors can be similar. The phoneme sequence and its initial timestamp of the target speech: K[k] - 0.1s, m[m] - 0.3s, Moonshot[muːnʃɑːt] - 0.8s,

[0145] Among them, phonemes It appears twice at 0.2s and 0.4s, and the timestamps are very close. In natural pronunciation, consecutive identical phonemes are usually merged into one phoneme. Therefore, the processor 620 can merge these two phonemes into one and retain the earlier timestamp. The adjusted phoneme sequence and its timestamp are as follows: K[k]-0.1s, m[m]-0.3s, Moonshot[mu:n∫ɑ:t]-0.8s,

[0146] in, The time stamp difference between {Moonshot[mu:n∫ɑ:t]} and {Moonshot[mu:n∫ɑ:t]} is large (0.1s), which may represent an unnatural pause in natural language. In order to make the speech flow smoother, the processor 620 can continue to adjust the time stamp of {Moonshot[mu:n∫ɑ:t]} forward to make it closer to The adjusted phoneme sequence and its timestamp are as follows: K[k]-0.1s, m[m]-0.3s, Moonshot[mu:n∫ɑ:t]-0.75s, Through the above steps, the phoneme sequence is made more compact and natural, and the timestamp is more in line with human pronunciation habits, so that the overall viewing effect of the subsequent interactive video is more natural.

[0147] Continue to see Figure 7 , step S330 may further include:

[0148] S332: Generate the target lip shape graph sequence based on the at least one key phoneme, the timestamp corresponding to the at least one key phoneme, and the lip shape graph set.

[0149] Specifically, for each key phoneme, step S332 may include:

[0150] S332-1: Determine frame index information corresponding to the key phoneme based on the total duration of the target speech, the target frame rate, and the timestamp corresponding to the key phoneme.

[0151] The total duration of the target speech can be obtained by checking the properties of the voice file or the duration of the waveform, or by using a function related to the duration of the audio file in a programming language. The target frame rate can be the frame rate of the interactive video, that is, the frame rate of the basic video. As previously mentioned, the processor 620 can determine the total number of frames of the interactive video based on the total duration of the target speech and the frame rate (target frame rate) of the basic video. The frame index information of the key phoneme can refer to the frame in which the utterance of the key phoneme appears.

[0152] For example, if the total duration of the target speech is 20 seconds and the target frame rate is 25FPS, the total number of frames of the corresponding interactive video is 500 frames. The timestamp of phoneme 1 is [200ms, 400ms]. Phoneme 1 appears or sounds in frames 5 to 10. The frame index information of phoneme 1 is frames 5 to 10. For another example, if the total duration of the target speech is 20 seconds and the target frame rate is 30FPS, the total number of frames of the corresponding interactive video is 600 frames. The timestamp of phoneme 2 is 1.5s. Phoneme 1 appears or sounds in frame 45. The frame index information of phoneme 2 is frame 45.

[0153] In some embodiments, the frame index information corresponding to the key phonemes can be determined by the AI ​​model. For example, by inputting the target speech into the frame index model, the frame index information corresponding to the key phonemes in the target speech can be determined. The frame index model can be obtained through training. The training data of the frame index model may include sample speech and its corresponding sample frame index information. The training goal of the frame index model may include constraining the difference between the frame index information output by it and the pre-marked frame index information to be within a first preset difference threshold. Among them, the first preset difference threshold can be determined based on experience or based on a model training process.

[0154] In some embodiments, the frame index model may be an Audio2Lip model.

[0155] S332-2: Obtain a standard lip shape sequence corresponding to the key phoneme from the lip shape set.

[0156] The standard lip shape graph sequence corresponding to each phoneme in the lip shape graph set may include one lip shape graph, two lip shape graphs, or multiple lip shape graphs.

[0157] As mentioned above, the lip shape diagram collection can be integrated into a Sprite diagram. The Sprite diagram includes M areas, and the M areas correspond to the M phonemes one by one. Among them, the multiple lip shape diagrams corresponding to each phoneme are located in the area corresponding to the phoneme. Therefore, in some embodiments, the standard lip shape diagram sequence corresponding to the key phoneme can be obtained from the lip shape diagram collection in the following way: locate the target area corresponding to the key phoneme from the Sprite diagram, and generate the standard lip shape diagram sequence based on the multiple lip shape diagrams in the target area. For example, phoneme 1 corresponds to area A, phoneme 2 corresponds to area B, and phoneme 3 corresponds to area C. If the key phoneme is phoneme 1, the target area is area A, and the standard lip shape diagram sequence corresponding to phoneme 1 includes the lip shape diagram in area A. If the key phoneme is phoneme 3, the target area is area C, and the standard lip shape diagram sequence corresponding to phoneme 3 includes the lip shape diagram in area C.

[0158] S332-3: Based on the frame index information and the standard lip shape graph sequence, generate an updated lip shape graph sequence corresponding to the key phoneme.

[0159] The frame index information can synchronize the actual speaking speed of the target voice or key phoneme with the frames in the video. The standard lip-shaped graph sequence is the lip-shaped graph of the key phoneme in the material library. The number of frames represented by the frame index information may be the same as or different from the number of lip-shaped graphs in the standard lip-shaped graph sequence.

[0160] In some embodiments, if the number of frames represented by the frame index information is equal to the number of lip shape images in the standard lip shape image sequence, the standard lip shape image sequence is used as the updated lip shape image sequence. For example, the frame index information of phoneme 1 is the 5th to 8th frames, and the number of lip shape images in the standard lip shape image sequence corresponding to phoneme 1 is 4. Then these 4 lip shape images can be directly used as the updated lip shape image sequence.

[0161] In some embodiments, if the number of frames represented by the frame index information is greater than the number of lip shape graphs in the standard lip shape graph sequence, the standard lip shape graph sequence is interpolated based on the frame index information to obtain the updated lip shape graph sequence. The number of frames being greater than the number of lip shape graphs indicates that the standard lip shape graph sequence cannot provide enough frames for the video. The processor 620 needs to generate new lip shape graphs by interpolation to make up the number of lip shape graphs to the number of frames. For example, the frame index information of phoneme 2 is the 10th frame to the 20th frame, and the number of lip shape graphs in the standard lip shape graph sequence corresponding to phoneme 2 is 8. The standard lip shape graph sequence is still missing 3 lip shape graphs. The processor 620 can generate 3 lip shape graphs by interpolation to obtain an updated lip shape graph sequence including 11 lip shape graphs.

[0162] In some embodiments, if the number of frames represented by the frame index information is less than the number of lip graphs in the standard lip graph sequence, the standard lip graph sequence is sampled based on the frame index information to obtain the updated lip graph sequence. The number of frames less than the number of lip graphs indicates that the lip graphs provided by the standard lip graph sequence exceed those required by the video. The processor 620 can select some lip graphs by sampling to reduce the number of lip graphs to the number of frames. For example, the frame index information of phoneme 3 is the 22nd to 24th frames, and the number of lip graphs in the standard lip graph sequence corresponding to phoneme 2 is 5. The standard lip graph sequence has 2 more lip graphs. The processor 620 can select 3 lip graphs by sampling to obtain an updated lip graph sequence including 3 lip graphs. For example, the processor 620 can select the first initial lip graph, the third transition lip graph, and the fifth final lip graph in the standard lip graph sequence.

[0163] The updated lip graph sequence is obtained through the above steps, ensuring that each phoneme in the interactive video can be synchronized with the correct frame, thereby creating a smooth audio-visual experience for the target user 100.

[0164] S332-4: Concatenate the updated lip shape graph sequences corresponding to the at least one key phoneme to obtain the target lip shape graph sequence.

[0165] By concatenating the updated lip graph sequences corresponding to all key phonemes, a target lip graph sequence having the same number of lip graphs as the total number of video frames can be obtained.

[0166] Continue to refer to Figure 3 The method S300 further includes:

[0167] S340: Fitting the target lip shape image sequence to the mouth area of ​​the digital human in the basic video to obtain a target video.

[0168] The mouth of the digital human 201 in the basic video may be in a closed state. The processor 620 replaces the mouth of the digital human 201 in the basic video with the lip shape image in the target lip shape image sequence, and obtains a target video with lip shape changes and body movements.

[0169] Specifically, step S340 may include the following steps:

[0170] S341: extracting a video segment from the basic video.

[0171] The number of image frames in the video clip is equal to the number of lip shape images in the target lip shape image sequence. That is, the duration of the video clip is equal to the duration of the target speech. In this way, the mouth areas in the video clip can be replaced, and the mouth areas in the video clip are all from the same mouth, thereby ensuring the coherence and harmony of the image of the digital human 201.

[0172] S342: Fit the target lip shape sequence to the mouth area of ​​the digital human in the video clip frame by frame to obtain the target video.

[0173] The processor 620 replaces the mouth area of ​​the digital human 201 frame by frame using the lip shape images in the target lip shape image sequence, thereby obtaining a target video with lip shape changes and body movements.

[0174] Figure 8 FIG. 4 shows a flow chart of obtaining a target video according to some embodiments of the present specification. Figure 8 As shown, for any natural number i, step S342 may include:

[0175] Step S342-1: Based on the inter-eye distance in the i-th frame of the lip shape image and the inter-eye distance in the i-th frame of the limb shape image, scaling the i-th frame of the lip shape image so that the lip shape in the scaled i-th frame of the lip shape image is adapted to the face in the i-th frame of the limb shape image.

[0176] In some embodiments, the eye spacing in the limb image may be the eye spacing corresponding to the aforementioned basic video. The eye spacing in the mouth image may be the eye spacing corresponding to the aforementioned sprite image. As mentioned above, the eye spacing is related to the size and proportion of the head (face), and is an important parameter for characterizing the size of the head (face). The face size in the mouth image may be different from the face size in the limb image, and direct fitting may result in an incoordination between the face and the limbs.

[0177] The processor 620 can calculate the size of each face based on the inter-eye distance in the lip shape map and the inter-eye distance in the limb map, thereby scaling the lip shape map so that the lip shape in the lip shape map matches the face in the limb map. For example, the processor 620 can calculate the ratio of the inter-eye distance in the lip shape map to the inter-eye distance in the limb map. If the ratios are consistent, the processor 620 does not need to scale the lip shape map. If the ratio is greater than 1, the processor 620 can scale the lip shape map until the ratio of the inter-eye distance between the two is 1. If the ratio is less than 1, the processor 620 can enlarge the lip shape map until the ratio of the inter-eye distance between the two is 1.

[0178] By adjusting the size of the lip shape map, the lip shape in the lip shape map is matched with the face in the body shape map, thereby avoiding the phenomenon that the digital human 201 is unnatural or uncoordinated due to the inconsistency of the sizes of the two combined materials.

[0179] Step S342-2: based on the mouth key point information corresponding to the i-th frame of the mouth shape image, cut out the mouth area from the scaled i-th frame of the mouth shape image to obtain a mouth sub-image.

[0180] The mouth key points corresponding to the mouth shape diagram may be the mouth key points corresponding to the aforementioned sprite diagram. The mouth key point information may be the coordinate information of the key points. The mouth key point information may also be the displacement of the key points relative to the fixed point. For example, how much is the displacement of the chin relative to the closed mouth state. Based on the mouth key point information, the processor 620 may determine the exact position and range of the mouth area in the mouth shape diagram, thereby extracting a mouth sub-diagram containing only the mouth area from the mouth shape diagram.

[0181] Step S342-3: Based on the mouth key point information corresponding to the i-th frame lip shape image and the mouth key point information corresponding to the i-th frame limb image, determine the mouth area of ​​the digital human in the i-th frame limb image.

[0182] In some embodiments, the processor 620 may process the mouth key point information corresponding to the limb map based on the mouth key point information corresponding to the mouth map. For example, as described above, the mouth key point information corresponding to the mouth map may be the displacement of the key point relative to the fixed point. For example, how much is the displacement of the chin relative to the closed mouth state. The processor 620 may add the displacement of the key point in the mouth map relative to the fixed point to the mouth key point corresponding to the limb map, thereby determining the mouth area corresponding to the size in the mouth map in the limb map, thereby ensuring the accuracy of subsequent fitting.

[0183] Step S342 - 4 : attaching the mouth sub-image to the mouth area of ​​the digital human 201 .

[0184] In some embodiments, the step of fitting the mouth sub-image to the mouth area of ​​the digital human 201 may include the following steps: performing Poisson fusion on the mouth sub-image and the mouth area of ​​the digital human 201 to fit the mouth sub-image to the mouth area of ​​the digital human 201. Using Poisson fusion, a seamless transition area can be created between the mouth sub-image and the mouth area, so that the synthesized image looks natural, thereby making the image of the digital human 201 more harmonious and natural, and providing a better experience for the target user 100.

[0185] In some embodiments, the mouth area fitting based on key points can be achieved through the AI ​​model. For example, by inputting the mouth shape map and the limb map into the fitting model, the mouth area in the mouth shape map can be fitted to the mouth area of ​​the limb map. The fitting model can be obtained through training. The training data of the fitting model may include sample mouth shape maps and sample limb maps. The training goal of the fitting model may include constraining the difference between the output fitted image and the pre-marked fitted image to be within a second preset difference threshold. Among them, the second preset difference threshold can be determined based on experience or based on the model training process.

[0186] In some embodiments, the frame index model may be a GAN model.

[0187] At this point, the processor 620 completes the generation of the target video.

[0188] Fig. 9 Some image frames in a target video provided according to some embodiments of the present specification are shown. Fig. 9 Figure 2 shows three frames of body images with mouth sub-images attached to the mouth area, Frame 1', Frame 2', and Frame 3'. The digital human 201 is uttering phonemes. In Frame 1', the lip shape of the digital human 201 is closed, and the body movement is both hands touching the abdomen. In Frame 2', the lip shape of the digital human 201 is slightly open, and the body movement is the right hand touching the abdomen and the left hand is raised. In Frame 3', the lip shape of the digital human 201 is lips protruding forward, and the body movement is the right hand touching the abdomen and the left hand is raised, and the angle of the left hand is higher than that in Frame 2'.

[0189] In the target video, the digital human 201 can well combine body movements (eg, hand movements, head movements) and lip shape changes, making the image of the digital human 201 more realistic and natural.

[0190] In some embodiments, when method S300 is executed by the terminal device 200, method S300 may further include the following steps: synchronously playing the target voice and the target video to present the interactive effect of the digital human 201. When the target voice and the target video are synchronously played, the changes in the lip shape of the digital human 201 in the target video match the changes in the factors in the target voice. For example, when the target voice is played to the point where the digital human 201 outputs the word "country", the lip shape of the digital human 201 is roughly circular and protrudes forward. Synchronously playing the target voice and the target video can ensure that the lip shape of the digital human 201 in the target voice and the target video is completely corresponding in time, thereby enhancing the realism and immersion of the interaction both auditorily and visually.

[0191] In some embodiments, after generating the target video, method S300 may further include the following steps: merging the target voice and the target video to generate the target audio and video. When method S300 is executed by the terminal device 200, after generating the target audio and video, the terminal device 200 may play the target audio and video to present the interactive effect of the digital human 201. When method S300 is executed by the server, after generating the target audio and video, the server may send the target audio and video to the terminal device 200 to present the interactive effect of the digital human 201 through the terminal device 200.

[0192] In summary, the interactive video generation method S300 provided in this specification can pre-construct a material library including a basic video and a lip shape map set. When an interactive video needs to be generated, the generation device can obtain the basic video and the lip shape map set from the material library, use the lip shape map set to generate a target lip shape map sequence that adapts to the phoneme changes of the target voice, and then fit the target lip shape map sequence to the basic video to obtain the interactive video. The above scheme avoids the real-time image rendering work during the interaction process, so it can reduce the resource consumption in the interactive video generation process. In this way, the digital human interactive product does not need to use or use less GPU resources, or only uses relatively low CPU resources to achieve the generation of digital human interactive videos. On the one hand, it can reduce the computational cost of generating interactive videos, and on the other hand, the digital human interactive product can be applied to various terminal devices to improve the wide application. In addition, the generation method S300 can ensure that the lip shape changes of the digital human match the voice content output by fitting the target lip shape map sequence to the mouth area of ​​the digital human in the basic video, thereby providing a good experience and immersion for the target user 100.

[0193] On the other hand, the present specification provides a non-transitory storage medium storing at least one set of executable instructions for interactive video generation. When the executable instructions are executed by the processor, the executable instructions instruct the processor to implement the steps of the interactive video generation method S300 described in the present specification. In some possible implementations, various aspects of the present specification can also be implemented in the form of a program product, which includes a program code. When the program product is run on the generating device 600, the program code is used to enable the generating device 600 to perform the steps of the interactive video generation method S300 described in the present specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on the generating device 600. However, the program product of the present specification is not limited to this. In the present specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with the instruction execution system. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the generation device 600, partially on the generation device 600, as a stand-alone software package, partially on the generation device 600 and partially on a remote computing device, or entirely on a remote computing device.

[0194] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0195] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0196] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "embodiments" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, particular features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0197] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0198] Each patent, patent application, publication of patent applications, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein, except to the extent that it is inconsistent or conflicting with this document or that has a limiting effect on the broadest scope of the claims, may be incorporated herein by reference and used for all purposes now or hereafter associated with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall prevail.

[0199] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A method for generating a digital human interactive video, comprising: Obtaining a target voice, wherein the target voice represents information that the digital human needs to express through voice during interaction; Obtaining a basic video and a set of lip shape graphs from a pre-built material library, wherein the basic video represents the body movement information of the digital human, and the set of lip shape graphs represents the lip shape change information involved in the pronunciation process of different phonemes in a preset phoneme set; Based on the lip shape graph set, generating a target lip shape graph sequence that matches the phoneme changes in the target speech; and The target lip shape sequence is fitted to the mouth area of ​​the digital human in the basic video to obtain a target video.

2. The method according to claim 1, wherein: The preset phoneme set includes M phonemes in the phoneme table that meet the target condition, wherein the target condition includes: the change range between the pronunciation mouth shape and the closed mouth shape of the phoneme is greater than a preset threshold, and M is an integer greater than 1; The mouth shape graph set includes a plurality of mouth shape graphs corresponding to each of the M phonemes, and the plurality of mouth shape graphs corresponding to each phoneme represent mouth shape change information of the phoneme during pronunciation.

3. The method according to claim 2, wherein: The step of generating a target lip shape graph sequence matching the phoneme changes in the target speech based on the lip shape graph set comprises: Determine at least one key phoneme in the target speech, and a timestamp corresponding to the at least one key phoneme, wherein the key phoneme belongs to the preset phoneme set; and The target lip graph sequence is generated based on the at least one key phoneme, the time stamp corresponding to the at least one key phoneme, and the lip graph set.

4. The method according to claim 3, wherein: The determining at least one key phoneme in the target speech and a timestamp corresponding to the at least one key phoneme comprises: Performing speech recognition on the target speech to obtain a plurality of text units and timestamps corresponding to the plurality of text units; Mapping the multiple text units to multiple phonemes, and obtaining timestamps corresponding to the multiple phonemes based on timestamps corresponding to the multiple text units; The at least one key phoneme is located from the multiple phonemes, and a timestamp corresponding to the at least one key phoneme is obtained.

5. The method according to claim 4, wherein: Mapping the plurality of text units into a plurality of phonemes comprises: Obtaining a target language used by the target speech; and The plurality of text units are mapped to the plurality of phonemes based on a mapping relationship corresponding to the target language, wherein the mapping relationship represents a mapping relationship between the text units and the phonemes in the target language.

6. The method according to claim 3, wherein: The step of generating the target lip shape graph sequence based on the at least one key phoneme, the timestamp corresponding to the at least one key phoneme, and the lip shape graph set comprises: For each key phoneme: Based on the total duration of the target speech, the target frame rate, and the timestamp corresponding to the key phoneme, frame index information corresponding to the key phoneme is determined, Obtaining a standard lip shape sequence corresponding to the key phoneme from the lip shape set, Based on the frame index information and the standard lip shape graph sequence, generating an updated lip shape graph sequence corresponding to the key phoneme; and The updated lip shape graph sequences corresponding to the at least one key phoneme are concatenated to obtain the target lip shape graph sequence.

7. The method according to claim 6, wherein: The lip shape graph set is integrated into a sprite graph, the sprite graph includes M regions, the M regions correspond to the M phonemes one by one, wherein the multiple lip shape graphs corresponding to each phoneme are located in the region corresponding to the phoneme; The step of obtaining the standard lip shape image sequence corresponding to the key phoneme from the lip shape image set includes: locating a target area corresponding to the key phoneme from the Sprite image, and generating the standard lip shape image sequence based on the multiple lip shape images in the target area.

8. The method according to claim 6, wherein: The step of generating an updated lip shape graph sequence corresponding to the key phoneme based on the frame index information and the standard lip shape graph sequence includes any one of the following: If the number of frames represented by the frame index information is equal to the number of lip shape graphs in the standard lip shape graph sequence, then using the standard lip shape graph sequence as the updated lip shape graph sequence; If the number of frames represented by the frame index information is greater than the number of lip shape graphs in the standard lip shape graph sequence, interpolating the standard lip shape graph sequence based on the frame index information to obtain the updated lip shape graph sequence; or If the number of frames represented by the frame index information is less than the number of lip shape images in the standard lip shape image sequence, the standard lip shape image sequence is sampled based on the frame index information to obtain the updated lip shape image sequence.

9. The method according to claim 1, wherein: The number of lip shape graphs in the target lip shape graph sequence is equal to the total duration of the target speech multiplied by the frame rate of the basic video, and the target video is obtained by fitting the target lip shape graph sequence to the mouth area of ​​the digital human in the basic video, including: Cutting out a video segment from the base video, wherein the number of image frames in the video segment is equal to the number of lip shape graphs in the target lip shape graph sequence; and The target lip shape sequence is fitted frame by frame to the mouth area of ​​the digital human in the video clip to obtain the target video.

10. The method according to claim 9, wherein: For any natural number i, the i-th frame of the lip shape image in the target lip shape image sequence is fitted to the mouth area of ​​the digital human in the i-th frame of the limb image of the video clip, comprising: Based on the inter-eye distance in the i-th frame of the mouth shape image and the inter-eye distance in the i-th frame of the limb image, scaling the i-th frame of the mouth shape image so that the mouth shape in the scaled i-th frame of the mouth shape image is adapted to the face in the i-th frame of the limb image; Based on the mouth key point information corresponding to the i-th frame of the mouth shape image, intercepting the mouth area from the scaled i-th frame of the mouth shape image to obtain a mouth sub-image; Based on the mouth key point information corresponding to the i-th frame of the mouth shape image and the mouth key point information corresponding to the i-th frame of the limb image, determining the mouth area of ​​the digital human in the i-th frame of the limb image; and The mouth sub-image is attached to the mouth area of ​​the digital human.

11. The method according to claim 10, wherein: The step of attaching the mouth sub-image to the mouth region of the digital human comprises: The mouth sub-image is Poisson-fused with the mouth region of the digital human, so as to fit the mouth sub-image to the mouth region of the digital human.

12. The method according to claim 2, wherein: For any natural number i, the multiple lip shape graphs corresponding to the i-th phoneme in the lip shape graph set are obtained in the following manner: Obtaining three-dimensional image parameters corresponding to the digital human; Obtaining initial facial parameters and pronunciation facial parameters corresponding to the i-th phoneme, wherein the initial facial parameters are facial parameters of the character in a closed-mouth state, and the pronunciation facial parameters are facial parameters of the character when pronouncing the i-th phoneme; Performing interpolation based on the initial facial parameters and the pronunciation facial parameters to obtain a facial parameter sequence of the i-th phoneme during the pronunciation process; and The three-dimensional image parameters and the facial parameter sequence are rendered by a rendering tool to obtain a plurality of lip shape diagrams corresponding to the i-th phoneme.

13. The method according to claim 1, wherein: The basic video is obtained in the following manner: Obtaining three-dimensional image parameters corresponding to the digital human; Obtain the corresponding limb movement parameters when the character performs the preset action; as well as The three-dimensional image parameters and the limb movement parameters are rendered by a rendering tool to obtain the basic video.

14. The method according to claim 1, wherein: The method further comprises: The target voice and the target video are played synchronously to present the interactive effect of the digital human.

15. The method according to claim 1, wherein: The method further comprises: Merging the target voice and the target video to generate target audio and video; The target audio and video are played to present the interactive effect of the digital human, or the target audio and video are sent to a terminal device to present the interactive effect of the digital human through the terminal device.

16. A device for generating a digital human interactive video, comprising: At least one storage medium storing at least one instruction set for generating a digital human interaction video; as well as at least one processor, in communication with the at least one storage medium, Wherein, when the generating device is running, the at least one processor reads the at least one instruction and executes the method according to any one of claims 1-15 according to the instructions of the at least one instruction set.

Images