Picture and voice based video synthesis method, system, device and storage medium

Abstract

The application belongs to the technical field of artificial intelligence, and specifically discloses a video synthesis method, system, device and storage medium based on pictures and voice, which acquires a face picture and audio data, constructs a single phoneme video segment of each phoneme according to the face picture, performs voice recognition according to the audio data, extracts pronunciation structured data, splices and adjusts the pronunciation structured data and the single phoneme video segment of each phoneme to obtain a corresponding synthesized video, and can realize intelligent video synthesis, effectively reduces the video synthesis time and computing resource requirement of the user end. The application can accurately simulate the mouth movement of the user, improve the naturalness and reality of the synthesized video, reduce the computing amount and time of the user end video synthesis, improve the video synthesis efficiency and stability, improve the fluency and presentation effect of the synthesized video, and has a wide application prospect.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method, system, device, and storage medium for video synthesis based on images and voice. Background Technology

[0002] In the field of video compositing, existing methods to improve the realism and expressiveness of synthesized videos are mainly divided into two categories: rule-based and machine learning-based methods. Rule-based methods require manually writing rules to control the video compositing process, which suffers from complex rules and difficulty in maintenance. In contrast, machine learning-based methods can automatically control the video compositing process by training models, but this requires a large amount of training data and computational resources. In general, existing video compositing technologies still have certain drawbacks: they are time-consuming, especially when compositing videos longer than a minute, which is unfavorable for human-computer interaction scenarios requiring real-time applications; secondly, they place extremely high demands on server performance, particularly on GPU and memory performance, which limits their widespread adoption. Therefore, there is an urgent need for a video compositing technology that can effectively reduce the time consumption and computational resource requirements of video compositing while improving the realism and expressiveness of synthesized videos. Summary of the Invention

[0003] The purpose of this invention is to provide a video synthesis method, system, device, and storage medium based on images and voice, in order to solve the aforementioned problems existing in the prior art.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] Firstly, it provides a video synthesis method based on images and speech, including:

[0006] Obtain facial images and audio data provided by the client;

[0007] The lip region is detected in a face image, and a set of mouth shapes for the pronunciation process of each phoneme is generated based on the lip region detection results. The set of mouth shapes for the pronunciation process contains several consecutive mouth shape images.

[0008] The pronunciation process of each phoneme is nested into a face image to generate a single phoneme video segment corresponding to each phoneme. The single phoneme video segment contains several pronunciation frames, and the pronunciation frame is composed of a face image with the corresponding pronunciation mouth shape image nested in it.

[0009] Each single-phoneme video segment is associated with its corresponding phoneme and then stored in a phoneme segment dictionary;

[0010] The audio data is processed by speech recognition to extract the corresponding pronunciation structure data. The pronunciation structure data includes a phoneme sequence composed of several continuous phonemes, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence.

[0011] The phoneme segment dictionary and pronunciation structured data are transmitted to the client, so that the client can retrieve the corresponding single-phoneme video segments from the phoneme segment dictionary based on the phoneme sequence, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence, splice and adjust them to obtain a synthesized video, and play the synthesized video synchronously with the audio data.

[0012] In one possible design, the step of detecting the lip region of a facial image and generating a set of mouth shapes for the pronunciation process of each phoneme based on the lip region detection results includes:

[0013] Perform lip detection on a face image to obtain lip detection results, wherein the lip detection results include lip images;

[0014] Based on the lip area image and the preset mouth shape configuration parameters corresponding to each phoneme, a set of mouth shapes for the pronunciation process of each phoneme is generated using a preset dynamic expression generation component.

[0015] In one possible design, the speech recognition processing of the audio data, extracting the corresponding pronunciation structured data, includes:

[0016] The audio data is preprocessed to obtain the Mel-spectral characteristics of the audio data;

[0017] The Mel-spectral features of the audio data are input into a pre-set speech recognition model for speech recognition processing to obtain the pronunciation structured data corresponding to the audio data.

[0018] In one possible design, the transmission of the phoneme fragment dictionary and pronunciation structured data to the client includes:

[0019] The phoneme fragment dictionary and pronunciation structured data are converted into corresponding JSON format files and then transmitted to the client.

[0020] In one possible design, the phoneme segment dictionary and the phonetic structured data are transmitted to the client, so that the client can retrieve the corresponding single-phoneme video segments from the phoneme segment dictionary based on the phoneme sequence, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence, and then splice and adjust them to obtain a synthesized video, including:

[0021] The phoneme segment dictionary and the pronunciation structured data are transmitted to the client, so that the client can retrieve the corresponding single phoneme video segments from the phoneme segment dictionary according to the phoneme sequence, and concatenate them sequentially. The duration of each single phoneme video segment is adjusted according to the duration of each phoneme in the phoneme sequence, and the transition duration between each single phoneme video segment is adjusted according to the pause duration between two adjacent phonemes in the phoneme sequence, so as to obtain the synthesized video.

[0022] In one possible design, the mouth shape images corresponding to the starting and ending pronunciation frames of the monophone video segments are both preset universal half-open mouth shapes. When the client adjusts the transition duration between each monophone video segment based on the pause duration between adjacent phonemes in the phoneme sequence, it also includes:

[0023] The last sound frame of the monophone video segment corresponding to the previous phoneme is used as a pause frame. Pause frames are used to synthesize pause segments. The duration of the pause segment is the pause duration between the previous phoneme and the next phoneme.

[0024] The pause segments are spliced ​​between the monophone video segments corresponding to the preceding phoneme and the monophone video segments corresponding to the following phoneme to obtain a composite video.

[0025] In one possible design, the method further includes:

[0026] Based on the lip position detection results, a set of pronunciation transition mouth shapes is generated between each phoneme, and the set of pronunciation transition mouth shapes contains several consecutive transition mouth shapes;

[0027] The transitional mouth shapes between each phoneme are nested on face images to generate transitional video segments between each phoneme. Each transitional video segment contains several transitional frames, and each transitional frame is composed of face images with corresponding transitional mouth shapes nested within them.

[0028] Based on the connection relationship between adjacent phonemes in the phoneme sequence, the corresponding transition video segments are extracted, and each transition video segment is associated with its corresponding adjacent phonemes and stored in the phoneme segment dictionary.

[0029] The phoneme segment dictionary and the pronunciation structured data are transmitted to the client, so that the client can retrieve the corresponding single-phoneme video segments from the phoneme segment dictionary according to the phoneme sequence and splice them sequentially. The duration of each single-phoneme video segment is adjusted according to the duration of each phoneme in the phoneme sequence. The corresponding transition video segment is retrieved from the phoneme segment dictionary according to the connection relationship between adjacent phonemes in the phoneme sequence and spliced ​​between the corresponding two single-phoneme video segments. The duration of the corresponding transition video segment is adjusted according to the pause duration between two adjacent phonemes in the phoneme sequence to obtain the synthesized video.

[0030] Secondly, a video synthesis system based on images and speech is provided, including a data acquisition unit, a lip-sync generation unit, a segment generation unit, a dictionary construction unit, an audio recognition unit, and a synthesis execution unit, wherein:

[0031] The data acquisition unit is used to acquire facial images and audio data provided by the client.

[0032] The mouth shape generation unit is used to detect the lip area of ​​a face image and generate a mouth shape set for the pronunciation process of each phoneme based on the lip area detection results. The mouth shape set for the pronunciation process contains several consecutive mouth shape images.

[0033] The segment generation unit is used to nest the mouth shape set of the pronunciation process of each phoneme on a face image to generate a single phoneme video segment corresponding to each phoneme. The single phoneme video segment contains several pronunciation frames, and the pronunciation frame is composed of a face image nested with the corresponding pronunciation mouth shape image.

[0034] The dictionary construction unit is used to associate each single-phoneme video segment with its corresponding phoneme and store them in the phoneme segment dictionary;

[0035] An audio recognition unit is used to perform speech recognition processing on audio data and extract the pronunciation structure data corresponding to the audio data. The pronunciation structure data includes a phoneme sequence composed of several continuous phonemes, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence.

[0036] The synthesis execution unit is used to transmit the phoneme segment dictionary and the pronunciation structured data to the client, so that the client can retrieve the corresponding single phoneme video segments from the phoneme segment dictionary according to the phoneme sequence, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence, splice and adjust them to obtain the synthesized video, and play the synthesized video synchronously with the audio data.

[0037] Thirdly, it provides video synthesis devices based on images and voice, including:

[0038] Memory, used to store instructions;

[0039] A processor is configured to read instructions stored in the memory and execute the method described in any one of the first aspects above, according to the instructions.

[0040] Fourthly, a computer-readable storage medium is provided, on which instructions are stored, which, when executed on a computer, cause the computer to perform any of the methods described in the first aspect. A computer program product containing instructions is also provided, which, when executed on a computer, cause the computer to perform any of the methods described in the first aspect.

[0041] Beneficial Effects: This invention acquires facial images and audio data, constructs single-phoneme video segments based on the facial images, performs speech recognition based on the audio data, extracts structured pronunciation data, and uses the structured pronunciation data and single-phoneme video segments to splice and adjust the resulting synthetic video. This achieves highly intelligent video synthesis, effectively reducing the time and computational resource requirements for video synthesis on the user end. Based on lip extraction from facial images, this invention can accurately simulate the user's mouth movements, improving the naturalness and realism of the synthesized video. By pre-generating single-phoneme video segments and structured pronunciation data on the backend, the computational load and time for video synthesis on the user end can be reduced, improving video synthesis efficiency and stability, enhancing the smoothness and presentation effect of the synthesized video, and demonstrating broad application prospects. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the steps in the method of Embodiment 1 of the present invention;

[0044] Figure 2 This is a schematic diagram of the system configuration in Embodiment 2 of the present invention;

[0045] Figure 3 This is a schematic diagram of the device configuration in Embodiment 3 of the present invention. Detailed Implementation

[0046] It should be noted that the descriptions of these embodiments are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. The specific structural and functional details disclosed herein are only for describing exemplary embodiments of the invention. However, the invention may be embodied in many alternative forms and should not be construed as being limited to the embodiments described herein.

[0047] It should be understood that, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments according to the specific circumstances.

[0048] Specific details are provided in the following description to provide a complete understanding of the exemplary embodiments. However, those skilled in the art will understand that the exemplary embodiments can be implemented without these specific details. For example, the system may be shown in block diagrams to avoid obscuring the example with unnecessary details. In other embodiments, well-known processes, structures, and techniques may be shown without non-essential details to avoid obscuring the embodiments.

[0049] Example 1:

[0050] This embodiment provides a video synthesis method based on images and audio, which can be applied to the server side of a corresponding video synthesis system. The video synthesis system includes a client and a server, and the client establishes a communication connection with the server, such as... Figure 1 As shown, the method includes the following steps:

[0051] S1. Obtain facial images and audio data provided by the client.

[0052] In practice, users can provide the server with a clear, unobstructed image of their face and an audio file of any length via the client.

[0053] S2. Detect the lips in the face image, and generate a set of mouth shapes for the pronunciation process of each phoneme based on the lip detection results. The set of mouth shapes for the pronunciation process contains several consecutive mouth shape images.

[0054] In practice, the server first performs lip detection on the face image to obtain lip detection results, which include lip images. Then, based on the lip images and pre-defined lip shape configuration parameters for each phoneme, a pre-defined dynamic expression generation component generates a set of lip shapes for the pronunciation process of each phoneme. The lip shape configuration parameters for each phoneme are derived from the classification of Chinese character pronunciation, as follows:

[0055] A a an ang ao;

[0056] B ba bai ban bang bao bei ben beng bi bian biao bie bin bing bo bu;

[0057] C ca cai can cang cao ce cen ceng cha chai chan chang chao che chencheng chi chong chou chu chuan chuang chui chun chuo ci cong cou cu cuan cuicun cuo chu chuan chuai;

[0058] D da dai dan dang dao de dei den deng di dia dian diao die ding diudong dou du duan dui dun duo;

[0059] E e ei en eng er;

[0060] F fa fan fang fei fen feng fo fou fu;

[0061] G ga gai gan gang gao ge gei gen geng gong gou gu gua guai guan guanggui gun guo;

[0062] H ha hai han hang hao he hei hen heng hong hou hu hua huai huanhuanghui hun huo;

[0063] J ji jia jian jiang jiao jie jin jing jiong jiu ju juan jue jun;

[0064] K ka kai kan kang kao ke kei ken keng kong kou ku kua kuai kuan kuangkui kun kuo;

[0065] L la lai lan lang lao le lei leng li lia lian liang liao lie lin lingliu lo long lou lu lu luan lue lun luo;

[0066] M ma mai man mang mao mei men meng mi mian miao mie min ming miu momou mu;

[0067] N na nai nan nang nei nen neng ni nia nian niang niao nie nin ningniu nong nou nu nu nuan nue nuo;

[0068] P pa pai pan pang pao pei pen peng pi pian piao pie pin ping po poupu;

[0069] Q qi qia qian qiang qiao qie qin qing qiong qiu qu quan que qun;

[0070] R ran rang rao re ren reng ri rong rou ru ruan rui run ruo;

[0071] S sa sai san sang sao se sen seng sha shai shan shang shao she sheishen sheng shi shou shu shua shuai shuan shuang shui shun shuo si song sou susuan sui sun suo;

[0072] T ta tai tan tang tao te teng ti tian tiao tie ting tong tou tu tuantui tun tuo;

[0073] W wa wai wan wang wei wen weng wo wu;

[0074] X xi xia xian xiang xiao xie xin xing xiong xiu xu xuan xue xun;

[0075] Y ya yan yang yao ye yi yin ying yong yo you yu yuan yue yun;

[0076] Z za zai zan zang zao ze zei zen zeng zha zhai zhan zhang zhao zhezhei zhen zheng zhi zhong zhou zhu zhua zhuai zhuan zhuang zhui zhun zhuo zizong zou zu zuan zui zun zuo.

[0077] All the above pronunciations can be roughly categorized as follows:

[0078] Labial consonants (all sounds with the initials b, p, and m) are pronounced by naturally closing the lips together and letting the airflow through. When producing sound, the lips naturally separate due to the airflow. The mouth shape transition during the pronunciation process is as follows: lips slightly open -> naturally closed -> small O-shaped mouth open - lips slightly open.

[0079] Labiodental consonants (all pronunciations of the initial consonant f) are produced by lightly touching the upper teeth with the lower lip, and the airflow is emitted between the lips and teeth. The mouth shape transition during the pronunciation process is as follows: lips slightly open -> lower lip lightly touches the upper teeth -> lips slightly open.

[0080] Apical consonants (all pronunciations of the initials l, n, d, t, except for the vowels o, u, uo, ü): When pronouncing these sounds, the tip of the tongue touches the upper gum or hard palate. The mouth shape transition during pronunciation is: lips slightly open -> open wider -> lips slightly open.

[0081] For palatal consonants (initials zh, ch, sh, r), the tongue is raised and brought close to the hard palate. The mouth shape transitions during pronunciation as follows: lips slightly open -> open wider -> lips slightly open.

[0082] For lingual consonants (initials z, c, s), the tip of the tongue touches the upper gum ridge, and the back of the tongue rises. The mouth shape transitions during the pronunciation process as follows: lips slightly open -> open wider -> lips slightly open.

[0083] For velar consonants (initials g, k, h), the back of the tongue touches the soft palate when pronouncing them, and the airflow is emitted through the space between the back of the tongue and the soft palate. The mouth shape transition during the pronunciation process is: lips slightly open -> open wider -> lips slightly open.

[0084] Alveolar consonants (initials j, q, x, y, i) are produced by placing the tip of the tongue against the upper gum ridge, with airflow passing between the tip of the tongue and the upper gum ridge. The mouth shape transition during pronunciation is as follows: lips slightly open -> open wider and lift the upper mouth shape slightly -> lips slightly open.

[0085] o, w, u, ü, uo (all consonants plus these vowel-like sounds) – when pronouncing these sounds, the mouth is rounded, the tongue is in the middle, the pronunciation time is long, and the mouth shape transitions during the pronunciation process as follows: lips slightly open -> wide open O mouth shape -> lips slightly open.

[0086] When pronouncing "e", the mouth is slightly open and the tongue is flat. The pronunciation is similar to "uh", but the mouth is opened wider. The mouth shape transition during the pronunciation process is: lips slightly open -> mouth opens wider -> lips slightly open.

[0087] Following this logic, based on the above general classification, the corresponding mouth shape configuration parameters for each phoneme are finally organized. For example, the vowel-consonant combinations of commonly used Chinese pronunciations can be defined. First, a list of mouth shape types containing commonly used Chinese pronunciations is created: pinyin_combinations = ['mouth shape 1', 'mouth shape 2', 'mouth shape 3', etc.]. These mouth shapes can be used to simulate the process of Chinese pronunciation.

[0088] S3. The mouth shape set of the pronunciation process of each phoneme is nested on the face image to generate a single phoneme video segment corresponding to each phoneme. The single phoneme video segment contains a number of pronunciation frames. The pronunciation frame is composed of a face image with the corresponding pronunciation mouth shape image nested in it.

[0089] In practice, the server can nest the mouth shape set of each phoneme's pronunciation process onto a face image, replace the lip area on the original face image, and generate a single phoneme video segment corresponding to each phoneme. The single phoneme video segment contains several pronunciation frames, and the pronunciation frame is composed of a face image with a nested corresponding pronunciation mouth shape image.

[0090] S4. Associate each single-phoneme video segment with its corresponding phoneme and store them in a phoneme segment dictionary.

[0091] In practice, the server associates each single-phoneme video segment with its corresponding phoneme and stores the information in a phoneme segment dictionary for subsequent use by the client in video synthesis. For example, an empty dictionary `pinyin_video_map` can be created.<key,value> The key stores elements from `pinyin_combinations`, and the value stores the corresponding pronunciation frame for that element, used to store the mapping relationship between vowels and consonants and their corresponding single-phoneme video segments. The `model.generateFrame(image,pinyin)` function is used to generate the pronunciation frame corresponding to each phoneme and store it in the variable `frame` to obtain the single-phoneme video segment. Then, the `frame` is added to the dictionary `pinyin_video_map`.

[0092] S5. Perform speech recognition processing on the audio data and extract the pronunciation structure data corresponding to the audio data. The pronunciation structure data includes a phoneme sequence composed of several continuous phonemes, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence.

[0093] In practice, the server can use the `preprocess_audio` function to preprocess the audio data, converting it into the feature representation required by the pre-defined speech recognition model. The `librosa.feature.melspec trogram` function extracts the Mel spectrum of the audio data, converting it into a time-frequency domain representation. The `librosa.power_to_db` function converts the Mel spectrum into a logarithmically scaled power spectrum, which better represents the energy distribution of the spectrum across different frequency ranges. The power spectrum features are then adjusted to the shape required by the speech recognition model, i.e., the corresponding Mel spectrum features. Two new dimensions can be added using `np.newaxis`, adjusting the feature shape to the form (1, 1, spectral feature dimension). The first dimension represents the number of samples (here, 1), the second dimension represents the number of channels (here, 1, because there is only one input feature channel), and the third dimension represents the spectral feature dimension. Finally, the Mel spectrum features of the audio data are input into the speech recognition model for speech recognition processing, obtaining the corresponding pronunciation structured data. The `exe.run` function can then be used to perform inference, obtaining the pronunciation structured data corresponding to the audio data. The structured pronunciation data includes a phoneme sequence consisting of several consecutive phonemes, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence.

[0094] S6. Transmit the phoneme fragment dictionary and the pronunciation structure data to the client, so that the client can retrieve the corresponding single-phoneme video fragments from the phoneme fragment dictionary according to the phoneme sequence, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence, splice and adjust them to obtain a synthesized video, and play the synthesized video synchronously with the audio data.

[0095] In practice, the server can transmit the dictionary `pinyin_video_map` to the client in JSON format and save it as a file on the client. The `jsonify` function can be used to convert the structured pronunciation data into JSON format and transmit it to the client. The client retrieves the corresponding single-phoneme video clips from the phoneme segment dictionary based on the phoneme sequence and uses the `concatenateVideoClips` function to concatenate them sequentially. The duration of each single-phoneme video clip is adjusted according to the duration of each phoneme in the phoneme sequence, and the transition duration between each single-phoneme video clip is adjusted according to the pause duration between adjacent phonemes in the phoneme sequence, resulting in a synthesized video. The specific process includes: using the last pronunciation frame of the monophone video segment corresponding to the preceding phoneme as a pause frame; synthesizing pause segments using these pause frames, the duration of which is the pause duration between the preceding and following phonemes; splicing the pause segments between the monophone video segments corresponding to the preceding and following phonemes to obtain a synthesized video; the mouth shape images corresponding to the starting and ending pronunciation frames of the monophone video segments are all set to a general half-open mouth shape. By using a general half-open mouth shape for pause transitions, the synthesized video can appear more realistic and coherent, ensuring that each pronunciation mouth shape image is continuous and avoiding any stuttering where the mouth closes after each word is spoken.

[0096] Alternatively, to make the synthesized video look more natural and smooth, the following steps S21-S24 can be used to achieve a smooth transition between individual phoneme video segments:

[0097] S21. Generate a set of transition mouth shapes between phonemes based on the lip position detection results. The set of transition mouth shapes contains several consecutive transition mouth shapes. The server can generate the set of transition mouth shapes between phonemes in the same way as in step S2 above.

[0098] S22. The lip-sync sets for the pronunciation transitions between each phoneme are nested onto face images to generate transition video segments between each phoneme. Each transition video segment contains several transition frames, and each transition frame is composed of face images with corresponding nested lip-sync shapes. The server can use the same method as in step S3 above to nest the lip-sync sets for the pronunciation transitions between each phoneme onto face images to generate transition video segments between each phoneme.

[0099] S23. Extract corresponding transition video segments based on the connection relationship between adjacent phonemes in the phoneme sequence, and store each transition video segment in the phoneme segment dictionary after associating it with its corresponding adjacent phonemes. After the server performs speech recognition on the audio data and obtains the corresponding pronunciation structured data, it can extract corresponding transition video segments based on the connection relationship between adjacent phonemes in the phoneme sequence, and synchronously store each extracted transition video segment in the phoneme segment dictionary after associating it with its corresponding adjacent phonemes.

[0100] S24. The phoneme segment dictionary and the pronunciation structured data are transmitted to the client, so that the client can retrieve the corresponding single-phoneme video segments from the phoneme segment dictionary according to the phoneme sequence, splice them sequentially, adjust the duration of each single-phoneme video segment according to the duration of each phoneme in the phoneme sequence, retrieve the corresponding transition video segments from the phoneme segment dictionary according to the connection relationship between adjacent phonemes in the phoneme sequence, splice them between the corresponding two single-phoneme video segments, and adjust the duration of the corresponding transition video segments according to the pause duration between two adjacent phonemes in the phoneme sequence to obtain the synthesized video.

[0101] After receiving the synthesized video, the client can synchronize the synthesized video with the audio data for playback. This process includes creating a Blob URL for the synthesized video and... <video>Playing within an element: Use URL.createObjectURL to convert the composite video into a Blob URL and assign that URL to <video>The element's src attribute is used, and then the play method is called to start playing the composite video.

[0102] This embodiment's method, based on lip extraction from facial images, can accurately simulate the user's mouth movements, improving the naturalness and realism of the synthesized video. By pre-generating single-phoneme video clips and pronunciation structured data on the backend, it can reduce the computational load and time for video synthesis on the user end, improve video synthesis efficiency and stability, and enhance the smoothness and presentation effect of the synthesized video, thus having broad application prospects.

[0103] Example 2:

[0104] This embodiment provides a video synthesis system based on images and audio, such as... Figure 2 As shown, it includes a data acquisition unit, a lip-sync generation unit, a segment generation unit, a dictionary construction unit, an audio recognition unit, and a synthesis execution unit, wherein:

[0105] The data acquisition unit is used to acquire facial images and audio data provided by the client.

[0106] The mouth shape generation unit is used to detect the lip area of ​​a face image and generate a mouth shape set for the pronunciation process of each phoneme based on the lip area detection results. The mouth shape set for the pronunciation process contains several consecutive mouth shape images.

[0107] The segment generation unit is used to nest the mouth shape set of the pronunciation process of each phoneme on a face image to generate a single phoneme video segment corresponding to each phoneme. The single phoneme video segment contains several pronunciation frames, and the pronunciation frame is composed of a face image nested with the corresponding pronunciation mouth shape image.

[0108] The dictionary construction unit is used to associate each single-phoneme video segment with its corresponding phoneme and store them in the phoneme segment dictionary;

[0109] An audio recognition unit is used to perform speech recognition processing on audio data and extract the pronunciation structure data corresponding to the audio data. The pronunciation structure data includes a phoneme sequence composed of several continuous phonemes, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence.

[0110] The synthesis execution unit is used to transmit the phoneme segment dictionary and the pronunciation structured data to the client, so that the client can retrieve the corresponding single phoneme video segments from the phoneme segment dictionary according to the phoneme sequence, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence, splice and adjust them to obtain the synthesized video, and play the synthesized video synchronously with the audio data.

[0111] Example 3:

[0112] This embodiment provides a video synthesis device based on images and voice, such as... Figure 3 As shown, at the hardware level, it includes:

[0113] The data interface is used to establish data communication between the processor and the client.

[0114] Memory, used to store instructions;

[0115] The processor is configured to read instructions stored in the memory and execute the video synthesis method in Embodiment 1 according to the instructions.

[0116] Optionally, the device also includes an internal bus. The processor, memory, and data interface can be interconnected via the internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0117] The memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0118] Example 4:

[0119] This embodiment provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the video synthesis method of Embodiment 1. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems.

[0120] This embodiment also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the video synthesis method of Embodiment 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system.

[0121] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / video> < / video>

Claims

1. A video synthesis method based on images and speech, characterized in that, include: Obtain facial images and audio data provided by the client; The lip region is detected in a face image, and a set of mouth shapes for the pronunciation process of each phoneme is generated based on the lip region detection results. The set of mouth shapes for the pronunciation process contains several consecutive mouth shape images. The pronunciation process of each phoneme is nested into a face image to generate a single phoneme video segment corresponding to each phoneme. The single phoneme video segment contains several pronunciation frames, and the pronunciation frame is composed of a face image with the corresponding pronunciation mouth shape image nested in it. Each single-phoneme video segment is associated with its corresponding phoneme and then stored in a phoneme segment dictionary; The audio data is processed by speech recognition to extract the corresponding pronunciation structure data. The pronunciation structure data includes a phoneme sequence composed of several continuous phonemes, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence. The phoneme segment dictionary and pronunciation structured data are transmitted to the client, so that the client can retrieve the corresponding single-phoneme video segments from the phoneme segment dictionary based on the phoneme sequence, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence, splice and adjust them to obtain a synthesized video, and play the synthesized video synchronously with the audio data.

2. The video synthesis method based on images and speech according to claim 1, characterized in that, The step of detecting the lip region in a facial image and generating a set of mouth shapes for the pronunciation process of each phoneme based on the lip region detection results includes: Perform lip detection on a face image to obtain lip detection results, wherein the lip detection results include lip images; Based on the lip area image and the preset mouth shape configuration parameters corresponding to each phoneme, a set of mouth shapes for the pronunciation process of each phoneme is generated using a preset dynamic expression generation component.

3. The video synthesis method based on images and speech according to claim 1, characterized in that, The process of performing speech recognition processing on the audio data and extracting the corresponding structured pronunciation data includes: The audio data is preprocessed to obtain the Mel-spectral characteristics of the audio data; The Mel-spectral features of the audio data are input into a pre-set speech recognition model for speech recognition processing to obtain the pronunciation structured data corresponding to the audio data.

4. The video synthesis method based on images and speech according to claim 1, characterized in that, The process of transmitting the phoneme fragment dictionary and pronunciation structured data to the client includes: The phoneme fragment dictionary and pronunciation structured data are converted into corresponding JSON format files and then transmitted to the client.

5. The video synthesis method based on images and speech according to claim 1, characterized in that, The process involves transmitting the phoneme segment dictionary and the phonetic structured data to the client, enabling the client to retrieve corresponding single-phoneme video segments from the phoneme segment dictionary based on the phoneme sequence, the duration of each phoneme in the phoneme sequence, and the pause duration between adjacent phonemes in the phoneme sequence. These segments are then spliced ​​and adjusted to obtain a synthesized video, including: The phoneme segment dictionary and the pronunciation structured data are transmitted to the client, so that the client can retrieve the corresponding single phoneme video segments from the phoneme segment dictionary according to the phoneme sequence, and concatenate them sequentially. The duration of each single phoneme video segment is adjusted according to the duration of each phoneme in the phoneme sequence, and the transition duration between each single phoneme video segment is adjusted according to the pause duration between two adjacent phonemes in the phoneme sequence, so as to obtain the synthesized video.

6. The video synthesis method based on images and speech according to claim 5, characterized in that, The lip-sync images corresponding to the starting and ending lip-sync frames of the monophone video segments are both set universal half-open lip-sync images. When the client adjusts the transition duration between each monophone video segment based on the pause duration between adjacent phonemes in the phoneme sequence, it also includes: The last sound frame of the monophone video segment corresponding to the previous phoneme is used as a pause frame. Pause frames are used to synthesize pause segments. The duration of the pause segment is the pause duration between the previous phoneme and the next phoneme. The pause segments are spliced ​​between the monophone video segments corresponding to the preceding phoneme and the monophone video segments corresponding to the following phoneme to obtain a composite video.

7. The video synthesis method based on images and speech according to claim 1, characterized in that, The method further includes: Based on the lip position detection results, a set of pronunciation transition mouth shapes is generated between each phoneme, and the set of pronunciation transition mouth shapes contains several consecutive transition mouth shapes; The transitional mouth shapes between each phoneme are nested on face images to generate transitional video segments between each phoneme. Each transitional video segment contains several transitional frames, and each transitional frame is composed of face images with corresponding transitional mouth shapes nested within them. Based on the connection relationship between adjacent phonemes in the phoneme sequence, the corresponding transition video segments are extracted, and each transition video segment is associated with its corresponding adjacent phonemes and stored in the phoneme segment dictionary. The phoneme segment dictionary and the pronunciation structured data are transmitted to the client, so that the client can retrieve the corresponding single-phoneme video segments from the phoneme segment dictionary according to the phoneme sequence and splice them sequentially. The duration of each single-phoneme video segment is adjusted according to the duration of each phoneme in the phoneme sequence. The corresponding transition video segment is retrieved from the phoneme segment dictionary according to the connection relationship between adjacent phonemes in the phoneme sequence and spliced ​​between the corresponding two single-phoneme video segments. The duration of the corresponding transition video segment is adjusted according to the pause duration between two adjacent phonemes in the phoneme sequence to obtain the synthesized video.

8. A video synthesis system based on images and speech, characterized in that, It includes a data acquisition unit, a lip-sync generation unit, a segment generation unit, a dictionary construction unit, an audio recognition unit, and a synthesis execution unit, wherein: The data acquisition unit is used to acquire facial images and audio data provided by the client. The mouth shape generation unit is used to detect the lip area of ​​a face image and generate a mouth shape set for the pronunciation process of each phoneme based on the lip area detection results. The mouth shape set for the pronunciation process contains several consecutive mouth shape images. The segment generation unit is used to nest the mouth shape set of the pronunciation process of each phoneme on a face image to generate a single phoneme video segment corresponding to each phoneme. The single phoneme video segment contains several pronunciation frames, and the pronunciation frame is composed of a face image nested with the corresponding pronunciation mouth shape image. The dictionary construction unit is used to associate each single-phoneme video segment with its corresponding phoneme and store them in the phoneme segment dictionary; An audio recognition unit is used to perform speech recognition processing on audio data and extract the pronunciation structure data corresponding to the audio data. The pronunciation structure data includes a phoneme sequence composed of several continuous phonemes, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence. The synthesis execution unit is used to transmit the phoneme segment dictionary and the pronunciation structured data to the client, so that the client can retrieve the corresponding single phoneme video segments from the phoneme segment dictionary according to the phoneme sequence, the duration of each phoneme in the phoneme sequence, and the pause duration between two adjacent phonemes in the phoneme sequence, splice and adjust them to obtain the synthesized video, and play the synthesized video synchronously with the audio data.

9. A video synthesis device based on images and speech, characterized in that, include: Memory, used to store instructions; A processor is configured to read instructions stored in the memory and execute the video synthesis method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the video synthesis method according to any one of claims 1-7.

Images