Video AI synthesis method and system for character and mouth shape matching
Through pronunciation feature analysis and database construction, the matching of text and mouth shape is achieved, the poor synchronization caused by polyphonic characters is solved, the video quality is improved, and the synthesis process is continuously optimized through user feedback, and the Chinese text and synthesis in the existing technology is solved. Polyphonic problems when matching lip shapes.
Patent Information
- Application Number
- CN202411163912.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art has failed to effectively solve the problem of polyphonic characters when matching Chinese characters with lip shapes, resulting in poor synchronization of Chinese characters, sound and picture in video synthesis.
Through pronunciation feature analysis, the pronunciation feature and semantic screening can be used to achieve matching text and lip shapes, solve the problem of multiphonic characters, and improve the video quality through post-processing, and use user feedback to update the synthesis process.
It achieves accurate matching of text and lips, solves the synchronization problem between text, sound and picture caused by polyphonic characters, improves the quality of video, and continuously optimizes the synthesis process through user feedback.
Smart Images

Figure CN119031202A_ABST
Abstract
Description
[0001] The present application relates to the field of video processing technology, and specifically to a video AI synthesis method and system for text and lip matching. Background Art
[0002] In the face of the current information explosion, people have accepted the use of short videos to transmit information. However, as a relatively difficult data type to modify, video recording requires a dedicated person to perform each video recording, which consumes labor and equipment costs. Therefore, it is necessary to study the method of matching text with lip movements to automatically generate corresponding videos.
[0003] Chinese patent number CN202311746822.X discloses a method for driving Chinese speech to accurately generate lip shape video in the field of digital humans, including: downloading the CMLR Chinese data set and preprocessing it to obtain the preprocessed CMLR Chinese data set; using the preprocessed CMLR Chinese data set as training data, training the wav2lip lip synchronization model according to the set training standard to obtain a new model; inputting the preprocessed video frame into the new model for lip reasoning, and outputting a new lip shape driven video. The present invention addresses the defect of poor support for Chinese speech in the existing model, and retrains it using the Chinese speech data set, so that the Chinese pronunciation lip shape is calibrated; avoiding the problem of the existing model opening and closing the lip shape too quickly when matching the Chinese speech. The invention fails to effectively solve the problem of polyphones in Chinese.
[0004] In summary, the prior art lacks a technical solution for video synthesis with text and lip matching. Summary of the invention
[0005] The purpose of this application is to provide a video AI synthesis method and system for text and lip matching to solve the technical problems raised in the above background technology.
[0006] To achieve the above objectives, this application discloses the following technical solutions:
[0007] In a first aspect, the present application discloses a video AI synthesis method for text and lip matching, the method comprising:
[0008] S1: Analyze the pronunciation characteristics of the input text content, where the pronunciation characteristics include phonemes, syllables and intonations;
[0009] S2: constructing a voice database, wherein the voice database includes various voice models and corresponding voice numbers;
[0010] S3: constructing a lip shape database, wherein the lip shape database includes various lip shape models and corresponding lip shape numbers;
[0011] S4: using the pronunciation features and the voice database, converting the text into video and sound data;
[0012] S5: using the sound data and the lip shape database, converting the text into video image data;
[0013] S6: Merge the sound data and the picture data to obtain a first video;
[0014] S7: performing post-processing on the first video to obtain a second video, and outputting the second video;
[0015] S8: Obtain the historical second video and the corresponding user feedback, and use the user feedback to update steps S1-S7.
[0016] Preferably, step S1 further comprises:
[0017] S101: Segment the text content into individual words;
[0018] S102: Divide the text content into a number of short sentences using punctuation marks;
[0019] S103: performing phoneme capture and syllable capture on the single word;
[0020] S104: Capturing the intonation of the short sentence based on the meaning of the punctuation marks and the semantics of the short sentence, wherein the intonation includes a rising tone, a falling tone, a flat tone and a melody, and updating the result of the phoneme capture and the result of the syllable capture in step S103;
[0021] S105: defining the result of the phoneme capture update, the result of the syllable capture update and the result of the intonation capture in step S104 as the pronunciation feature.
[0022] Preferably, step S2 further comprises:
[0023] The voice model is configured to store the sound data corresponding to the combination results of various phonemes, various syllables and various intonations of the single word; the voice number is Word j,k , wherein Word is the single word, j is the number of the single word in the speech database, and k is the number of the combination result of phoneme, syllable and intonation.
[0024] Preferably, step S3 further comprises:
[0025] The lip shape model is configured as follows: the picture data corresponding to the various lip shapes of the single word are stored; the lip shape number is MS j,k , wherein MS is the mouth shape.
[0026] Preferably, step S4 further comprises:
[0027] S401: Generate a first voice number matrix using the pronunciation features of the single word, and select k randomly when generating the first voice number matrix;
[0028] S402: Using the pronunciation features of the short sentence to update the first voice number matrix, the updating process is specifically as follows:
[0029] A1: parsing the pronunciation features of the short sentence to obtain the semantics of the short sentence;
[0030] A2: Update k using the semantics of the short sentence;
[0031] S403: Obtain a second voice number matrix based on the result of updating the first voice number matrix, and output the second voice number matrix as the video sound data.
[0032] Preferably, step S5 further comprises:
[0033] S501: parsing the video and sound data to obtain the second voice number matrix;
[0034] S502: parsing the second voice number matrix to obtain the corresponding voice number;
[0035] S503: Generate a lip shape number matrix using the correspondence between the voice number and the lip shape number;
[0036] S504: Output the lip number matrix as the video picture data.
[0037] Preferably, the speech model is further configured to store the sound duration of the sound data of the single word and represent it as Word j,k_SD , where SD represents the sound duration of the sound data of the single word with the voice number j,k;
[0038] The lip shape model is also configured to store the screen duration of the screen data of the lip shape and is represented as MS j,k_PD , where PD represents the screen duration of the lip shape screen data with voice number j,k;
[0039] Step S6 also includes:
[0040] S600: using the video sound data and the video picture data, respectively obtaining the corresponding sound duration and picture duration;
[0041] S601: compare the size of SD and PD, when SD>PD, execute step S602, when SD=PD, execute step S603, when SD<PD, execute step S604;
[0042] S602: Completing the video screen data so that the duration of the completed screen is equal to the duration of the sound. The process of completing the screen data is as follows:
[0043] B1: Split the picture data into several picture frames and obtain the picture frame duration;
[0044] B2: without changing the order of the picture frames, a blurred picture frame is inserted between any two picture frames except the first and last picture frames, and the duration of the blurred picture frame is equal to the duration of the picture frame;
[0045] B3: Calculate the picture duration of the picture data after the blurred picture frame is inserted, and when the completed picture duration is equal to the sound duration, output the completed video picture data and execute step S605;
[0046] S603: do nothing and go to step S605;
[0047] S604: Completing the video sound data so that the duration of the completed sound is equal to the duration of the picture. The process of completing the sound data is as follows:
[0048] C1: Split the sound data into several sound frames and obtain the duration of the sound frames;
[0049] C2: without changing the order of the sound frames, inserting fuzzy sound frames between any two sound frames except the first and last sound frames, and the duration of the fuzzy sound frames is equal to the duration of the sound frames;
[0050] C3: Calculate the sound duration of the sound data after the blurred sound frame is inserted, and when the completed sound duration is equal to the picture duration, output the completed video sound data and execute step S605;
[0051] S605: Fusing the sound data and the picture data of any one of the completed video picture data and the video sound data, the video picture data and the video sound data, and the video picture data and the completed video sound data based on the correspondence between the second speech label matrix and the lip shape label matrix to obtain a first video.
[0052] Preferably, step S7 further comprises:
[0053] S701: Performing sound smoothing processing on the sound data of the completed video, and performing picture smoothing processing on the picture data of the completed video;
[0054] S702: Update the first video using any one of the sound data of the completed video after sound smoothing and the picture data of the completed video after picture smoothing, define the updated result as the second video and output it.
[0055] Preferably, step S8 further comprises:
[0056] Obtain the second historical video and the corresponding user feedback, and use the user feedback to update steps S1-S7
[0057] S801: Pushing a user feedback information collection form to the user after the user watches the second video, wherein the user feedback information collection form is used to collect the problems of asynchronism or errors among words, sounds and pictures found by the user in the second video;
[0058] S802: receiving the problem of asynchronism or error among words, sounds and pictures found by the user in the second video, and replaying steps S1-S7 in the process of obtaining the second video, marking the problematic steps where errors occur;
[0059] S803: Update steps S1-S7 using the problem step.
[0060] In a second aspect, the present application discloses a video AI synthesis system for text and lip matching, the system is applicable to the video AI synthesis method for text and lip matching as described in any one of the above items, and the system comprises:
[0061] A pronunciation feature analysis module, wherein the pronunciation feature analysis module is configured to: perform pronunciation feature analysis on the input text content, wherein the pronunciation features include phonemes, syllables and intonations;
[0062] A voice database construction module, wherein the voice database construction module is configured to: construct a voice database, wherein the voice database includes various voice models and corresponding voice numbers;
[0063] A lip shape database construction module, wherein the lip shape database construction module is configured to: construct a lip shape database, wherein the lip shape database includes various lip shape models and corresponding lip shape numbers;
[0064] A video and sound data acquisition module, wherein the video and sound data acquisition module is configured to: convert text into video and sound data using the pronunciation features and the voice database;
[0065] A video screen data acquisition module, wherein the video screen data acquisition module is configured to: convert text into video screen data using the sound data and the lip shape database;
[0066] A first video acquisition module, wherein the first video acquisition module is configured to: merge the sound data and the picture data to obtain a first video;
[0067] A second video acquisition module, wherein the second video acquisition module is configured to: perform post-processing on the first video to obtain a second video, and output the second video;
[0068] An update module, wherein the update module is configured to obtain a historical second video and corresponding user feedback, and use the user feedback to update the above module.
[0069] Beneficial effects: The video AI synthesis method and system for matching text and lip shape of the present application provide a data basis for the screening of voice numbers and lip shape numbers through pronunciation feature analysis; by constructing a voice database and a lip shape database, it realizes the investigation of polyphones; through pronunciation features, a voice database, and a lip shape database, it realizes the matching of text and lip shape, and solves the synchronization between the text, sound and picture of polyphones through semantic screening; improves the quality of the video through post-processing; and realizes the update of the video AI synthesis process through user feedback. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0071] Figure 1 A flowchart of a video AI synthesis method for text and lip matching provided in an embodiment of the present application;
[0072] Figure 2 A schematic block diagram of a video AI synthesis system for text and lip matching provided in an embodiment of the present application. DETAILED DESCRIPTION
[0073] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.
[0074] In this article, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprising..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0075] The first aspect of this embodiment discloses Figure 1 A video AI synthesis method for text and lip matching is shown, the method comprising:
[0076] S1: Analyze the pronunciation characteristics of the input text content, and the pronunciation characteristics include phonemes, syllables and intonations;
[0077] S2: Build a voice database, which includes various voice models and corresponding voice numbers;
[0078] S3: construct a lip shape database, which includes various lip shape models and corresponding lip shape numbers;
[0079] S4: Using pronunciation features and voice database, convert text into video sound data;
[0080] S5: Convert text into video data using the sound data and lip shape database;
[0081] S6: Merge the sound data and the picture data to obtain a first video;
[0082] S7: performing post-processing on the first video to obtain a second video, and outputting the second video;
[0083] S8: Obtain the historical second video and the corresponding user feedback, and use the user feedback to update steps S1-S7.
[0084] The analysis of pronunciation features provides a data basis for the screening of voice numbers and lip shape numbers. By building a voice database and a lip shape database, polyphones are identified. Through pronunciation features, voice database, and lip shape database, the matching of text and lip shape is achieved, and the synchronization between text, sound, and picture of polyphones is solved through semantic screening. The quality of the video is improved through post-processing. The video AI synthesis process is updated through user feedback.
[0085] Specifically, step S1 also includes:
[0086] S101: Segment the text content into individual words;
[0087] S102: Divide the text content into a number of short sentences using punctuation marks;
[0088] S103: performing phoneme capture and syllable capture on a single word;
[0089] S104: Capturing the intonation of the short sentence based on the meaning of the punctuation marks and the semantics of the short sentence, the intonation includes rising tone, falling tone, flat tone and melody, and updating the result of the phoneme capture and the result of the syllable capture in step S103. The punctuation marks in this embodiment, for example, exclamation marks represent intonations such as surprise, so capturing the intonation includes using punctuation marks, and acquiring the semantics of the short sentence is any kind of semantic analysis in the existing natural language processing technology;
[0090] S105: The result of the phoneme capture update, the result of the syllable capture update and the result of the intonation capture in step S104 are defined as pronunciation features.
[0091] Specifically, step S2 also includes:
[0092] The voice model is configured as follows: it stores the sound data corresponding to the combination of various phonemes, syllables and intonations of a single word; the voice number is Word j,k , where Word is a single word, j is the number of the single word in the speech database, and k is the number of the combination of phonemes, syllables and intonations. This step realizes the acquisition of various pronunciations of polyphones.
[0093] Specifically, step S3 also includes:
[0094] The configuration of the lip shape model is as follows: the picture data corresponding to various lip shapes of single words are stored; the lip shape number is MS j,k , where MS is the mouth shape.
[0095] Specifically, step S4 also includes:
[0096] S401: Generate a first voice number matrix using pronunciation features of a single word, and select k randomly when generating the first voice number matrix;
[0097] S402: Using the pronunciation features of the short sentence to update the first voice number matrix, the updating process is specifically as follows:
[0098] A1: Analyze the pronunciation features of short sentences to obtain the semantics of short sentences;
[0099] A2: Update k using the semantics of the short sentence. This step enables the determination of the pronunciation of polyphonetic characters in a specific language environment.
[0100] S403: Obtain a second voice number matrix based on the result of updating the first voice number matrix, and output the second voice number matrix as video sound data.
[0101] Specifically, step S5 also includes:
[0102] S501: parsing video and sound data to obtain a second voice number matrix;
[0103] S502: parsing the second voice number matrix to obtain corresponding voice numbers;
[0104] S503: Generate a lip number matrix using the correspondence between the voice number and the lip number. In this embodiment, the lip number and the voice number use the same numbering logic, which effectively improves the accuracy of the matching process.
[0105] S504: Output the lip number matrix as video picture data.
[0106] Specifically, the speech model is also configured to store the sound duration of the sound data of a single word and represent it as Word j,k_SD , where SD represents the sound duration of the sound data of the single word with the voice number j,k;
[0107] The lip model is also configured to store the screen duration of the lip-shaped screen data and is represented as MS j,k_PD , where PD represents the screen duration of the lip shape screen data with voice number j,k;
[0108] Step S6 also includes:
[0109] S600: using the video sound data and the video picture data, respectively obtaining the corresponding sound duration and picture duration;
[0110] S601: compare the size of SD and PD, when SD>PD, execute step S602, when SD=PD, execute step S603, when SD<PD, execute step S604;
[0111] S602: Completing the video picture data so that the duration of the completed picture is equal to the duration of the sound. The specific process of completing the picture data is as follows:
[0112] B1: Split the picture data into several picture frames and obtain the picture frame duration;
[0113] B2: The order of the picture frames is not changed, and blurred picture frames are inserted between any two picture frames except the first and last picture frames, and the duration of the blurred picture frames is equal to the duration of the picture frames;
[0114] B3: Calculate the picture duration of the picture data after the blurred picture frame is inserted. When the completed picture duration is equal to the sound duration, output the completed video picture data and execute step S605. It should be noted that the insertion of the blurred picture frame in this embodiment does not affect the picture quality of the output second video. The acquisition of the blurred picture frame is any one of the video frame insertion technologies in the existing video processing technology.
[0115] S603: do nothing and go to step S605;
[0116] S604: Completing the audio data of the video so that the duration of the completed audio is equal to the duration of the picture. The specific process of completing the audio data is as follows:
[0117] C1: Split the sound data into several sound frames and obtain the duration of the sound frames;
[0118] C2: The order of the sound frames is not changed, and a fuzzy sound frame is inserted between every two sound frames except the first and last sound frames, and the duration of the fuzzy sound frame is equal to the duration of the sound frame;
[0119] C3: Calculate the sound duration of the sound data after the blurred sound frame is inserted. When the sound duration after completion is equal to the picture duration, output the completed video sound data and execute step S605. It should be noted that the insertion of the blurred sound frame in this embodiment does not affect the sound quality of the output second video. The acquisition of the blurred sound frame is any one of the audio frame insertion technologies in the existing audio processing technology.
[0120] S605: Fusing the sound data and the picture data of any one of the completed video picture data and video sound data, the video picture data and video sound data, and the video picture data and the completed video sound data based on the correspondence between the second voice label matrix and the lip shape label matrix to obtain the first video.
[0121] Specifically, step S7 also includes:
[0122] S701: Performing sound smoothing processing on the sound data of the completed video, and performing picture smoothing processing on the picture data of the completed video;
[0123] S702: Update the first video using any one of the sound data of the completed video after sound smoothing and the picture data of the completed video after picture smoothing, define the updated result as the second video and output it.
[0124] Specifically, step S8 also includes:
[0125] Get the second historical video and the corresponding user feedback, and use the user feedback to update steps S1-S7
[0126] S801: Pushing a user feedback information collection form to the user after the user watches the second video, where the user feedback information collection form is used to collect the problems of asynchronism or errors between words, sounds and pictures found by the user in the second video;
[0127] S802: receiving the problem of asynchrony or error between words, sounds and pictures found by the user in the second video, and replaying steps S1-S7 in the process of obtaining the second video, marking the problematic steps where errors occur;
[0128] S803: Steps S1-S7 are updated using the problem steps. It should be noted that the present embodiment presets the judgment of polyphones that are over-scaled in different places in step S2. However, based on the characteristics of Chinese polyphones, relying on the preset to realize the judgment of exhaustive polyphones is not of practical significance. Therefore, the judgment of polyphones is accumulated through continuous feedback in the actual operation process.
[0129] The second aspect of this embodiment discloses Figure 2 A video AI synthesis system for text and lip matching is shown, the system is applicable to the video AI synthesis method for text and lip matching as described in any one of the above items, and the system includes:
[0130] The pronunciation feature analysis module is configured to: perform pronunciation feature analysis on the input text content, and the pronunciation features include phonemes, syllables and intonations;
[0131] The voice database construction module is configured to: construct a voice database, the voice database includes various voice models and corresponding voice numbers;
[0132] A lip shape database construction module, the lip shape database construction module is configured to: construct a lip shape database, the lip shape database includes various lip shape models and corresponding lip shape numbers;
[0133] The video and sound data acquisition module is configured to: convert text into video and sound data using pronunciation features and a voice database;
[0134] The video screen data acquisition module is configured to: convert text into video screen data using sound data and lip shape database;
[0135] A first video acquisition module, the first video acquisition module is configured to: merge the sound data and the picture data to obtain a first video;
[0136] A second video acquisition module, the second video acquisition module is configured to: perform post-processing on the first video to obtain a second video, and output the second video;
[0137] Update module, the update module is configured to: obtain the historical second video and the corresponding user feedback, and use the user feedback to update the above module.
[0138] In summary, the video AI synthesis method and system for matching text and lip shape of the present embodiment provides a data basis for the screening of voice numbers and lip shape numbers through pronunciation feature analysis; by constructing a voice database and a lip shape database, it realizes the investigation of polyphones; through pronunciation features, a voice database, and a lip shape database, it realizes the matching of text and lip shape, and solves the synchronization between the text, sound and picture of polyphones through semantic screening; improves the quality of the video through post-processing; and realizes the update of the video AI synthesis process through user feedback.
[0139] In the embodiments provided in the present application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code or any appropriate combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: application specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein or their combination. For software implementation, part or all of the flow of the embodiment can be completed by instructing the relevant hardware through a computer program. When implemented, the above program can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a computer can access. The computer-readable storage medium can include but is not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer.
[0140] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A video AI synthesis method for text and lip matching, characterized in that: The method includes: S1: Analyze the pronunciation characteristics of the input text content, where the pronunciation characteristics include phonemes, syllables and intonations; S2: constructing a voice database, wherein the voice database includes various voice models and corresponding voice numbers; S3: constructing a lip shape database, wherein the lip shape database includes various lip shape models and corresponding lip shape numbers; S4: using the pronunciation features and the voice database, converting the text into video and sound data; S5: using the sound data and the lip shape database, converting the text into video image data; S6: Merge the sound data and the picture data to obtain a first video; S7: performing post-processing on the first video to obtain a second video, and outputting the second video; S8: Obtain the historical second video and the corresponding user feedback, and use the user feedback to update steps S1-S7.
2. The video AI synthesis method for text and lip matching according to claim 1, characterized in that: Step S1 also includes: S101: Segment the text content into individual words; S102: Divide the text content into a number of short sentences using punctuation marks; S103: performing phoneme capture and syllable capture on the single word; S104: Capturing the intonation of the short sentence based on the meaning of the punctuation marks and the semantics of the short sentence, wherein the intonation includes a rising tone, a falling tone, a flat tone and a melody, and updating the result of the phoneme capture and the result of the syllable capture in step S103; S105: defining the result of the phoneme capture update, the result of the syllable capture update and the result of the intonation capture in step S104 as the pronunciation feature.
3. The video AI synthesis method for text and lip matching according to claim 2, characterized in that: Step S2 also includes: The voice model is configured to store the sound data corresponding to the combination results of various phonemes, various syllables and various intonations of the single word; the voice number is Word j,k , wherein Word is the single word, j is the number of the single word in the speech database, and k is the number of the combination result of phoneme, syllable and intonation.
4. The video AI synthesis method for text and lip matching according to claim 3, characterized in that: Step S3 also includes: The lip shape model is configured as follows: the picture data corresponding to the various lip shapes of the single word are stored; the lip shape number is MS j,k , wherein MS is the mouth shape.
5. The video AI synthesis method for text and lip matching according to claim 4, characterized in that: Step S4 also includes: S401: Generate a first voice number matrix using the pronunciation features of the single word, and select k randomly when generating the first voice number matrix; S402: Using the pronunciation features of the short sentence to update the first voice number matrix, the updating process is specifically as follows: A1: parsing the pronunciation features of the short sentence to obtain the semantics of the short sentence; A2: Update k using the semantics of the short sentence; S403: Obtain a second voice number matrix based on the result of updating the first voice number matrix, and output the second voice number matrix as the video sound data.
6. The video AI synthesis method for text and lip matching according to claim 5, characterized in that: Step S5 also includes: S501: parsing the video and sound data to obtain the second voice number matrix; S502: parsing the second voice number matrix to obtain the corresponding voice number; S503: Generate a lip shape number matrix using the correspondence between the voice number and the lip shape number; S504: Output the lip number matrix as the video picture data.
7. The video AI synthesis method for text and lip matching according to claim 6, characterized in that: The speech model is also configured to store the sound duration of the sound data of the single word and represent it as Word j,k_SD , where SD represents the sound duration of the sound data of the single word with the voice number j,k; The lip shape model is also configured to store the screen duration of the screen data of the lip shape and is represented as MS j,k_PD , where PD represents the screen duration of the lip shape screen data with voice number j,k; Step S6 also includes: S600: using the video sound data and the video picture data, respectively obtaining the corresponding sound duration and picture duration; S601: compare the size of SD and PD, and when SD>PD, execute step S602; when SD=PD, execute step S603; when SD<PD, execute step S604; S602: Completing the video picture data so that the duration of the completed picture is equal to the duration of the sound. The process of completing the picture data is as follows: B1: Split the picture data into several picture frames and obtain the picture frame duration; B2: without changing the order of the picture frames, a blurred picture frame is inserted between any two picture frames except the first and last picture frames, and the duration of the blurred picture frame is equal to the duration of the picture frame; B3: Calculate the picture duration of the picture data after the blurred picture frame is inserted, and when the completed picture duration is equal to the sound duration, output the completed video picture data and execute step S605; S603: Do nothing and go to step S605; S604: Completing the video sound data so that the duration of the completed sound is equal to the duration of the picture. The process of completing the sound data is as follows: C1: Split the sound data into several sound frames and obtain the duration of the sound frames; C2: without changing the order of the sound frames, inserting fuzzy sound frames between any two sound frames except the first and last sound frames, and the duration of the fuzzy sound frames is equal to the duration of the sound frames; C3: Calculate the sound duration of the sound data after the blurred sound frame is inserted, and when the completed sound duration is equal to the picture duration, output the completed video sound data and execute step S605; S605: Fusing the sound data and the picture data of any one of the completed video picture data and the video sound data, the video picture data and the video sound data, and the video picture data and the completed video sound data based on the correspondence between the second speech label matrix and the lip shape label matrix to obtain a first video.
8. The video AI synthesis method for text and lip matching according to claim 7, characterized in that: Step S7 also includes: S701: Performing sound smoothing processing on the sound data of the completed video, and performing picture smoothing processing on the picture data of the completed video; S702: Update the first video using any one of the sound data of the completed video after sound smoothing and the picture data of the completed video after picture smoothing, define the updated result as the second video and output it.
9. The video AI synthesis method for text and lip matching according to claim 8, characterized in that: Step S8 also includes: Obtain the second historical video and the corresponding user feedback, and use the user feedback to update steps S1-S7 S801: Pushing a user feedback information collection form to the user after the user watches the second video, wherein the user feedback information collection form is used to collect the problems of asynchronism or errors among words, sounds and pictures found by the user in the second video; S802: receiving the problem of asynchronism or error among words, sounds and pictures found by the user in the second video, and replaying steps S1-S7 in the process of obtaining the second video, marking the problematic steps where errors occur; S803: Update steps S1-S7 using the problem step.
10. A video AI synthesis system for text and lip matching, the system being applicable to the video AI synthesis method for text and lip matching as claimed in any one of claims 1 to 9, characterized in that: The system includes: A pronunciation feature analysis module, wherein the pronunciation feature analysis module is configured to: perform pronunciation feature analysis on the input text content, wherein the pronunciation features include phonemes, syllables and intonations; A voice database construction module, wherein the voice database construction module is configured to: construct a voice database, wherein the voice database includes various voice models and corresponding voice numbers; A lip shape database construction module, wherein the lip shape database construction module is configured to: construct a lip shape database, wherein the lip shape database includes various lip shape models and corresponding lip shape numbers; A video and sound data acquisition module, wherein the video and sound data acquisition module is configured to: convert text into video and sound data using the pronunciation features and the voice database; A video screen data acquisition module, wherein the video screen data acquisition module is configured to: convert text into video screen data using the sound data and the lip shape database; A first video acquisition module, wherein the first video acquisition module is configured to: merge the sound data and the picture data to obtain a first video; A second video acquisition module, wherein the second video acquisition module is configured to: perform post-processing on the first video to obtain a second video, and output the second video; An update module, wherein the update module is configured to obtain a historical second video and corresponding user feedback, and use the user feedback to update the above module.
Citation Information
Patent Citations
Live broadcast method and device based on artificial intelligence, equipment and storage medium
CN111010586A
Virtual object control method and device, electronic equipment and medium
CN113538636A
Mouth shape image generation method and device
CN117372588A
Mobile communication terminal for outputting voice received text message to voice using avatar and Method thereof
KR1020090051851A
