A method and system for synthesizing virtual human video clips for intelligent video generation
By establishing classification models and multimodal data generation technology, the problem of insufficient authenticity and emotional expression in virtual human video clip synthesis is solved, efficient and low-cost virtual human video generation is achieved, and the authenticity of virtual human videos is improved and Emotional expression.
Patent Information
- Application Number
- CN202411107818.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-13
AI Technical Summary
In the synthesis of virtual human video clips, it is difficult to effectively improve the authenticity and emotional expression of virtual human videos, and traditional video production requires a lot of post-editing work, resulting in inefficiency and high cost.
By establishing a classification model based on text density, role interaction and emotional richness, we predict whether video storyboard scripts are suitable for using virtual human broadcasts, and use multimodal input data to generate virtual human broadcasts, combining matching video clips in the database Merge into a full video.
It improves the authenticity of virtual human video synthesis and the accuracy of emotional expression, reduces the workload of post-editing, improves efficiency and reduces costs.
Smart Images

Figure CN118660117B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent video generation, and in particular to a method and system for synthesizing virtual human video clips for intelligent video generation. Background Art
[0002] With the rapid development of computer vision, deep learning and artificial intelligence technology, the generation and interaction of virtual characters has become a hot research field. The core of this technology is to automatically create realistic virtual characters through algorithms and enable these characters to show natural behaviors and movements in videos. Traditional video production requires a lot of post-editing work, including actor performances, shooting, editing and other steps, while virtual human video clip synthesis technology can achieve these functions through computational means, greatly improving efficiency and reducing costs. The development of these technologies has made virtual people have broad application prospects in entertainment, education, customer service and other fields. For example, online education platforms can use virtual teachers to provide personalized teaching; e-commerce platforms can use virtual models to display products; social software can develop virtual avatars to enhance users' immersive experience, etc. With the advancement of technology, the virtual human video clip synthesis method and system will be further improved to provide users with a richer and more diverse digital experience. Summary of the invention
[0003] The present invention provides a method for synthesizing virtual human video clips for intelligent video generation, comprising:
[0004] S110, the user inputs video requirement parameters, and a storyboard segment of the video is generated according to the parameters;
[0005] S120, establishing a training classification model to predict whether a storyboard script needs to be reported by a virtual human;
[0006] S130, for a storyboard script with a virtual person broadcasting tag, generating a virtual person broadcasting segment;
[0007] S140, for a storyboard without a virtual person broadcasting tag, searching a database for a matching video clip;
[0008] S150: Merge the virtual human broadcast segment and the matching video segment into a complete video.
[0009] A method for synthesizing virtual human video clips for intelligent video generation as described above is characterized in that it predicts whether a video storyboard script is suitable for virtual human broadcasting, and quantifies it from three aspects: text density, character interaction, and emotional richness. Text density refers to the number of words of dialogue or narration per minute in the script, character interaction refers to the frequency of interaction between characters in the script, and emotional richness refers to the number and intensity of emotional types expressed in the script. For text density and character interaction, high values mean that more information transmission and interaction are required, which is more suitable for virtual human broadcasting. For emotional richness, low values are more suitable for virtual human broadcasting.
[0010] The method for synthesizing virtual human video clips for intelligent video generation as described above is characterized in that the method for establishing the classification training model specifically includes the following sub-steps:
[0011] Calculate text density based on storyboard script data;
[0012] Calculate the degree of character interaction based on storyboard script data;
[0013] Calculate the emotional richness based on the storyboard script data;
[0014] The comprehensive probability value is calculated according to the text density, character interaction and emotional richness, and the training classification model is established based on the comprehensive probability value.
[0015] The method for synthesizing virtual human video clips for intelligent video generation as described above is characterized in that the method for generating virtual human broadcast clips specifically includes the following sub-steps:
[0016] Use multimodal input data to provide a data source for the generator;
[0017] Generate virtual human broadcast segments based on multimodal input data;
[0018] Use a multi-scale discriminator to test the generated virtual human broadcast clips.
[0019] The method for synthesizing virtual human video clips for intelligent video generation as described above is characterized in that the method for searching for matching video clips specifically includes the following sub-steps:
[0020] Conduct text analysis on storyboards to extract key information;
[0021] Generate a series of features based on key information;
[0022] Using features as query conditions, we search in a database containing rich video clips, and use feature matching algorithms to find video clips that match the script content.
[0023] The present invention also provides a virtual human video clip synthesis system for intelligent video generation, including: an input and generation storyboard module, a classification model training module, a virtual human broadcast clip generation module, a matching video clip generation module, and a merging module.
[0024] Input and generate storyboard module: the user inputs the video requirement parameters and generates the storyboard segments of the video according to the parameters;
[0025] Classification model training module: establish a training classification model to predict whether the storyboard script needs to be reported by a virtual human;
[0026] Virtual human broadcast segment generation module: for storyboards with virtual human broadcast tags, generate virtual human broadcast segments;
[0027] Matching video clip generation module: for storyboards without virtual human broadcasting tags, search for matching video clips in the database;
[0028] Merging module: merges the virtual human broadcast segment and the matching video segment into a complete video.
[0029] The virtual human video segment synthesis system for intelligent video generation as described above is characterized in that the method for generating the virtual human broadcast segment specifically includes the following sub-steps:
[0030] Use multimodal input data to provide a data source for the generator;
[0031] Generate virtual human broadcast segments based on multimodal input data;
[0032] Use a multi-scale discriminator to test the generated virtual human broadcast clips.
[0033] The beneficial effects achieved by the present invention are as follows: through intelligent video generation technology, the authenticity of virtual human video synthesis and the accuracy of emotional expression are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0035] Figure 1 This is a flow chart of a method for synthesizing virtual human video clips for intelligent video generation provided in Example 1 of the present application;
[0036] Figure 2This is a schematic diagram of a virtual human video clip synthesis system for intelligent video generation provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION
[0037] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention. Embodiment 1
[0038] like Figure 1 As shown, Embodiment 1 of the present application provides a method for synthesizing virtual human video clips for intelligent video generation.
[0039] Step S110: The user inputs video requirement parameters, and generates storyboard segments of the video according to the parameters;
[0040] The user inputs the video requirement parameters, including video theme, duration, key content points, etc., and uses the large language model to generate the video storyboard. First, the overall structure and main content of the video are determined according to the input parameters, and then the video duration is evenly distributed to each scene to generate the time parameter. Secondly, the scene description parameters are combined to generate a specific description of each scene. Finally, the storyboard containing the scene description and time nodes is output. Specifically, the formula is used: Generate a storyboard, where Represents the generated storyboard script. represents the large language model algorithm, Represents a set of input parameters, including video topics, key content points, etc. Indicates the video duration. Represents scene description parameters, such as the scene's environment, roles, and actions. Indicates the start and end time of each scene.
[0041] Step S120: Establishing a training classification model to predict whether a storyboard script needs to be reported by a virtual human;
[0042] Predict whether a video storyboard is suitable for virtual human broadcasting. The three aspects of text density, character interaction and emotional richness are quantified. Text density refers to the number of words per minute in the script for dialogue or narration. Character interaction refers to the frequency of interaction between characters in the script, such as the number of dialogue turns or action interactions. Emotional richness refers to the number and intensity of emotional types expressed in the script. For text density and character interaction, high values mean that more information transmission and interaction are required, which is more suitable for virtual human broadcasting. For emotional richness, low values are more suitable for virtual human broadcasting because it is difficult for virtual humans to fully capture and express complex emotions. The specific sub-steps include the following:
[0043] Step S121: Calculating text density according to storyboard script data;
[0044] Text density is calculated by calculating the importance of each word in the storyboard data, then summing the importance of all words and comparing it with the total length of the video. The importance of words is calculated by word frequency and inverse document frequency. Word frequency refers to the frequency of a word appearing in a text. The more frequently a word appears in a text, the more important it is to the content of the text. Inverse document frequency is used to measure the general importance of a word. If a word appears in many texts, its distinguishing ability is poor and its inverse document frequency value is low. On the contrary, if a word appears in only a few texts, its inverse document frequency is high. Specifically, the formula is used: Calculate the text density, where represents the text density of the j-th storyboard, represents the number of occurrences of word i in the text of the j-th storyboard, represents the total number of words in the text of the j-th storyboard, Indicates the total number of storyboards, j indicates the subscript of the storyboard, and the value of j is 1-M, i indicates the subscript of the word in the text of the current storyboard, and the value of i is 1-n, It means counting the total number of words i in the text of all storyboards. Represents the total video length of the j-th storyboard.
[0045] Step S122: Calculating the role interaction degree according to the storyboard script data;
[0046] The role interaction is calculated by analyzing the number and type of dialogues between characters in the storyboard. The dialogue turns, the number of times a character mentions other characters, and the direct action interaction parameter data between characters are extracted from the storyboard. Specifically, the formula is used: Calculate the role interaction, where represents the role interaction degree of the j-th storyboard, represents the total number of dialogue turns of character i in the jth storyboard, represents the total number of dialogue turns of character k in the jth storyboard, It is the number of times role i mentions other role k. represents the number of direct action interactions performed by character i on character j, They represent the conversation rounds, the number of times a character mentions other characters, and the weights of the direct action interaction parameters between characters. It represents the total video duration of the j-th storyboard. i and k represent the subscripts of the characters, with values ranging from 1 to n.
[0047] Step S123: Calculating the emotional richness according to the storyboard script data;
[0048] The calculation of emotional richness involves text sentiment analysis and consideration of the diversity and intensity of emotions depicted in the video script. Emotional richness is expressed as the diversity of the types and degrees of different emotions expressed in the script. According to the results of sentiment analysis, emotions are divided into multiple categories. Positive emotions can be subdivided into happiness, excitement, hope, etc.; negative emotions can be subdivided into sadness, anger, fear, etc. The emotional richness is calculated by counting the frequency and intensity of each emotion category. Specifically, the formula is used: Calculate the emotional richness, where represents the emotional richness of the j-th storyboard, represents the i-th emotion category of the j-th storyboard, represents the probability of the occurrence of the i-th emotion, represents the average intensity of the i-th emotion, by Multiply by the sentiment probability An index that accurately reflects the diversity and intensity of emotions in the script is obtained. The higher the emotional intensity and the more evenly distributed the emotions are, the greater the emotional richness is. Conversely, if only one or a few emotions are dominant, even if there are many types of emotions, their intensity is generally low, and the emotional richness is relatively small.
[0049] Step S124: calculating a comprehensive probability value according to the text density, the role interaction degree and the emotion richness, and establishing a training classification model according to the comprehensive probability value;
[0050] The comprehensive probability value is calculated based on the text density, role interaction and emotional richness. Specifically, the formula is used: Calculate the comprehensive probability value, where represents the comprehensive probability value of the j-th storyboard script, represents the normalization function, , , Represent the text density, character interaction and emotional richness of the j-th storyboard, respectively. , , They respectively represent the influencing factors of the text density, character interaction and emotional richness of the j-th storyboard.
[0051] Until the comprehensive probability values of all storyboard scripts are calculated, each video script with a comprehensive probability value exceeding 0.5 is marked with a label that requires virtual human broadcasting, and the training classification model is completed.
[0052] Step S130: for the storyboard script with the virtual person broadcasting tag, generate a virtual person broadcasting segment;
[0053] For the storyboard segments that the classification model predicts need to be broadcast by a virtual person, a virtual person broadcast segment is generated. The virtual person broadcast segment includes audio (voice content, sound type, speaking speed, intonation), picture (facial expression, lip shape, clothing appearance) and action. It is divided into three modules: multimodal input, generator and discriminator, which respectively realize the functions of storyboard data preprocessing, virtual person generation and result verification. The method for generating virtual person broadcast segments specifically includes the following sub-steps:
[0054] Step S131: using multimodal input data to provide a data source for the generator;
[0055] In order to improve the emotional expression and accuracy of the generated images, a multimodal input data method is adopted, which includes audio-driven data, picture features and action features. By analyzing the text emotion, tone, speaking speed and other information of the storyboard script, the text is converted into a speech signal. In this process, the appropriate voice type can be selected, and the speaking speed and tone can be adjusted as needed. The emotional state of the character can be inferred based on the text content, and then the audio-driven data can be obtained; facial images that meet specific emotions and expressions are generated according to the emotional state, corresponding lip animations are generated according to the voice content, and specific clothing appearances are generated according to the description in the script, so as to obtain picture features; corresponding behavioral actions are generated through text descriptions, ensuring that these actions are coordinated with the voice content and facial expressions, and then the action features are obtained. Specifically, the formula is used: Represents multimodal input data, where represents a multimodal input data set, Represents audio driver data, Indicates the picture feature data, Represents action feature data.
[0056] Step S132: Generate a virtual person broadcast segment based on the multi-modal input data;
[0057] In order to improve the expressiveness of the virtual person, a variety of technologies and methods are used to optimize its voice type, speech rate, tone and emotional expression, facial expression and posture, lip synchronization, and clothing and appearance customization. First, the audio driving data is analyzed through a deep learning model to extract and classify different voice type features, including timbre (such as male, female, child, etc.), pitch and sound quality. Then, the virtual person's speech rate is adjusted according to the scene requirements and user preferences, and a prediction model is established by learning the speech rate distribution in a large number of broadcast videos. In addition, an emotion recognition model is used to extract emotional features from speech data to generate intonation changes under different emotional states, such as happiness, sadness or anger. At the same time, computer vision technology is used to analyze facial expressions and posture changes in video data, and corresponding animation models are generated to enhance the visual effect during broadcasting and ensure that the virtual person's lip shape is synchronized with the generated speech. Finally, through image generation technology and user input, the virtual person's clothing and appearance can be customized according to the scene and user preferences.
[0058] The audio driving data in the multimodal input data is converted into speech using text-to-speech technology. Specifically, the formula is adopted: Generate speech, where stands for text-to-speech technology. Represents audio driver data, Represents the generated speech; the image feature data in the multimodal input data is converted into a picture using image generation technology. Specifically, the formula is used: Generate a screen, where represents image generation technology, Indicates the picture feature data, Represents the generated picture; uses artificial intelligence action recognition technology to convert the action feature data in the multimodal input data into actions. Specifically, the formula is used: Generate actions where It represents artificial intelligence action recognition technology. Represents action feature data, Indicates the generated action and the generated speech , screen ,action Composition of virtual human broadcast segments .
[0059] Step S133: using a multi-scale discriminator to test the generated virtual human broadcast segment;
[0060] A multi-scale discriminator is introduced to judge the authenticity of the generated virtual human broadcast clips at different scales, thereby improving the detail fidelity of the generated images. Three discriminators of different scales are defined, including mouth shape, facial expression, and key point alignment. The tasks of the three discriminators are to distinguish between real images and generated images, to distinguish between local areas and global images, and to distinguish between image contrast and hue. Specifically, the formula is used: To test, represents the loss function, Indicates a segment broadcast by a virtual person. They represent three discriminators: mouth shape, facial expression, and key point alignment. It means to expect the joint distribution of the real image x and the generated image y. Representation Discriminator The judgment of the input images x and y returns a real value between 0 and 1, indicating the discriminator The probability that the input images x and y match is If it is close to 1, it means that the discriminator thinks that the pair of inputs is matched, and the loss function value of the first term is small. It means to find the expectation of the joint distribution of the real image x and random noise z. Representation Discriminator For the input image x and the random image The judgment, when When it is close to 1, it means that the discriminator mistakenly generates random images. It is judged to be a real image. At this time, the loss function value of the second item is large. By minimizing the value of the loss function, the virtual human broadcast segment is gradually improved. quality, making it increasingly difficult for the discriminator to distinguish.
[0061] Step S140: For the storyboard without the virtual person broadcasting tag, searching for a matching video clip in the database;
[0062] For storyboards without the virtual person announcement tag, search for matching video clips in the database using the following steps:
[0063] 1. Perform text analysis on the storyboard script to extract key information, such as emotional state, scene description, dialogue content, etc.
[0064] 2. Generate a series of features based on these key information, such as emotional features, scene features, dialogue features, etc.
[0065] 3. Using these features as query conditions, search in a database containing rich video clips, and use feature matching algorithms to find video clips that match the script content. Specifically, the formula is used: Perform feature matching, where represents the feature matching algorithm, Represents a collection of storyboard features, Represents a collection of storyboard features The i-th eigenvector in , Represents the feature set of database video material, Represents the feature set of database video material The i-th eigenvector of , i is a subscript with a value of 1-n, and n is the number of eigenvectors in the storyboard feature set. Represented as the final matched video segment.
[0066] Step S150: merging the virtual person broadcast segment and the matching video segment into a complete video;
[0067] The virtual person broadcast clip generated by the above steps and matching video clips , merge them into a complete video, ensure that the virtual human broadcast segment and the matching video segment are aligned on the timeline, add smooth transition effects between different segments, such as fade in and fade out, transition animation, etc., to ensure the continuity and viewing experience of the video, adjust the audio volume balance, enhance the color saturation of the video, add background music or sound effects, etc., to further improve the quality of the video. Finally, conduct a comprehensive quality check to ensure that the audio and video synchronization of the video is correct, and the overall visual effect and auditory experience meet the expected standards. Embodiment 2
[0068] like Figure 2 As shown, Embodiment 1 of the present application provides a virtual human video segment synthesis system for intelligent video generation.
[0069] Input and generate storyboard module 21: the user inputs video requirement parameters, and generates storyboard segments of the video according to the parameters;
[0070] The user inputs the video requirement parameters, including video theme, duration, key content points, etc., and uses the large language model to generate the video storyboard. First, the overall structure and main content of the video are determined according to the input parameters, and then the video duration is evenly distributed to each scene to generate the time parameter. Secondly, the scene description parameters are combined to generate a specific description of each scene. Finally, the storyboard containing the scene description and time nodes is output. Specifically, the formula is used: Generate a storyboard, where Represents the generated storyboard script. represents the large language model algorithm, Represents a set of input parameters, including video topics, key content points, etc. Indicates the video duration. Represents scene description parameters, such as the scene's environment, roles, and actions. Indicates the start and end time of each scene.
[0071] Classification model training module 22: establishing a training classification model to predict whether a storyboard script needs to be reported by a virtual person;
[0072] Predict whether a video storyboard is suitable for virtual human broadcasting. The text density, character interaction, and emotional richness are quantified. Text density refers to the number of words per minute in the script for dialogue or narration. Character interaction refers to the frequency of interaction between characters in the script, such as the number of dialogue turns or action interactions. Emotional richness refers to the number and intensity of emotions expressed in the script. For text density and character interaction, high values mean that more information transmission and interaction are required, which is more suitable for virtual human broadcasting. For emotional richness, low values are more suitable for virtual human broadcasting because it is difficult for virtual humans to fully capture and express complex emotions. Specifically, it includes the following submodules:
[0073] Text density calculation submodule: calculates text density based on storyboard script data;
[0074] Text density is calculated by calculating the importance of each word in the storyboard data, then summing the importance of all words and comparing it with the total length of the video. The importance of words is calculated by word frequency and inverse document frequency. Word frequency refers to the frequency of a word appearing in a text. The more frequently a word appears in a text, the more important it is to the content of the text. Inverse document frequency is used to measure the general importance of a word. If a word appears in many texts, its distinguishing ability is poor and its inverse document frequency value is low. On the contrary, if a word appears in only a few texts, its inverse document frequency is high. Specifically, the formula is used: Calculate the text density, where represents the text density of the j-th storyboard, represents the number of occurrences of word i in the text of the j-th storyboard, represents the total number of words in the text of the j-th storyboard, Indicates the total number of storyboards, j indicates the subscript of the storyboard, and the value of j is 1-M, i indicates the subscript of the word in the text of the current storyboard, and the value of i is 1-n, It means counting the total number of words i in the text of all storyboards. Represents the total video length of the j-th storyboard.
[0075] Role interaction calculation submodule: calculates role interaction based on storyboard script data;
[0076] The role interaction is calculated by analyzing the number and type of dialogues between characters in the storyboard. The dialogue turns, the number of times a character mentions other characters, and the direct action interaction parameter data between characters are extracted from the storyboard. Specifically, the formula is used: Calculate the role interaction, where represents the role interaction degree of the j-th storyboard, represents the total number of dialogue turns of character i in the jth storyboard, represents the total number of dialogue turns of character k in the jth storyboard, It is the number of times role i mentions other role k. represents the number of direct action interactions performed by character i on character j, They represent the conversation rounds, the number of times a character mentions other characters, and the weights of the direct action interaction parameters between characters. It represents the total video duration of the j-th storyboard. i and k represent the subscripts of the characters, with values ranging from 1 to n.
[0077] Emotional richness calculation submodule: calculates emotional richness based on storyboard script data;
[0078] The calculation of emotional richness involves text sentiment analysis and consideration of the diversity and intensity of emotions depicted in the video script. Emotional richness is expressed as the diversity of the types and degrees of different emotions expressed in the script. According to the results of sentiment analysis, emotions are divided into multiple categories. Positive emotions can be subdivided into happiness, excitement, hope, etc.; negative emotions can be subdivided into sadness, anger, fear, etc. The emotional richness is calculated by counting the frequency and intensity of each emotion category. Specifically, the formula is used: Calculate the emotional richness, where represents the emotional richness of the j-th storyboard, represents the i-th emotion category of the j-th storyboard, represents the probability of the occurrence of the i-th emotion, represents the average intensity of the i-th emotion, by Multiply by the sentiment probability An index that accurately reflects the diversity and intensity of emotions in the script is obtained. The higher the emotional intensity and the more evenly distributed the emotions are, the greater the emotional richness is. Conversely, if only one or a few emotions are dominant, even if there are many types of emotions, their intensity is generally low, and the emotional richness is relatively small.
[0079] Comprehensive probability value calculation submodule: calculates the comprehensive probability value according to the text density, role interaction and emotional richness, and establishes a training classification model based on the comprehensive probability value;
[0080] The comprehensive probability value is calculated based on the text density, role interaction and emotional richness. Specifically, the formula is used: Calculate the comprehensive probability value, where represents the comprehensive probability value of the j-th storyboard script, represents the normalization function, , , Represent the text density, character interaction and emotional richness of the j-th storyboard, respectively. , , They respectively represent the influencing factors of the text density, character interaction and emotional richness of the j-th storyboard.
[0081] Until the comprehensive probability values of all storyboard scripts are calculated, each video script with a comprehensive probability value exceeding 0.5 is marked with a label that requires virtual human broadcasting, and the training classification model is completed.
[0082] Virtual person broadcast segment generating module 23: for a storyboard script with a virtual person broadcast tag, generating a virtual person broadcast segment;
[0083] For the storyboard segments that the classification model predicts need to be reported by virtual humans, the virtual human reporting segments are generated. The virtual human reporting segments include audio (voice content, sound type, speaking speed, intonation), images (facial expressions, lip shape, clothing appearance) and actions. They are divided into three sub-modules: multimodal input, generator and discriminator, which respectively realize the functions of storyboard data preprocessing, virtual human generation and result verification, and generate virtual human reporting segments. Specifically, it includes the following sub-modules:
[0084] Multimodal input submodule: uses multimodal input data to provide a data source for the generator;
[0085] Multimodal input data includes audio-driven data, picture features, and action features. By analyzing the textual emotions, tone, speech speed, and other information of the storyboard script, the text is converted into a speech signal. In this process, the appropriate voice type can be selected, and the speech speed and tone can be adjusted as needed. The emotional state of the character can be inferred based on the text content, and the audio-driven data can be obtained; facial images that match specific emotions and expressions are generated based on the emotional state, corresponding lip animations are generated based on the speech content, and specific clothing appearances are generated based on the description in the script, and the picture features are obtained; corresponding behavioral actions are generated through text descriptions, ensuring that these actions are coordinated with the speech content and facial expressions, and the action features are obtained. Specifically, the formula is used: Represents multimodal input data, where represents a multimodal input data set, Represents audio driver data, Indicates the picture feature data, Represents action feature data.
[0086] Generation submodule: Generates virtual human broadcast segments based on multimodal input data;
[0087] In order to improve the expressiveness of the virtual person, it is necessary to optimize its voice type, speaking speed, tone and emotional expression, facial expression and posture, lip synchronization, and clothing and appearance customization. First, the audio driving data is analyzed through a deep learning model to extract and classify different voice type features, including timbre (such as male, female, child, etc.), pitch and sound quality. Then, the speaking speed of the virtual person is adjusted according to the scene requirements and user preferences, and a prediction model is established by learning the speaking speed distribution in a large number of broadcast videos. In addition, an emotion recognition model is used to extract emotional features from the voice data to generate intonation changes under different emotional states, such as happiness, sadness or anger. At the same time, computer vision technology is used to analyze the facial expressions and posture changes in the video data, and the corresponding animation model is generated to enhance the visual effect during the broadcast and ensure that the virtual person's lip shape is synchronized with the generated speech. Finally, through image generation technology and user input, the clothing and appearance of the virtual person can be customized according to the scene and user preferences.
[0088] The audio driving data in the multimodal input data is converted into speech using text-to-speech technology. Specifically, the formula is adopted: Generate speech, where It stands for text-to-speech technology. Represents audio driver data, Represents the generated speech; the image feature data in the multimodal input data is converted into a picture using image generation technology. Specifically, the formula is used: Generate a screen, where represents the image generation technology, Indicates the picture feature data, Represents the generated picture; uses artificial intelligence action recognition technology to convert the action feature data in the multimodal input data into actions. Specifically, the formula is used: Generate actions where It represents artificial intelligence action recognition technology. Represents action feature data, Indicates the generated action and the generated speech , screen ,action Composition of virtual human broadcast segments .
[0089] Discriminator submodule: Use a multi-scale discriminator to test the generated virtual human broadcast clips;
[0090] A multi-scale discriminator is introduced to judge the authenticity of the generated virtual human broadcast clips at different scales, thereby improving the detail fidelity of the generated images. Three discriminators of different scales are defined, including mouth shape, facial expression, and key point alignment. The tasks of the three discriminators are to distinguish between real images and generated images, to distinguish between local areas and global images, and to distinguish between image contrast and hue. Specifically, the formula is used: To test, represents the loss function, Indicates a segment broadcast by a virtual person. They represent three discriminators: mouth shape, facial expression, and key point alignment. It means to expect the joint distribution of the real image x and the generated image y. Representation Discriminator The judgment of the input images x and y returns a real value between 0 and 1, indicating the discriminator The probability that the input images x and y match is If it is close to 1, it means that the discriminator thinks that the pair of inputs is matched, and the loss function value of the first term is small. It means to find the expectation of the joint distribution of the real image x and random noise z. Representation Discriminator For the input image x and the random image The judgment, when When it is close to 1, it means that the discriminator mistakenly generates random images. It is judged to be a real image. At this time, the loss function value of the second item is large. By minimizing the value of the loss function, the virtual human broadcast segment is gradually improved. quality, making it increasingly difficult for the discriminator to distinguish.
[0091] The matching video segment generating module 24: for the storyboard without the virtual person broadcasting tag, searching the database for the matching video segment;
[0092] For storyboards without the avatar announcement tag, search for matching video clips in the database:
[0093] 1. Perform text analysis on the storyboard script to extract key information, such as emotional state, scene description, dialogue content, etc.
[0094] 2. Generate a series of features based on these key information, such as emotional features, scene features, dialogue features, etc.
[0095] 3. Using these features as query conditions, search in a database containing rich video clips, and use feature matching algorithms to find video clips that match the script content. Specifically, the formula is used: Perform feature matching, where represents the feature matching algorithm, Represents a collection of storyboard features, Represents a collection of storyboard features The i-th eigenvector in , Represents the feature set of database video material, Represents the feature set of database video material The i-th eigenvector of , i is a subscript with a value of 1-n, and n is the number of eigenvectors in the storyboard feature set. Represented as the final matched video segment.
[0096] Merging module 25: merging the virtual human broadcasting segment and the matching video segment into a complete video;
[0097] The generated virtual person broadcasts the clip and matching video clips , merge them into a complete video, ensure that the virtual human broadcast segment and the matching video segment are aligned on the timeline, add smooth transition effects between different segments, such as fade-in and fade-out, transition animation, etc., to ensure the continuity and viewing experience of the video, adjust the audio volume balance, enhance the color saturation of the video, add background music or sound effects, etc., to further improve the quality of the video. Finally, conduct a comprehensive quality check to ensure that the audio and video synchronization of the video is correct, and the overall visual effect and auditory experience meet the expected standards.
[0098] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for synthesizing virtual human video clips for intelligent video generation, characterized in that: include: S110, the user inputs video requirement parameters, and a storyboard script of the video is generated according to the parameters; S120, establishing a training classification model to predict whether a storyboard script needs to be reported by a virtual human; S130, for a storyboard script with a virtual person broadcasting tag, generating a virtual person broadcasting segment; S140, for a storyboard without a virtual person broadcasting tag, searching a database for a matching video clip; S150, merging the virtual human broadcast segment and the matching video segment into a complete video; For storyboards without the virtual person announcement tag, search for matching video clips in the database using the following steps: Conduct text analysis on storyboards to extract key information, including emotional state, scene description, and dialogue content; Generate a series of features based on these key information, including emotional features, scene features, and dialogue features; Using these features as query conditions, we searched in a database containing rich video clips and used feature matching algorithms to find video clips that matched the script content. Specifically, the formula is used: Perform feature matching, where represents the feature matching algorithm, Represents a collection of storyboard features, Represents a collection of storyboard features The i-th eigenvector in , Represents the feature set of database video material, Represents the feature set of database video material The i-th eigenvector of , i is a subscript with a value of 1-n, and n is the number of eigenvectors in the storyboard feature set. Represented as the final matched video segment.
2. A method for synthesizing virtual human video clips for intelligent video generation as claimed in claim 1, characterized in that: To predict whether a video storyboard script is suitable for virtual human broadcasting, we quantify it from three aspects: text density, character interaction, and emotional richness. Text density refers to the number of words per minute in the script for dialogue or narration, character interaction refers to the frequency of interaction between characters in the script, and emotional richness refers to the number and intensity of emotional types expressed in the script. For text density and character interaction, high values mean that more information transmission and interaction are required, which is more suitable for virtual human broadcasting. For emotional richness, low values are more suitable for virtual human broadcasting.
3. A method for synthesizing virtual human video clips for intelligent video generation as claimed in claim 2, characterized in that: The method for establishing a training classification model specifically includes the following sub-steps: Calculate text density based on storyboard script data; Calculate the degree of character interaction based on storyboard script data; Calculate the emotional richness based on the storyboard script data; The comprehensive probability value is calculated according to the text density, character interaction and emotional richness, and the training classification model is established based on the comprehensive probability value.
4. The method for synthesizing virtual human video clips for intelligent video generation according to claim 1, characterized in that: The method for generating a virtual person broadcast segment specifically includes the following sub-steps: Use multimodal input data to provide a data source for the generator; Generate virtual human broadcast segments based on multimodal input data; Use a multi-scale discriminator to test the generated virtual human broadcast clips.
5. The method for synthesizing virtual human video clips for intelligent video generation according to claim 1, characterized in that: The method for searching for matching video clips specifically includes the following sub-steps: Conduct text analysis on storyboards to extract key information; Generate a series of features based on key information; Using features as query conditions, we search in a database containing rich video clips, and use feature matching algorithms to find video clips that match the script content.
6. A virtual human video clip synthesis system for intelligent video generation, characterized in that: include: Input and generate storyboard module: the user inputs the video requirement parameters and generates the storyboard script of the video according to the parameters; Training classification model building module: Building a training classification model to predict whether a storyboard script needs to be reported by a virtual human; Virtual human broadcast segment generation module: for storyboards with virtual human broadcast tags, generate virtual human broadcast segments; Matching video clip generation module: for storyboards without virtual human broadcasting tags, search for matching video clips in the database; Merging module: merges the virtual human broadcast segment and the matching video segment into a complete video; For storyboards without the virtual person announcement tag, search for matching video clips in the database using the following steps: Conduct text analysis on storyboards to extract key information, including emotional state, scene description, and dialogue content; Generate a series of features based on these key information, including emotional features, scene features, and dialogue features; Using these features as query conditions, we searched in a database containing rich video clips and used feature matching algorithms to find video clips that matched the script content. Specifically, the formula is used: Perform feature matching, where represents the feature matching algorithm, Represents a collection of storyboard features, Represents a collection of storyboard features The i-th eigenvector in , Represents the feature set of database video material, Represents the feature set of database video material The i-th eigenvector of , i is a subscript with a value of 1-n, and n is the number of eigenvectors in the storyboard feature set. Represented as the final matched video segment.
7. A virtual human video segment synthesis system for intelligent video generation as claimed in claim 6, characterized in that: The training classification model building module includes the following submodules: A text density calculation submodule, used to calculate text density based on storyboard script data; The role interaction degree calculation submodule is used to calculate the role interaction degree according to the storyboard script data; The emotion richness calculation submodule is used to calculate the emotion richness according to the storyboard script data; The comprehensive probability value calculation submodule is used to calculate the comprehensive probability value according to the text density, role interaction and emotion richness, and establish a training classification model according to the comprehensive probability value.
8. A virtual human video segment synthesis system for intelligent video generation as claimed in claim 6, characterized in that: The virtual human broadcast segment generation module includes the following submodules: The multimodal input submodule is used to use multimodal input data to provide a data source for the generator; A generation submodule is used to generate a virtual human broadcast segment based on multi-modal input data; The discriminator submodule is used to use a multi-scale discriminator to verify the generated virtual human broadcast clips.
