Video generation method and device, storage medium and computer program product
By generating image-derived text, text reading audio and virtual characters, multi-modal conversion from image to video is realized, solving the problem of single and fixed image description in the prior art, and improving the diversity of user experience and application scenarios.
Patent Information
- Application Number
- CN202411406708.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing picture-viewing and speaking technology is limited to text form and cannot realize the real image-to-multi-modal information conversion, resulting in a poor user experience and cannot meet personalized needs.
By generating image-derived text, the text reads audio and virtual characters aloud, the multi-modal conversion from image to video is realized, and a variety of information types are generated to enrich the expression form of images.
It realizes multi-dimensional and multi-modal presentation of images, enriches the application scenarios of image information, and improves user interaction experience and application diversity.
Smart Images

Figure CN118918212A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, device, storage medium and computer program product. Background Art
[0002] With the development and popularization of artificial intelligence technology, users can use generative artificial intelligence (AIGC) to analyze and describe images at any time, that is, use artificial intelligence to realize picture description.
[0003] Conventional picture description technology is limited to text form, and stops at generating a text description that meets the requirements based on a given image. It uses text to describe the image or tell a story derived from the image, but does not truly realize the "speaking" function. The application scenarios are relatively single and fixed, and it is difficult to meet the user's personalized picture description needs, resulting in a poor user experience.
[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are conventional technologies. Summary of the invention
[0005] The main purpose of this application is to provide a video generation method, device, storage medium and computer program product, aiming to enhance user experience.
[0006] To achieve the above object, the present application provides a video generation method, which includes: Generate image-derived text based on the acquired original image; Generating text reading audio according to the image derived text; generating a virtual character according to the original image; A text reading video is generated according to the virtual character and the text reading audio.
[0007] In one embodiment, the step of generating image-derived text based on the acquired original image includes: Acquire a labeled sample, wherein the labeled sample includes a plurality of sample images and a description text for describing each of the sample images; Fine-tune a preset multimodal large language model using the labeled samples to obtain a fine-tuned multimodal large language model; The fine-tuned multimodal large language model is applied to recognize the original image and generate image-derived text.
[0008] In one embodiment, the step of generating a virtual character according to the original image includes: Generate a description text of the virtual character according to the original image and the image-derived text; The virtual character is generated according to the description text.
[0009] In one embodiment, the step of generating a virtual character according to the description text includes: Extracting keywords from the description text to determine character image feature words corresponding to each of a plurality of preset character images; Determine a character image whose similarity between the corresponding character image feature word and the keyword exceeds a preset similarity threshold as a candidate image; Evaluate the clarity of image details of each of the candidate images, and determine the candidate image with the highest clarity of image details as the target image; A virtual character is generated according to the description text and the target image.
[0010] In one embodiment, the step of generating text reading audio according to the image-derived text includes: Identifying the textual semantics of the image-derived text, extracting environmental semantic words from the textual semantics, and determining environmental sound effects that match the environmental semantic words; The text reading audio is generated according to the image-derived text and the environmental sound effect.
[0011] In one embodiment, the step of generating text reading audio according to the image-derived text includes: recognizing the textual semantics of the image-derived text, and determining the voice characteristics of a character for reading the image-derived text according to the textual semantics; Acquire human voice audio matching the sound feature, and generate the text reading audio based on the human voice audio and the image-derived text.
[0012] In one embodiment, the step of generating a text reading video according to the virtual character and the text reading audio includes: Identifying the emotion of the text reading audio to obtain a first emotion label; Extracting a second emotion tag of a preset character video, and calculating the emotion similarity between the first emotion tag and the second emotion tag; Selecting a character video of the second emotion label with the greatest emotion similarity as the target video; The text reading video is generated according to the target video, the virtual character and the text reading audio.
[0013] In addition, to achieve the above-mentioned purpose, the embodiment of the present application further proposes a video generating device, the video generating device comprising: A text generation module is used to generate image-derived text based on the acquired original image; An audio generation module, used to derive text from the image and generate text reading audio; An image generation module, used to generate a virtual character according to the original image; The video generation module is used to generate a text reading video according to the virtual character and the text reading audio.
[0014] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also proposes a video generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video generation method described above.
[0015] In addition, to achieve the above-mentioned purpose, an embodiment of the present application further proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video generation method described above are implemented.
[0016] In addition, to achieve the above-mentioned purpose, an embodiment of the present application further provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the video generation method described above are implemented.
[0017] The present application provides a video generation method. The embodiment of the present application first generates image-derived text based on the acquired original image, providing an accurate text script for subsequent audio and video generation; generates text reading audio based on the image-derived text, converts text into speech, crosses the visual boundary, and enriches the landing scene of the image from the auditory dimension; generates a virtual character based on the original image to provide material for subsequent video generation; generates a text reading video based on the virtual character and the text reading audio, completing the transformation from static images to dynamic audio-visual content. The present application realizes multi-dimensional and multi-modal presentation of images by converting images into multiple information types such as text, audio and video, enriches the expression form and application scenarios of image information, and enhances the user's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0020] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the video generation method of the present application; Figure 2 A schematic diagram of an original image provided in Example 1 of the video generation method of the present application; Figure 3 A schematic diagram of a virtual character provided in Example 1 of the video generation method of the present application; Figure 4 This is a flow chart of a relatively complete embodiment of the video generation method of the present application; Figure 5 This is a schematic diagram of the module structure of the video generating device according to an embodiment of the present application; Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the video generation method in the embodiment of the present application.
[0021] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0022] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0023] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0024] It should be noted that in this application, "first" and "second" are used to distinguish the order of the instructions in the current context and do not represent their importance or uniqueness.
[0025] The main solution of the embodiment of the present application is: generate image-derived text based on the acquired original image; generate text reading audio based on the image-derived text; generate a virtual character based on the original image; generate a text reading video based on the virtual character and the text reading audio.
[0026] In this embodiment, for the convenience of description, the following description is made with the terminal as the execution subject.
[0027] Since conventional picture description technology usually focuses on single-modal information processing, it is unable to obtain key information in the image through cross-modal understanding of images and texts. The generated image description text is not accurate enough, making it difficult to provide users with a better interactive experience. Conventional technical solutions are limited to converting visual images into written text and only generate text descriptions that match the image, which limits the diversity of application scenarios of image information, fails to meet users' demand for personalized experience, and reduces users' interactive experience.
[0028] The present application provides a solution that enriches the application scenarios of image information by generating text, audio, and video based on the original image; at the same time, with the help of the model's cross-modal information understanding ability, it can better understand the relationship between images and text, obtain more accurate and reasonable text descriptions, and achieve effective interaction. By obtaining labeled samples containing specific fields or specific tasks, the MLLM model is fine-tuned so that it can more accurately answer tasks in the field and promote effective interaction. Furthermore, through the interaction between images and text, the conversion of images to text, audio, and video is realized, thereby realizing multi-modal interactions such as image-to-text, text-to-audio, text-to-image, and image-to-audio video, realizing effective interaction of multi-modal information of images, and providing users with a better interactive experience.
[0029] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes the terminal as the execution subject as an example to illustrate this embodiment and the following embodiments.
[0030] Based on this, the embodiment of the present application provides a video generation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the video generation method of the present application.
[0031] In this embodiment, the video generation method includes steps S10 to S40: Step S10, generating image-derived text according to the acquired original image; It should be noted that the original image refers to the image provided by the user to the model, hoping that the model will analyze and describe it. Image-derived text refers to the text generated based on user requirements or model settings after content analysis of the original image, including image scene description and image content extension; generation refers to the process of converting the visual information in the original image into text information based on a series of algorithms and technical means.
[0032] Exemplarily, multimodal large language models (MLLM) can be used to understand the complex information in the image and convert it into natural language descriptions. MLLM includes VisualGLM-6B (Visual General Language Model with 6 Billion parameters), BLIP-2 model (Bootstrapped Language Image Pretraining) and CLIP-2 (Contrastive Language Image Pretraining) model.
[0033] For example, Figure 2 The original image schematic diagram provided in the embodiment of the present application is as follows: Figure 2 As shown in the figure, a bird is sitting on a branch. The bird image uploaded by the user (i.e., the original image) is obtained, and the prompt command "Describe this image" entered by the user is received. The image content is analyzed by VisualGLM-6B, the appearance of the bird and the environment information are identified, and a descriptive text is generated: "This image shows a bird sitting on a branch. It has gray and white feathers, a black beak, and brown eyes."
[0034] For example, the CLIP-2 model is used to generate image-derived text, and a bird picture uploaded by a user is received as the original image, such as Figure 2 , receiving the prompt instruction input by the user "Based on this picture, compile a vivid fairy tale. The protagonist of the story is Benben", analyzing the picture content and the prompt instruction through the CLIP-2 model, integrating the information of the two, and generating a fairy tale with the picture as the theme "A long time ago, there was a cute elf named Benben. He had gray feathers, a black beak and brown eyes, and liked to play on tree branches. One day, he found a small insect on the branch, so he decided to take the insect back to share with other animals" as the image-derived text.
[0035] Step S20, generating text reading audio according to the image derived text; It should be noted that text reading audio refers to the use of text-to-speech (TTS) conversion technology to convert input text information into audible voice output. Models and tools that use TTS technology to implement text-to-speech audio include the Bark model, MockingBird model and Adobe Audition tool.
[0036] Exemplarily, image-derived text input by the user or generated by the model is obtained, the text content is analyzed through the Bark model, the text is converted into a voice signal, and the generated voice signal is integrated into a complete audio file (text reading audio), and the user can choose to download or play the audio file.
[0037] It can be understood that by converting images into text and converting text descriptions into text reading audio, the combination of the two realizes the conversion from images to text and audio modes, enriches the application scenarios of image information, and thus enhances the user's interactive experience.
[0038] Step S30, generating a virtual character according to the original image; It should be noted that virtual characters refer to images of non-real people, animals or objects created through computer graphics technology, which can be two-dimensional images, three-dimensional models or animated characters. Generation refers to the process of creating new virtual character images based on the information of the original image using computer graphics technology and algorithms. The virtual character can be a character that reads the text derived from the image, or a character corresponding to each object in the image. The generation process can be achieved by using the Wensheng graph model, which includes the SD (Stable Diffusion) model, the FLUX model and the Midjourney model.
[0039] For example, a deep learning model is trained, which can generate a text describing the image of each object in a picture based on the picture provided by the user, and then generate a virtual character suitable for reading the text based on the text describing the image. When using the model to generate a virtual character, the original image provided by the user containing a puppy and a kitten is obtained, the puppy in the image has pure white hair, and the kitten has black and white spots, and the virtual characters of the puppy and kitten with corresponding character features are generated.
[0040] Exemplarily, the original image uploaded by the user is obtained, and the image-generated text model is used to generate a virtual character description text suitable for reading the derived text of the original image, or the user writes a virtual character description text based on his or her own imagination, such as "a beautiful young lady who is very good at telling fairy tales". The virtual character description text is obtained, and the text content is read and analyzed by the SD image-generated text tool, and the key features of the virtual character are extracted from the text to generate the virtual character required by the user. Since the SD image-generated text tool has better support for English, the descriptive text can be translated into English and then input into the SD image-generated text tool.
[0041] For example, after seeing the first version of the virtual character generated by the SD Wenshengtu tool, the user believes that there are still areas for improvement, and the user enters an improvement description: "We need this lady's face to face us, she has big eyes and beautiful lips, and her face is located in the middle of the entire screen." The description text is received, and the SD Wenshengtu tool is used to extract key information from it, and the original virtual character is adjusted to generate a virtual character that better meets the user's requirements. Figure 3 A schematic diagram of a virtual character provided in an embodiment of the present application is provided, such as Figure 3 As shown, the lady's face is located in the middle of the picture, with big eyes and beautiful lips. Optionally, if the user is not satisfied with the generated virtual character, the model can be instructed to regenerate until a virtual character that the user is satisfied with is generated.
[0042] Step S40, generating a text reading video according to the virtual character and the text reading audio.
[0043] It should be noted that the text reading video refers to a dynamic image composed of a series of continuous frames, each of which is a static image. The virtual character in the video can make corresponding lip movements, facial expressions and even body language according to the voice content in the audio. This process can be achieved through audio-driven image models such as the Hallo model, SadTalker model and Wav2Lip model.
[0044] Exemplarily, the preset audio-driven image model is the SadTalker model. The generated virtual character and text reading video are obtained, and feature information is extracted from the input text reading audio through the SadTalker model, and the parameters in the virtual character image are adjusted accordingly to generate a dynamic image sequence matching the audio content, which is combined with the text reading audio to generate a video with synchronized audio and video, and the user can choose to play, save and share it.
[0045] This embodiment provides a method for generating video from images. By analyzing the image content and generating image-derived text, it can more accurately describe the image and generate reasonable and fluent text for subsequent steps. Then, the text is converted into speech, enriching the implementation scenarios of picture-based speech. By generating virtual characters based on the descriptive text of the virtual characters, it provides materials for subsequent video generation. By using the audio-driven image model combined with the above-generated audio and virtual characters to generate videos, the transformation from static images to dynamic audio-visual content is completed. The entire process converts images into different forms such as text, audio, and video, realizes effective multi-modal interaction of images, provides rich application scenarios for images, and enhances the user's interactive experience.
[0046] In a feasible implementation, step S10 may include steps S11 to S13: Step S11, obtaining a labeled sample, wherein the labeled sample includes a plurality of sample images and a description text for describing each sample image; It should be noted that a marked sample refers to a set of sample data that has been manually or automatically labeled with specific information, which is used to guide the training and learning process of the model.
[0047] For example, the user hopes that the model can describe the background content of medical images more accurately and naturally. The user finds a large number of medical images and describes the background content of the found medical images. The annotated samples provided by the user include medical images, corresponding background content descriptions, and the prompt information "What is in the background of this picture?".
[0048] Step S12, fine-tuning the preset multimodal large language model using the labeled samples to obtain a fine-tuned multimodal large language model; It should be noted that the multimodal large language model MLLM is an advanced artificial intelligence model that can process multi-modal data (such as text, images, videos, etc.) and generate coherent language output. This type of model uses deep learning technology and is trained with large-scale data sets. It can capture rich semantic and contextual information, and can also understand complex information in images and convert it into natural language descriptions.
[0049] In addition, it should be noted that fine-tuning refers to the targeted optimization of pre-trained models through specific domain data to improve the adaptability and performance of the model for specific data sets or tasks. In multimodal scenarios, annotated samples can include images and their corresponding text descriptions. MLLM can learn how to extract the key information needed by users from images by repeatedly learning the pairing of these images and descriptions.
[0050] Exemplarily, the VisualGLM-6B model is fine-tuned so that the VisualGLM-6B model can more accurately obtain the background content of medical images. Obtain an annotated sample with background content provided by the user, which contains a medical image and its corresponding background content introduction, and let the VisualGLM-6B model learn the sample and optimize the parameters of the background content introduction. After that, when the prompt information input by the user contains "background content", the VisualGLM-6B model can more accurately identify the background content of the medical field, so as to output a professional text description that better meets the user's needs.
[0051] It is understandable that by learning sample data for specific tasks, the multimodal large language model MLLM can better adapt to and complete tasks in specific fields and generate more professional descriptions, thereby improving the effectiveness of multimodal interaction and enhancing the user's interactive experience.
[0052] Step S13: Apply the fine-tuned multimodal large language model to recognize the original image and generate image-derived text.
[0053] For example, a user works in the field of art and wants to generate text describing works of art. Obtain images of a series of works of art collected by the user and annotated samples of professional reviews, and use these samples to fine-tune the VisualGLM-6B model so that it can understand and generate professional descriptions in the field of art. After training, obtain new images of works of art uploaded by the user, and apply the trained VisualGLM-6B model to generate image-derived text describing the work.
[0054] In this implementation, by using labeled samples of specific fields or specific tasks to train a multimodal large language model MLLM, the model can be better adapted to the needs and features of specific fields, extract key information from images required by users, generate more professional and personalized texts, and improve the professionalism of text generation based on images, thereby enhancing the user's interactive experience.
[0055] In a feasible implementation, step S30 may include steps S31-S32: Step S31, generating a description text of the virtual character according to the original image and the image-derived text; It should be noted that description text refers to the text content used to describe the image characteristics of the virtual character in detail, including appearance, clothing and expressions. Since cross-modal processing of text and image information is required, you can train your own model to generate descriptions for specific image characteristics, or you can use a multimodal large language model (MLLM) to generate description text.
[0056] Exemplarily, an original image provided by a user containing a forest, a stream and a leisurely deer and a fairy tale generated based on the image (i.e., image-derived text) are obtained, and a prompt message input by the user "Based on the given image and the fairy tale generated above, generate a descriptive text suitable for reciting the story" is received. The VisualGLM-6B model is used to simultaneously analyze and understand the image content and text content in the above text, and generate the descriptive text required by the user.
[0057] It is understandable that users can choose to apply MLLM's contextual understanding capabilities and cross-modal understanding of image and text information to generate descriptive text, which saves users the step of manually entering descriptive text, realizes effective interaction between images, and improves the user's interactive experience.
[0058] Step S32, generating a virtual character according to the description text.
[0059] For example, the description text of the virtual character is obtained, "Youyou is a little deer living deep in the forest. It has soft and delicate fur and gentle eyes. Its personality is docile and sensitive." The text content is read and analyzed by the SD text image tool to generate the virtual character required by the user.
[0060] In this embodiment, by combining the information of the original image and the image-derived text, the model can better capture the characteristics of the object in the original image, enrich the image of the virtual character, and promote effective interaction compared to analyzing a certain information alone. Compared with manual input, using the model to generate descriptive text can more comprehensively summarize the characteristics of the virtual character and save time and effort. At the same time, by generating virtual characters through text and images, multi-modal effective interaction of images is achieved, which improves the user's interactive experience.
[0061] In a feasible implementation, step S32 may include steps A31 to A34: Step A31, extracting keywords from the description text, and determining character image characteristic words corresponding to each of a plurality of preset character images; It should be noted that character images refer to pre-stored images of multiple different characters that actually exist in reality. Each character image has its own unique appearance, clothing, expression and other image features, which can be used as a reference for generating virtual characters. Character image feature words refer to words that can reflect the characteristics of character images extracted from character images using image recognition technology.
[0062] For example, the description text "a knight in silver armor, holding a long sword" is obtained, and the keywords "silver armor", "knight" and "long sword" can be extracted from it through the natural language processing model. At the same time, the image recognition model is used to analyze the locally stored character images, extract the character image feature words, and store the character images and their corresponding character image feature words in association.
[0063] Step A32, determining the character images whose similarity between the corresponding character image feature words and the keywords exceeds a preset similarity threshold as candidate images; It should be noted that similarity is a metric to measure the degree of similarity between two objects (such as text, images, etc.), which can be evaluated by calculating the distance or similarity coefficient of the feature vectors between the objects, and is used to compare the degree of match between the character image feature words and the keywords. Candidate images refer to a set of character images that are related to user queries or system requirements after screening. These images have a high degree of similarity with the keywords in terms of character image features and are potential objects for further processing or display.
[0064] For example, the similarity between the character image feature words of each character graphic and the extracted keywords is calculated. For example, the character image feature words of the bird picture are "long beak", "brown eyes" and "gray feathers", while the extracted text keywords are "gray feathers", "long beak" and "brown eyes". The similarity between the two is calculated to be 90%. The calculated similarity is then compared with the preset similarity threshold of 80%. It is found that the calculated similarity is greater than the similarity threshold, so the bird picture is determined as a candidate object.
[0065] It can be understood that by comparing the character image feature words and keywords of the character images, character images that match the description text can be quickly screened out, reducing the steps for users to blindly search through a large number of character images, facilitating the subsequent generation of virtual characters that meet the user's description, thereby achieving effective interaction and enhancing the user's interactive experience.
[0066] Step A33, evaluating the clarity of image details of each candidate image, and determining the candidate image with the highest clarity of image details as the target image; It should be noted that image details refer to small elements or areas in the image that correspond to the characteristic words of the character's image, such as eyes, lip shape, skin color, etc.; clarity refers to the degree to which the image details are clearly presented, which can be quantified by calculating parameters such as image contrast, sharpness or high-frequency components.
[0067] Exemplarily, the first candidate image and the second candidate image are obtained by screening according to the similarity threshold, and the clarity of the first candidate image is calculated to be 85%, and the clarity of the second candidate image is calculated to be 95%. The second candidate image has the highest clarity, so the second candidate image is determined as the target image.
[0068] It can be understood that by selecting the image with the highest clarity as the target image, it helps to reduce recognition errors caused by image blur, improve the overall recognition accuracy, and further promote effective interaction.
[0069] Step A34, generating a virtual character according to the description text and the target image.
[0070] Exemplarily, the description text of the virtual character "a bird with gray-white feathers, a long beak and brown eyes" is obtained. After similarity and clarity screening, an image of a bird with character image feature words "long beak", "brown eyes" and "gray feathers" is determined as the target image, and the basic form and appearance features of the target image are learned. At the same time, the feathers of the virtual bird are automatically adjusted to gray-white feathers according to the text description, and finally a virtual bird (virtual character) with gray-white feathers similar to the target image is generated.
[0071] In this embodiment, by screening the character images according to the similarity between the keywords in the text description and the keywords of the character image characteristics of the character images, the character images that match the text description can be quickly matched; by using the character image with the highest definition as the target image, the accuracy of the initial generation of the virtual character is effectively improved, and the virtual character is adjusted and optimized with the help of the text description to ensure that the generated virtual character meets the user's needs. By cross-modally understanding the information contained in the image and text, the model can more effectively extract image information and generate virtual character images that better meet the user's needs, completing the conversion from image to image, meeting the user's needs for diversified image application scenarios, and improving the user's interactive experience.
[0072] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. On this basis, step S20 includes: Step A21, identifying the textual semantics of the image-derived text, extracting environmental semantic words from the textual semantics, and determining environmental sound effects that match the environmental semantic words; It should be noted that text semantics refers to the meaning expressed by the text obtained by parsing and understanding the text through semantic analysis technology; on this basis, keywords or phrases that describe or reflect the characteristics of the environment are extracted from the text, which can be called environmental semantic words. Environmental sound effects refer to the sound effects that exist naturally in a specific environment or scene, which are used to enhance the realism or atmosphere of the scene.
[0073] For example, the image-derived text obtained is "waves lapping against the shore". The text is analyzed using a natural language processing model, and the two environmental semantic words "waves" and "shore" are identified and extracted. The sound effect library is searched for environmental sound effects that match "waves" and "shore", and the gentle sound of waves is found.
[0074] Step A22, generating text reading audio according to the image-derived text and environmental sound effects.
[0075] For example, the text semantics of the acquired image-derived text is identified and analyzed, and the relevant parameters of the generated text reading audio, such as intonation, speech speed, etc., are adjusted according to the content and emotional tendency of the text, and the environmental sound effects matching the environmental semantic words are searched. For example, when synthesizing the sentence "waves gently hit the shore", the intonation is lowered, the speech speed is slowed down, and the gentle sound of the waves is integrated as background sound. Finally, the integrated audio data is encoded and compressed to generate the text reading audio, which is output to a specified storage location for subsequent playback or use.
[0076] In this embodiment, by integrating technical means such as natural language processing and speech synthesis, voice presentation of image-derived text is achieved; by adding environmental sound effects, the realism and immersion of the text reading audio description scene are enhanced, the content expressiveness of the text reading audio is improved, and the application boundaries of image information are broadened, thereby improving the user's interactive experience.
[0077] In a feasible implementation manner, step S20 further includes: Step A23, identifying the text semantics of the image-derived text, and determining the voice characteristics of the character used to read the image-derived text according to the text semantics; It should be noted that voice features refer to voice attributes that can reflect different personalities and emotional states, including timbre, speaking speed and accent, which are used to express different emotions, characters or scenes. Determination refers to the process of selecting or calculating the most suitable voice features of the character who reads the text based on the semantics of the text derived from the recognized image through certain rules or models.
[0078] For example, the image-derived text obtained is "warm sunset, golden light shining on the calm lake". The text is analyzed using natural language processing technology, and key semantic words such as "warmth", "glow" and "calm" are identified. It is inferred that the text conveys a tranquil and warm emotion, and the voice characteristics suitable for reading the text are determined to be "soft" and "middle-aged female voice".
[0079] Step A24, obtaining human voice audio that matches the sound features, deriving text based on the human voice audio and the image, and generating text reading audio.
[0080] It should be noted that vocal audio refers to audio files containing human speech, which can be professional dubbing, reading or other forms of sound performance.
[0081] Exemplarily, based on the image-derived text "warm sunset, golden light shining on the calm lake", the sound features suitable for reading the text are determined to be "soft" and "middle-aged female voice", and a warm and soft female voice is searched in the human voice audio library. Using text-to-speech TTS technology, the warm and soft female voice is used to read the image-derived text to generate text reading audio for users to listen to online or download.
[0082] In this embodiment, by combining sound recognition, text recognition and TTS technology, the function of generating personalized text reading audio based on the descriptive content of the image-derived text and human voice audio suitable for reading aloud is realized, thereby enhancing the diversity of application scenarios, meeting the personalized and diversified usage needs of users, and improving the user's interactive experience.
[0083] In a feasible implementation manner, step S40 further includes: Step A41, identifying the emotion of the text reading audio to obtain a first emotion label; It should be noted that emotion recognition refers to the use of natural language processing and speech processing technology to analyze the energy intensity, zero-crossing rate, jitter and other characteristics of audio signals to determine the emotional state expressed by the speaker; the emotion label is the result output of the emotion recognition process, which is a classification identifier of the emotional state expressed in the text reading audio, and is used to indicate the dominant emotion type in the audio.
[0084] For example, the acquired text reading audio is a news broadcast, and the announcer's voice may reveal serious or urgent emotions. The audio emotion recognition model can capture this emotional feature and classify it into a first emotional label such as "serious" or "urgent".
[0085] Step A42, extracting a second emotion tag of a preset character video, and calculating the emotion similarity between the first emotion tag and the second emotion tag; It should be noted that character videos refer to continuous image sequences captured from real scenes, which record relevant information about real characters in the real world, including actions, expressions and sound information. Emotional similarity can be quantified by calculating a certain distance or similarity score between two emotion labels, which is used to compare the similarity of emotions conveyed by audio and video.
[0086] Exemplarily, a video emotion recognition model is used to process character videos in a video library to extract emotion-related features in the video, including visual features and audio features. Features extracted from different modalities are fused to obtain a second emotion tag for the character video. The character video and its corresponding emotion tag are associated and stored, and the second emotion tag can then be extracted at any time to perform similarity calculations with the first emotion tag extracted from the audio.
[0087] Step A43, selecting a character video with a second emotion label having the greatest emotion similarity as a target video; Step A44, generating a text reading video according to the target video, the virtual character and the text reading audio.
[0088] For example, the reading audio (text reading audio) of the story text with the first emotion label "happy" is obtained, and the similarity between the first emotion label and the second emotion label of all the character videos in the video library is calculated. The character video with the second emotion label with the greatest emotion similarity is a funny interview video (target video). The SadTalker model is used to extract the facial expressions in the key frames of the funny interview video, especially the lip shape and facial muscle movements, and the lip shape and expression changes of the virtual character are driven according to the emotional changes expressed in the text reading audio, and a series of virtual character image sequences with expression changes are generated, and the generated virtual character image sequence is combined with the text reading audio to generate the text reading video.
[0089] For example, the SadTalker model is first used to combine the text reading audio and the virtual character to generate an initial video, in which the virtual character's lip shape changes with the text reading audio. Then, based on the character video with the highest emotional similarity to the text reading audio, machine learning or deep learning technology is applied to allow the virtual character to learn the expression changes in the character video, and the expression of the virtual character in the initial video is adjusted according to the emotional changes in the text reading audio, combined with the original audio and lip shape animation, to generate a more vivid and realistic text reading video.
[0090] In this embodiment, by allowing the virtual character to learn the lip shape and facial expressions of the real character according to emotional changes, the final generated video is made more natural, and the realism and interactivity of the video content generated based on the audio and virtual character are improved, thereby improving the user's interactive experience.
[0091] In order to help understand the above technical solution, a more complete embodiment of a video generation method is provided for illustration. Figure 4 ,include: Step A101, obtaining the original image.
[0092] Step A102, based on the original image, apply image processing technology to identify key information in the image and generate pattern-derived text.
[0093] Step A103, based on the image-derived text, apply the text-to-speech (TTS) technology to read the image-derived text and generate text reading audio; or integrate the initial audio generated by the TTS technology based on the acquired environmental sound effects to generate text reading audio; or adjust the tone, speech speed, volume and other sound features of the initial audio generated by the TTS technology based on the acquired human voice audio to generate text reading audio.
[0094] In step A104, a trained graph-based model may be applied to generate a virtual character based on the original image; or a virtual character may be generated based on a description text of the virtual character and a character image of a real object.
[0095] Step A105, based on the text reading audio and the virtual character, the audio-driven image model is applied to generate a text reading video; or based on the acquired character video, the virtual character is allowed to learn the expression changes in the character video, and the expression changes of the virtual character and the text reading audio are integrated to generate a text reading video.
[0096] Step A106, fine-tune the multimodal large language model MLLM using labeled samples in a specific field, and use the fine-tuned MLLM to generate image-derived text based on the original image.
[0097] Step A107, based on the principle image and the image-derived text, the image and text information are cross-modally processed to generate a description text of the virtual character suitable for reading the image-derived text. The fine-tuned MLLM can also be used to generate the description text.
[0098] Step A108, obtaining the character image that is used as a reference for learning when generating a virtual character, by extracting keywords from the description text, calculating the similarity between the character image feature words and the preset character image, selecting the character image whose similarity exceeds the preset similarity threshold as the candidate image, and then determining the candidate graphic with the highest clarity of the image details as the target image, that is, the reference learned character image.
[0099] Step A109, obtaining the environmental sound effects that match the image-derived text, by identifying the textual semantics of the image-derived text, extracting environmental semantic words therefrom, and searching the sound effects library for the environmental sound effects that match the environmental semantic words.
[0100] Step A110, obtaining human voice audio suitable for reading the image-derived text, by identifying the textual semantics of the image-derived text, determining the voice features of the character suitable for reading the image-derived text, and searching for human voice audio matching the voice features in a human voice audio library.
[0101] Step A111, obtaining the role video by extracting the emotion tag of the text reading audio and the emotion tag of the preset role video, calculating the emotion similarity between the two, and selecting the role video with the greatest similarity as the role video for the virtual character to learn.
[0102] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the video generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0103] This application also provides a video generation device, please refer to Figure 5 , the video generating device comprises: The text generation module 10 is used to generate image-derived text according to the acquired original image; An audio generation module 20, used to derive text from the image and generate text reading audio; An image generation module 30, used to generate a virtual character based on an original image; The video generation module 40 is used to generate a text reading video according to the virtual character and the text reading audio.
[0104] Optionally, the text generation module 10 is further used for: Acquire a labeled sample, wherein the labeled sample includes a plurality of sample images and a description text for describing each sample image; Fine-tune the preset multimodal large language model using the labeled samples to obtain a fine-tuned multimodal large language model; Apply the fine-tuned multimodal large language model to recognize original images and generate image-derived text.
[0105] Optionally, the image generation module 30 is further used for: Generate description text of the virtual character based on the original image and the image-derived text; Generate a virtual character based on the description text.
[0106] Optionally, the image generation module 30 is further used for: Extracting keywords from the description text, and determining the character image characteristic words corresponding to each of the plurality of preset character images; A character image whose similarity between the corresponding character image feature word and the keyword exceeds a preset similarity threshold is determined as a candidate image; Evaluate the clarity of image details of each candidate image, and determine the candidate image with the highest clarity of image details as the target image; Generate a virtual character based on the description text and the target image.
[0107] Optionally, the audio generation module 20 is further configured to: Identify the textual semantics of the image-derived text, extract the environmental semantic words in the textual semantics, and determine the environmental sound effects that match the environmental semantic words; Generate text reading audio based on image-derived text and environmental sound effects.
[0108] Optionally, the audio generation module 20 is further configured to: identifying the textual semantics of the image-derived text, and determining the voice characteristics of the character for reading the image-derived text according to the textual semantics; Acquire human voice audio that matches the sound features, derive text based on the human voice audio and image, and generate text reading audio.
[0109] Optionally, the video generation module 40 is further used for: Identify the emotion of the text reading audio and obtain the first emotion label; Extracting a second emotion tag of a preset character video, and calculating the emotion similarity between the first emotion tag and the second emotion tag; Select the character video with the second emotional label with the greatest emotional similarity as the target video; Generate a text reading video based on the target video, virtual character and text reading audio.
[0110] The video generation device provided by the present application adopts the video generation method in the above embodiment, which can solve the technical problem of how to achieve effective multi-modal interaction of images. Compared with the prior art, the beneficial effects of the video generation device provided by the present application are the same as the beneficial effects of the video generation method provided by the above embodiment, and other technical features in the video generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0111] The present application provides a video generating device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video generating method in the above-mentioned embodiment 1.
[0112] Reference below Figure 6 , which shows a schematic diagram of the structure of a video generating device suitable for implementing an embodiment of the present application. The video generating device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The video generating device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0113] like Figure 6As shown, the video generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the xxx device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the video generation device to communicate with other devices wirelessly or by wire to exchange data. Although the video generation device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0114] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0115] The video generation device provided by the present application adopts the video generation method in the above embodiment, which can solve the technical problem of how to achieve effective multi-modal interaction of images. Compared with the prior art, the beneficial effects of the video generation device provided by the present application are the same as the beneficial effects of the video generation method provided by the above embodiment, and the other technical features in the video generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0116] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0117] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0118] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the video generation method in the above-mentioned embodiment.
[0119] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.
[0120] The computer-readable storage medium may be included in the video generating device; or may exist independently without being assembled into the video generating device.
[0121] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the video generating device, the video generating device: generates image-derived text based on the acquired original image; generates text reading audio based on the image-derived text; generates a virtual character based on the original image; and generates a text reading video based on the virtual character and the text reading audio.
[0122] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0123] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0124] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0125] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video generation method, and can solve the technical problem of how to achieve effective multi-modal interaction of images. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the video generation method provided in the above-mentioned embodiment, and will not be described in detail here.
[0126] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video generation method when executed by a processor.
[0127] The computer program product provided by this application can solve the technical problem of how to achieve effective multi-modal interaction of images. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the video generation method provided by the above embodiment, which will not be repeated here.
[0128] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A video generation method, characterized in that: The video generation method comprises: Generate image-derived text based on the acquired original image; Generating text reading audio according to the image derived text; generating a virtual character according to the original image; A text reading video is generated according to the virtual character and the text reading audio.
2. The video generation method according to claim 1, characterized in that: The step of generating image-derived text according to the acquired original image includes: Acquire a labeled sample, wherein the labeled sample includes a plurality of sample images and a description text for describing each of the sample images; Fine-tune a preset multimodal large language model using the labeled samples to obtain a fine-tuned multimodal large language model; The fine-tuned multimodal large language model is applied to recognize the original image and generate image-derived text.
3. The video generation method according to claim 1, characterized in that: The step of generating a virtual character according to the original image comprises: Generate a description text of the virtual character according to the original image and the image-derived text; The virtual character is generated according to the description text.
4. The video generation method according to claim 3, characterized in that: The step of generating a virtual character according to the description text comprises: Extracting keywords from the description text to determine character image feature words corresponding to each of a plurality of preset character images; Determine a character image whose similarity between the corresponding character image feature word and the keyword exceeds a preset similarity threshold as a candidate image; Evaluate the clarity of image details of each of the candidate images, and determine the candidate image with the highest clarity of image details as the target image; A virtual character is generated according to the description text and the target image.
5. The video generation method according to claim 1, characterized in that: The step of generating text reading audio according to the image derived text comprises: Identifying the textual semantics of the image-derived text, extracting environmental semantic words from the textual semantics, and determining environmental sound effects that match the environmental semantic words; The text reading audio is generated according to the image-derived text and the environmental sound effect.
6. The video generation method according to claim 1, characterized in that: The step of generating text reading audio according to the image derived text comprises: recognizing the textual semantics of the image-derived text, and determining the voice characteristics of a character for reading the image-derived text according to the textual semantics; Acquire human voice audio matching the sound feature, and generate the text reading audio based on the human voice audio and the image-derived text.
7. The video generation method according to claim 1, characterized in that: The step of generating a text reading video according to the virtual character and the text reading audio comprises: Identifying the emotion of the text reading audio to obtain a first emotion label; Extracting a second emotion tag of a preset character video, and calculating the emotion similarity between the first emotion tag and the second emotion tag; Selecting a character video of the second emotion label with the greatest emotion similarity as the target video; The text reading video is generated according to the target video, the virtual character and the text reading audio.
8. A video generating device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video generation method according to any one of claims 1 to 7.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video generation method according to any one of claims 1 to 7 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the video generation method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Virtual image generation method and device, terminal and storage medium
CN110827379A
Video generation method and related device
CN111669515A
Video generation method and device, equipment and storage medium
CN114242070A
Audio book manufacturing method and device and storage medium
CN116403561A
Virtual image generation method and device, electronic equipment and storage medium
CN116704085A
Cited By
Speech generation method and device of script, equipment and medium
CN120726988A
System and method for automatically generating audiobooks with animation and computer-readable storage medium thereof
TWI929632B
Method and apparatus for generating video, and device and storage medium
WO2026157632A1
