Method and device for generating video based on copywriting, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, and discloses a method and device for generating a video based on a copywriting, equipment and a medium, and the method comprises the steps: obtaining a to-be-generated video copywriting, extracting a corresponding key semantic feature through a preset semantic understanding model, converting the to-be-generated video copywriting into corresponding audio data through a preset voice conversion model, and storing the audio data in a database; the method comprises the following steps: acquiring key semantic features of audio data, marking timestamps on the audio data by adopting a text-to-speech algorithm, matching the key semantic features with a preset material library to obtain picture materials, generating an image-text video with the timestamps, performing preset mouth shape synchronous rendering on the audio data by adopting a preset dynamic conversation face video model, and generating a virtual human video with the timestamps. And according to the timestamp of the image-text video and the timestamp of the virtual human video, the image-text video and the virtual human video are merged, and a final video is generated and uploaded, so that the technical problem that the customer experience is affected due to the fact that too templated video generation cannot meet the requirements of different customers at present is solved.

Description

Technical Field

[0001] The present invention relates to the fields of financial technology and artificial intelligence, and in particular to a method, device, equipment and medium for generating videos based on text. Background Art

[0002] In the digital age, video, as one of the attractive media, has become a powerful tool for conveying information, telling stories and creating influence. High-quality video production often requires expensive software and professional skills. With the development of artificial intelligence technology, users can generate videos through simple text descriptions or upload pictures, which improves the efficiency of creation. These tools provide a variety of templates and special effects libraries, support a variety of styles and background music. With the development of deep learning, video generation has been greatly improved in terms of picture quality, length, and coherence. Technological advances have made the characteristics of video generation tasks that contain a lot of picture information, high complexity, strong randomness of actions, and also need to consider factors such as space and time. Although video generation technology has made progress, the generation effect is not ideal. Video generation products that can reach commercial levels in the three dimensions of video length, fidelity, and coherence are still lacking.

[0003] However, there are still some problems with the existing video intelligent generation technology. For example, the generated video content is too templated, lacks innovation and personalization, and cannot meet the needs of different customers; the quality of the generated video content is unstable, sometimes too rough, sometimes too fancy, affecting the customer experience; in addition, the speed of generating videos is slow and cannot meet the real-time requirements.

[0004] Currently, insurance sales mainly rely on offline agent visits, which is inefficient and cannot meet customers' demand for personalized insurance products. In addition, with the development of Internet technology, customers' behavior patterns are also changing. They are more inclined to obtain information, compare products, and complete transactions online.

[0005] Therefore, how to generate scenario videos based on copywriting to meet the needs of personalized insurance products while reducing the workload of agents and improving efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present invention provides a method, device, computer equipment and medium for generating videos based on text, so as to solve the technical problem of low efficiency caused by the merchants needing to fill in cumbersome forms when the same financial products have different categories, prices and other differentiated demands for different B-ends (merchants).

[0007] In a first aspect, a method for generating a video based on a copy is provided, comprising:

[0008] Obtain the video text to be generated and extract the corresponding key semantic features through the preset semantic understanding model;

[0009] The video text to be generated is converted into corresponding audio data by using a preset speech conversion model, and the audio data is timestamped by using a text-to-speech algorithm;

[0010] Match the key semantic features with the preset material library to obtain image materials, and generate graphic and text videos with timestamps;

[0011] Using a preset dynamic talking face video model to perform preset lip-sync rendering on the audio data to generate a virtual human video with a timestamp;

[0012] According to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged to generate a final video and upload it.

[0013] In a second aspect, a device for generating a video based on a copy is provided, comprising:

[0014] An acquisition module is used to acquire the video text to be generated and extract the corresponding key semantic features through a preset semantic understanding model;

[0015] An audio data module, used to convert the video text to be generated into corresponding audio data through a preset speech conversion model, and mark the audio data with a timestamp using a text-to-speech algorithm;

[0016] A picture, text and video module, used to match the key semantic features with a preset material library to obtain picture materials and generate picture, text and video with timestamps;

[0017] A virtual human video module, used for performing preset lip-sync rendering on the audio data using a preset dynamic talking human face video model, and generating a virtual human video with a timestamp;

[0018] The video merging module is used to merge the graphic video and the virtual human video according to the timestamp of the graphic video and the timestamp of the virtual human video, generate a final video and upload it.

[0019] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for generating a video based on text when executing the computer program.

[0020] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned method for generating a video based on a text are implemented.

[0021] In the scheme implemented by the above-mentioned method, device, computer equipment and storage medium for generating video based on text, the video text to be generated can be obtained, and the corresponding key semantic features can be extracted through a preset semantic understanding model, the video text to be generated can be converted into corresponding audio data through a preset speech conversion model, and the audio data can be timestamped by a text-to-speech algorithm, the key semantic features can be matched with a preset material library to obtain image materials, and a graphic video with a timestamp can be generated, and a preset dynamic conversation face video model can be used to perform preset lip synchronization rendering on the audio data to generate a virtual human video with a timestamp, and the graphic video and the virtual human video can be merged according to the timestamp of the graphic video and the timestamp of the virtual human video to generate a final video and upload it, thereby solving the technical problem that the current existing video intelligent generation technology cannot meet the needs of different customers due to being too templated, thus affecting the customer experience.

[0022] At the same time, according to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged, which solves the technical problem of low efficiency of the current insurance sales mainly relying on offline agent visits.

[0023] Furthermore, the audio features are input into a preset dynamic conversation face video model, the virtual human head movements and facial feature movements are associated, and preset lip synchronization rendering is performed to generate a virtual human video with a timestamp, thereby achieving personalized video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0025] Figure 1 is a schematic diagram of an application environment of a method for generating a video based on a copy in an embodiment of the present invention;

[0026] Figure 2 It is a flow chart of a method for generating a video based on a text in one embodiment of the present invention;

[0027] Figure 3 is a structural schematic diagram of a device for generating a video based on a copy in one embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0029] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The method for generating a video based on a text provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, wherein the video text to be generated is obtained through the client, and the server extracts the corresponding key semantic features through a preset semantic understanding model; the video text to be generated is converted into corresponding audio data through a preset speech conversion model, and the audio data is timestamped by a text-to-speech algorithm; the key semantic features are matched with a preset material library to obtain image materials, and a graphic video with a timestamp is generated; a preset dynamic conversation face video model is used to perform preset lip synchronization rendering on the audio data to generate a virtual human video with a timestamp; according to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged to generate a final video and upload it. In the present invention, computer vision technology and artificial intelligence technology are used to generate high-quality video content in real time. The quality of the generated video content is evaluated and optimized to ensure the stability of the quality of the video content. The generated video content is transmitted to customers via the Internet to achieve online sales and promotion. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 As shown, Figure 2 A flowchart of a method for generating a video based on a text provided by an embodiment of the present invention includes the following steps:

[0033] S10: Obtain the video text to be generated, and extract the corresponding key semantic features through a preset semantic understanding model;

[0034] In this embodiment, the step of obtaining the video text to be generated and extracting the corresponding key semantic features through a preset semantic understanding model includes:

[0035] Performing preprocessing on the obtained video text to be generated by removing redundancy and colloquialism;

[0036] The key semantic features corresponding to the preprocessed video text to be generated are extracted through a preset semantic understanding model.

[0037] In one embodiment, the script input by the user or the generated script may be subjected to basic redundancy removal, and then TTS generates audio. TTS is the abbreviation of Text-to-Speech, which is a technology that converts text information into voice signals. The TTS system usually includes three steps: text preprocessing, speech synthesis, and speech post-processing. In the text preprocessing stage, the system will perform operations such as word segmentation, part-of-speech tagging, and grammatical analysis on the input text to identify words and phrases in the text. Then, in the speech synthesis stage, the preprocessed text will be converted into a speech signal, and speech will be generated through a speech synthesizer. Finally, in the speech post-processing stage, the system will perform operations such as pitch adjustment, sound quality improvement, and noise elimination on the generated speech to improve the naturalness and clarity of the speech.

[0038] Get the video text to be generated and extract the corresponding key semantic features through the preset semantic understanding model. Previously, it also included:

[0039] Adopting the retrieval enhancement generation algorithm to retrieve the copywriting information in the preset knowledge base;

[0040] The video copy to be generated that is content-associated is generated according to the copy information.

[0041] It is understandable that large-model text generation based on retrieval-augmented generation can ensure the accuracy of the generated content. The large-model text generation technology based on retrieval-augmented generation (RAG) is a method that combines information retrieval technology and large language models to improve the ability to handle knowledge-intensive tasks. This technology retrieves relevant information from an external knowledge base and provides it as input to a large language model to enhance the model's ability to understand and answer specific questions. The RAG model was first proposed by the Facebook AI Research (FAIR) team in 2020 and quickly became a popular solution in large-model applications.

[0042] The key to RAG technology is its ability to utilize external data, thereby getting rid of the limitations of model training data and improving the accuracy and richness of generated content. It mainly consists of two core modules: the retrieval module (Retriever) and the generation module (Generator). The retrieval module is responsible for retrieving information related to the problem from the pre-established knowledge base, while the generation module uses the retrieved information as context input to generate text content that meets user needs and context.

[0043] In practical applications, RAG can be used in a variety of scenarios such as question-answering systems, text summarization, and content generation, significantly improving content production efficiency and coverage. For example, in the fields of media publishing and e-commerce, RAG technology can help automate or assist in the manual production of various text content, such as news, financial reports, weather forecasts, abstracts, meeting minutes, reviews, official documents, product descriptions, advertising copy, dialogue replies, comments, etc.

[0044] In addition, RAG technology also involves some technical challenges, such as knowledge cutoff problem, hallucination problem, etc., which may cause the model to confidently make wrong responses. In order to solve these problems, researchers are constantly exploring new methods and technologies, such as introducing multimodal information such as knowledge graphs and sentiment analysis to improve the accuracy and diversity of text generation.

[0045] It should be further explained that obtaining the video text to be generated and extracting the corresponding key semantic features through the preset semantic understanding model also includes:

[0046] For the pre-acquired image data, the attention mechanism is used to extract key content features;

[0047] The key content features are marked with text content tags to generate the preset material library.

[0048] For example, we first need to build a rich material library. The construction of the material library involves the generation of materials and the labeling and storage of materials. Stable diffusion is used to generate static materials. Because the business has accumulated rich video creation data and rich pictures and audio materials, these videos or materials will also be understood and split. For example, TTS obtains high-quality script texts and establishes feature extractors to extract valuable image information from videos and pictures.

[0049] Stable Diffusion is an advanced text-to-image generation model that generates high-quality static images by using diffusion models. These models iteratively shape random noise to transform it into AI-generated artistic images, a process known as denoising diffusion. The core technologies of Stable Diffusion include variational autoencoders (VAE), U-Net networks, and a text encoder, where VAE is used to compress images into a latent space, and U-Net is responsible for generating images from the latent space. This model is able to handle complex text prompts and generate images with high realism and rich details.

[0050] In addition, Stable Diffusion 3 uses a new Multimodal Diffusion Transformer (MMDiT) architecture that uses independent sets of weights to process image and language representations, improving text understanding and spelling capabilities. When generating images, the Stable Diffusion model is able to generate images with rich semantic information based on text prompts, a process that usually requires 30 to 50 iterations of denoising.

[0051] For example, parsing text into machine-readable labels or instructions, describing elements in the image, such as "cat", "suitcase", and the processed visual content itself. In image processing, encoders are usually used to convert images into a format that the model can process. A cat laying in a suitcase is used to generate or recognize images. In the image generation model, the decoder is responsible for converting the latent space representation into a visual image. The interactive encoder is used to process user interaction and feedback components, and the interactive information is encoded into a format that the model can understand. Finally, the image and "cat" and "suitcase" are labeled, including the collection, labeling, storage and development of programmable interfaces of various materials. The core of this link is to understand various types of video materials, label them with content and style, and extract various attributes such as shape, size, color, and duration.

[0052] S11, converting the to-be-generated video text into corresponding audio data through a preset speech conversion model, and marking a timestamp on the audio data using a text-to-speech algorithm;

[0053] In this embodiment, a human voice audio sample is obtained, acoustic features are extracted, and a sound cloning algorithm is used to generate the audio data corresponding to the acoustic features. In the process of generating audio, the agent audio data can be collected, and audio with the agent's timbre can be generated through sound cloning (technically, encoding and decoding are performed based on the transformer model architecture, key acoustic features are extracted based on the input voice sample, and then the decoder generates a new voice output based on these features, which can effectively capture and reproduce the voice characteristics of the speaker. In terms of training, a large amount of voice data is first used for pre-training to master the basic characteristics and structure of the voice, and then a small number of voice samples of a specific person are used for fine-tuning to learn to imitate the specific voice characteristics of the person. In this way, the agent only needs to record three or five sentences to achieve a permanent simulated voice generation effect)

[0054] Specifically, for example, voice sample input: First, a voice sample of a specific person needs to be input. These samples will be used to extract key acoustic features and serve as training data. Feature extraction: Use a deep learning model, such as WavLM or Whisper, to extract features from the input voice sample. The WavLM model is pre-trained based on 94,000 hours of unsupervised English data. The decoder generates voice output: The extracted features are fed into the decoder, which generates new voice output based on these features. This process can effectively capture and reproduce the voice characteristics of the speaker. Pre-training and fine-tuning: In training, a large amount of voice data is first used for pre-training to master the basic features and structure of the voice. Then a small amount of voice samples of a specific person are used for fine-tuning so that the model learns to imitate the specific voice characteristics of that person. Computing resource requirements: Since these models are usually large, they require relatively high computing resources. For example, the WavLM model may require a GPU with 16GB or more video memory to run. The Whisper model has similar resource requirements. Datasets: Training these models usually requires a large dataset. For example, WavLM was pre-trained using 94,000 hours of data, while the specific dataset for the Whisper model was not explicitly mentioned in the search results. Fine-tuning techniques: Some projects, such as VITS-fast-fine-tuning, provide fast fine-tuning methods that allow users to add their own voices to pre-trained models. Model optimization: For example, the Whisper model has optimized versions, such as Whisper large-v3-turbo.

[0055] S12, matching the key semantic features with a preset material library to obtain image materials, and generating a graphic video with a timestamp;

[0056] include:

[0057] Matching the key semantic features with the video templates in the preset material library;

[0058] Matching and extracting the associated picture material in the preset material library according to the key semantic features;

[0059] Fill the image material into the video template to generate the graphic video with a timestamp.

[0060] It is understandable that generating graphic and text videos requires semantic understanding, retrieving suitable image materials in the material library for splicing and layout, and then matching the appropriate images with the corresponding audio frames.

[0061] For example, semantic understanding: First, the system needs to understand the semantics of the input text, which usually involves natural language processing (NLP) technology. The purpose is to enable the computer to understand the meaning of the text and extract key information to prepare for subsequent speech synthesis. Material library retrieval: According to the results of semantic understanding, retrieve picture materials that match the text content in the material library. This step requires image recognition and matching algorithms to ensure the relevance and applicability of the material. Picture material splicing and layout: The retrieved picture materials are spliced ​​and laid out to form the visual content of the video. This process may involve image editing and typesetting technology, as well as elements of creative design. Audio matching: Select appropriate audio materials and match these audios with each frame in the video. This usually requires audio processing and timeline alignment technology to ensure synchronization of audio and visual content. Speech synthesis: Use speech synthesis technology (such as TTS) to convert the processed text into speech and synchronize it with the corresponding part in the video. This may involve deep learning models such as WaveNet or Tacotron. Video generation: Merge the processed visual content and audio to generate the final text-and-picture video. This step may involve video encoding and compression technology to ensure the quality and smoothness of the video. Optimization and testing: Optimize and test the generated video to ensure the synchronization of video and audio, as well as the overall viewing experience. This may involve user feedback collection and iterative improvement.

[0062] S13, using a preset dynamic conversation face video model to perform preset lip synchronization rendering on the audio data to generate a virtual human video with a timestamp;

[0063] In this embodiment, it specifically includes:

[0064] Using a Mel frequency algorithm to extract audio features corresponding to the audio data;

[0065] The audio features are input into a preset dynamic conversation face video model, the head movements of the virtual person and the facial feature movements are associated, and preset lip synchronization rendering is performed to generate a virtual person video with a timestamp.

[0066] It should be noted that video generation technology mainly relies on deep learning and machine learning algorithms, which learn features and patterns in videos by analyzing large amounts of video data. For example, generative adversarial networks (GANs) and variational autoencoders (VAEs) are used to generate realistic video content. Core algorithm architecture: The core algorithm architecture of virtual human template video includes learning based on a small amount of video data, Chinese deep feature extraction based on MFCC, 3D feature extraction based on 3DMM, integrated sound encoding and decoding, and the combination of head movements, blinking, and lip movements. The video rendering network is also part of the core algorithm, which is responsible for rendering the generated video frames into a coherent video output. Video generation process: The video generation process usually involves a video compression network, which compresses the video content into a more compact form for more efficient processing. Next, the video is decomposed into small blocks containing spatial and temporal information, which are called spatial-temporal latent patches. Finally, a Transformer model receives these patches and text prompts and decides how to transform or combine these fragments to generate the final video. Multimodal learning: Multimodal learning is an important aspect in AI video generation, which involves processing and integrating information from different data sources, such as text, images, audio, and video. In text-to-video generation, multimodal learning models combine text input with the video content generation process through joint representation learning, cross-modal translation, and end-to-end training. Advanced video generation architectures often combine diffusion models and Transformers to generate high-quality videos with smooth motion and reasonable physical effects. The development of multimodal learning techniques is crucial to improving the quality and accuracy of text-to-video. AI video generators: AI video generators use advanced machine learning algorithms to analyze text input, select appropriate visual effects, generate realistic avatars and sounds, and combine these elements into a coherent video. These tools are revolutionizing content creation, enabling individuals and businesses to quickly and affordably produce engaging video content. Tools typically include functions such as text-to-video generation, image-to-video conversion, video editing, frame interpolation, and loop video generation.

[0067] In summary, the implementation process of AI video generation technology involves multiple complex steps, including video data analysis, feature extraction, model training, video rendering, and multimodal learning. These technological advances provide new possibilities for content creation, making the conversion from text description to video content more efficient and accurate.

[0068] It can be understood that 3D Face Reconstruction, which generally refers to the process of reconstructing a three-dimensional facial model from a two-dimensional image or video using computer vision technology. This process may involve the following aspects: 3D Face: refers to the generated three-dimensional facial model, which may include facial geometry, feature points, texture, etc. Reconstruction: refers to the process of recovering the three-dimensional structure from a two-dimensional image. Identity: In the context of facial reconstruction, this may refer to the personal identity represented by the reconstructed facial model, or the visual consistency of the model with the real person. Texture: refers to the surface details of the facial model, such as skin texture, wrinkles, etc., which increase the realism of the model. Lighting: refers to the lighting effect simulated when rendering a three-dimensional facial model, which is crucial to making the model look more realistic.

[0069] In summary, it is necessary to combine deep learning and computer graphics methods, such as convolutional neural networks (CNN) and generative adversarial networks (GAN), to achieve high-quality facial reconstruction.

[0070] FACIAL-GAN is an advanced generative adversarial network for generating videos with dynamic talking faces. It can synthesize realistic face videos with synchronized lip movements and personalized head movements and blinks under audio-driven conditions. This technology contains a time-dependent generator and a local speech generator, which can jointly learn speech, context, and personalized information, and generate contextual latent space vectors to generate videos with implicit attributes (such as head pose, blink information) and explicit attributes (such as lip movements). During the training process, FACIAL-GAN is first pre-trained with a large amount of speech data to master the basic features and structure of speech, and then fine-tuned with a small number of speech samples of a specific person to learn to imitate the specific voice characteristics of that person. In addition, FACIAL-GAN can also predict blink information and embed it into the final rendering module through attention maps to output realistic eye movement information. The quality of the videos generated by this method is significantly better than the existing advanced methods, and can effectively capture and reproduce the voice characteristics of the speaker.

[0071] MFCC (Mel Frequency Cepstral Coefficients) is a feature extraction method widely used in audio signal processing, especially in speech recognition and speaker recognition. MFCC is designed based on the human hearing characteristics, simulating the human ear's perception of sound frequency, and extracting information reflecting its essential characteristics from the speech signal through a series of processing steps.

[0072] The calculation process of MFCC includes the following key steps: Pre-emphasis: amplify the high frequency part through the pre-emphasis filter, increase some high frequency energy, and reduce the impact of high frequency loss in signal transmission. Framing and windowing: divide the signal into short time frames and apply a window function to each frame to reduce the discontinuity between frames. Fast Fourier transform (FFT): perform FFT on the signal of each frame to convert it from the time domain to the frequency domain. Mel filter bank: pass the frequency domain signal through a set of Mel filter banks to simulate the auditory perception characteristics of the human ear and convert the frequency domain signal to the Mel scale frequency domain. Logarithmic operation: perform logarithmic operation on the frequency domain signal after passing through the Mel filter bank to enhance the dynamic range of the speech signal. Discrete cosine transform (DCT): perform DCT on the logarithmic spectrum obtained in the previous step and convert it to the cepstral space. Normalization: normalize the cepstral coefficients to enhance the stability of the speech signal. Feature splicing: splice the cepstral coefficients of adjacent frames to form the final MFCC feature vector.

[0073] MFCC features usually include the first 12 coefficients, plus the energy of the current frame, for a total of 13 features. The more forward coefficients contain more information about the fundamental frequency and formant. In addition, the first-order and second-order differences of these coefficients are calculated to capture the temporal variation characteristics of speech. The advantage of MFCC is that it describes the information of the time-spectrum graph with less data, has lower correlation between features, has better distinguishability, and can extract information that characterizes the fundamental frequency and formant. However, it is relatively computationally intensive, not robust to noise, and is ineffective for speech synthesis because there is no inverse transform from MFCC features to audio signals.

[0074] S14. Merge the graphic video and the virtual human video according to the timestamp of the graphic video and the timestamp of the virtual human video to generate a final video and upload it.

[0075] In this embodiment, the process includes rough cutting of the video (recognition of long silent segments, addition of subtitles, addition of covers, etc., matching of corresponding stickers, GIFs, etc. at appropriate locations, cutting of the video, addition of transition effects, etc.). After the video is generated, there are mainly the following steps to complete the packaging and distribution of the video: Video post-processing: including adding subtitles, generating static covers, and adding watermarks to the video; Video review process: including script compliance review, material compliance review, and material copyright review. Since the materials are extracted from the top regular copyright websites, the copyright review is automatically passed, and the compliance review of the materials will also be automatically passed. Video distribution: upload the corresponding storage according to customer needs to complete the distribution of the video.

[0076] Use computer vision and artificial intelligence technologies to generate high-quality video content in real time. Evaluate and optimize the quality of the generated video content to ensure the stability of the quality of the video content. Transmit the generated video content to customers via the Internet to achieve online sales and promotion.

[0077] In one embodiment, for example, when an insurance agent generates a promotional video, for the agent's private domain business customers, by generating video content online, the agent can introduce insurance products to customers more quickly and easily, reduce intermediate links, and improve sales efficiency. In addition, the agent can also automatically generate innovative and personalized video content based on the customer's personal information and insurance needs, improve customer experience, and attract more customers. For agents operating public domain short video accounts, automatically generating innovative and personalized video content can improve the professionalism and influence of agents and promote the improvement of IP. For customers, the improved sales method can provide better quality, personalized, and convenient services. By generating video content online, customers can have a more comprehensive understanding of insurance products, quickly compare different products, and improve the efficiency of purchasing decisions. In addition, automatically generating innovative and personalized video content based on the customer's personal information and insurance needs can meet the customer's personalized needs and improve customer experience.

[0078] In addition, the generated videos are used for public / private domain operations, and from the perspective of platform functions, they provide compliance monitoring based on insurance scenarios and connect major content platforms to support one-click distribution and observation of subsequent data.

[0079] The method for generating videos based on text provided in the present embodiment can obtain the text of the video to be generated, extract the corresponding key semantic features through a preset semantic understanding model, convert the video text to be generated into corresponding audio data through a preset speech conversion model, and use a text-to-speech algorithm to mark the timestamp on the audio data, match the key semantic features with a preset material library to obtain image materials, and generate a text-and-graphic video with a timestamp, use a preset dynamic conversation face video model to perform preset lip synchronization rendering on the audio data, generate a virtual human video with a timestamp, merge the text-and-graphic video with the virtual human video according to the timestamp of the text-and-graphic video and the timestamp of the virtual human video, generate a final video and upload it, thereby solving the technical problem that the existing video intelligent generation technology cannot meet the needs of different customers due to being too templated, thus affecting the customer experience.

[0080] At the same time, according to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged, which solves the technical problem of low efficiency of the current insurance sales mainly relying on offline agent visits.

[0081] Furthermore, the audio features are input into a preset dynamic conversation face video model, the virtual human head movements and facial feature movements are associated, and preset lip synchronization rendering is performed to generate a virtual human video with a timestamp, thereby achieving personalized video generation.

[0082] In one embodiment, a device for generating a video based on a text is provided, and the device for generating a video based on a text corresponds to the method for generating a video based on a text in the above embodiment. Figure 3 As shown, the device for generating video based on text includes an acquisition module 101, an audio data module 102, a graphic video module 103, a virtual human video module 104, and a video merging module 105. The functional modules are described in detail as follows:

[0083] The acquisition module 101 is used to acquire the video text to be generated and extract the corresponding key semantic features through a preset semantic understanding model;

[0084] In one embodiment, the acquisition module 101 is specifically used for the following steps:

[0085] Performing preprocessing on the obtained video text to be generated by removing redundancy and colloquialism;

[0086] The key semantic features corresponding to the preprocessed video text to be generated are extracted through a preset semantic understanding model.

[0087] The audio data module 102 is used to convert the video text to be generated into corresponding audio data through a preset speech conversion model, and mark the audio data with a timestamp using a text-to-speech algorithm;

[0088] The image-text-video module 103 is used to match the key semantic features with a preset material library to obtain image materials and generate image-text-video with a timestamp;

[0089] In one embodiment, the text-image module 103 is specifically used for the following steps:

[0090] Matching the key semantic features with the video templates in the preset material library;

[0091] Matching and extracting the associated picture material in the preset material library according to the key semantic features;

[0092] Fill the image material into the video template to generate the graphic video with a timestamp.

[0093] A virtual human video module 104 is used to perform preset lip-sync rendering on the audio data using a preset dynamic talking human face video model to generate a virtual human video with a timestamp;

[0094] In one embodiment, the virtual human video module 104 is specifically used for the following steps:

[0095] Using a Mel frequency algorithm to extract audio features corresponding to the audio data;

[0096] The audio features are input into a preset dynamic conversation face video model, the head movements of the virtual person and the facial feature movements are associated, and preset lip synchronization rendering is performed to generate a virtual person video with a timestamp.

[0097] The video merging module 105 is used to merge the graphic video with the virtual human video according to the timestamp of the graphic video and the timestamp of the virtual human video, generate a final video and upload it.

[0098] In this embodiment, it also includes:

[0099] The human voice audio module is used to obtain human voice audio samples, extract acoustic features, and use a sound cloning algorithm to generate the audio data corresponding to the acoustic features.

[0100] The text retrieval module is used to use a retrieval enhancement generation algorithm to retrieve text information in a preset knowledge base, and generate the video text to be generated that is content-related according to the text information.

[0101] The material library module is used to extract key content features from pre-acquired image data using an attention mechanism, mark the key content features with text content tags, and generate the preset material library.

[0102] The device for generating video based on text provided in the present embodiment can obtain the text of the video to be generated, and extract the corresponding key semantic features through a preset semantic understanding model, convert the video text to be generated into corresponding audio data through a preset speech conversion model, and use a text-to-speech algorithm to mark the timestamp on the audio data, match the key semantic features with a preset material library to obtain image materials, and generate a graphic video with a timestamp, use a preset dynamic conversation face video model to perform preset lip synchronization rendering on the audio data to generate a virtual human video with a timestamp, merge the graphic video and the virtual human video according to the timestamp of the graphic video and the timestamp of the virtual human video, generate a final video and upload it, so as to solve the technical problem that the existing video intelligent generation technology cannot meet the needs of different customers due to being too templated, thus affecting the customer experience.

[0103] At the same time, according to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged, which solves the technical problem of low efficiency of the current insurance sales mainly relying on offline agent visits.

[0104] Furthermore, the audio features are input into a preset dynamic conversation face video model, the virtual human head movements and facial feature movements are associated, and preset lip synchronization rendering is performed to generate a virtual human video with a timestamp, thereby achieving personalized video generation.

[0105] For the specific definition of the device for generating video based on text, please refer to the definition of the method for generating video based on text above, which will not be repeated here. Each module in the above-mentioned device for generating video based on text can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0106] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a method for generating a video based on a text are implemented.

[0107] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a method for generating a video based on a text are implemented.

[0108] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0109] Obtain the video text to be generated and extract the corresponding key semantic features through the preset semantic understanding model;

[0110] The video text to be generated is converted into corresponding audio data by using a preset speech conversion model, and the audio data is timestamped by using a text-to-speech algorithm;

[0111] Match the key semantic features with the preset material library to obtain image materials, and generate graphic and text videos with timestamps;

[0112] Using a preset dynamic talking face video model to perform preset lip-sync rendering on the audio data to generate a virtual human video with a timestamp;

[0113] According to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged to generate a final video and upload it.

[0114] The computer device provided in this embodiment can obtain the video text to be generated, and extract the corresponding key semantic features through a preset semantic understanding model, convert the video text to be generated into corresponding audio data through a preset speech conversion model, and use a text-to-speech algorithm to mark the audio data with a timestamp, match the key semantic features with a preset material library to obtain image materials, and generate a graphic video with a timestamp, use a preset dynamic conversation face video model to perform preset lip synchronization rendering on the audio data to generate a virtual human video with a timestamp, merge the graphic video and the virtual human video according to the timestamp of the graphic video and the timestamp of the virtual human video, generate a final video and upload it, so as to solve the technical problem that the current existing video intelligent generation technology cannot meet the needs of different customers due to being too templated, thus affecting the customer experience.

[0115] At the same time, according to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged, which solves the technical problem of low efficiency of the current insurance sales mainly relying on offline agent visits.

[0116] Furthermore, the audio features are input into a preset dynamic conversation face video model, the virtual human head movements and facial feature movements are associated, and preset lip synchronization rendering is performed to generate a virtual human video with a timestamp, thereby achieving personalized video generation.

[0117] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0118] Obtain the video text to be generated and extract the corresponding key semantic features through the preset semantic understanding model;

[0119] The video text to be generated is converted into corresponding audio data by using a preset speech conversion model, and the audio data is timestamped by using a text-to-speech algorithm;

[0120] Match the key semantic features with the preset material library to obtain image materials, and generate graphic and text videos with timestamps;

[0121] Using a preset dynamic talking face video model to perform preset lip-sync rendering on the audio data to generate a virtual human video with a timestamp;

[0122] According to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged to generate a final video and upload it.

[0123] The computer-readable storage medium provided in this embodiment can obtain the video text to be generated, and extract the corresponding key semantic features through a preset semantic understanding model, convert the video text to be generated into corresponding audio data through a preset speech conversion model, and use a text-to-speech algorithm to mark the audio data with a timestamp, match the key semantic features with a preset material library to obtain image materials, and generate a graphic video with a timestamp, use a preset dynamic conversation face video model to perform preset lip synchronization rendering on the audio data, generate a virtual human video with a timestamp, merge the graphic video and the virtual human video according to the timestamp of the graphic video and the timestamp of the virtual human video, generate a final video and upload it, so as to solve the technical problem that the current existing video intelligent generation technology cannot meet the needs of different customers due to being too templated, thus affecting the customer experience.

[0124] At the same time, according to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged, which solves the technical problem of low efficiency of the current insurance sales mainly relying on offline agent visits.

[0125] Furthermore, the audio features are input into a preset dynamic conversation face video model, the virtual human head movements and facial feature movements are associated, and preset lip synchronization rendering is performed to generate a virtual human video with a timestamp, thereby achieving personalized video generation.

[0126] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0128] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

[0130] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A method for generating a video based on a copy, characterized in that: include: Obtain the video text to be generated, and extract the corresponding key semantic features through the preset semantic understanding model; The video text to be generated is converted into corresponding audio data by using the preset speech conversion model, and the audio data is timestamped by using a text-to-speech algorithm; Match the key semantic features with the preset material library to obtain image materials, and generate graphic and text videos with timestamps; Using a preset dynamic talking face video model to perform preset lip-sync rendering on the audio data to generate a virtual human video with a timestamp; According to the timestamp of the graphic video and the timestamp of the virtual human video, the graphic video and the virtual human video are merged to generate a final video and upload it.

2. The method for generating a video based on a copy as claimed in claim 1, characterized in that: The method of obtaining the video text to be generated and extracting the corresponding key semantic features through a preset semantic understanding model also includes: Adopting the retrieval enhancement generation algorithm to retrieve the copywriting information in the preset knowledge base; The video copy to be generated that is content-associated is generated according to the copy information.

3. The method for generating a video based on a copy as claimed in claim 1, characterized in that: The method of obtaining the video text to be generated and extracting the corresponding key semantic features through a preset semantic understanding model also includes: Use the attention mechanism to extract key content features from pre-acquired image data; The key content features are marked with text content tags to generate the preset material library.

4. The method for generating a video based on a copy as claimed in claim 1, characterized in that: The step of obtaining the video text to be generated and extracting the corresponding key semantic features through a preset semantic understanding model includes: Performing pre-processing on the obtained video text to be generated by removing redundancy and colloquialism; The key semantic features corresponding to the preprocessed video text to be generated are extracted through a preset semantic understanding model.

5. The method for generating a video based on a copywriting according to claim 1, characterized in that: The key semantic features are matched with the preset material library to obtain the image material, and a graphic video with a timestamp is generated, including: Matching the key semantic features with the video templates in the preset material library; Matching and extracting the associated picture material in the preset material library according to the key semantic features; Fill the image material into the video template to generate the graphic video with a timestamp.

6. The method for generating a video based on a text as claimed in claim 1, characterized in that: The preset dynamic conversation face video model is used to perform preset lip synchronization rendering on the audio data to generate a virtual human video with a timestamp, including: Using a Mel frequency algorithm to extract audio features corresponding to the audio data; The audio features are input into a preset dynamic conversation face video model, the head movements of the virtual person and the facial feature movements are associated, and preset lip synchronization rendering is performed to generate a virtual person video with a timestamp.

7. The method for generating a video based on a copy as claimed in claim 1, characterized in that: The video text to be generated is converted into corresponding audio data by using a preset voice conversion model, and then the method further includes: Obtain human voice audio samples and extract acoustic features; The audio data corresponding to the acoustic features is generated using a sound cloning algorithm.

8. A device for generating a video based on a copy, characterized in that: include: An acquisition module is used to acquire the video text to be generated and extract the corresponding key semantic features through a preset semantic understanding model; An audio data module, used to convert the video text to be generated into corresponding audio data through a preset speech conversion model, and mark the audio data with a timestamp using a text-to-speech algorithm; A picture, text and video module, used to match the key semantic features with a preset material library to obtain picture materials and generate picture, text and video with timestamps; A virtual human video module, used for performing preset lip-sync rendering on the audio data using a preset dynamic talking human face video model, and generating a virtual human video with a timestamp; The video merging module is used to merge the graphic video and the virtual human video according to the timestamp of the graphic video and the timestamp of the virtual human video, generate a final video and upload it.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generating a video based on a text as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a video based on a text as described in any one of claims 1 to 7 are implemented.

Images