A video synthesis method, device and readable storage medium

Abstract

The application discloses a video synthesis method, device and readable storage medium. The method comprises the following steps: when an original text is acquired, acquiring a text to be processed in the original text; acquiring a text semantic vector of the text to be processed; searching for a candidate picture vector with the same semantics as the text semantic vector in a candidate picture vector corresponding to a candidate picture; determining a text mapping picture corresponding to the text to be processed based on the candidate picture corresponding to the searched candidate picture vector; when a speech conversion parameter associated with the text to be processed is acquired, converting the text to be processed into a text mapping speech conforming to the speech conversion parameter; based on the text to be processed, the text mapping picture and the text mapping speech, generating an audio frame and a video frame used for synthesizing a target video; and performing video synthesis processing on the generated audio frame and video frame to obtain a target video corresponding to the original text. By using the application, the audio-visual display effect of the text can be enriched in the process of converting the text into a video.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a video synthesis method, apparatus, and readable storage medium. Background Technology

[0002] In existing technologies, for a given piece of text, different display effects can be achieved by adjusting the text display style (such as font, color, etc.) or manually adding images. However, these display effects are still relatively simple and often cannot meet the diverse needs of business users. In addition, these processing methods require a lot of manpower and time, resulting in low efficiency in displaying multimedia data (such as text, images, etc.). Summary of the Invention

[0003] This application provides a video synthesis method, apparatus, and readable storage medium, which can enrich the audiovisual display effect of text and improve the display efficiency of multimedia data during the text-to-video conversion process.

[0004] One embodiment of this application provides a video synthesis method, including:

[0005] When the original text is obtained, the text to be processed is obtained from the original text; the length of the text to be processed is less than the length of the original text.

[0006] Obtain the text semantic vector of the text to be processed, find the candidate image vector with the same semantics as the text semantic vector in the candidate image vector corresponding to the candidate image, and determine the text mapping image corresponding to the text to be processed based on the candidate image corresponding to the found candidate image vector.

[0007] When the speech conversion parameters associated with the text to be processed are obtained, the text to be processed is converted into text-mapped speech that matches the speech conversion parameters;

[0008] Based on the text to be processed, the text-mapped image, and the text-mapped speech, audio frames and video frames are generated for synthesizing the target video. The generated audio frames and video frames are then processed for video synthesis to obtain the target video corresponding to the original text.

[0009] One embodiment of this application provides a video synthesis apparatus, including:

[0010] The text acquisition module is used to extract the text to be processed from the original text when the original text is acquired; the length of the text to be processed is less than the length of the original text.

[0011] The image search module is used to obtain the text semantic vector of the text to be processed, search for candidate image vectors with the same semantics as the text semantic vector in the candidate image vectors corresponding to the candidate images, and determine the text mapping image corresponding to the text to be processed based on the candidate images corresponding to the found candidate image vectors.

[0012] The speech conversion module is used to convert the text to be processed into a text-mapped speech that matches the speech conversion parameters when the speech conversion parameters associated with the text to be processed are obtained.

[0013] The video generation module is used to generate audio and video frames for synthesizing the target video based on the text to be processed, the text-mapped image, and the text-mapped speech. The generated audio and video frames are then processed to synthesize the target video corresponding to the original text.

[0014] The aforementioned text acquisition module includes:

[0015] The first segmentation unit is used to identify the first type of segmentation symbols in the original text when the original text is acquired, and to perform text segmentation on the original text based on the identified first type of segmentation symbols. The transition text of the segmented original text is used as the first transition text. The text length of the first transition text is less than the text length of the original text.

[0016] The text classification unit is used to classify the transition text with a length greater than a length threshold as the text to be detected and the transition text with a length less than or equal to the length threshold as the first text to be processed in the first transition text.

[0017] The second segmentation unit is used to identify the second type of segmentation symbols in the text to be detected, and to segment the text based on the identified second type of segmentation symbols. The transition text of the segmented text to be detected is used as the second transition text. The i-th transition text and the j-th transition text are obtained from any two adjacent transition texts of the second transition text; j = i + 1; i is a positive integer less than K, and K is the number of transition texts of the segmented text to be detected.

[0018] The correlation detection unit is used to perform text correlation detection on the i-th transition text and the j-th transition text, obtain the text correlation detection result between the i-th transition text and the j-th transition text, obtain the second text to be processed based on the text correlation detection result, and use the first text to be processed and the second text to be processed as the original text to be processed.

[0019] The aforementioned relevant detection units include:

[0020] The first conversion subunit is used to perform vector conversion on the i-th transition text to obtain the first text conversion vector corresponding to the i-th transition text, and to perform vector conversion on the j-th transition text to obtain the second text conversion vector corresponding to the j-th transition text.

[0021] The model detection subunit is used to input the first text transformation vector and the second text transformation vector into the relevance detection model, and to perform text relevance detection on the first text transformation vector and the second text transformation vector through the relevance detection model to obtain the text relevance detection result between the i-th transition text and the j-th transition text.

[0022] The first determining subunit is used to take the synthesized text composed of the i-th transition text and the j-th transition text as the second text to be processed if the text relevance detection result indicates that there is a relevance between the i-th transition text and the j-th transition text.

[0023] The second determining subunit is used to treat both the i-th and j-th transition texts as the second texts to be processed if the text relevance detection result indicates that there is no relevance between the i-th and j-th transition texts.

[0024] The aforementioned image search module includes:

[0025] The text encoding unit is used to input the text to be processed into the text encoding model, and the text encoding model encodes the text to be processed to obtain the text semantic vector of the text to be processed.

[0026] The image encoding unit is used to input candidate images into the image encoding model, and to encode the candidate images through the image encoding model to obtain the candidate image vector corresponding to the candidate images.

[0027] The vector search unit is used to find the candidate image vector with the minimum vector distance to the text semantic vector among the candidate image vectors corresponding to the candidate image, and use it as the candidate image vector with the same semantics as the text semantic vector.

[0028] The image determination unit is used to determine the text mapping image corresponding to the text to be processed based on the candidate image corresponding to the found candidate image vector.

[0029] The aforementioned text encoding unit includes:

[0030] The second transformation subunit is used to perform vector transformation on the text to be processed, so as to obtain the third text transformation vector corresponding to the text to be processed.

[0031] The text encoding subunit is used to input the third text transformation vector into the text encoding network in the text encoding model. The text encoding network encodes the third text transformation vector to obtain the text semantic vector of the text to be processed.

[0032] Specifically, the second conversion subunit is used to perform word segmentation on the text to be processed, obtain the text word segmentation of the text to be processed, perform word vector conversion on the text word segmentation, obtain the word vector corresponding to the text word segmentation, obtain the global semantic vector corresponding to the text to be processed and the position vector associated with the text position of the text word segmentation in the text to be processed, and perform vector fusion on the word vector, global semantic vector and position vector to obtain the third text conversion vector corresponding to the text to be processed.

[0033] The text encoding model contains M text encoding networks, including text encoding network E. m and text encoding network E m+1 Text encoding network E m+1 For text encoding network E m The next text encoding network, where m is a positive integer less than M;

[0034] The aforementioned text encoding subunit is specifically used to input the third text transformation vector into the text encoding network in the text encoding model, and then through the text encoding network E... m Output the historical text encoding vector and input the historical text encoding vector into the text encoding network E. m+1 In text encoding network E m+1 In the middle, based on historical text encoding vectors and text encoding network E m+1 The first intermediate encoding vector is obtained by using a multi-head self-attention network in the text encoding network. Then, residual concatenation and normalization are performed on the first intermediate encoding vector and the historical text encoding vector to obtain the second intermediate encoding vector. This second intermediate encoding vector is then input into the text encoding network E. m+1 The fully connected layer in the first layer yields the third intermediate encoding vector; residual connections and normalization are then performed on the third and second intermediate encoding vectors to obtain the text encoding network E. m+1 The target text encoding vector is used to obtain the text semantic vector of the text to be processed.

[0035] The aforementioned image encoding unit includes:

[0036] The image segmentation subunit is used to segment candidate images and use the segmented images as region images.

[0037] The feature extraction subunit is used to input the region image into the feature extraction network in the image coding model, extract features from the region image through the feature extraction network, obtain the image features corresponding to the region image, and obtain the image feature sequence based on the image features corresponding to the region image.

[0038] The image encoding subunit is used to input the image feature sequence into the image encoding network in the image encoding model. The image encoding network encodes the image feature sequence to obtain the candidate image vector corresponding to the candidate image.

[0039] Wherein, if the number of candidate image vectors found is N, then the number of candidate images corresponding to the found candidate image vectors is N; N is a positive integer greater than 1;

[0040] The aforementioned image determination unit includes:

[0041] The parameter configuration subunit is used to obtain the N candidate images found and configure the image arrangement order and image switching frequency for the N candidate images;

[0042] The image synthesis subunit is used to synthesize N candidate images based on the configured image arrangement order and image switching frequency, and use the synthesized image as the text mapping image corresponding to the text to be processed.

[0043] Among them, the speech conversion parameters are obtained by the business object associated with the original text by triggering the voiceprint in the voiceprint list; the voiceprint list is output by the business object by triggering the voiceprint selection control, or the voiceprint list is output when keywords carrying business object attributes used to characterize the business object are identified in the text to be processed.

[0044] The aforementioned speech conversion module includes:

[0045] The phoneme conversion unit is used to perform phoneme conversion on the text to be processed based on the speech conversion parameters associated with the text to be processed when the speech conversion parameters associated with the text to be processed are obtained, so as to obtain the phoneme sequence corresponding to the text to be processed, and obtain the phoneme parameters corresponding to the phonemes in the phoneme sequence; the phoneme parameters are determined by the speech conversion parameters.

[0046] The speech generation unit is used to generate text-mapped speech that matches the speech conversion parameters based on phoneme sequences and phoneme parameters.

[0047] The aforementioned video generation module includes:

[0048] The preprocessing unit is used to perform image preprocessing on the text mapping image to obtain a preprocessed image with the target pixels;

[0049] The adjustment unit is used to determine the speech attribute label of the text-mapped speech based on the speech conversion parameters, adjust the image tone of the preprocessed image based on the speech attribute label, use the preprocessed image with adjusted image tone as the target image, and adjust the text display style of the text to be processed based on the speech attribute label, use the text display style of the text to be processed as the target text.

[0050] The audio frame generation unit is used to perform audio encoding on the text-mapped speech to obtain audio frames for synthesizing the target video.

[0051] The video frame generation unit is used to generate video frames associated with audio frames based on the speech duration, target image, and target text mapped from text to speech.

[0052] The aforementioned video frame generation unit includes:

[0053] The first subunit is determined based on the speech duration of the text-to-speech mapping and the first time interval to determine the display start timestamp of the target text, and based on the speech duration and the second time interval to determine the display end timestamp of the target text. The duration between the display start timestamp and the display end timestamp is taken as the text display duration of the target text.

[0054] The second time determination subunit is used to determine the playback start time stamp of the target image based on the display start time stamp and the third time interval, and to determine the playback end time stamp of the target image based on the display end time stamp and the fourth time interval, and to take the duration between the playback start time stamp and the playback end time stamp as the playback duration of the target image.

[0055] The image-text alignment subunit is used to align target text with text display duration and target images with image playback duration, and generate video frames associated with audio frames based on the aligned target text and target images.

[0056] The aforementioned video generation module includes:

[0057] The audio-video alignment unit is used to align audio frames and video frames based on the duration of the audio frames to obtain audio-video frames for synthesizing the target video.

[0058] The video generation unit is used to generate the target video corresponding to the original text based on audio and video frames.

[0059] One embodiment of this application provides a computer device, including: a processor and a memory;

[0060] The processor is connected to a memory, which stores a computer program. When the computer program is executed by the processor, it causes the computer device to perform the method provided in the embodiments of this application.

[0061] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in this application.

[0062] One embodiment of this application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in this application embodiment.

[0063] In this embodiment, when the original text is obtained, the text to be processed can be extracted from it, where the length of the text to be processed is less than the length of the original text. Further, the text semantic vector of the text to be processed can be obtained, and candidate image vectors with the same semantic meaning as the text semantic vector can be searched among the candidate image vectors corresponding to the candidate images. Then, the text mapping image corresponding to the text to be processed can be determined based on the candidate image corresponding to the found candidate image vector. Simultaneously, when the speech conversion parameters associated with the text to be processed are obtained, the text to be processed can be converted into text mapping speech that matches the speech conversion parameters. Then, based on the text to be processed, the text mapping image, and the text mapping speech, audio frames and video frames for synthesizing the target video can be generated. Subsequently, the generated audio frames and video frames can be subjected to video synthesis processing to obtain the target video corresponding to the original text. Therefore, the embodiments of this application can quickly obtain the text mapping image and text mapping speech corresponding to the text to be processed in the original text, and can generate the corresponding target video based on the obtained text to be processed, text mapping image and text mapping speech. Compared with the method of simply displaying text or image, the embodiments of this application can enrich the audiovisual display effect of text by converting text into video. In addition, since images and voice-over can be intelligently added to the text in this process, the display efficiency of multimedia data (such as videos containing text, images and speech) can also be improved. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0066] Figure 2 This is a schematic diagram of a video synthesis scene provided in an embodiment of this application;

[0067] Figure 3 This is a schematic flowchart of a video synthesis method provided in an embodiment of this application;

[0068] Figure 4 This is a schematic flowchart of a video synthesis method provided in an embodiment of this application;

[0069] Figure 5 This is a schematic diagram of the structure of a text encoding model provided in an embodiment of this application;

[0070] Figure 6 This is a schematic diagram of the structure of an image encoding model provided in an embodiment of this application;

[0071] Figure 7 This is a schematic diagram of a video synthesis scene provided in an embodiment of this application;

[0072] Figure 8 This is a schematic diagram of the structure of a video synthesis device provided in an embodiment of this application;

[0073] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0074] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0075] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0076] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0077] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies various theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language, that is, the language people use in daily life, and thus it has a close relationship with linguistic research. NLP technologies typically include text processing, semantic understanding, machine translation, question answering, knowledge graphs, information retrieval, and dialogue systems. In the embodiments of this application, NLP technology can be used to obtain the text semantic vector of the text to be processed.

[0078] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include data processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition. In this embodiment, computer vision technology can be used to obtain candidate image vectors corresponding to candidate images and to search for candidate image vectors that have the same semantic meaning as the text semantic vector.

[0079] Key technologies in speech processing include automatic speech recognition, speech synthesis, and voiceprint recognition. Text-to-speech (TTS), also known as text-to-speech conversion, is a technology that converts computer-generated or externally input text information into understandable, highly natural-sounding speech output. This is equivalent to giving a machine an artificial mouth, allowing it to express itself using different timbres. Speech synthesis technology involves multiple disciplines such as acoustics, linguistics, digital signal processing, and computer science. In the embodiments of this application, speech synthesis technology can be used to convert text to be processed into text-mapped speech.

[0080] The solutions provided in this application belong to the field of machine learning (ML) within artificial intelligence. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. In this application embodiment, the video synthesis model (including a relevance detection model, text encoding model, image encoding model, speech conversion model, etc.) is an AI model based on machine learning technology, which can be used to convert raw text into corresponding target videos.

[0081] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. For example... Figure 1As shown, the network architecture may include a service server 100 and a terminal cluster. The terminal cluster may include multiple user terminals, and this embodiment does not limit the number of user terminals included in the terminal cluster. For example, the terminal cluster may specifically include: user terminal 200a, user terminal 200b, user terminal 200c, ..., user terminal 200n. Communication connections may exist between the terminal cluster members; for example, there is a communication connection between user terminal 200a and user terminal 200b, and between user terminal 200a and user terminal 200c. Simultaneously, any user terminal in the terminal cluster may have a communication connection with the service server 100; for example, there is a communication connection between user terminal 200a and service server 100. The above communication connection is not limited to a specific method; it can be directly or indirectly connected via wired communication, directly or indirectly connected via wireless communication, or through other methods. This application does not impose any limitations on this method.

[0082] It should be understood that, such as Figure 1 Each user terminal in the terminal cluster shown can have an application client installed. When the application client runs on each user terminal, it can interact with the aforementioned... Figure 1 The business servers 100 shown interact with each other. The application clients can be multimedia clients (e.g., short video clients), social clients, information clients (e.g., news clients), entertainment clients (e.g., reading clients, game clients), shopping clients, in-vehicle clients, smart home clients, browsers, or other application clients capable of displaying text, images, audio, and video data. These application clients can be standalone clients or embedded sub-clients integrated into other clients (e.g., social clients); this is not limited here. The business servers 100 can be a collection of multiple servers, such as backend servers and data processing servers corresponding to the application clients. Therefore, each user terminal can transmit data with the business servers 100 through the installed application clients; for example, each user terminal can use the business servers 100 to synthesize uploaded text into corresponding videos.

[0083] The business server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud databases, cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The user terminal can be a smart terminal capable of running the aforementioned application client with video synthesis capabilities, such as a smartphone, tablet, laptop, desktop computer, PDA, mobile internet device (MID), wearable device (e.g., smartwatch, smart bracelet), smart computer, or smart vehicle. The user terminal and the business server can be connected directly or indirectly via wired or wireless means; this embodiment does not impose any limitations on this connection.

[0084] It should be understood that, in order to enrich the audiovisual presentation effect, this application provides a text-to-video method that can automatically and quickly convert text into video. For ease of subsequent understanding and explanation, this application can refer to the text to be converted as the original text. This original text can originate from any file, webpage, or text scraped from a business platform. For example, the original text can be obtained from a document or an article circulating online; alternatively, the text entered by a business object through text input operations (e.g., text edited when posting comments on a network platform) can also be used as the original text. This application does not limit the source of the original text. Furthermore, this application can refer to the user associated with the original text (e.g., the user who uploaded a document to be processed) as the business object.

[0085] For ease of understanding, the following will be used as an example. Figure 1The following explanation uses the business server 100 and user terminal 200a as examples. User terminal 200a can have one of the aforementioned application clients (e.g., a short video client) installed and running. Therefore, a business object associated with user terminal 200a can upload an original file (or webpage) containing original text to the application client. The application client can then send the uploaded original file to the business server 100, which has a communication connection with user terminal 200a. Optionally, the original file may also contain other types of data besides text, such as images or audio; this embodiment does not limit this. After receiving the original file, the business server 100 can extract text from it and use the extracted text as the original text. Then, it can obtain the text to be processed from the original text. It is understood that the length of the text to be processed is less than the length of the original text; that is, the text to be processed is text segmented from the original text. This effectively controls the length of the text to be processed, providing a basis for subsequent processing. For example, assuming the original text is an article, text segmentation can yield multiple sentences from that article (i.e., the text to be processed).

[0086] Optionally, the user terminal 200a can extract the text from the original file to obtain the original text, which can then be sent to the business server 100.

[0087] Furthermore, the business server 100 can obtain the text semantic vector of the text to be processed, and search for candidate image vectors with the same semantic meaning as the text semantic vector in the candidate image vectors corresponding to the candidate images. Then, it can determine the text mapping image corresponding to the text to be processed based on the candidate images corresponding to the found candidate image vectors. The candidate images provided here can come from an image candidate library composed of a large number of images pre-collected from the Internet, or from the local cache of the user terminal 200a (e.g., photos stored in the local photo album), or from images carried in the original file (or webpage), or from historical images generated when the business object accesses the application client (e.g., images historically viewed, used, shared, or collected by the business object). Meanwhile, when the business server 100 obtains the speech conversion parameters associated with the text to be processed, it can convert the text to be processed into text-mapped speech that matches the speech conversion parameters. These speech conversion parameters can be used to indicate the characteristics of the converted text-mapped speech (e.g., pitch, intensity, duration, timbre, etc.), and are associated with the target voiceprint. This embodiment allows the business object to select the target voiceprint itself, or to determine the target voiceprint based on a configured voiceprint selection strategy. Therefore, the timbre, rhythm, etc., of the final presented text-mapped speech can be personalized. For example, the business object can choose to play the obtained text-mapped speech using a "gentle female voice." Furthermore, the business server 100 can generate audio frames and video frames for synthesizing the target video based on the obtained text to be processed, text-mapped image, and text-mapped speech. Subsequently, the generated audio frames and video frames can be processed for video synthesis to obtain the target video corresponding to the original text. Finally, the business server 100 can return the obtained target video to the user terminal 200a for playback.

[0088] It is understandable that the above method can be implemented by calling a pre-trained video synthesis model based on deep learning technology.

[0089] It is understood that the methods provided in the embodiments of this application can be executed by a computer device, which includes, but is not limited to, a user terminal (e.g., Figure 1 (any user terminal in the user terminal cluster shown) or service server (e.g., Figure 1 (The business server 100 shown).

[0090] Optionally, it is understood that the system architecture may include multiple business servers. A user terminal can be connected to a business server, and each business server can obtain the original text (e.g., text in a webpage) provided by the user terminal it is connected to, and then call the video synthesis model to convert the original text into the target video.

[0091] Optionally, and understandably, the user terminal can also obtain the original text, and thus call the video synthesis model to convert the original text into the target video.

[0092] It is understood that the business scenarios applicable to the above network architecture may include: video production, video recommendation, video search, image search, image and text matching, speech synthesis, and subtitle production, etc. Specific business scenarios will not be listed here.

[0093] For example, in a video production scenario, a business object (e.g., object X1) can pre-edit the original text needed for video production (e.g., video script A1). This video script A1 can then be uploaded to an application client on a user terminal (e.g., user terminal 200a mentioned above). A computer device (e.g., business server 100 mentioned above) can perform the aforementioned text segmentation, image and text retrieval, speech synthesis, and video synthesis processes on the received video script A1, thereby obtaining the video corresponding to video script A1 (e.g., video B1). This video B1 can then be returned to the application client for presentation. It is understandable that if object X1 is dissatisfied with the presentation of video B1, it can update video script A1 to obtain video script A2. Through a similar processing procedure, the video corresponding to video script A2 (e.g., video B2) can ultimately be obtained.

[0094] For example, in a video recommendation scenario, a computer device (e.g., the aforementioned business server 100) can generate a video (e.g., video B3) corresponding to the original text (e.g., article A3) read or browsed by a business object (e.g., object X2), and then push video B3 to the application client corresponding to object X2, thereby presenting richer audiovisual effects to object X2 when it browses the text.

[0095] For example, in a video search scenario, a computer device (e.g., the aforementioned business server 100) can determine the original text (e.g., text A4) based on the search content entered by the business object (e.g., object X3) when performing a video search, and push the video (e.g., video B4) generated based on the text A4 to the object X3 as the video search result.

[0096] For example, in an image search scenario, a computer device (e.g., the aforementioned business server 100) can determine the original text (e.g., text A5) based on the search content input by the business object (e.g., object X4) when performing an image search. It can then obtain the text semantic vector (e.g., vector C1) of text A5, search for candidate image vectors with the same semantics as vector C1 among multiple candidate image vectors, and determine the image corresponding to text A5 (e.g., image D1) based on the candidate image corresponding to the found candidate image vector. Subsequently, the obtained image D1 can be pushed to object X4 as the image search result. For instance, when object X4 is creating an article, it can use sentences from the article as search content, thereby efficiently obtaining suitable images and improving the efficiency of multimodal content production.

[0097] For example, in an image-text matching scenario, a computer device (e.g., the aforementioned business server 100) can obtain an image (e.g., image D2) uploaded by a business object (e.g., object X5) and can obtain corresponding text (e.g., text A6) for that image D2. For instance, when object X5 posts an image update on a web platform, relevant text can be automatically added.

[0098] For example, in a speech synthesis scenario, a computer device (e.g., the aforementioned business server 100) can obtain text (e.g., text A7) uploaded by a business object (e.g., object X6) through an application client on a user terminal (e.g., the aforementioned user terminal 200a), and obtain the speech conversion parameters associated with the text A7. It can then convert the text A7 into speech (e.g., speech E) that matches the speech conversion parameters and return speech E to the application client corresponding to object X6 for playback. Object X6 can adjust the speech conversion parameters as needed to achieve personalized auditory effect display.

[0099] For ease of understanding, please refer to the following: Figure 2 , Figure 2 This is a schematic diagram of a video synthesis scene provided in an embodiment of this application. The scene can be implemented by a computer device (e.g., computer device 20, not shown in the diagram), which can be the aforementioned... Figure 1 In the corresponding embodiment, the business server 100 or any user terminal in the user terminal cluster (e.g., user terminal 200a) is associated with the computer device.

[0100] like Figure 2As shown, computer device 20 can first obtain the original text (e.g., text 201). Here, text 201 can be text obtained from the local cache of computer device 20, text obtained from a business platform (e.g., information platform), or text uploaded by business object A (e.g., text edited by business object A through the corresponding application client). This application embodiment does not limit the source of text 201. In addition, this application embodiment does not limit the format of text 201.

[0101] It is understood that text 201 may contain various text data such as characters and symbols (e.g., punctuation marks). These characters and symbols can form different sentences, paragraphs, or chapters. To improve the efficiency and effect of video synthesis, computer device 20 can perform text segmentation on text 201 to obtain the text to be processed in text 201 (e.g., multiple sentences obtained from segmentation). Optionally, computer device 20 can recognize segmentation symbols (e.g., punctuation marks) in text 201, and then perform text segmentation on text 201 based on the recognized segmentation symbols. For example, it can obtain the segmented sentences in text 201 based on recognized punctuation marks such as periods, question marks, and exclamation marks. Figure 2 As shown, assuming that after the computer device 20 performs text segmentation on the text 201, it can obtain m texts (m is a positive integer greater than 1) to be processed. The m texts to be processed can specifically include text 201a, text 201b, ..., text 201m. The length of each text to be processed is less than the length of the original text. For example, the number of characters in text 201a is less than the number of characters in text 201.

[0102] It should be understood that, in order to obtain a text mapping image corresponding to each text to be processed, the computer device 20 can determine the text mapping image corresponding to the text to be processed based on candidate images found in the candidate images that are semantically similar to a text to be processed. Specifically, the computer device 20 can calculate the similarity between the text to be processed and the candidate images from a vector perspective. Specifically, the computer device 20 can perform text encoding on each text to be processed obtained above, thereby obtaining a text semantic vector corresponding to each text to be processed. For example, as... Figure 2 As shown, computer device 20 performs text encoding on text 201a to obtain text semantic vector 202a of text 201a; performs text encoding on text 201b to obtain text semantic vector 202b of text 201b; ...; performs text encoding on text 201m to obtain text semantic vector 202m of text 201m. Figure 2As shown, this application embodiment provides multiple (e.g., n, where n is a positive integer greater than 1) candidate images for retrieval. These candidate images can be images obtained by the computer device 20 from the Internet, images stored in the local cache of the computer device 20, or historical images associated with the historical object behavior of business object A. This application embodiment does not limit these. The n candidate images can specifically include candidate image 203a, candidate image 203b, candidate image 203c, ..., candidate image 203n. Furthermore, the computer device 20 can encode each of the n candidate images to obtain a candidate image vector corresponding to each candidate image. For example, encoding candidate image 203a yields candidate image vector 204a; encoding candidate image 203b yields candidate image vector 204b; encoding candidate image 203c yields candidate image vector 204c; ...; encoding candidate image 203n yields candidate image vector 204n. It can be understood that the image encoding of the candidate images can be pre-executed by the computer device 20.

[0103] Furthermore, the computer device 20 can search among the aforementioned n candidate image vectors for candidate image vectors that have the same semantic meaning as each text semantic vector. Therefore, it can determine the text mapping image corresponding to each text to be processed based on the candidate images corresponding to the found candidate image vectors. For example, taking text 201a as an example, the computer device 20 can find candidate image vectors with the same semantic meaning as text semantic vector 202a among the n candidate image vectors. Assuming the number of found candidate image vectors is n1 (n1 is a positive integer), the computer device 20 can determine the text mapping image corresponding to text 201a based on the n1 candidate images corresponding to these n1 candidate image vectors. It is understandable that, optionally, when n1 equals 1, the candidate image (e.g., candidate image 203a) corresponding to the found candidate image vector (e.g., candidate image vector 204a) can be directly used as the text mapping image corresponding to text 201a; optionally, when n1 is greater than 1, the composite image obtained by combining the candidate images (e.g., candidate images 203a and 203b) corresponding to these n1 (e.g., 2) candidate image vectors (e.g., candidate image vectors 204a and 204b) can be used as the text mapping image corresponding to text 201a. For example, these n1 candidate images can be combined into a dynamic composite image (e.g., an animated image). The process of computer device 20 obtaining text mapping images corresponding to other texts to be processed can refer to the above process of obtaining the text mapping image corresponding to text 201a, and will not be repeated here. Figure 2 As shown, assuming that after the above retrieval process, the text mapping image corresponding to text 201a is image 205a, the text mapping image corresponding to text 201b is image 205b, ..., and the text mapping image corresponding to text 201m is image 205m.

[0104] Furthermore, computer device 20 can convert the text to be processed into text-mapped speech, wherein the text-mapped speech can be determined by speech conversion parameters (e.g., speech conversion parameter 206) obtained by computer device 20. Here, speech conversion parameter 206 can be obtained by business object A through voiceprint selection, or it can be obtained by a voiceprint selection strategy configured by computer device 20; this embodiment does not limit this. For example, as... Figure 2 As shown, assuming that business object A selects voiceprint B as the target voiceprint, the computer device can obtain the speech conversion parameter 206 associated with the voiceprint B, and can convert the above m texts to be processed into text-mapped speech that matches the speech conversion parameter 206. For example, text 201a is converted into speech 207a, text 201b is converted into speech 207b, ..., text 201m is converted into speech 207m.

[0105] Furthermore, the computer device 20 can generate corresponding audio frames based on the aforementioned m text-mapped speech. For example, the computer device 20 can perform audio encoding on speech 207a to obtain audio frame 208a corresponding to speech 207a; perform audio encoding on speech 207b to obtain audio frame 208b corresponding to speech 207b; ...; perform audio encoding on speech 207m to obtain audio frame 208m corresponding to speech 207m. Each audio frame 208a, audio frame 208b, ..., and audio frame 208m can include one or more audio frames. The specific number of frames is determined by the audio encoding method adopted by the computer device 20, and this embodiment does not limit this. In addition, the computer device 20 can also generate corresponding video frames based on the above-mentioned text to be processed and text mapping images. For example, based on text 201a and image 205a, video frame 209a is generated; based on text 201b and image 205b, video frame 209b is generated; ...; based on text 201m and image 205m, video frame 209m is generated.

[0106] Finally, computer device 20 can perform video synthesis processing on the generated audio and video frames to obtain the target video corresponding to the original text. For example... Figure 2 As shown, computer device 20 can align each audio frame (e.g., audio frame 208a) with the corresponding video frame (e.g., video frame 209a) to obtain audio and video frames for synthesizing the target video, and then generate the target video (e.g., video 210) corresponding to the original text (e.g., text 201) based on the obtained audio and video frames (e.g., m audio and video frames).

[0107] It is understandable that, compared to simply displaying text 201, playing the resulting video 210 can provide business object A with richer audiovisual display effects. For example, in addition to seeing the text presented as subtitles in the video, users can also see images that match the semantics of each piece of text to be processed, and hear dubbing played using the target voiceprint. Furthermore, the computer device 20 can call a pre-trained video synthesis model to intelligently add images and dubbing to the text and automatically convert it into video, thus improving the production and display efficiency of multimedia data (such as videos containing text, images, and audio).

[0108] The specific implementation method of computer device 20 acquiring the text to be processed, the corresponding text mapping image, and the text mapping speech from the original text, and generating the target video based on the acquired text to be processed, text mapping image, and text mapping speech, can be found below. Figures 3-7 The description in the corresponding embodiments.

[0109] Please see Figure 3 , Figure 3 This is a flowchart illustrating a video synthesis method provided in an embodiment of this application. This video synthesis method can be executed by a computer device, which may include, but is not limited to, a user terminal or a business server running a video synthesis model. For ease of understanding, this embodiment of the application uses the execution of the method by a business server as an example for illustration. Figure 3 As shown, the video synthesis method may include at least the following steps S101-S104:

[0110] Step S101: When the original text is obtained, the text to be processed in the original text is obtained;

[0111] Specifically, the business server can extract text from original files, web pages, etc., to obtain the original text. After obtaining the original text, the business server can identify the first type of segmentation symbols in the original text, and then segment the original text based on the identified first type of segmentation symbols. The transition text obtained from the segmentation of the original text can be used as the first transition text, wherein the length of the first transition text is less than the length of the original text. This application embodiment does not limit the length of the first transition text, nor does it limit the number of transition texts of the original text. For example, a long article (i.e., the original text) can be segmented into multiple sentences (i.e., transition text). The first type of segmentation symbols here can include punctuation marks such as periods, question marks, exclamation marks, and quotation marks, which can be used at the end of sentences or other specific positions. In actual applications, the specific symbols included in the first type of segmentation symbols can be specified according to business needs. This application embodiment does not limit this.

[0112] It is understandable that, due to the variability of various language expressions, when the length of a certain first transition text is relatively long, even if the first transition text does not contain the first type of segmentation symbols, a second text segmentation is performed according to semantic needs to control the length of the final text to be processed. Based on this, in the first transition text, the business server can use transition text with a length greater than a length threshold as the text to be detected, and transition text with a length less than or equal to the length threshold as the first text to be processed. The length threshold can be adjusted as needed, and this embodiment does not limit this. Further, the business server can identify the second type of segmentation symbols in the text to be detected, and then perform text segmentation on the text to be detected based on the identified second type of segmentation symbols, and can use the transition text of the segmented text to be detected as the second transition text. The length of the second transition text is less than the length of the text to be detected, and this embodiment does not limit the length of the second transition text. The second type of segmentation symbols can include commas, semicolons, and other punctuation marks that can be used in the middle of a sentence or other specific positions. In practical applications, the specific symbols included in the second type of segmentation symbols can be specified according to business needs, and this embodiment does not limit this. For example, please refer to the above again. Figure 2 Assuming Figure 2 The text 201a shown is the first transitional text segmented from text 201. When the text length of text 201a is greater than the length threshold (e.g., 20 characters), text segmentation of text 201a can continue.

[0113] In this application, embodiments can employ regular expressions, function lookups, or table lookups to identify delimiters (including first-type and second-type delimiters) in text. This application does not limit the specific identification method used. Regular expressions (also called rule expressions) describe a string matching pattern and can be used to check if a string contains a certain substring, replace matched substrings, or extract substrings that meet certain conditions from a string. Therefore, different regular expressions can be used to identify first-type delimiters (e.g., periods, exclamation marks, question marks, etc.) and second-type delimiters (e.g., commas, semicolons, etc.) to achieve text segmentation of the original text.

[0114] Furthermore, the business server can detect whether there is a correlation between any two adjacent transition texts in the second transition text, thereby determining whether to use these two adjacent transition texts as subsequent texts to be processed. For ease of understanding, the i-th and j-th transition texts of the second transition text are used as an example for explanation. In this embodiment, the business server can obtain the i-th and j-th transition texts from any two adjacent transition texts of the second transition text, where j = i + 1, i is a positive integer less than K, and K is the number of transition texts in the segmented text to be detected. This embodiment does not limit the specific value of K. Furthermore, the business server can perform text correlation detection on the i-th and j-th transition texts to obtain the text correlation detection result between the i-th and j-th transition texts, and then obtain the second text to be processed based on the text correlation detection result. Finally, the first and second texts to be processed can be used as the texts to be processed from the original text.

[0115] In this embodiment, a relevance detection model can be used to detect the text relevance between the i-th transition text and the j-th transition text. Specifically, the service server can perform vector transformation on the i-th transition text to obtain a first text transformation vector corresponding to the i-th transition text, and can also perform vector transformation on the j-th transition text to obtain a second text transformation vector corresponding to the j-th transition text. Further, the first and second text transformation vectors can be input into the relevance detection model. By performing text relevance detection on the first and second text transformation vectors through the relevance detection model, the text relevance detection result between the i-th and j-th transition texts can be obtained.

[0116] Optionally, if the text relevance detection result indicates that there is a relevance between the i-th transition text and the j-th transition text, the synthesized text composed of the i-th transition text and the j-th transition text can be used as the second text to be processed. Here, the synthesized text can be obtained by concatenating the i-th transition text and the j-th transition text.

[0117] Optionally, if the text relevance detection result indicates that there is no relevance between the i-th transition text and the j-th transition text, then both the i-th and j-th transition texts can be used as the second texts to be processed. In other words, the relevance detection model can determine whether there is a relevance between the i-th and j-th transition texts. If there is no relevance, it means that the secondary segmentation of these two transition texts is appropriate.

[0118] For example, please refer to the above again. Figure 2Suppose that text 201a is segmented based on the identified second type of segmentation symbol (e.g., comma) to obtain text 2011a and text 2012a. Then, computer device 20 can call a pre-trained relevance detection model to perform text relevance detection on text 2011a and text 2012a. If a relevance is detected between text 2011a and text 2012a, the composite text (i.e., text 201a) corresponding to text 2011a and text 2012a can be used as the second text to be processed. Conversely, if no relevance is detected between text 2011a and text 2012a, both text 2011a and text 2012a can be used as the second text to be processed.

[0119] The business server can utilize a text database with massive amounts of text to train a deep neural network to obtain a relevance detection model. Optionally, a relevance detection model can be built using networks such as BERT (Bidirectional Encoder Representations from Transformers), Transformer, or ELMo (Embeddings from Language Models). This application embodiment does not limit the network structure used in the relevance detection model.

[0120] As can be seen from the above, the embodiments of this application can use a method of symbol recognition + text relevance detection based on a relevance detection model to achieve text segmentation of the original text. Therefore, it can effectively divide the semantics of long texts (e.g., long sentences with a large number of characters) and control the text length of the text to be processed, thereby providing a basis for subsequent steps.

[0121] Step S102: Obtain the text semantic vector of the text to be processed, find the candidate image vector with the same semantics as the text semantic vector in the candidate image vector corresponding to the candidate image, and determine the text mapping image corresponding to the text to be processed based on the candidate image corresponding to the found candidate image vector.

[0122] In this embodiment, the business server can perform cross-modal semantic retrieval based on the similarity between vectors to obtain the text mapping image corresponding to the text to be processed. Specifically, the business server can input the text to be processed into a text encoding model, and by encoding the text through the text encoding model, the text semantic vector of the text to be processed can be obtained. In addition, the business server can input candidate images into an image encoding model, and by encoding the candidate images through the image encoding model, the candidate image vector corresponding to the candidate image can be obtained. It can be understood that since the candidate images are prepared in advance, the business server can also perform image encoding on the candidate images when it obtains them (e.g., offline processing), thereby improving the efficiency of cross-modal semantic retrieval.

[0123] Furthermore, the business server can search for the candidate image vector with the minimum vector distance to the text semantic vector among the candidate image vectors corresponding to the candidate images. This candidate image vector has the same semantic meaning as the text semantic vector, and the text mapping image corresponding to the text to be processed can be determined based on the candidate image corresponding to the found candidate image vector. Here, the vector distance can be a parameter that can characterize the similarity between the candidate image vector and the text semantic vector, such as cosine distance, Hamming distance, or Euclidean distance; this embodiment does not limit this. For example, please refer again to the above... Figure 2 ,like Figure 2 As shown, assuming that the candidate image vector found by computer device 20 among n candidate image vectors that has the minimum vector distance (e.g., minimum cosine distance) to text semantic vector 202a is candidate image vector 204b, then the candidate image 203b corresponding to candidate image vector 204b can be used as the text mapping image corresponding to text 201a.

[0124] It should be noted that the candidate images in this application embodiment may originate from a large-scale image candidate library composed of images pre-collected from the Internet, or from the local cache of the business server, or from images in the original files or web pages associated with the original text, or from historical images generated when the business object accesses the corresponding application client. The business server can obtain the historical object data of the business object, and then obtain historical images associated with the business object from the historical object data. The historical object data is used to record historical object behaviors (editing, accessing, browsing, searching, clicking, sharing, etc.) generated when the business object accesses the application client (e.g., a short video client). These historical object behaviors can include behaviors related to images, such as images uploaded, accessed, browsed, edited, and searched by the business object, as well as images published, saved, favorited, recommended, liked, forwarded, and commented on by the business object; all of these belong to historical images.

[0125] It is understood that in the specific implementation of this application, data related to historical objects, historical object behavior, and historical images are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0126] It should be understood that the specific process of text encoding of the text to be processed can be as follows: the business server can perform vector transformation on the text to be processed to obtain the third text transformation vector corresponding to the text to be processed, and then input the third text transformation vector into the text encoding network in the text encoding model. By performing text encoding on the third text transformation vector through the text encoding network, the text semantic vector of the text to be processed can be obtained.

[0127] It should be understood that the specific process of image encoding for candidate images can be as follows: the business server can perform image segmentation on the candidate images and use the segmented images as region images. Image segmentation of the candidate images can yield multiple region images, and each region image can have the same image size. Therefore, the business server can perform uniform segmentation on the candidate images. Specifically, when the image size of a candidate image is not suitable for uniform segmentation, the business server can enlarge or reduce the candidate image before performing image segmentation to obtain an image to be segmented with a suitable size. Then, the business server performs image segmentation on this image. This application embodiment does not limit the number or size of the segmented region images. Furthermore, the business server can input the region images into the feature extraction network in the image encoding model. Through the feature extraction network, features corresponding to the region images can be extracted, and then an image feature sequence can be obtained based on the image features corresponding to the region images. Furthermore, the business server can input the image feature sequence into the image encoding network in the image encoding model. Through the image encoding network, the image feature sequence can be encoded to obtain the candidate image vector corresponding to the candidate image.

[0128] It is understood that the embodiments of this application can construct an image-text retrieval model for cross-modal semantic retrieval based on the aforementioned text encoding model, image encoding model, and search module. The business server can utilize a multimedia database with massive amounts of multimedia data (including image data and image title data, for example, data collected from the internet) to train a deep neural network to obtain the image-text retrieval model. It should be noted that the text encoding model and image encoding model can be built and trained independently, or they can be built and trained jointly; this embodiment of the application does not limit this. Optionally, the text encoding model can be built using networks such as BERT, ELMo, Transformer, and DAN (Deep Averaging Network); optionally, the image encoding model can be built using networks such as Vision Transformer, Hybrid (a hybrid model of convolution and Transformer), and ResNet (residual neural network). This embodiment of the application does not limit the network structure used for the text encoding model and image encoding model.

[0129] It should be understood that if the number of candidate image vectors found is N, then the number of candidate images corresponding to each candidate image vector is N. Optionally, if N equals 1, the business server can use the candidate image corresponding to this single candidate image vector as the text mapping image corresponding to the text to be processed. Optionally, if N is a positive integer greater than 1, the business server can synthesize the N candidate images corresponding to the N candidate image vectors found into a text mapping image corresponding to the text to be processed. The specific process can be as follows: the business server obtains the N candidate images found, and can configure the image arrangement order and image switching frequency for the N candidate images. Then, based on the configured image arrangement order and image switching frequency, it can synthesize the N candidate images, and subsequently use the synthesized image as the text mapping image corresponding to the text to be processed. For example, suppose the business server finds three candidate images, namely candidate image 1, candidate image 2, and candidate image 3. The business server can configure the image arrangement order (e.g., in the order of candidate image 2-candidate image 1-candidate image 3) and the image switching frequency (e.g., switching one image every 1 second) for these three candidate images. Finally, the dynamic image (referred to as GIF) obtained by combining these three candidate images can be used as the corresponding text mapping image.

[0130] As can be seen from the above, compared with traditional retrieval algorithms, the image-text retrieval model adopted in this application embodiment can directly understand and encode text and images, mapping the representation features of both to the same space, and can realize direct cross-modal retrieval of "text-image" and "image-text". Therefore, it can improve the efficiency and accuracy of cross-modal semantic retrieval, and at the same time improve the efficiency of business objects in creating and sharing multimodal content (e.g., making videos or creating articles).

[0131] Step S103: When the speech conversion parameters associated with the text to be processed are obtained, the text to be processed is converted into text-mapped speech that matches the speech conversion parameters.

[0132] In this embodiment, the speech conversion parameters are obtained by the business object associated with the original text by triggering the voiceprint in the voiceprint list. It can be understood that during text-to-speech conversion, optionally, the application client corresponding to the business object can respond to the business object's trigger operation on the voiceprint selection control and output a voiceprint list; that is, the voiceprint list here is output by the business object by triggering the voiceprint selection control. Optionally, the application client can respond to the speech input operation on the voiceprint list, obtain the object's voice input by the business object, and send the object's voice to the business server. The business server can then perform voiceprint recognition on the received object's voice to obtain the business object's voiceprint and add the business object's voiceprint to the voiceprint list. Optionally, the business object (e.g., business object 1) can also add the voiceprints of other business objects (e.g., business object 2) to the voiceprint list. For example, business object 1 can upload a voice recording of business object 2 (e.g., stored locally on the user terminal corresponding to business object 1) to the application client. Through similar processing, the voiceprint of business object 2 can ultimately be added to the voiceprint list. Alternatively, the business server can identify keywords in the text to be processed that represent the attributes of the business object, and then generate a business notification message based on the identified keywords. This business notification message is then returned to the application client corresponding to the business object, so that the application client outputs a voiceprint list when it receives the business notification message. In other words, the voiceprint list here is output when keywords carrying the attributes of the business object are identified in the text to be processed.

[0133] Among them, business object attributes can be used to characterize the emotions or attribute features of business objects. For example, emotions can include happiness, excitement, sadness, etc., and attribute features can include the gender of business objects, etc. Each type of business object attribute can be configured with one or more keywords. For example, when the business object attribute is the emotion of happiness, its corresponding keywords can be set to "hahahaha" or "so happy", etc. This application embodiment does not limit this.

[0134] It is understood that in the specific implementation of this application, data related to business object attributes and voiceprints of business objects are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0135] It is understood that the voiceprint list can contain one or more voiceprints, and this application embodiment does not limit the number of voiceprints in the voiceprint list. For example, in response to a business object's selection operation on the voiceprint list, the application client can use the voiceprint corresponding to the selection operation as the target voiceprint, and then send the target voiceprint to the business server so that the business server can obtain the speech conversion parameters corresponding to the target voiceprint. Each voiceprint in the voiceprint list can correspond to a set of speech conversion parameters. Optionally, the business server can also configure a corresponding voiceprint selection strategy to determine the target voiceprint in the voiceprint list. For example, when a target keyword is identified in the text to be processed, it can be determined that the text to be processed meets the voiceprint selection strategy, so the business server can automatically select the voiceprint associated with the target keyword as the target voiceprint. Optionally, the business server can also detect the text length of the text to be processed. For example, when the text length of the text to be processed is detected to reach a target length threshold, it can be determined that the text to be processed meets the voiceprint selection strategy, so the business server can automatically select the voiceprint associated with the target length threshold as the target voiceprint.

[0136] It can be understood that the business server can convert the text to be processed into text-mapped speech that matches the speech conversion parameters. Specifically, upon obtaining the speech conversion parameters associated with the text to be processed, the business server can perform phoneme conversion on the text based on these parameters, obtaining the phoneme sequence corresponding to the text. It can also obtain the phoneme parameters corresponding to the phonemes in the phoneme sequence, where the phoneme parameters are determined by the speech conversion parameters. A phoneme is the smallest unit of sound defined from a sound quality perspective. Furthermore, the business server can generate text-mapped speech that matches the speech conversion parameters based on the phoneme sequence and phoneme parameters.

[0137] It should be understood that the embodiments of this application can use text-to-speech (TTS) technology to convert the text to be processed into text-mapped speech. For example, the business server can input the text to be processed into the speech conversion model, and the text processing network in the speech conversion model can perform text processing on the text to be processed, that is, convert the text to be processed into a phoneme sequence based on the speech conversion parameters, and can mark the phoneme parameters corresponding to each phoneme in the phoneme sequence (e.g., start and end time, frequency changes, etc.). In the text processing process, it can also handle difficult problems such as distinguishing words with the same spelling but different pronunciations, handling abbreviations, and determining the pause position. Then, the speech conversion network in the speech conversion model can generate text-mapped speech based on the phoneme sequence and the phoneme parameters corresponding to each marked phoneme. In the embodiments of this application, text-to-speech conversion can be performed using methods such as splicing, parametric methods, and duct simulation methods. Among them, the splicing method selects basic units from pre-recorded audio segments for splicing, the parametric method trains phoneme parameters to convert them into waveforms, and the duct simulation method directly converts the text into physical sound signals.

[0138] It is understandable that the business server can utilize a database containing massive amounts of text and speech data to train a deep neural network to obtain a speech conversion model. Optionally, networks such as Transformer, WaveNet (a technique that uses neural networks to model the original audio waveform), Tacotron (an end-to-end speech synthesis model that directly synthesizes speech from text), Tacotron2 (an improved version of Tacotron), and DeepVoice3 (a fully convolutional neuron speech synthesis system based on an attention mechanism) can be used to construct the speech conversion model. This application does not limit the network structure used in the speech conversion model.

[0139] Optionally, the business server can also insert blank sounds or other background sounds (e.g., background music) into the text-mapped speech.

[0140] Step S104: Based on the text to be processed, the text-mapped image, and the text-mapped speech, generate audio frames and video frames for synthesizing the target video. Perform video synthesis processing on the generated audio frames and video frames to obtain the target video corresponding to the original text.

[0141] Specifically, the business server can perform image preprocessing (including scaling, alignment, etc.) on the text-mapped images to obtain preprocessed images with the target pixels. For example, the Pillow tool can be used to perform a series of preprocessing steps such as scaling and alignment on the text-mapped images, thereby processing each text-mapped image into a text-mapped image with the same pixel size. Pillow is a fork of PIL (Python Imaging Library) that provides many image processing functions.

[0142] Furthermore, the business server can determine the speech attribute label of the text-mapped speech based on the speech conversion parameters. This speech attribute label characterizes the speech attributes of the text-mapped speech, which can refer to the style or emotion exhibited by the text-mapped speech. Further, the business server can adjust the image tone of the preprocessed image based on this speech attribute label and use the adjusted preprocessed image as the target image. It can also adjust the text display style of the text to be processed based on the speech attribute label and use the adjusted text display style as the target text. Here, image tone refers to the image style of the preprocessed image, and text display style can include the font, font size, color, and transition effects of the text to be processed. For example, when the speech attribute label is "soothing," the business server can adjust the image tone of the preprocessed image to match "soothing" (e.g., adjust the color tone of the preprocessed image to warm colors) and adjust the text display style of the text to match "soothing" (e.g., adjust the color of the text to warm colors).

[0143] Furthermore, the service server can perform audio encoding on the text-mapped speech to obtain audio frames for synthesizing the target video. For example, the service server can use a specific audio encoding method to encode the text-mapped speech. It is understood that different audio encoding methods may have different encoding parameters (such as sampling rate, sample size, bit rate, etc.), so the final audio frames may also be different. Commonly used audio encoding standards include G.711 (a voice compression standard developed by the International Telecommunication Union), AAC (Advanced Audio Coding, an audio encoding technology based on MPEG-2 launched in 1997), and Opus (a lossy audio encoding format). Audio files composed of audio frames can be encapsulated into MP3 (Moving Picture Experts Group Audio Layer III), OGG (OGGVobis (oggVorbis)), AAC, and other file formats. This application does not limit the audio encoding method used by the service server, nor does it limit the number and size of the audio frames obtained by the service server based on the encoding parameters when performing audio encoding on the text-mapped speech.

[0144] Furthermore, the service server can generate video frames associated with audio frames based on the speech duration of the text-to-speech mapping, the target image, and the target text. Specifically, the service server can determine the display start timestamp of the target text based on the speech duration of the text-to-speech mapping and a first time interval, and can determine the display end timestamp of the target text based on the speech duration and a second time interval. The duration between the display start timestamp and the display end timestamp can be used as the text display duration of the target text. Further, the service server can determine the playback start timestamp of the target image based on the display start timestamp and a third time interval, and can determine the playback end timestamp of the target image based on the display end timestamp and a fourth time interval. The duration between the playback start timestamp and the playback end timestamp can be used as the image playback duration of the target image. Further, the service server aligns the target text with the text display duration and the target image with the image playback duration, thereby generating video frames associated with audio frames based on the aligned target text and target image. In this embodiment, the specific sizes of the first time interval, second time interval, third time interval, and fourth time interval are not limited. It is understood that this application embodiment does not limit the relationship between voice duration, text display duration, and image playback duration. For example, optionally, voice duration can be set to equal text display duration to equal image playback duration. Furthermore, the business object can adjust the aforementioned timestamps or durations as needed. It should be noted that the business server can use specific video encoding methods to encode the target image and specific text encoding methods (such as SubRip, ASS (Advanced SubStation Alpha), etc.) to encode the target text, ultimately obtaining encoded video frames. Commonly used video encoding standards include H.265 (H.265-HEVC (High Efficiency Video Coding), an efficient video coding standard adopted by the ITU in 2013), H.264 (a highly compressed digital video codec standard jointly proposed by the ITU and the International Organization for Standardization), and MPEG-4 (Moving Picture Experts Group 4, a scheme introduced by the Moving Picture Experts Group in 1999 suitable for low-rate applications), etc. This application does not limit the video encoding method and text encoding method used by the business server.

[0145] It is understood that in practical applications, audio and video may become asynchronous due to various reasons, such as differences in the decoding and rendering time of each video frame, or that video frames with rich colors may be decoded and rendered more slowly than those with simple colors. This application's embodiments can support multiple audio and video synchronization methods, such as video synchronization to audio, audio synchronization to video, and audio and video synchronization to an external clock. For example, for video synchronization to audio, the service server can align the audio frame and the corresponding video frame based on the duration of the audio frame to obtain the audio and video frames used to synthesize the target video. Then, the target video corresponding to the original text can be generated based on the audio and video frames. For example, see the above. Figure 2 ,like Figure 2 As shown, computer device 20 can align audio frame 208a with video frame 209a to obtain audio / video frame 1; align audio frame 208b with video frame 209b to obtain audio / video frame 2; ...; align audio frame 208m with video frame 209m to obtain audio / video frame m. Finally, video 210 can be generated based on audio / video frames 1, 2, ..., and m. Here, audio / video synchronization allows for a certain delay, meaning the delay must be within an acceptable range. This acts as a feedback mechanism: when the video is slower than the audio, the video playback speed should be increased, and frames can be dropped appropriately to catch up with the audio. If there is already a delay, it can be reduced; conversely, the video playback speed should be decreased.

[0146] Optionally, the MoviePy tool can be used to composite target videos with specific formats (e.g., MKV (Matroska VideoFile), AVI (Audio Video Interleaved), MP4 (an abbreviation for MPEG-4), etc.). MoviePy is a Python module for video editing that can be used for basic video operations (such as cutting, splicing, and title insertion), video compositing (also known as non-linear editing), video processing, or creating advanced effects.

[0147] The aforementioned relevance detection model, text encoding model, image encoding model, and speech conversion model can be jointly constructed into a comprehensive video synthesis model. Therefore, the business server can directly call the trained video synthesis model to convert the original text into the target video. It should be noted that the relevance detection model, text encoding model, image encoding model, and speech conversion model can be built and trained individually, or they can be built and trained together. This application embodiment does not limit this.

[0148] As described above, the embodiments of this application can quickly obtain the text mapping image and text mapping speech corresponding to the text to be processed in the original text, and can generate the corresponding target video based on the obtained text to be processed, text mapping image and text mapping speech. Compared with the method of simply displaying text or image, the embodiments of this application can enrich the audiovisual display effect of text by converting text into video. In addition, since images and voice-over can be intelligently added to the text in this process, the display efficiency of multimedia data (such as videos containing text, images and speech) can also be improved.

[0149] Please see Figure 4 , Figure 4 This is a flowchart illustrating a video synthesis method provided in an embodiment of this application. This video synthesis method can be executed by a computer device, which may include, but is not limited to, a user terminal or a business server running a video synthesis model. For ease of understanding, this embodiment of the application uses the execution of the method by a business server as an example for illustration. Figure 4 As shown, the video synthesis method may include at least the following steps:

[0150] Step S201: When the original text is obtained, the original text is segmented, and the text to be processed is determined based on the transition text obtained from the segmentation.

[0151] The specific process of the business server performing text segmentation on the original text can be found in the above. Figure 3 Step S101 in the corresponding embodiment will not be described again here.

[0152] Here, we will still use the i-th and j-th transition texts as examples for explanation. The process of vector transformation of the i-th and j-th transition texts by the business server can be as follows: The business server can perform word segmentation on the i-th transition text to obtain the first word segment of the i-th transition text, and then perform word vector transformation on the first word segment to obtain the first word vector corresponding to the first word segment. Further, the first semantic vector corresponding to the i-th transition text and the first position vector associated with the text position of the first word segment in the i-th transition text are obtained. Then, the first word vector, the first semantic vector, and the first position vector can be fused to obtain the first text transformation vector corresponding to the i-th transition text. Similarly, the business server can perform word segmentation on the j-th transition text to obtain the second word segment of the j-th transition text. Then, it can perform word vector conversion on the second word segment to obtain the second word vector corresponding to the second word segment. Furthermore, it can obtain the second semantic vector corresponding to the j-th transition text and the second position vector associated with the text position of the second word segment in the j-th transition text. Then, it can perform vector fusion of the second word vector, the second semantic vector, and the second position vector to obtain the second text conversion vector corresponding to the j-th transition text.

[0153] Furthermore, the first and second text transformation vectors can be input into the relevance detection model. The relevance detection model performs text relevance detection on the first and second text transformation vectors to obtain the text relevance detection result between the i-th transition text and the j-th transition text. For example, when the text relevance detection result is 1, it means that there is no relevance between the i-th and j-th transition texts; when the text relevance detection result is 0, it means that there is a relevance between the i-th and j-th transition texts.

[0154] Optionally, embodiments of this application may use a pre-trained BERT model as the correlation detection model. The specific network structure of the BERT model can be found in subsequent descriptions. Figure 5 The network structure shown.

[0155] Step S202: Input the text to be processed into the text encoding model, and encode the text to be processed through the text encoding model to obtain the text semantic vector of the text to be processed;

[0156] Specifically, the business server can perform word segmentation on the text to be processed, obtaining the text segments, and then perform word vector conversion on the text segments to obtain the corresponding word vectors. Further, the business server can obtain the global semantic vector corresponding to the text to be processed and the position vectors associated with the text positions of the text segments in the text to be processed. It can also fuse the word vectors, global semantic vector, and position vectors to obtain the third text conversion vector corresponding to the text to be processed.

[0157] Furthermore, the third text transformation vector can be input into the text encoding network in the text encoding model for text encoding. Optionally, in this embodiment, a trained BERT model can be used as the text encoding model (i.e., TextEncoder). Please refer to [link to relevant documentation]. Figure 5 , Figure 5 This is a schematic diagram of the structure of a text encoding model provided in an embodiment of this application. For example... Figure 5 As shown, assuming a text encoding model (e.g., Figure 5 The text encoding model 503 shown has M text encoding networks, including text encoding network E. m and text encoding network E m+1 Text encoding network E m+1 For text encoding network E m The next text encoding network, where m is a positive integer less than M. For example, the M text encoding networks in text encoding model 503 can specifically include text encoding network 1, text encoding network 2, ..., text encoding network M, text encoding network E. m and text encoding network E m+1 It can be any two adjacent text encoding networks from text encoding network 1, text encoding network 2, ..., text encoding network M. Optionally, each text encoding network has the same network structure, and can adopt the network structure of the encoder (i.e., Transformer Encoder) in Transformer. Then the specific process of text encoding can be: the business server converts the third text into a vector (e.g., ...). Figure 5 The vector 502 shown (which is obtained by vector transformation of text 501) is input into the text encoding model (e.g., Figure 5 The text encoding network in the text encoding model 503 shown, and through the text encoding network E m (For example, Figure 5 The text encoding network 1 shown outputs historical text encoding vectors, which can then be input into the text encoding network E. m+1 (For example, Figure 5 The text encoding network shown is 2). In text encoding network Em+1 In this context, it is possible to base it on historical text encoding vectors and text encoding network E. m+1 The first intermediate encoding vector is obtained through a multi-head self-attention network. Then, residual concatenation and normalization are performed on the first intermediate encoding vector and the historical text encoding vector to obtain the second intermediate encoding vector, which can be input into the text encoding network E. m+1 The fully connected layer in the network yields the third intermediate encoding vector; further, residual connections and normalization can be performed on the third and second intermediate encoding vectors to obtain the text encoding network E. m+1 The target text encoding vector can be used to ultimately obtain the text semantic vector (i.e., TextVector) of the text to be processed. For example, the last text encoding network (e.g., Figure 5 The text encoding vector output by the text encoding network M shown (e.g., Figure 5 The vector 504 shown is used as the text semantic vector of the text to be processed (e.g., text 501).

[0158] For example, in one implementation, a BERT-based text encoding model can be used to encode a text of length L (e.g., text 1). First, text 1 can be transformed into a vector at the character level (i.e., tokenization and embedding) to obtain a one-dimensional sequence of length L+2 (i.e., the third text transformation vector). Then, the one-dimensional sequence can be encoded using a text encoding network in the text encoding model (e.g., using a Transformer Encoder) to obtain the representation vector of text 1 (i.e., the text semantic vector).

[0159] Step S203: Input the candidate image into the image encoding model, and encode the candidate image through the image encoding model to obtain the candidate image vector corresponding to the candidate image;

[0160] Optionally, embodiments of this application may use a pre-trained Vision Transformer model as the image encoding model (i.e., ImageEncoder). Please refer to [link / reference needed]. Figure 6 , Figure 6 This is a schematic diagram of the structure of an image encoding model provided in an embodiment of this application. For example... Figure 6 As shown, the business server can perform image segmentation on a candidate image to obtain multiple (e.g., z) region images. For example, the z (e.g., 9) region images can specifically include region image A1, ..., region image A x Image A (area) y Image A (area) zThe z regions are all the same size. These z regions can then be input into the feature extraction network of an image encoding model (e.g., ...). Figure 6 The feature extraction network 601 shown above extracts features from these z regions of the image, resulting in image features 1, ..., 2 corresponding to region image A1. x Corresponding image features x, ..., region image A y Corresponding image features y, ..., region image A z The corresponding image feature z, and the image feature sequence 602 can be determined based on image feature 1, ..., image feature x, ..., image feature y, ..., and image feature z. Furthermore, the image feature sequence 602 can be input into the image coding network in the image coding model (e.g., ...). Figure 6 The image encoding network 603 shown can be an encoder in Transformer. By encoding the image feature sequence 602 through the image encoding network 603, a candidate image vector 604 (i.e., ImageVector) can be obtained.

[0161] For example, in one implementation, a candidate image (e.g., candidate image w) can be divided into H×H (e.g., 3×3) regions of the same size, resulting in H×H region images with the same image size. Then, features can be extracted from each region image using a feature extraction network in the image encoding model to obtain an H×H one-dimensional sequence (i.e., image feature sequence). Finally, the image encoding network in the image encoding model can encode the image feature sequence to obtain the representation vector (i.e., candidate image vector) of the candidate image w.

[0162] Step S204: Find the candidate image vector with the minimum vector distance to the text semantic vector among the candidate image vectors corresponding to the candidate image, and use it as the candidate image vector with the same semantics as the text semantic vector.

[0163] It is understood that the business server can retrieve the candidate image vector with the closest vector distance (e.g., closest cosine distance) from a large number of candidate image vectors corresponding to the text semantic vector, and can use the retrieved candidate image vector as a candidate image vector with the same semantic meaning as the text semantic vector. In other words, it can find the image that best matches the semantic meaning as the accompanying image for the video. Optionally, an efficient and reliable retrieval algorithm (e.g., the Faiss (Facebook AI Similarity Search) algorithm) can be used to implement the above retrieval process. This application embodiment does not limit the retrieval algorithm used.

[0164] The specific process for this step can be found above. Figure 3 Step S102 in the corresponding embodiment will not be described again here.

[0165] Step S205: Determine the text mapping image corresponding to the text to be processed based on the candidate images corresponding to the found candidate image vectors;

[0166] The specific process for this step can be found above. Figure 3 Step S102 in the corresponding embodiment will not be described again here.

[0167] Step S206: Based on the obtained speech conversion parameters associated with the text to be processed, perform phoneme conversion on the text to be processed to obtain the phoneme sequence corresponding to the text to be processed, and obtain the phoneme parameters corresponding to the phonemes in the phoneme sequence.

[0168] The specific process for this step can be found above. Figure 3 Step S103 in the corresponding embodiment will not be described again here.

[0169] Step S207: Generate text-mapped speech that matches the speech conversion parameters based on the phoneme sequence and phoneme parameters;

[0170] The specific process for this step can be found above. Figure 3 Step S103 in the corresponding embodiment will not be described again here.

[0171] Step S208: Perform image preprocessing on the text mapping image to obtain a preprocessed image with target pixels;

[0172] The specific process for this step can be found above. Figure 3 Step S104 in the corresponding embodiment will not be described again here.

[0173] Step S209: Determine the speech attribute labels of the text-mapped speech based on the speech conversion parameters, adjust the image tone of the preprocessed image based on the speech attribute labels to obtain the target image, and adjust the text display style of the text to be processed based on the speech attribute labels to obtain the target text.

[0174] The specific process for this step can be found above. Figure 3 Step S104 in the corresponding embodiment will not be described again here.

[0175] Step S210: Audio encoding is performed on the text-mapped speech to obtain audio frames used for synthesizing the target video;

[0176] The specific process for this step can be found above. Figure 3 Step S104 in the corresponding embodiment will not be described again here.

[0177] Step S211: Based on the speech duration, target image, and target text of the text-to-speech mapping, generate a video frame associated with the audio frame.

[0178] The specific process for this step can be found above. Figure 3 Step S104 in the corresponding embodiment will not be described again here.

[0179] Step S212: Based on the duration of the audio frame, align the audio frame and the video frame to obtain the audio-video frame used to synthesize the target video, and generate the target video corresponding to the original text based on the audio-video frame.

[0180] The specific process for this step can be found above. Figure 3 Step S104 in the corresponding embodiment will not be described again here.

[0181] As described above, the embodiments of this application can quickly obtain the text mapping image and text mapping speech corresponding to the text to be processed in the original text, and can generate the corresponding target video based on the obtained text to be processed, text mapping image and text mapping speech. Compared with the method of simply displaying text or image, the embodiments of this application can enrich the audiovisual display effect of text by converting text into video. In addition, since images and voice-over can be intelligently added to the text in this process, the display efficiency of multimedia data (such as videos containing text, images and speech) can also be improved.

[0182] Please see also Figure 7 , Figure 7 This is a schematic diagram of a video synthesis scene provided in an embodiment of this application. The scene can be implemented using a computer device, which can be the aforementioned... Figure 1 The corresponding embodiment refers to either the service server 100 or any user terminal in the user terminal cluster (e.g., user terminal 200a). Figure 7 The sentence segmentation module, image and text retrieval module, speech synthesis module, and video generation module shown can be modules in a text-to-video system based on large-scale image and text retrieval (corresponding to the video synthesis model mentioned above). For example... Figure 7As shown, the computer device can acquire the raw text input by the business object, and then input the raw text into the sentence segmentation module 701 (which may include the aforementioned relevance detection model). The sentence segmentation module 701 can automatically segment the raw text without requiring manual segmentation by the business object, thus quickly obtaining the text to be processed from the raw text. Further, the computer device can input the text to be processed into the image-text retrieval module 702 (corresponding to the aforementioned image-text retrieval model, including a text encoding model and an image encoding model). The image-text retrieval module 702 can retrieve semantically similar text-mapped images (e.g., image P1, image P2, ..., image Pn) from the candidate images based on the text to be processed, and use them as accompanying images, without requiring the business object to manually search for accompanying images from the internet. Simultaneously, the computer device can input the text to be processed into the speech synthesis module 703 (corresponding to the aforementioned speech conversion model). The speech synthesis module 703 can automatically convert the text to be processed into text-mapped speech (e.g., speech Q1, speech Q2, ..., speech Qn), and use it as narration, thus eliminating the need for manual narration by the business object. Furthermore, the computer device can set timestamps related to the subtitles (i.e., display start timestamp and display end timestamp) based on the speech duration of the text-to-speech mapping, and convert them together with the text to be processed into subtitles with a specific format (e.g., SRT, ASS, SMI, SSA, or SUB format, etc.), such as Subtitle 1, Subtitle 2, ..., Subtitle n. Finally, the computer device can input the obtained subtitles, images, and voice-over into the video generation module 704, which can automatically generate a playable video file (e.g., video 705) based on the subtitles, images, and voice-over. The specific implementation process of this scenario can be found above. Figures 3-4 The description in the corresponding embodiments.

[0183] As described above, this application embodiment can quickly convert text input by a business user into a video containing subtitles and voice-over (and even background music) through a text-to-video system. Since the entire process is fully automated and requires no technical skills from the video creator (i.e., the business user), it effectively lowers the barrier to video creation and improves video production efficiency. Furthermore, compared to simply displaying text or images, this application embodiment enriches the audiovisual presentation of the text by displaying the generated video.

[0184] Please see Figure 8 This is a schematic diagram of a video compositing apparatus provided in an embodiment of this application. The video compositing apparatus can be a computer program (including program code) running on a computer device; for example, the video compositing apparatus is an application software. This apparatus can be used to execute corresponding steps in the video compositing method provided in the embodiments of this application. Figure 8As shown, the video synthesis device 1 may include: a text acquisition module 11, an image search module 12, a speech conversion module 13, and a video generation module 14;

[0185] The text acquisition module 11 is used to acquire the text to be processed from the original text when the original text is acquired; the length of the text to be processed is less than the length of the original text.

[0186] The text acquisition module 11 may include: a first segmentation unit 111, a text classification unit 112, a second segmentation unit 113, and a correlation detection unit 114;

[0187] The first segmentation unit 111 is used to identify the first type of segmentation symbols in the original text when the original text is acquired, and to perform text segmentation on the original text based on the identified first type of segmentation symbols, and to take the transition text of the segmented original text as the first transition text; the text length of the first transition text is less than the text length of the original text.

[0188] The text classification unit 112 is used to classify the transition text with a text length greater than a length threshold as the text to be detected, and the transition text with a text length less than or equal to the length threshold as the first text to be processed.

[0189] The second segmentation unit 113 is used to identify the second type of segmentation symbols in the text to be detected, and to segment the text to be detected based on the identified second type of segmentation symbols. The transition text of the segmented text to be detected is used as the second transition text. The i-th transition text and the j-th transition text are obtained from any two adjacent transition texts of the second transition text; j = i + 1; i is a positive integer less than K, and K is the number of transition texts of the segmented text to be detected.

[0190] The relevant detection unit 114 is used to perform text relevance detection on the i-th transition text and the j-th transition text, obtain the text relevance detection result between the i-th transition text and the j-th transition text, obtain the second text to be processed based on the text relevance detection result, and use the first text to be processed and the second text to be processed as the text to be processed of the original text.

[0191] The relevant detection unit 114 may include: a first conversion subunit 1141, a model detection subunit 1142, a first determination subunit 1143, and a second determination subunit 1144;

[0192] The first conversion subunit 1141 is used to perform vector conversion on the i-th transition text to obtain the first text conversion vector corresponding to the i-th transition text, and to perform vector conversion on the j-th transition text to obtain the second text conversion vector corresponding to the j-th transition text.

[0193] The model detection subunit 1142 is used to input the first text transformation vector and the second text transformation vector into the relevance detection model, and to perform text relevance detection on the first text transformation vector and the second text transformation vector through the relevance detection model to obtain the text relevance detection result between the i-th transition text and the j-th transition text.

[0194] The first determining subunit 1143 is used to take the synthesized text composed of the i-th transition text and the j-th transition text as the second text to be processed if the text relevance detection result indicates that there is a relevance between the i-th transition text and the j-th transition text.

[0195] The second determining subunit 1144 is used to treat both the i-th and j-th transition texts as the second texts to be processed if the text relevance detection result indicates that there is no relevance between the i-th and j-th transition texts.

[0196] The specific implementation methods of the first conversion subunit 1141, the model detection subunit 1142, the first determination subunit 1143, and the second determination subunit 1144 can be found above. Figure 3 The description of step S101 in the corresponding embodiment, or you can refer to the above. Figure 4 The description of step S201 in the corresponding embodiments will not be repeated here.

[0197] The specific implementation methods of the first segmentation unit 111, the text classification unit 112, the second segmentation unit 113, and the correlation detection unit 114 can be found above. Figure 3 The description of step S101 in the corresponding embodiment, or you can refer to the above. Figure 4 The description of step S201 in the corresponding embodiments will not be repeated here.

[0198] Image search module 12 is used to obtain the text semantic vector of the text to be processed, search for candidate image vectors with the same semantics as the text semantic vector in the candidate image vectors corresponding to the candidate images, and determine the text mapping image corresponding to the text to be processed based on the candidate images corresponding to the found candidate image vectors.

[0199] The image search module 12 may include: a text encoding unit 121, an image encoding unit 122, a vector search unit 123, and an image determination unit 124.

[0200] The text encoding unit 121 is used to input the text to be processed into the text encoding model, and to encode the text to be processed through the text encoding model to obtain the text semantic vector of the text to be processed.

[0201] The text encoding unit 121 may include: a second conversion subunit 1211 and a text encoding subunit 1212;

[0202] The second conversion subunit 1211 is used to perform vector conversion on the text to be processed, so as to obtain the third text conversion vector corresponding to the text to be processed.

[0203] The second conversion subunit 1211 is specifically used to perform word segmentation on the text to be processed, obtain the text word segmentation of the text to be processed, perform word vector conversion on the text word segmentation, obtain the word vector corresponding to the text word segmentation; obtain the global semantic vector corresponding to the text to be processed and the position vector associated with the text position of the text word segmentation in the text to be processed, and perform vector fusion on the word vector, global semantic vector and position vector to obtain the third text conversion vector corresponding to the text to be processed.

[0204] The text encoding subunit 1212 is used to input the third text transformation vector into the text encoding network in the text encoding model, and to perform text encoding on the third text transformation vector through the text encoding network to obtain the text semantic vector of the text to be processed.

[0205] The text encoding model contains M text encoding networks, including text encoding network E. m and text encoding network E m+1 Text encoding network E m+1 For text encoding network E m The next text encoding network, where m is a positive integer less than M;

[0206] The text encoding subunit 1212 is specifically used to input the third text transformation vector into the text encoding network in the text encoding model, and through the text encoding network E m Output the historical text encoding vector and input the historical text encoding vector into the text encoding network E. m+1 In text encoding network E m+1 In the middle, based on historical text encoding vectors and text encoding network E m+1 The first intermediate encoding vector is obtained by using a multi-head self-attention network in the text encoding network. Then, residual concatenation and normalization are performed on the first intermediate encoding vector and the historical text encoding vector to obtain the second intermediate encoding vector. This second intermediate encoding vector is then input into the text encoding network E. m+1 The fully connected layer in the first layer yields the third intermediate encoding vector; residual connections and normalization are then performed on the third and second intermediate encoding vectors to obtain the text encoding network E. m+1 The target text encoding vector is used to obtain the text semantic vector of the text to be processed.

[0207] The specific implementation methods of the second conversion subunit 1211 and the text encoding subunit 1212 can be found in the above description. Figure 3 The description of step S102 in the corresponding embodiment, or you can refer to the above. Figure 4 The description of step S202 in the corresponding embodiments will not be repeated here.

[0208] Image encoding unit 122 is used to input candidate images into image encoding model, and to encode the candidate images through image encoding model to obtain candidate image vectors corresponding to the candidate images.

[0209] The image encoding unit 122 may include: an image segmentation subunit 1221, a feature extraction subunit 1222, and an image encoding subunit 1223;

[0210] Image segmentation subunit 1221 is used to segment candidate images and use the segmented images as region images.

[0211] The feature extraction subunit 1222 is used to input the region image into the feature extraction network in the image coding model, extract features from the region image through the feature extraction network, obtain the image features corresponding to the region image, and obtain the image feature sequence based on the image features corresponding to the region image.

[0212] Image encoding subunit 1223 is used to input the image feature sequence into the image encoding network in the image encoding model, and to encode the image feature sequence through the image encoding network to obtain the candidate image vector corresponding to the candidate image.

[0213] The specific implementation methods of the image segmentation subunit 1221, feature extraction subunit 1222, and image encoding subunit 1223 can be found above. Figure 3 The description of step S102 in the corresponding embodiment, or you can refer to the above. Figure 4 The description of step S203 in the corresponding embodiments will not be repeated here.

[0214] The vector search unit 123 is used to search for the candidate image vector that has the minimum vector distance to the text semantic vector in the candidate image vector corresponding to the candidate image, and use it as the candidate image vector that has the same semantics as the text semantic vector.

[0215] Image determination unit 124 is used to determine the text mapping image corresponding to the text to be processed based on the candidate image corresponding to the found candidate image vector;

[0216] Wherein, if the number of candidate image vectors found is N, then the number of candidate images corresponding to the found candidate image vectors is N; N is a positive integer greater than 1;

[0217] The image determination unit 124 may include: a parameter configuration subunit 1241 and an image composition subunit 1242;

[0218] The parameter configuration subunit 1241 is used to obtain the N candidate images found and configure the image arrangement order and image switching frequency for the N candidate images;

[0219] Image synthesis subunit 1242 is used to synthesize N candidate images based on the configured image arrangement order and image switching frequency, and use the synthesized image as the text mapping image corresponding to the text to be processed.

[0220] The specific implementation methods of the parameter configuration subunit 1241 and the image compositing subunit 1242 can be found above. Figure 3 The description of step S102 in the corresponding embodiments will not be repeated here.

[0221] The specific implementation methods of the text encoding unit 121, image encoding unit 122, vector lookup unit 123, and image determination unit 124 can be found above. Figure 3 The description of step S102 in the corresponding embodiment, or you can refer to the above. Figure 4 The descriptions of steps S202-S205 in the corresponding embodiments will not be repeated here.

[0222] The speech conversion module 13 is used to convert the text to be processed into a text-mapped speech that matches the speech conversion parameters when the speech conversion parameters associated with the text to be processed are obtained.

[0223] Among them, the speech conversion parameters are obtained by the business object associated with the original text by triggering the voiceprint in the voiceprint list; the voiceprint list is output by the business object by triggering the voiceprint selection control, or the voiceprint list is output when keywords carrying business object attributes used to characterize the business object are identified in the text to be processed.

[0224] The speech conversion module 13 may include: a phoneme conversion unit 131 and a speech generation unit 132;

[0225] The phoneme conversion unit 131 is used to perform phoneme conversion on the text to be processed based on the speech conversion parameters associated with the text to be processed when the speech conversion parameters associated with the text to be processed are obtained, to obtain the phoneme sequence corresponding to the text to be processed, and to obtain the phoneme parameters corresponding to the phonemes in the phoneme sequence; the phoneme parameters are determined by the speech conversion parameters.

[0226] The speech generation unit 132 is used to generate text-mapped speech that matches the speech conversion parameters based on the phoneme sequence and phoneme parameters.

[0227] The specific implementation methods of the phoneme conversion unit 131 and the speech generation unit 132 can be found in the above description. Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0228] The video generation module 14 is used to generate audio frames and video frames for synthesizing target videos based on the text to be processed, text-mapped images, and text-mapped speech. The generated audio frames and video frames are then processed to synthesize the target video corresponding to the original text.

[0229] The video generation module 14 may include: a preprocessing unit 141, an adjustment unit 142, an audio frame generation unit 143, a video frame generation unit 144, an audio-video alignment unit 145, and a video generation unit 146.

[0230] The preprocessing unit 141 is used to perform image preprocessing on the text mapping image to obtain a preprocessed image with target pixels;

[0231] The adjustment unit 142 is used to determine the speech attribute label of the text-mapped speech based on the speech conversion parameters, adjust the image tone of the preprocessed image based on the speech attribute label, use the preprocessed image with adjusted image tone as the target image, and adjust the text display style of the text to be processed based on the speech attribute label, use the text display style of the text to be processed as the target text.

[0232] The audio frame generation unit 143 is used to perform audio encoding on the text-mapped speech to obtain audio frames for synthesizing the target video.

[0233] The video frame generation unit 144 is used to generate video frames associated with audio frames based on the speech duration, target image, and target text mapped from text to speech.

[0234] The video frame generation unit 144 may include: a first time determination subunit 1441, a second time determination subunit 1442, and an image-text alignment subunit 1443;

[0235] The first time determination subunit 1441 is used to determine the display start timestamp of the target text based on the speech duration of the text-mapped speech and the first time interval, and to determine the display end timestamp of the target text based on the speech duration and the second time interval, and to use the duration between the display start timestamp and the display end timestamp as the text display duration of the target text.

[0236] The second time determination subunit 1442 is used to determine the playback start time stamp of the target image based on the display start time stamp and the third time interval, and to determine the playback end time stamp of the target image based on the display end time stamp and the fourth time interval, and to use the duration between the playback start time stamp and the playback end time stamp as the image playback duration of the target image.

[0237] The image-text alignment subunit 1443 is used to align target text with text display duration and target image with image playback duration, and generate a video frame associated with the audio frame based on the aligned target text and target image.

[0238] The specific implementation methods of the first-time determination subunit 1441, the second-time determination subunit 1442, and the image-text alignment subunit 1443 can be found above. Figure 3 The description of step S104 in the corresponding embodiments will not be repeated here.

[0239] The audio-video alignment unit 145 is used to align audio frames and video frames based on the duration of the audio frames to obtain audio-video frames for synthesizing the target video.

[0240] The video generation unit 146 is used to generate the target video corresponding to the original text based on the audio and video frames.

[0241] The specific implementation methods of the preprocessing unit 141, adjustment unit 142, audio frame generation unit 143, video frame generation unit 144, audio-video alignment unit 145, and video generation unit 146 can be found above. Figure 3 The description of step S104 in the corresponding embodiments will not be repeated here.

[0242] The specific implementation methods of the text acquisition module 11, image search module 12, speech conversion module 13, and video generation module 14 can be found above. Figure 3 The description of steps S101-S104 in the corresponding embodiments, or you can refer to the above. Figure 4 The descriptions of steps S201-S212 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0243] Please see Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 9As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1004 may be high-speed RAM or non-volatile memory, such as at least one disk drive. The memory 1005 may optionally be at least one storage device located remotely from the processor 1001. Figure 9 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0244] In such Figure 9 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to execute the aforementioned... Figure 3 , Figure 4 The description of the video synthesis method in any corresponding embodiment will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0245] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned video synthesis apparatus 1. The computer program includes program instructions, and when the processor executes the program instructions, it can execute the aforementioned... Figure 3 , Figure 4 The description of the video synthesis method in any corresponding embodiment is already provided, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0246] The aforementioned computer-readable storage medium can be the video compositing apparatus provided in any of the foregoing embodiments or the internal storage unit of the aforementioned computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0247] Furthermore, it should be noted that this application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned... Figure 3 , Figure 4 The method is provided in any of the corresponding embodiments. Furthermore, the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program products or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0248] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0249] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0250] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A video synthesis method, characterized in that, include: When the original text is obtained, the text to be processed in the original text is obtained; The length of the text to be processed is less than the length of the original text; The text to be processed is vector-transformed to obtain a third text-transformed vector. This third text-transformed vector is then input into the text encoding network of the text encoding model. The text encoding model contains M text encoding networks, including text encoding network E. m and text encoding network E m+1 The text encoding network E m+1 For the text encoding network E m The next text encoding network, where m is a positive integer less than M; Through the text encoding network E m Output the historical text encoding vector, and input the historical text encoding vector into the text encoding network E. m+1 ; In the text encoding network E m+1 In this process, based on the historical text encoding vector and the text encoding network E m+1 The first intermediate encoding vector is obtained from the multi-head self-attention network in the middle; The first intermediate encoding vector and the historical text encoding vector are residually concatenated and normalized to obtain the second intermediate encoding vector, which is then input into the text encoding network E. m+1 The fully connected layer in the middle is used to obtain the third intermediate encoding vector; The text encoding network E is obtained by performing residual concatenation and normalization on the third intermediate encoding vector and the second intermediate encoding vector. m+1 The target text encoding vector; Based on the target text encoding vector, the text semantic vector of the text to be processed is obtained. In the candidate image vector corresponding to the candidate image, a candidate image vector with the same semantics as the text semantic vector is searched. Based on the candidate image corresponding to the searched candidate image vector, the text mapping image corresponding to the text to be processed is determined. When the speech conversion parameters associated with the text to be processed are obtained, the text to be processed is converted into a text-mapped speech that matches the speech conversion parameters; Based on the text to be processed, the text-mapped image, and the text-mapped speech, audio frames and video frames for synthesizing the target video are generated. The generated audio frames and video frames are then subjected to video synthesis processing to obtain the target video corresponding to the original text.

2. The method according to claim 1, characterized in that, The step of obtaining the text to be processed from the original text when the original text is obtained includes: When the original text is obtained, the first type of segmentation symbol in the original text is identified, and the original text is segmented based on the identified first type of segmentation symbol. The transition text of the segmented original text is taken as the first transition text; the text length of the first transition text is less than the text length of the original text. In the first transition text, transition text with a text length greater than a length threshold is taken as the text to be detected, and transition text with a text length less than or equal to the length threshold is taken as the first text to be processed; The second type of segmentation symbol in the text to be detected is identified, and the text to be detected is segmented based on the identified second type of segmentation symbol. The transition text of the segmented text to be detected is taken as the second transition text. The i-th transition text and the j-th transition text are obtained from any two adjacent transition texts of the second transition text; j=i+1; i is a positive integer less than K, and K is the number of transition texts of the segmented text to be detected. Text relevance detection is performed on the i-th transition text and the j-th transition text to obtain the text relevance detection result between the i-th transition text and the j-th transition text. Based on the text relevance detection result, a second text to be processed is obtained. The first text to be processed and the second text to be processed are used as the text to be processed of the original text.

3. The method according to claim 2, characterized in that, The step involves performing text relevance detection on the i-th transition text and the j-th transition text to obtain a text relevance detection result between the i-th transition text and the j-th transition text, and obtaining a second text to be processed based on the text relevance detection result, including: Perform vector transformation on the i-th transition text to obtain the first text transformation vector corresponding to the i-th transition text, and perform vector transformation on the j-th transition text to obtain the second text transformation vector corresponding to the j-th transition text; The first text transformation vector and the second text transformation vector are input into the relevance detection model. The relevance detection model is used to perform text relevance detection on the first text transformation vector and the second text transformation vector to obtain the text relevance detection result between the i-th transition text and the j-th transition text. If the text relevance detection result indicates that there is a relevance between the i-th transition text and the j-th transition text, then the synthesized text composed of the i-th transition text and the j-th transition text will be used as the second text to be processed. If the text relevance detection result indicates that there is no relevance between the i-th transition text and the j-th transition text, then both the i-th transition text and the j-th transition text will be used as the second text to be processed.

4. The method according to claim 1, characterized in that, The step of searching for candidate image vectors with the same semantic meaning as the text semantic vector in the candidate image vectors corresponding to the candidate images, and determining the text mapping image corresponding to the text to be processed based on the candidate images corresponding to the found candidate image vectors, includes: The candidate image is input into the image encoding model, and the candidate image is encoded by the image encoding model to obtain the candidate image vector corresponding to the candidate image. Find the candidate image vector that has the minimum vector distance to the text semantic vector among the candidate image vectors corresponding to the candidate image, and use it as the candidate image vector that has the same semantics as the text semantic vector; The text mapping image corresponding to the text to be processed is determined based on the candidate image corresponding to the found candidate image vector.

5. The method according to claim 1, characterized in that, The step of performing vector transformation on the text to be processed to obtain a third text transformation vector corresponding to the text to be processed includes: The text to be processed is segmented into words to obtain the text segments of the text to be processed. The text segments are then converted into word vectors to obtain the word vectors corresponding to the text segments. Obtain the global semantic vector corresponding to the text to be processed and the position vector associated with the text position of the text segmentation in the text to be processed. Then, perform vector fusion on the word vector, the global semantic vector and the position vector to obtain the third text transformation vector corresponding to the text to be processed.

6. The method according to claim 4, characterized in that, The step of inputting candidate images into an image encoding model and encoding the candidate images using the image encoding model to obtain candidate image vectors corresponding to the candidate images includes: The candidate images are segmented, and the segmented images are used as region images; The region image is input into the feature extraction network in the image encoding model. The feature extraction network extracts features from the region image to obtain the image features corresponding to the region image. Based on the image features corresponding to the region image, an image feature sequence is obtained. The image feature sequence is input into the image encoding network in the image encoding model, and the image encoding network encodes the image feature sequence to obtain the candidate image vector corresponding to the candidate image.

7. The method according to claim 4, characterized in that, If the number of candidate image vectors found is N, then the number of candidate images corresponding to the found candidate image vectors is N; N is a positive integer greater than 1. The step of determining the text mapping image corresponding to the text to be processed based on the candidate image corresponding to the found candidate image vector includes: Obtain the N candidate images found, and configure the image arrangement order and image switching frequency for the N candidate images; Based on the configured image arrangement order and image switching frequency, the N candidate images are synthesized, and the synthesized image is used as the text mapping image corresponding to the text to be processed.

8. The method according to claim 1, characterized in that, The speech conversion parameters are obtained by the business object associated with the original text by triggering the voiceprint in the voiceprint list; the voiceprint list is output by the business object by triggering the voiceprint selection control, or the voiceprint list is output when keywords carrying business object attributes used to characterize the business object are identified in the text to be processed. The step of converting the text to be processed into a text-mapped speech that matches the speech conversion parameters when the speech conversion parameters associated with the text to be processed are obtained includes: When the speech conversion parameters associated with the text to be processed are obtained, the text to be processed is converted into phonemes based on the speech conversion parameters to obtain the phoneme sequence corresponding to the text to be processed, and the phoneme parameters corresponding to the phonemes in the phoneme sequence are obtained; the phoneme parameters are determined by the speech conversion parameters. Based on the phoneme sequence and the phoneme parameters, a text-mapped speech that matches the speech conversion parameters is generated.

9. The method according to claim 1, characterized in that, The step of generating audio frames and video frames for synthesizing the target video based on the text to be processed, the text-mapped image, and the text-mapped speech includes: The text-mapped image is preprocessed to obtain a preprocessed image with the target pixels; Based on the speech conversion parameters, the speech attribute labels of the text-mapped speech are determined, the image tone of the preprocessed image is adjusted based on the speech attribute labels, the preprocessed image with adjusted image tone is used as the target image, and the text display style of the text to be processed is adjusted based on the speech attribute labels, the text to be processed with adjusted text display style is used as the target text. The text-mapped speech is audio encoded to obtain audio frames used for synthesizing the target video; Based on the speech duration of the text-mapped speech, the target image, and the target text, a video frame associated with the audio frame is generated.

10. The method according to claim 9, characterized in that, The process of generating a video frame associated with the audio frame based on the speech duration of the text-mapped speech, the target image, and the target text includes: Based on the speech duration of the text-mapped speech and the first time interval, the display start timestamp of the target text is determined, and based on the speech duration and the second time interval, the display end timestamp of the target text is determined. The duration between the display start timestamp and the display end timestamp is taken as the text display duration of the target text. Based on the display start timestamp and the third time interval, the playback start timestamp of the target image is determined, and based on the display end timestamp and the fourth time interval, the playback end timestamp of the target image is determined. The duration between the playback start timestamp and the playback end timestamp is taken as the playback duration of the target image. The target text with the specified text display duration and the target image with the specified image playback duration are aligned, and a video frame associated with the audio frame is generated based on the aligned target text and target image.

11. The method according to claim 9, characterized in that, The step of performing video compositing on the generated audio and video frames to obtain the target video corresponding to the original text includes: Based on the duration of the audio frame, the audio frame and the video frame are aligned to obtain an audio-video frame for synthesizing the target video. The target video corresponding to the original text is generated based on the audio and video frames.

12. A video synthesis apparatus, characterized in that, include: The text acquisition module is used to acquire the text to be processed from the original text when the original text is acquired; The length of the text to be processed is less than the length of the original text; The image search module is used to perform vector transformation on the text to be processed to obtain a third text transformation vector corresponding to the text to be processed, and input the third text transformation vector into the text encoding network in the text encoding model; wherein, the number of text encoding networks in the text encoding model is M, and the M text encoding networks include text encoding network E m and text encoding network E m+1 The text encoding network E m+1 For the text encoding network E m The next text encoding network, where m is a positive integer less than M; The image search module is also used to access the text encoding network E. m Output the historical text encoding vector, and input the historical text encoding vector into the text encoding network E. m+1 ; The image search module is also used in the text encoding network E m+1 In this process, based on the historical text encoding vector and the text encoding network E m+1 The first intermediate encoding vector is obtained from the multi-head self-attention network in the middle; The image search module is further configured to perform residual concatenation and normalization on the first intermediate encoding vector and the historical text encoding vector to obtain a second intermediate encoding vector, and input the second intermediate encoding vector into the text encoding network E. m+1 The fully connected layer in the middle is used to obtain the third intermediate encoding vector; The image search module is further configured to perform residual connection and normalization processing on the third intermediate encoding vector and the second intermediate encoding vector to obtain the text encoding network E. m+1 The target text encoding vector; The image search module is further configured to obtain the text semantic vector of the text to be processed based on the target text encoding vector, search for candidate image vectors with the same semantics as the text semantic vector in the candidate image vectors corresponding to the candidate images, and determine the text mapping image corresponding to the text to be processed based on the candidate images corresponding to the found candidate image vectors. The speech conversion module is used to convert the text to be processed into a text-mapped speech that matches the speech conversion parameters when the speech conversion parameters associated with the text to be processed are obtained. The video generation module is used to generate audio frames and video frames for synthesizing a target video based on the text to be processed, the text mapping image, and the text mapping speech, and to perform video synthesis processing on the generated audio frames and video frames to obtain the target video corresponding to the original text.

13. A computer device, characterized in that, include: Processor and memory; The processor is connected to the memory, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to cause the computer device to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method according to any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, the computer instructions being adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

Images