Verified video method and device, computer equipment and storage medium

Abstract

The invention relates to a text video method and device, computer equipment and a storage medium. The method comprises the following steps: segmenting a sample video into a plurality of semantically coherent video clips, performing speaking object detection on each video clip, and screening out the video clip containing a clear and visible speaking object as a screened clip, so as to ensure that a face sample participating in model training is relatively high in definition and natural in face posture; the problem of face flicker in a traditional data set is avoided, face proportion adjustment is carried out on speaking objects in the screening fragments to obtain the cut fragments, consistency of face proportions in all sample videos is ensured, the situation that faces in the traditional data set are too large or too small is avoided, actual application scenes are better met, and the user experience is improved. According to the embodiment of the invention, the cut fragments and the audio data of the speaking object are utilized to participate in training of the text video model, so that the mouth shape of the speaking object can be synchronized with the audio, and the trained text video model is enabled to have good face expression in the video content generated for the text video request.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for generating video. Background Technology

[0002] With the rapid development of artificial intelligence technology, text-to-video (TTV), a technology that can automatically generate video content based on text descriptions, is gradually becoming a research hotspot in the fields of computer vision and natural language processing. In recent years, thanks to the widespread application of the Transformer architecture, TTV technology has made significant progress, with the emergence of a series of excellent models such as OpenAI's Sora and Runway's Gen-2. However, current TTV models on the market still have significant shortcomings in face generation, particularly in terms of facial expression naturalness, lip-sync, and face stability. These shortcomings stem from numerous issues with the face quality of existing TTV training datasets, which severely restrict the widespread application of TTV technology in film and television production, virtual anchors, and education and training. Summary of the Invention

[0003] This application provides a text-based video method, apparatus, computer device, and storage medium to address the problem that existing text-based video models perform poorly because text-based video training datasets cannot provide high-quality face training data.

[0004] Firstly, this application provides a text-based video generation method, the method comprising:

[0005] The acquired sample video is segmented into multiple semantically coherent video segments;

[0006] The speaker recognition result in each video segment is determined, and the video segment corresponding to the speaker recognition result that meets the preset face condition is used as the filter segment, wherein the preset face condition is used to ensure that there is a clearly visible speaker in the filter segment;

[0007] The selected segments are cropped based on a preset cropping algorithm to obtain cropped segments, wherein the preset cropping algorithm is used to ensure that the proportion of faces in the cropped segments is a preset proportion of faces;

[0008] The initial text-based video model is trained using each of the cropped segments and the corresponding audio data as face training data to obtain the preset text-based video model.

[0009] Upon receiving a text-based video request, the preset text-based video model is used to generate the target video corresponding to the text-based video request, wherein the text-based video request is used to request the generation of a target video containing a face object according to the description text.

[0010] Secondly, this application provides a video recording device, the device comprising:

[0011] The segmentation module is used to divide the acquired sample video into multiple semantically coherent video segments;

[0012] The filtering module is used to determine the recognition result of the speaking object in each of the video segments, and to select the video segments corresponding to the recognition result of the speaking object that meets the preset face conditions as the filtering segments, wherein the preset face conditions are used to ensure that there is a clearly visible speaking object in the filtering segments;

[0013] The cropping module is used to crop the selected segments based on a preset cropping algorithm to obtain cropped segments, wherein the preset cropping algorithm is used to ensure that the proportion of faces in the cropped segments is a preset proportion of faces;

[0014] The training module is used to train the initial text-based video model using each of the cropped segments and the corresponding audio data as face training data to obtain the preset text-based video model.

[0015] The generation module is used to generate a target video corresponding to the text-based video request using the preset text-based video model when a text-based video request is received, wherein the text-based video request is used to request the generation of a target video containing a face object according to the description text.

[0016] Thirdly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described text-to-video method.

[0017] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the above-described text-to-video method.

[0018] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application divides the acquired sample video into multiple semantically coherent video segments; determines the recognition result of the speaking object in each video segment, and uses the video segment corresponding to the recognition result of the speaking object that meets the preset face conditions as the filtering segment, wherein the preset face conditions are used to ensure that there is a clearly visible speaking object in the filtering segment; the filtering segment is cropped based on a preset cropping algorithm to obtain a cropped segment, wherein the preset cropping algorithm is used to ensure that the proportion of faces in the cropped segment is a preset face proportion; the initial text-based video model is trained using each cropped segment and the audio data corresponding to each cropped segment as face training data to obtain a preset text-based video model; when a text-based video request is received, the target video corresponding to the text-based video request is generated using the preset text-based video model, wherein the text-based video request is used to request the generation of a target video containing face objects according to the description text.

[0019] Based on the above method, after segmenting the sample video into multiple semantically coherent video segments, the speaker is detected in each video segment. Video segments containing clearly visible speakers are selected as filter segments to ensure that the face samples used for model training have high clarity and natural facial poses, avoiding the face flickering problem in traditional datasets. The proportion of the face in the filter segments is adjusted to obtain cropped segments, ensuring the consistency of the face proportion in all sample videos and avoiding faces that are too large or too small in traditional datasets, which is more in line with actual application scenarios. Finally, the cropped segments and the audio data of the speakers are used to participate in the training of the text-based video model to ensure that the lip movements of the speakers are synchronized with the audio. This allows the trained text-based video model to generate video content with good face performance for text-based video requests, thus solving the problem that existing text-based video models perform poorly because text-based video training datasets cannot provide high-quality face training data. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0022] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0023] Figure 1 An application environment diagram of a text-to-video generation method provided in this application embodiment;

[0024] Figure 2 A flowchart illustrating a text-to-video generation method provided in an embodiment of this application;

[0025] Figure 3 A structural block diagram of a text-based video device provided in this application embodiment;

[0026] Figure 4 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0029] Figure 1 This is a diagram illustrating the application environment of the Chinese video generation method in one embodiment. (Refer to...) Figure 1 This text-to-video method is applied to a text-to-video system. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; a mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers.

[0030] In one embodiment, Figure 2This is a flowchart illustrating a text-to-video method in one embodiment, with reference to... Figure 2 This paper provides a method for text-to-video generation. This embodiment primarily applies this method to the aforementioned... Figure 1 Taking server 120 as an example, the video generation method specifically includes the following steps:

[0031] Step S210: The obtained sample video is segmented into multiple semantically coherent video segments.

[0032] Specifically, the sample videos can be long or short. By segmenting the sample videos according to their semantics, we can obtain multiple semantically coherent video segments. For example, a video segment may correspond to a line of dialogue or a speech by a speaker.

[0033] Step S220: Determine the speech object recognition result in each of the video segments, and select the video segments corresponding to the speech object recognition results that meet the preset face conditions as the filter segments, wherein the preset face conditions are used to ensure that there are clearly visible speech objects in the filter segments.

[0034] Specifically, speech object detection is performed on each video segment to determine the speech object recognition result. The speech object recognition result includes the speech object, the clarity of the speech object's face, the proportion of the face, the face pose, and the display brightness of the face region. The speech object recognition result is compared with preset face conditions to select video segments that meet the preset face conditions as filter segments. In this way, video segments with high face quality are selected, thereby improving the quality of the Wensheng video training dataset.

[0035] Step S230: The selected segment is cropped based on a preset cropping algorithm to obtain a cropped segment, wherein the preset cropping algorithm is used to ensure that the proportion of faces in the cropped segment is a preset proportion of faces.

[0036] Specifically, a preset cropping algorithm is used to crop the selected segments to ensure that the proportion of the speaker's face in the cropped segment is the preset proportion. This adjusts the different face proportions in different selected segments to the face proportions corresponding to the unified preset proportion, thus avoiding the presence of excessively large or small faces in the Wensheng video training dataset.

[0037] Step S240: Train the initial text-based video model using each cropped segment and the corresponding audio data as face training data to obtain the preset text-based video model.

[0038] Specifically, the cropped segments with adjusted face proportions and the corresponding audio data are input into the initial text-based video model to be trained. Multimodal data is generated using video segments with appropriate face proportions and audio data to participate in model training. This data is used to learn the synchronization between lip movements and audio when the face is speaking. Multimodal data can improve the performance of face generation during model training, such as face stability, naturalness of expression, and synchronization between lip movements and audio. Once the model is trained to meet the model requirements, the pre-trained text-based video model can be obtained.

[0039] Step S250: Upon receiving a text-based video request, the target video corresponding to the text-based video request is generated using the preset text-based video model, wherein the text-based video request is used to request the generation of a target video containing a face object according to the description text.

[0040] Specifically, a text-based video request can be a request initiated by terminal 110 to server 120, or a request triggered by a user on server 120. This request requests the generation of a target video containing facial objects according to the description text. The text-based video request includes video content description text and video format description text. The video content description text includes video description text and audio description text. The video description text indicates the character and plot requirements, while the audio description text indicates the background sound requirements. The video format description text includes descriptions of at least one parameter, such as video format, video duration, video clarity, video resolution, and video screen size. The preset text-based video model outputs the corresponding target video based on the text-based video request. The target video simultaneously meets the audio and video requirements corresponding to the video content description text and the video format requirements corresponding to the video format description text. Furthermore, the facial image quality of the facial objects in the target video is good, the expression is natural, and the lip movements are synchronized with the audio.

[0041] Based on the above method, after segmenting the sample video into multiple semantically coherent video segments, the speaker is detected in each video segment. Video segments containing clearly visible speakers are selected as filter segments to ensure that the face samples used for model training have high clarity and natural facial poses, avoiding the face flickering problem in traditional datasets. The proportion of the face in the filter segments is adjusted to obtain cropped segments, ensuring the consistency of the face proportion in all sample videos and avoiding faces that are too large or too small in traditional datasets, which is more in line with actual application scenarios. Finally, the cropped segments and the audio data of the speakers are used to participate in the training of the text-based video model to ensure that the lip movements of the speakers are synchronized with the audio. This allows the trained text-based video model to generate video content with good face performance for text-based video requests, thus solving the problem that existing text-based video models perform poorly because text-based video training datasets cannot provide high-quality face training data.

[0042] In one embodiment, before segmenting the acquired sample video into multiple semantically coherent video segments, the method further includes:

[0043] The original videos that meet the preset video quality conditions are selected from the video database as the sample videos. The preset video quality conditions include video resolution greater than or equal to the preset resolution, image clarity greater than or equal to the preset clarity, the number of video content types reaching at least a first preset number, and the number of video characters reaching at least a second preset number.

[0044] Specifically, the video database is a database provided by different video platforms. The video database includes long videos and / or short videos of different video qualities. Original videos with video resolution greater than or equal to a preset resolution, image clarity greater than or equal to a preset clarity, number of video content types greater than or equal to a first preset number, and number of video characters greater than or equal to a second preset number are selected from the video database as sample videos. In this way, original videos with high video quality and sufficient facial diversity are selected for subsequent facial training data extraction, which can improve the quality of facial training data extracted based on sample videos.

[0045] Video content types include various scene types, emotional types, and different lighting conditions. A greater number of video content types indicates higher content diversity, which is used to improve the scene diversity of facial training data. Similarly, a greater number of video characters indicates higher character diversity, which is used to improve the character diversity of facial training data.

[0046] In one embodiment, segmenting the acquired sample video into multiple semantically coherent video segments includes:

[0047] The sample video is converted to multiple video frames in a color space, and the color difference value between each adjacent video frame is calculated.

[0048] The moving average of the color difference values ​​between adjacent video frames within the sliding window is calculated by using a sliding window.

[0049] The video frame corresponding to the moving average value that is greater than the preset color difference value is determined as the segment segmentation point;

[0050] The sample video is divided into multiple semantically coherent video segments according to each of the segment segmentation points.

[0051] Specifically, using the HSL color space-based segmentation technique, long videos are divided into semantically coherent short segments. First, the OpenCV library is used to read each frame of the sample video and convert it to the HSL color space. Then, the color difference value between adjacent video frames is calculated. The formula for calculating the color difference value is as follows: Where F i and F i+1 This represents two adjacent video frames, where ΔH, ΔS, and ΔL represent the average differences in hue, saturation, and luminance channels between the adjacent video frames, respectively. H w S w L The weighting coefficient can be customized according to the actual application scenario. In this embodiment, w H w S w L Based on experience, the values ​​were set to 0.5, 0.3, and 0.2 respectively.

[0052] Then, a sliding window with a preset sliding length of N is used to calculate the moving average of the color difference values ​​between adjacent video frames within each sliding window. For example, if N is 3, the color difference value between video frame 1 and video frame 2 is 0.7, and the color difference value between video frame 2 and video frame 3 is 0.8, then the moving average value corresponding to this sliding window is (0.7 + 0.8) / 2 = 0.75. As the sliding window slides, the video frames within the sliding window change, therefore the moving average value of the sliding window is also different at different sliding times. That is, the moving average value corresponding to the sliding window is... When D avg (i) When the color difference exceeds the preset color difference value T, the i-th video frame is determined to be the shot switching point, which is the segment segmentation point. The sample video is then divided into multiple semantically coherent video segments according to the segment segmentation point.

[0053] The preset color difference value is determined by the video type of the sample video. Different video types correspond to different preset color difference values, and the preset color difference value ranges from 0.15 to 0.25.

[0054] In one embodiment, segmenting the sample video into multiple semantically coherent video segments according to each of the segment segmentation points includes:

[0055] The sample video is divided into multiple semantically coherent segments according to each of the segment segmentation points.

[0056] If the playback duration of the segment corresponding to the current segment is less than the preset duration, the current segment will be merged with the adjacent preceding or following segment to form a video segment; or,

[0057] When the playback duration of the segment corresponding to the current segment is greater than or equal to the preset duration, the current segment is treated as a video segment.

[0058] Specifically, the sample video is segmented into multiple semantically coherent segments based on the identified segmentation points. To avoid segments being too short, it is necessary to determine whether the playback duration of each segment is greater than or equal to a preset duration. In other words, the minimum duration of a segment must reach the preset duration. If the playback duration of the current segment is less than the preset duration, the current segment is merged with the preceding or following segment to form a single video segment. If the duration of the current segment reaches or exceeds the preset duration, it is treated as a single video segment. The segmentation method for audio segments is exactly the same as for video segments. Audio segments are obtained by segmenting the audio data corresponding to the sample video according to the segmentation points. The timestamp of the audio segment corresponding to the video segment is the same as the timestamp of the video segment.

[0059] The preset duration can be customized according to actual application needs; in this embodiment, the preset duration is set to 2 seconds. By merging excessively short segments to reduce the number of segments, the resource waste caused by performing speaker detection on excessively short segments can also be avoided.

[0060] In one embodiment, determining the speaker identification result in each of the video segments includes:

[0061] The face detection algorithm is used to detect the facial features of different face objects in each video frame of the video segment, wherein the facial features include face bounding boxes and facial features;

[0062] A face tracking algorithm is used to track the facial features of each face object in multiple consecutive video frames to obtain a face image sequence, wherein the face image sequence includes face images in multiple video frames, and each face image includes the facial features of at least one face object.

[0063] Audio features are extracted from the audio segments corresponding to the video segments according to the timestamps corresponding to each face image in the face image sequence;

[0064] The face image sequence and the audio features are input into a preset speaking object recognition model, and the output of the preset speaking object recognition model is determined as the speaking weight value of each face object in the video segment;

[0065] The face object corresponding to the speaking weight value that is greater than the preset weight value is determined as the speaking object in the video segment, wherein the speaking object recognition result includes the speaking object.

[0066] Specifically, face recognition is performed on each video segment to detect the bounding boxes and facial features of different face objects in each video frame. Each face object corresponds to one bounding box, and a video frame may contain the bounding boxes of one or more face objects. Facial features include eye features, nose tip features, and lip features.

[0067] By using a face tracking algorithm to track the facial features of a face object in multiple consecutive video frames, the facial movement trajectory of the face object in different video frames can be obtained, that is, a face image sequence is obtained. The face image sequence includes face images from multiple consecutive video frames, and each face image includes at least one facial feature corresponding to a face object.

[0068] Audio features corresponding to the timestamps of each face image in the face image sequence are extracted from the corresponding audio segments. The audio features of the video segments and the face image sequence are then input into a pre-defined speaker recognition model. This model includes a 3D-CNN module for processing face images, an LSTM module for processing audio features, and a fully connected layer for feature fusion. Based on the pre-defined speaker recognition model, the probability that different face objects in the video segment are speaking objects is calculated, thus obtaining the speaking weight value for each face object to determine which face object is speaking. The formula for calculating the speaking weight value is P(speak|F). i A i )=σ(W v ·CNN(F i )+W a ·LSTM(A i )+b), where F i Let A represent the face image in the i-th video frame. i W represents the corresponding audio feature. v W a Let b be the weight matrix corresponding to the 3D-CNN module and the LSTM module, b be the bias term, and σ be the sigmoid activation function.

[0069] When the speech weight value is greater than the preset weight value, it indicates that the face object corresponding to that speech weight value is the speaker in the video segment, that is, the speaker's lip movements match the audio features. In this embodiment, the preset weight value is set to 0.5. Based on the data from two different modalities, namely face recognition results and audio features, the speaker can be accurately detected.

[0070] In one embodiment, the step of using the video segment corresponding to the speech object recognition result that meets the preset facial conditions as the filter segment includes:

[0071] Based on the speech object recognition results of the video segment, determine the speech object, the number of speech objects, the facial clarity, the object display brightness, the speech object's facial pose yaw angle, and the facial pose pitch angle;

[0072] The video segments selected are those with one speaker, a face clarity greater than a preset clarity, a face yaw angle and a face pitch angle both less than a preset angle, and a display brightness within a preset brightness range.

[0073] Specifically, from multiple video clips, video clips with only one speaker, facial clarity greater than a preset clarity, facial pose yaw angle and facial pose pitch angle both less than preset angles, and display brightness within a preset brightness range are selected as filter clips. Facial clarity can be represented by the Laplacian response value, while facial pose yaw angle and facial pose pitch angle both less than preset angles indicate that the facial pose is close to frontal. By selecting video clips with high facial clarity, near-frontal facial features, and good lighting conditions based on the above selection criteria, these clips can be used to participate in model training, thereby improving the facial quality of the facial training data.

[0074] In one embodiment, the selected segment is cropped based on a preset cropping algorithm to obtain a cropped segment, including:

[0075] Obtain the bounding box of the face of the speaking object in each video frame of the selected segment;

[0076] Determine the union region of the facial bounding boxes of the speaking object in different video frames in the selected segment;

[0077] The cropping size corresponding to the speaking object in the filtered segment is determined based on the preset face proportion and the average area of ​​multiple face bounding box regions corresponding to the speaking object in the filtered segment.

[0078] The cropping region corresponding to the speaking object in the filtered segment is determined based on the cropping size and the average center coordinates of the bounding box region of the face corresponding to the speaking object in the filtered segment.

[0079] When the cropping region is greater than or equal to the union region, and the cropping region is located within the video frame range corresponding to the selected segment, the selected segment is cropped according to the cropping region to obtain the cropped segment corresponding to the speaking object in the selected segment.

[0080] Specifically, for each video segment, the bounding box of the speaker's face is obtained in each frame. The bounding box can represent not only the area occupied by the face but also the position of the face. The union of the regions where the bounding boxes of the faces appear in different video frames is obtained. The union region is used to reflect the range of facial movement of the speaker in different video frames. Generally, the range of facial movement of the speaker is smaller in shorter video segments, that is, the amplitude of facial swaying is smaller.

[0081] Determining the cropping size of the speaking object in the selected segment based on the preset face proportion and the average area of ​​multiple face bounding box regions corresponding to the speaking object in the selected segment specifically includes: calculating the average area S of the face bounding box regions of the speaking object in different video frames. face The cropping size is obtained by taking the square root of the ratio of the average area to the preset face proportion. Where R target This indicates the preset face proportion, which ranges from 0.08 to 0.12.

[0082] Determining the cropping region corresponding to the speaking object in the selected segment based on the cropping size and the average center coordinates of the face bounding box region corresponding to the speaking object in the selected segment specifically includes: calculating the average center coordinates based on the center coordinates of the face bounding box of the speaking object in different video frames of the selected segment, where the average center coordinates are (x... center ,y center The average center coordinate system includes the center x-coordinate and the center y-coordinate. The center x-coordinate minus half the cutting size is used to determine the cutting x-coordinate, and the center y-coordinate minus half the cutting size is used to determine the cutting y-coordinate. The cutting x-coordinate is x. crop =x center -d crop / 2, the cropping ordinate is y crop =y center -d crop / 2, the area selected by the cropping horizontal and vertical coordinates is defined as the cropping region.

[0083] However, in order to ensure that the cropping area can cover all positions of the speaking object in the video segment, it is necessary to limit the cropping area to be greater than or equal to the union area, and to ensure that the cropping area is within the range of the video frame. That is, when the cropping area meets the above conditions, each video frame in the selected segment is cropped according to the cropping area to obtain the cropped segment.

[0084] When the cropping area exceeds the video frame's range, the cropping area is shifted back to within the video frame's range, and then the selected segment is cropped according to the shifted cropping area, i.e., x is limited.crop =max(0,min(x) center -d crop / 2,WS crop )), and y crop =max(0,min(y) center -d crop / 2,HS crop )).

[0085] After cropping the selected segments according to the cropping area, the cropped segments are quality checked to ensure that the face is clearly visible, the expression is natural, and the lip movements are synchronized with the audio. The cropped segments used in model training can be the segments that have passed the quality check, which further improves the face quality of the face training data, thereby improving the performance of the text-based video model in face generation.

[0086] Figure 2 This is a flowchart illustrating a Chinese video generation method in one embodiment. It should be understood that, although... Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0087] In one embodiment, such as Figure 3 As shown, a video recording device is provided, comprising:

[0088] The segmentation module 310 is used to segment the acquired sample video into multiple semantically coherent video segments;

[0089] The filtering module 320 is used to determine the recognition result of the speaking object in each of the video segments, and to use the video segments corresponding to the recognition result of the speaking object that meets the preset face conditions as the filtering segments, wherein the preset face conditions are used to ensure that there is a clearly visible speaking object in the filtering segments;

[0090] The cropping module 330 is used to crop the selected segment based on a preset cropping algorithm to obtain a cropped segment, wherein the preset cropping algorithm is used to ensure that the proportion of faces in the cropped segment is a preset proportion of faces;

[0091] Training module 340 is used to train the initial text-based video model using each of the cropped segments and the audio data corresponding to each of the cropped segments as face training data, so as to obtain a preset text-based video model.

[0092] The generation module 350 is used to generate a target video corresponding to the textual video request using the preset textual video model when a textual video request is received, wherein the textual video request is used to request the generation of a target video containing a face object according to the description text.

[0093] In one embodiment, the segmentation module 310 is further configured to:

[0094] The original videos that meet the preset video quality conditions are selected from the video database as the sample videos. The preset video quality conditions include video resolution greater than or equal to the preset resolution, image clarity greater than or equal to the preset clarity, the number of video content types reaching at least a first preset number, and the number of video characters reaching at least a second preset number.

[0095] In one embodiment, the segmentation module 310 is further configured to:

[0096] The sample video is converted to multiple video frames in a color space, and the color difference value between each adjacent video frame is calculated.

[0097] The moving average of the color difference values ​​between adjacent video frames within the sliding window is calculated by using a sliding window.

[0098] The video frame corresponding to the moving average value that is greater than the preset color difference value is determined as the segment segmentation point;

[0099] The sample video is divided into multiple semantically coherent video segments according to each of the segment segmentation points.

[0100] In one embodiment, the segmentation module 310 is further configured to:

[0101] The sample video is divided into multiple semantically coherent segments according to each of the segment segmentation points.

[0102] If the playback duration of the segment corresponding to the current segment is less than the preset duration, the current segment will be merged with the adjacent preceding or following segment to form a video segment; or,

[0103] When the playback duration of the segment corresponding to the current segment is greater than or equal to the preset duration, the current segment is treated as a video segment.

[0104] In one embodiment, the filtering module 320 is further configured to:

[0105] The face detection algorithm is used to detect the facial features of different face objects in each video frame of the video segment, wherein the facial features include face bounding boxes and facial features;

[0106] A face tracking algorithm is used to track the facial features of each face object in multiple consecutive video frames to obtain a face image sequence, wherein the face image sequence includes face images in multiple video frames, and each face image includes the facial features of at least one face object.

[0107] Audio features are extracted from the audio segments corresponding to the video segments according to the timestamps corresponding to each face image in the face image sequence;

[0108] The face image sequence and the audio features are input into a preset speaking object recognition model, and the output of the preset speaking object recognition model is determined as the speaking weight value of each face object in the video segment;

[0109] The face object corresponding to the speaking weight value that is greater than the preset weight value is determined as the speaking object in the video segment, wherein the speaking object recognition result includes the speaking object.

[0110] In one embodiment, the filtering module 320 is further configured to:

[0111] Based on the speech object recognition results of the video segment, determine the speech object, the number of speech objects, the facial clarity, the object display brightness, the speech object's facial pose yaw angle, and the facial pose pitch angle;

[0112] The video segments selected are those with one speaker, a face clarity greater than a preset clarity, a face yaw angle and a face pitch angle both less than a preset angle, and a display brightness within a preset brightness range.

[0113] In one embodiment, the cutting module 330 is further configured to:

[0114] Obtain the bounding box of the face of the speaking object in each video frame of the selected segment;

[0115] Determine the union region of the facial bounding boxes of the speaking object in different video frames in the selected segment;

[0116] The cropping size corresponding to the speaking object in the filtered segment is determined based on the preset face proportion and the average area of ​​multiple face bounding box regions corresponding to the speaking object in the filtered segment.

[0117] The cropping region corresponding to the speaking object in the filtered segment is determined based on the cropping size and the average center coordinates of the bounding box region of the face corresponding to the speaking object in the filtered segment.

[0118] When the cropping region is greater than or equal to the union region, and the cropping region is located within the video frame range corresponding to the selected segment, the selected segment is cropped according to the cropping region to obtain the cropped segment corresponding to the speaking object in the selected segment.

[0119] like Figure 4 As shown, this application provides a computer device including a processor 711, a communication interface 712, a memory 713, and a communication bus 714, wherein the processor 711, the communication interface 712, and the memory 713 communicate with each other through the communication bus 714.

[0120] Memory 713 is used to store computer programs;

[0121] When the processor 711 executes the program stored in the memory 713, it implements the text-based video method provided in any of the foregoing method embodiments.

[0122] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] In one embodiment, the video recording device provided in this application can be implemented as a computer program, which can be implemented in the form of, for example, Figure 4 It runs on the computer device shown. The computer device's memory can store the various program modules that make up the video recording device, for example, Figure 3 The diagram shows a segmentation module 310, a filtering module 320, a cropping module 330, a training module 340, and a generation module 350. The computer program comprised of these modules causes a processor to execute the text-based video generation methods of the various embodiments of this application described in this specification.

[0124] Figure 4 The computer device shown can be used as follows Figure 3The segmentation module 310 in the illustrated text-based video device divides the acquired sample video into multiple semantically coherent video segments. The computer device can use the filtering module 320 to determine the speech object recognition results in each video segment and select the video segments corresponding to speech object recognition results that meet preset face conditions as filtered segments. The preset face conditions are used to ensure that there are clearly visible speech objects in the filtered segments. The computer device can use the cropping module 330 to crop the filtered segments based on a preset cropping algorithm to obtain cropped segments. The preset cropping algorithm is used to ensure that the proportion of faces in the cropped segments is a preset face proportion. The computer device can use the training module 340 to train an initial text-based video model using each cropped segment and the corresponding audio data as face training data to obtain a preset text-based video model. The computer device can use the generation module 350 to generate a target video corresponding to a text-based video request when a text-based video request is received, using the preset text-based video model. The text-based video request is used to request the generation of a target video containing face objects according to the descriptive text.

[0125] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the text-to-video method as provided in any of the foregoing method embodiments.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server 120, or network device, etc.) to execute the text-based video method described in various embodiments or some parts of embodiments.

[0128] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also mean including the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that alternatives or substitutions may be used.

[0129] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating text-based videos, characterized in that, The method includes: The acquired sample video is segmented into multiple semantically coherent video segments; The speaker recognition result in each video segment is determined, and the video segment corresponding to the speaker recognition result that meets the preset face condition is used as the filter segment, wherein the preset face condition is used to ensure that there is a clearly visible speaker in the filter segment; The selected segments are cropped based on a preset cropping algorithm to obtain cropped segments, wherein the preset cropping algorithm is used to ensure that the proportion of faces in the cropped segments is a preset proportion of faces; The initial text-based video model is trained using each of the cropped segments and the corresponding audio data as face training data to obtain the preset text-based video model. Upon receiving a text-based video request, the preset text-based video model is used to generate the target video corresponding to the text-based video request, wherein the text-based video request is used to request the generation of a target video containing a face object according to the description text.

2. The text-to-video method according to claim 1, characterized in that, Before segmenting the acquired sample video into multiple semantically coherent video segments, the method further includes: The original videos that meet the preset video quality conditions are selected from the video database as the sample videos. The preset video quality conditions include video resolution greater than or equal to the preset resolution, image clarity greater than or equal to the preset clarity, the number of video content types reaching at least a first preset number, and the number of video characters reaching at least a second preset number.

3. The text-to-video method according to claim 1, characterized in that, The process of segmenting the acquired sample video into multiple semantically coherent video segments includes: The sample video is converted to multiple video frames in a color space, and the color difference value between each adjacent video frame is calculated. The moving average of the color difference values ​​between adjacent video frames within the sliding window is calculated by using a sliding window. The video frame corresponding to the moving average value that is greater than the preset color difference value is determined as the segment segmentation point; The sample video is divided into multiple semantically coherent video segments according to each of the segment segmentation points.

4. The text-based video generation method according to claim 3, characterized in that, The sample video is segmented into multiple semantically coherent video segments according to each of the aforementioned segmentation points, including: The sample video is divided into multiple semantically coherent segments according to each of the segment segmentation points. If the playback duration of the segment corresponding to the current segment is less than the preset duration, the current segment will be merged with the adjacent preceding or following segment to form a video segment; or, When the playback duration of the segment corresponding to the current segment is greater than or equal to the preset duration, the current segment is treated as a video segment.

5. The text-to-video method according to claim 1, characterized in that, The determination of the speaker identification result in each of the video segments includes: The face detection algorithm is used to detect the facial features of different face objects in each video frame of the video segment, wherein the facial features include face bounding boxes and facial features; A face tracking algorithm is used to track the facial features of each face object in multiple consecutive video frames to obtain a face image sequence, wherein the face image sequence includes face images in multiple video frames, and each face image includes the facial features of at least one face object. Audio features are extracted from the audio segments corresponding to the video segments according to the timestamps corresponding to each face image in the face image sequence; The face image sequence and the audio features are input into a preset speaking object recognition model, and the output of the preset speaking object recognition model is determined as the speaking weight value of each face object in the video segment; The face object corresponding to the speaking weight value that is greater than the preset weight value is determined as the speaking object in the video segment, wherein the speaking object recognition result includes the speaking object.

6. The text-to-video method according to claim 5, characterized in that, The step of using the video segments corresponding to the speech object recognition results that meet preset facial conditions as the selected segments includes: Based on the speech object recognition results of the video segment, determine the speech object, the number of speech objects, the facial clarity, the object display brightness, the speech object's facial pose yaw angle, and the facial pose pitch angle; The video segments selected are those with one speaker, a face clarity greater than a preset clarity, a face yaw angle and a face pitch angle both less than a preset angle, and a display brightness within a preset brightness range.

7. The text-to-video method according to claim 1, characterized in that, The selected segments are cropped based on a preset cropping algorithm to obtain cropped segments, including: Obtain the bounding box of the face of the speaking object in each video frame of the selected segment; Determine the union region of the facial bounding boxes of the speaking object in different video frames in the selected segment; The cropping size corresponding to the speaking object in the filtered segment is determined based on the preset face proportion and the average area of ​​multiple face bounding box regions corresponding to the speaking object in the filtered segment. The cropping region corresponding to the speaking object in the filtered segment is determined based on the cropping size and the average center coordinates of the bounding box region of the face corresponding to the speaking object in the filtered segment. When the cropping region is greater than or equal to the union region, and the cropping region is located within the video frame range corresponding to the selected segment, the selected segment is cropped according to the cropping region to obtain the cropped segment corresponding to the speaking object in the selected segment.

8. A text-based video device, characterized in that, The device includes: The segmentation module is used to divide the acquired sample video into multiple semantically coherent video segments; The filtering module is used to determine the recognition result of the speaking object in each of the video segments, and to select the video segments corresponding to the recognition result of the speaking object that meets the preset face conditions as the filtering segments, wherein the preset face conditions are used to ensure that there is a clearly visible speaking object in the filtering segments; The cropping module is used to crop the selected segments based on a preset cropping algorithm to obtain cropped segments, wherein the preset cropping algorithm is used to ensure that the proportion of faces in the cropped segments is a preset proportion of faces; The training module is used to train the initial text-based video model using each of the cropped segments and the corresponding audio data as face training data to obtain the preset text-based video model. The generation module is used to generate a target video corresponding to the text-based video request using the preset text-based video model when a text-based video request is received, wherein the text-based video request is used to request the generation of a target video containing a face object according to the description text.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the text-based video method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text-based video method according to any one of claims 1 to 7.

Images