Short video generation method and device, computer device, and storage medium

Abstract

The present application relates to the field of speech semantics, and particularly relates to a short video generation method and device, computer equipment and a storage medium. The method comprises: performing speech feature extraction on audio data based on a mel-frequency cepstral coefficient algorithm to obtain speech features of the audio data; performing action prediction processing on the speech features and the audio data through an adversarial network model to obtain predicted action features corresponding to the audio data; performing face recognition on face video data through a face recognition model to obtain target face features corresponding to the face video data; and embedding the predicted action features and the target face features into a video generator for rendering to generate a short video containing virtual face movements. The short video generated by the present application contains virtual face movements corresponding to the audio data and the face video data, and also contains the audio data, so that there is no lag, word forgetting or the like, the video effect is good, the needs of users can be met, and user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speech semantics, and in particular to a short video generation method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the rapid development of video technology, in order to better carry out information dissemination and propaganda, various industries will usually choose to disseminate information through short videos, live broadcasts and other video media.

[0003] However, since most of the staff of the enterprise are not professional video workers, a lot of time needs to be spent on shooting videos, resulting in that the finally generated short videos are often not ideal. For example, shooting errors such as shooting lag and forgetting words occur during shooting. And the existing video generation technology is difficult to completely eliminate the lag, forgetting words and other errors in the video, leaving no trace. Therefore, the generated video effect is poor. SUMMARY

[0004] Therefore, it is necessary to provide a short video method, device, computer equipment and storage medium to solve the problem that the existing video generation technology is difficult to eliminate the lag, forgetting words and other errors in the video, resulting in poor video effect.

[0005] A short video generation method comprises:

[0006] obtaining audio data, performing speech feature extraction on the audio data based on a mel-frequency cepstral coefficient algorithm to obtain speech features of the audio data;

[0007] performing action prediction processing on the speech features and the audio data through an adversarial network model to obtain predicted action features corresponding to the audio data;

[0008] obtaining face video data, performing face recognition on the face video data through a face recognition model to obtain target face features corresponding to the face video data;

[0009] embedding the predicted action features and the target face features into a video generator for rendering to generate a short video containing virtual face movements.

[0010] A short video generation device comprises:

[0011] a speech feature module configured to obtain audio data, perform speech feature extraction on the audio data based on a mel-frequency cepstral coefficient algorithm to obtain speech features of the audio data;

[0012] a predicted action feature module configured to perform action prediction processing on the speech features and the audio data through an adversarial network model to obtain predicted action features corresponding to the audio data;

[0013] a target face feature module, configured to obtain face video data, perform face recognition on the face video data by using a face recognition model, and obtain target face features corresponding to the face video data;

[0014] a short video module, configured to embed the predicted action features and the target face features into a video generator for rendering, and generate a short video containing virtual face movements.

[0015] A computer device includes a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, and the processor implements the short video generation method described above when executing the computer readable instructions.

[0016] One or more readable storage media storing computer readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the short video generation method described above.

[0017] The short video generation method, device, computer device, and storage medium described above obtain audio data, perform speech feature extraction on the audio data based on a mel-frequency cepstral coefficient algorithm to obtain speech features of the audio data, perform action prediction processing on the speech features and the audio data by using a generative adversarial network model to obtain predicted action features corresponding to the audio data, obtain face video data, perform face recognition on the face video data by using a face recognition model to obtain target face features corresponding to the face video data, embed the predicted action features and the target face features into a video generator for rendering, and generate a short video containing virtual face movements. According to the obtained audio data and face video data, the short video can be generated. The short video contains virtual face movements corresponding to the audio data and face video data, and also contains the audio data, so there is no lag, word forgetting, or the like, the video effect is good, the needs of users can be met, and user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 is an application environment schematic diagram of the short video generation method in an embodiment of the present application;

[0020] Figure 2is a flowchart of a short video generation method in an embodiment of the present application;

[0021] Figure 3 is a structural diagram of a short video generation device in an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0024] The short video generation method provided in the embodiment can be applied in an application environment such as Figure 1 , in which a client communicates with a server. The client includes but is not limited to various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0025] In an embodiment, as shown in Figure 2 , a short video generation method is provided, which is applied to a server in Figure 1 for example, and includes the following steps:

[0026] S10, obtaining audio data, performing speech feature extraction on the audio data based on a mel-frequency cepstral coefficient algorithm to obtain speech features of the audio data.

[0027] Understandably, the audio data refers to speech data containing speaking sound. For example, speech data recorded by a user himself. In the field of speech processing, the mel-frequency cepstral coefficient algorithm is used for sound recognition, simulates the processing characteristics of the human ear to speech, and can improve the effect of speech recognition. The speech features refer to the timbre, volume, tone, and other speech-related features contained in the audio data.

[0028] S20, performing action prediction processing on the speech features and the audio data through a generative adversarial network model to obtain predicted action features corresponding to the audio data.

[0029] Understandably, the adversarial network model is a deep learning model that learns through mutual game playing of at least two modules (a generation model and a discrimination model) to produce a fairly good output result. Here, the adversarial network model can be a FACIAL-GAN (facial implicit feature learning model). The FACIAL-GAN is composed of three basic parts: a time-dependent generator, a local speech generator, and a discriminator. Among them, the time-dependent generator is used to build the context relationship; the local speech generator is used to extract the feature of each frame of audio data; and the discriminator is used to judge the true or false of the generated attribute. The FACIAL-GAN can well capture implicit features, context, and personalized information. Among them, the implicit features refer to blinking information, head posture, lip movement, and the like. By learning the speech features and audio data through the FACIAL-GAN, the implicit features such as blinking information, head posture, lip movement, and the like contained in the audio data can be predicted, i.e., the action features are predicted.

[0030] S30, obtaining face video data, performing face recognition on the face video data through a face recognition model to obtain target face features corresponding to the face video data.

[0031] Understandably, the face video data refers to video data containing a face. For example, video data containing a face recorded by a user. The face recognition model is used to recognize a target face in the face video data and extract features of the target face, i.e., target face features. Specifically, the face key point detection is performed on each frame of video image contained in the face video data through the face recognition model to obtain face region video data. Further, the face feature extraction is performed on the face region video data frame by frame to obtain the target face features corresponding to the face video data. The target face features include identity features, portrait texture features, light features, and the like.

[0032] S40, embedding the predicted action features and the target face features into a video generator for rendering to generate a short video containing virtual face movement.

[0033] Understandably, the video generator can generate corresponding video data according to the input data. The predicted action features such as blinking information, head posture, lip movement contained in the predicted audio data and the target face features such as identity features, portrait texture features, light features in the face video data are merged and input into the video generator. The video generator renders the predicted action features in the input audio data and the target face features in the face video data, and generates a short video containing virtual face movement and audio data. Among them, the virtual face movement refers to the movement of the virtual face according to the predicted action features. For example, blinking, head turning, lip movement, etc. The virtual face is generated by the video generator according to the target face features. That is, the virtual face movement contains both the predicted action features of the audio data and the target face features of the face video data.

[0034] In steps S10-S40, by obtaining audio data, performing speech feature extraction on the audio data based on the mel-frequency cepstral coefficient algorithm to obtain speech features of the audio data; by the action prediction processing of the adversarial network model on the speech features and the audio data, the predicted action features corresponding to the audio data are obtained; obtaining face video data, performing face recognition on the face video data by the face recognition model to obtain target face features corresponding to the face video data; embedding the predicted action features and the target face features into the video generator for rendering to generate a short video containing virtual face movement. According to the obtained audio data and face video data, this embodiment can generate a short video. The short video contains both the virtual face movement corresponding to the audio data and the face video data, and the audio data, without stuttering, forgetting words, etc. The video effect is good, which can meet the needs of users and improve the user experience.

[0035] Optionally, in step S10, that is, the speech feature extraction on the audio data based on the mel-frequency cepstral coefficient algorithm to obtain the speech features of the audio data, comprising:

[0036] S101, preprocessing the audio data to obtain preprocessed audio data;

[0037] S102, performing Fourier transform processing on the preprocessed audio data to obtain Fourier spectrum data;

[0038] S103, filtering the Fourier spectrum data by a mel filter to obtain filtered spectrum data;

[0039] S104, performing discrete cosine transform processing on the filtered spectrum data to obtain the speech features of the audio data.

[0040] It can be understood that, in order to improve the accuracy of the obtained speech features, the audio data is preprocessed by emphasizing, that is, the high-frequency information in the audio data is strengthened to generate preprocessed audio data. Further, the preprocessed audio data is subjected to Fourier transform processing to convert the audio data from time domain information to frequency domain information to obtain Fourier spectrum data. In order to simulate the hearing mechanism of the human ear, the obtained Fourier spectrum data is subjected to filtering processing by using a Mel filter to obtain filtered spectrum data, so that the obtained speech features are more consistent with the hearing of the human ear. Since the coefficients of the filtered spectrum data obtained by the Mel filter have high correlation, the filtered spectrum data is subjected to dimension reduction processing based on discrete cosine transform, and finally the speech features of the audio data are obtained.

[0041] Preferably, before the Fourier transform processing of the preprocessed audio data, the preprocessed audio data can be subjected to framing and windowing processing to ensure that the audio data is accurately converted to frequency domain information and reduce the loss of time domain information.

[0042] Optionally, in step S30, that is, the face recognition of the face video data by the face recognition model to obtain the target face features corresponding to the face video data, comprises:

[0043] S301, face key point detection is performed on each frame of video image contained in the face video data by the face recognition model to obtain face region video data;

[0044] S302, face feature extraction is performed on the face region video data frame by frame to obtain the target face features corresponding to the face video data.

[0045] It can be understood that, by performing face key point detection on each frame of video image contained in the face video data by the face recognition model, the face region in a plurality of video image frames containing faces is obtained. Wherein, the face key point detection refers to the detection of whether the video image frame contains face key points. If the detected video image frame contains face key points, the face region in the video image frame is marked according to the detected face key points, and the marked face region is separated from the face video data. By performing face key point detection on each frame of video image, face region video data composed of a plurality of face regions can be obtained. Further, face feature extraction is performed on the face region video data frame by frame to obtain the target face features.

[0046] In this embodiment, the face region is segmented from the face video data, so that the generated face region video data only contains face regions. The extraction of the target face features based on the face region video data containing only face regions can improve the realism of the short video picture containing virtual face motion generated finally, that is, improve the display effect of the short video.

[0047] Optionally, in step S302, the face feature extraction on the face region video data frame by frame to obtain the target face feature corresponding to the face video data comprises:

[0048] S3021, face feature extraction on the face region video data frame by frame through a face recognition model to obtain the initial face feature corresponding to the face video data;

[0049] S3022, three-dimensional face reconstruction based on the initial face feature using a three-dimensional face reconstruction technology to obtain a three-dimensional face model;

[0050] S3023, obtaining the target face feature corresponding to the face video data according to the three-dimensional face model.

[0051] Understandably, the initial face feature refers to the face feature in the face region video data. The three-dimensional face reconstruction technology can reconstruct a three-dimensional model of a face from one or more images containing the face, that is, reconstruct a three-dimensional model of the face from the face features in one or more images containing the face. Here, three-dimensional face reconstruction is performed based on the initial face feature obtained from the face region video data, and then the three-dimensional face feature corresponding to the target face is obtained from the reconstructed three-dimensional face model, that is, the target face feature.

[0052] In this embodiment, since the target face feature is obtained based on the three-dimensional face model, the target face feature contains more complete face-related features than the initial face feature, improving the accuracy of face feature acquisition.

[0053] Optionally, after step S40, that is, after generating the short video containing virtual face motion, comprising:

[0054] S401, obtaining a text corresponding to the audio data and the appearance time of the text in the audio data;

[0055] S402, generating a subtitle file according to the text and the appearance time;

[0056] S403, adding the subtitle file in the short video to generate a subtitle short video.

[0057] Understandably, the text corresponding to the audio data can be obtained by voice recognition technology, that is, the audio data is converted into text to obtain the text. The appearance time refers to the time period of the voice corresponding to the text appearing in the audio data time axis, and the time period is recorded as the appearance time of the text in the audio data. The subtitle file refers to the text obtained by sorting a plurality of text according to the time sequence of the appearance time of the text. After obtaining the subtitle file, the subtitle file is input into the video generator to render the short video containing virtual face movement to generate a short video containing both subtitles and virtual face movement, that is, a subtitle short video. In this embodiment, the generated subtitle short video increases the letters and improves the video display effect. At the same time, it is more conducive to user watching and improves the user experience.

[0058] Optionally, after step S403, after adding the subtitle file in the short video to generate a subtitle short video, the method further comprises:

[0059] S4031, performing tone analysis and atmosphere analysis on the subtitle file by a content analysis model to obtain video atmosphere data;

[0060] S4032, rendering the subtitle short video according to the atmosphere data to generate an atmosphere short video; the atmosphere short video contains atmosphere special effects corresponding to the atmosphere data.

[0061] Understandably, the content analysis model refers to a neural network model for identifying and analyzing tone keywords and atmosphere keywords in the subtitle file, and further obtaining video atmosphere data according to the results of tone analysis and atmosphere analysis. The content analysis model includes tone analysis algorithm and atmosphere analysis algorithm. The video atmosphere data includes atmosphere content and atmosphere time corresponding to the atmosphere content. The atmosphere content refers to the text content in the subtitle file corresponding to the atmosphere keyword. The atmosphere data is input into the video generator, and the subtitle short video is rendered by the video generator to add atmosphere special effects corresponding to the atmosphere data to the subtitle short video. In this embodiment, the generated atmosphere short video increases the atmosphere special effects and improves the video display effect.

[0062] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0063] In an embodiment, a short video generation device is provided, which corresponds to the short video generation method in the above embodiments. As shown in Figure 3As shown, the short video generation apparatus includes a voice feature module 10, a predicted action feature module 20, a target face feature module 30, and a short video module 40. The functions of each module are described in detail as follows:

[0064] The voice feature module 10 is configured to obtain audio data, perform voice feature extraction on the audio data based on a mel-frequency cepstral coefficient algorithm, and obtain voice features of the audio data.

[0065] The predicted action feature module 20 is configured to perform action prediction processing on the voice features and the audio data through an adversarial network model, and obtain predicted action features corresponding to the audio data.

[0066] The target face feature module 30 is configured to obtain face video data, perform face recognition on the face video data through a face recognition model, and obtain target face features corresponding to the face video data.

[0067] The short video module 40 is configured to embed the predicted action features and the target face features into a video generator for rendering, and generate a short video containing virtual face movements.

[0068] Optionally, the voice feature module 10 includes:

[0069] A preprocessed audio data unit is configured to preprocess the audio data to obtain preprocessed audio data.

[0070] A Fourier spectrum data unit is configured to perform Fourier transform processing on the preprocessed audio data to obtain Fourier spectrum data.

[0071] A filtered spectrum data unit is configured to perform filtering processing on the Fourier spectrum data through a mel filter to obtain filtered spectrum data.

[0072] A voice feature unit is configured to perform discrete cosine transform processing on the filtered spectrum data to obtain voice features of the audio data.

[0073] Optionally, the target face feature module 30 includes:

[0074] A face region video data unit is configured to perform face key point detection on each frame of video image contained in the face video data through the face recognition model to obtain face region video data.

[0075] A target face feature unit is configured to extract face features from the face region video data frame by frame to obtain target face features corresponding to the face video data.

[0076] Optionally, the target face feature unit includes:

[0077] An initial face feature unit is configured to extract initial face features from the face region video data frame by frame through a face recognition model to obtain initial face features corresponding to the face video data.

[0078] A three-dimensional face model unit is configured to perform three-dimensional face reconstruction based on the initial face features through a three-dimensional face reconstruction technology to obtain a three-dimensional face model.

[0079] A target face feature unit is further configured to obtain target face features corresponding to the face video data according to the three-dimensional face model.

[0080] Optionally, after the short video module 40, the short video generation apparatus comprises:

[0081] A text information acquisition module is configured to acquire text corresponding to the audio data and an appearance time of the text in the audio data.

[0082] A subtitle file module is configured to generate a subtitle file according to the text and the appearance time.

[0083] A subtitle short video module is configured to add the subtitle file in the short video to generate a subtitle short video.

[0084] Optionally, after the subtitle short video module, the short video generation apparatus comprises:

[0085] A video atmosphere data unit is configured to perform tone analysis and atmosphere analysis on the subtitle file through a content analysis model to obtain video atmosphere data.

[0086] An atmosphere short video unit is configured to render the subtitle short video according to the atmosphere data to generate an atmosphere short video; the atmosphere short video comprises atmosphere special effects corresponding to the atmosphere data.

[0087] The specific limitations of the short video generation apparatus can refer to the limitations of the short video generation method described above, which will not be repeated here. Each module in the short video generation apparatus described above can be realized by software, hardware and a combination thereof in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0088] In one embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in Figure 4As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system and computer readable instructions. The internal memory provides an environment for the operation of the operating system and computer readable instructions in the readable storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer readable instructions are executed by the processor to implement a short video generation method. The readable storage medium provided by the embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0089] In one embodiment, a computer device is provided, comprising a memory, a processor, and computer readable instructions stored on the memory and executable on the processor, wherein the processor executes the computer readable instructions to implement the following steps:

[0090] Obtaining audio data, performing speech feature extraction on the audio data based on a mel-frequency cepstral coefficient algorithm to obtain speech features of the audio data;

[0091] Performing action prediction processing on the speech features and the audio data through an adversarial network model to obtain predicted action features corresponding to the audio data;

[0092] Obtaining face video data, performing face recognition on the face video data through a face recognition model to obtain target face features corresponding to the face video data;

[0093] Embedding the predicted action features and the target face features into a video generator for rendering to generate a short video containing virtual face movements.

[0094] In one embodiment, one or more computer readable storage media having computer readable instructions stored thereon are provided. The readable storage medium provided by the embodiment includes a non-volatile readable storage medium and a volatile readable storage medium. The computer readable instructions are stored on the readable storage medium, and the computer readable instructions are executed by one or more processors to implement the following steps:

[0095] Obtaining audio data, performing speech feature extraction on the audio data based on a mel-frequency cepstral coefficient algorithm to obtain speech features of the audio data;

[0096] Performing action prediction processing on the speech features and the audio data through an adversarial network model to obtain predicted action features corresponding to the audio data;

[0097] Obtain face video data, perform face recognition on the face video data through a face recognition model, and obtain target face features corresponding to the face video data;

[0098] Embed the predicted action features and the target face features into a video generator for rendering, and generate a short video containing virtual face movements.

[0099] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer readable instructions are executed, the processes of the above-mentioned embodiments can be included. Any reference to a memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0101] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for generating short videos, characterized in that, include: Acquire audio data, and extract speech features from the audio data based on the Mel-Cepstral Coefficient algorithm to obtain the speech features of the audio data; An adversarial network model is used to perform action prediction processing on the speech features and audio data to obtain predicted action features corresponding to the audio data. The adversarial network model refers to a face implicit feature learning model. The face implicit feature learning model consists of three basic parts: a temporal correlation generator, a local speech generator, and a discriminator. Among them, the temporal correlation generator is used to construct contextual relationships; the local speech generator is used to extract features from each frame of the audio data; and the discriminator is used to determine the authenticity of the generated attributes. Acquire facial video data, perform facial recognition on the facial video data using a facial recognition model, and obtain the target facial features corresponding to the facial video data; The predicted action features of blinking, head posture and lip movement contained in the predicted audio data are combined with the target face features of identity features, human texture features and lighting features in the face video data and input into the video generator. Then, the predicted action features of the input audio data and the target face features of the face video data are rendered to generate a short video containing virtual face movement and audio data. The step of performing face recognition on face video data using a face recognition model to obtain target face features corresponding to the face video data includes: The face recognition model is used to detect facial key points in each frame of the video image contained in the face video data to obtain face region video data. The facial features are extracted frame by frame from the video data of the facial region to obtain the target facial features corresponding to the facial video data; The step of extracting facial features frame by frame from the facial region video data to obtain the target facial features corresponding to the facial video data includes: The face features are extracted frame by frame from the face region video data using a face recognition model to obtain the initial face features corresponding to the face video data. Based on the initial facial features, a three-dimensional facial reconstruction is performed using three-dimensional facial reconstruction technology to obtain a three-dimensional facial model; Based on the three-dimensional face model, the target face features corresponding to the face video data are obtained.

2. The short video generation method as described in claim 1, characterized in that, The process of extracting speech features from the audio data using the Mel-Cepstral Coefficient algorithm to obtain the speech features of the audio data includes: The audio data is preprocessed to obtain preprocessed audio data; The preprocessed audio data is subjected to Fourier transform to obtain Fourier spectrum data; The Fourier spectrum data is filtered using a Mel filter to obtain filtered spectrum data. The filtered spectrum data is subjected to discrete cosine transform to obtain the speech features of the audio data.

3. The short video generation method as described in claim 1, characterized in that, After generating the short video containing virtual face motion and audio data, the process includes: Obtain the text corresponding to the audio data and the time when the text appears in the audio data; Generate a subtitle file based on the text and the time of appearance; Add the subtitle file to the short video to generate a subtitled short video.

4. The short video generation method as described in claim 3, characterized in that, After adding the subtitle file to the short video to generate the subtitled short video, the process includes: The subtitle file is analyzed for tone and atmosphere using a content analysis model to obtain video atmosphere data. The subtitle short video is rendered based on the ambient data to generate an ambient short video; the ambient short video includes ambient effects corresponding to the ambient data.

5. A short video generation device, characterized in that, include: The speech feature module is used to acquire audio data, extract speech features from the audio data based on the Mel-Cepstral Coefficient algorithm, and obtain the speech features of the audio data. The action feature prediction module is used to perform action prediction processing on the speech features and audio data through an adversarial network model to obtain predicted action features corresponding to the audio data. The adversarial network model refers to a face implicit feature learning model. The face implicit feature learning model consists of three basic parts: a temporal correlation generator, a local speech generator, and a discriminator. Among them, the temporal correlation generator is used to construct contextual relationships; the local speech generator is used to extract features from each frame of the audio data; and the discriminator is used to determine the authenticity of the generated attributes. The target face feature module is used to acquire face video data, perform face recognition on the face video data through a face recognition model, and obtain the target face features corresponding to the face video data. The short video module is used to merge the blink information, head posture and lip movement prediction action features contained in the predicted audio data with the target face features in the face video data, such as identity features, human texture features and lighting features, and input them into the video generator. Then, the predicted action features of the input audio data and the target face features of the face video data are rendered to generate a short video containing virtual face movement and audio data. The target facial feature module includes: The face region video data unit is used to perform face key point detection on each frame of video image contained in the face video data through the face recognition model to obtain face region video data. The target face feature unit is used to extract face features frame by frame from the face region video data to obtain the target face features corresponding to the face video data. The target facial feature unit includes: An initial face feature unit is used to extract face features frame by frame from the face region video data using a face recognition model to obtain initial face features corresponding to the face video data. The three-dimensional face model unit is used to reconstruct a three-dimensional face based on the initial face features using three-dimensional face reconstruction technology to obtain a three-dimensional face model. The target face feature unit is used to obtain the target face features corresponding to the face video data based on the three-dimensional face model.

6. The short video generation device as described in claim 5, characterized in that, The speech feature module includes: A preprocessing audio data unit is used to preprocess the audio data to obtain preprocessed audio data; The Fourier spectrum data unit is used to perform Fourier transform processing on the preprocessed audio data to obtain Fourier spectrum data. The filtered spectrum data unit is used to filter the Fourier spectrum data using a Mel filter to obtain filtered spectrum data. The speech feature unit is used to perform discrete cosine transform processing on the filtered spectrum data to obtain the speech features of the audio data.

7. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the short video generation method as described in any one of claims 1 to 4.

8. A readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the short video generation method as described in any one of claims 1 to 4 to be performed.

Images