A video generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a video generation method and device, electronic equipment and storage medium, input a target video frame containing a face image of a target object in a to-be-processed video into a pre-trained face recognition model, determine the face feature of the target object in the target video frame as a target face feature; determine the expression feature vector of the to-be-processed video based on the target face feature in each target video frame; for each to-be-processed audio, process the to-be-processed audio based on a pre-trained beat point prediction model to obtain a target beat feature vector of the to-be-processed audio; calculate the similarity of the expression feature vector and the target beat feature vector as the matching degree of the to-be-processed video and the to-be-processed audio; and perform synthesis processing on the to-be-processed video and the target audio in each to-be-processed audio to obtain a target video; the matching degree of the target audio and the to-be-processed video is maximum. Based on this, the generation efficiency of the video can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to a video generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of Internet technology, users can access more and more resources, for example, users can browse videos (such as short videos) in the client. Short videos refer to videos with a relatively short duration (for example, 2 minutes). When making short videos, users can edit the captured short videos by editing operations such as trimming, adding special effects, and adding text, and then upload the edited short videos to the client for other users to browse.

[0003] In related technologies, a user can select a piece of music as background music for a short video to be shot. During the shooting of the short video, the person adjusts his / her expression multiple times according to the beat changes of the background music, so that the expression changes of the person in the shot short video are synchronized with the beat changes of the background music.

[0004] As can be seen, in the above process, the person needs to adjust his / her expression multiple times to match the beat changes of the background music, which is tedious for the user and reduces the efficiency of video generation. SUMMARY

[0005] Embodiments of the present application aim to provide a video generation method, device, electronic equipment and storage medium to improve the efficiency of video generation. The specific technical solutions are as follows:

[0006] In a first aspect of the present application, a video generation method is first provided, which comprises:

[0007] inputting a target video frame containing a face image of a target object in a to-be-processed video into a pre-trained face recognition model to determine a face feature of the target object in the target video frame as a target face feature; wherein the face recognition model is obtained by training based on sample video frames and feature labels representing face features of sample objects in the sample video frames;

[0008] determining an expression feature vector of the to-be-processed video based on the target face features in each target video frame; wherein the elements in the expression feature vector correspond one-to-one to each video frame in the to-be-processed video; one element represents the expression feature of the target object in the corresponding video frame; and the expression feature vector represents the amplitude of the expression feature of the target object in the to-be-processed video changing over time;

[0009] For each to-be-processed audio, the to-be-processed audio is processed based on a pre-trained beat point prediction model to obtain a beat feature vector of the to-be-processed audio as a target beat feature vector; wherein elements in the target beat feature vector correspond to audio frames in the to-be-processed audio one by one, and one element represents whether the corresponding audio frame is a beat point; the target beat feature vector represents the amplitude of the beat of the to-be-processed audio changing over time; the beat point prediction model is obtained by training based on sample audios and sample beat labels representing whether each audio frame in the sample audios is a beat point;

[0010] The similarity between the expression feature vector and the target beat feature vector is calculated as the matching degree of the to-be-processed video and the to-be-processed audio;

[0011] The to-be-processed video and the target audio in each to-be-processed audio are synthesized to obtain a target video; wherein the target audio has the maximum matching degree with the to-be-processed video.

[0012] Optionally, the target facial feature includes an organ state of at least one organ in the face of the target object;

[0013] The expression feature vector of the to-be-processed video is determined based on the target facial feature in each target video frame, including:

[0014] For each target video frame, an organ state value representing an organ state of at least one organ in the face of the target object in the target video frame is obtained; based on the obtained organ state value, an expression feature value corresponding to the target video frame is determined; based on the respective expression feature values corresponding to each target video frame, the expression feature vector of the to-be-processed video is generated; wherein the expression feature values corresponding to the video frames other than the target video frames in the expression feature vector are preset values.

[0015] Optionally, the expression feature value corresponding to the target video frame is determined based on the obtained organ state value, including:

[0016] In the case where the organ state value is multiple, a weighted sum of the organ state values is calculated as the expression feature value corresponding to the target video frame.

[0017] Optionally, before the method further includes:

[0018] For each target video frame, in a preset correspondence between organ states and emotion labels, an emotion label corresponding to an organ state of at least one organ in a face of a target object in the target video frame is determined as an emotion label corresponding to the target video frame; emotion labels of the to-be-processed video are determined based on the emotion labels corresponding to the respective target video frames; for each preset audio, a similarity between an emotion label of the preset audio and an emotion label of the to-be-processed video is calculated; and to-be-processed audios are determined based on the respective preset audios, wherein the similarity between the emotion label of the to-be-processed audio and the emotion label of the to-be-processed video is greater than a similarity threshold.

[0019] Optionally, the determining the to-be-processed audios based on the respective preset audios comprises:

[0020] From the respective preset audios, a preset audio corresponding to an emotion label whose similarity with the emotion label of the to-be-processed video is greater than the similarity threshold is determined as a candidate audio; for each candidate audio, if a time length of the candidate audio is the same as a time length of the to-be-processed video, the candidate audio is determined as a to-be-processed audio; and if the time length of the candidate audio is greater than the time length of the to-be-processed video, an audio segment with the same time length as the to-be-processed video is extracted from the candidate audio to obtain a to-be-processed audio.

[0021] Optionally, before the inputting the target video frame containing the face image of the target object in the to-be-processed video into the pre-trained face recognition model to determine the face feature of the target object in the target video frame as a target face feature, the method further comprises:

[0022] performing target detection on each video frame in the to-be-processed video to obtain a video frame containing a face image of the target object as a target video frame;

[0023] or,

[0024] performing target detection on each video frame in the to-be-processed video to obtain a video frame containing a face image of the target object; and sampling a video frame containing a face image of the target object in the to-be-processed video to obtain a target video frame.

[0025] Optionally, the training process of the face recognition model comprises:

[0026] Obtain sample video frames containing facial images of the sample object, and feature labels representing the facial features of the sample object in the sample video frames; input the sample video frames into a face recognition model with an initial structure to obtain the facial features of the sample object in the sample video frames as predicted facial features; calculate a first loss function value representing the difference between the feature labels and the predicted facial features; adjust the model parameters of the face recognition model with the initial structure based on the first loss function value until a first preset convergence condition is reached to obtain a trained face recognition model.

[0027] Optionally, the training process of the beat point prediction model includes:

[0028] Acquire sample audio and sample beat labels indicating whether each audio frame in the sample audio is a beat point; process the sample audio based on the initial structure of the beat point prediction model to obtain predicted beat labels containing the probability of whether each audio frame in the sample audio is a beat point; calculate a second loss function value representing the difference between the sample beat labels and the predicted beat labels; adjust the model parameters of the initial structure of the beat point prediction model based on the second loss function value until a second preset convergence condition is reached to obtain a trained beat point prediction model.

[0029] In a second aspect of the invention, a video generation apparatus is also provided, the apparatus comprising:

[0030] The facial feature prediction module is used to input a target video frame containing a facial image of a target object from the video to be processed into a pre-trained facial recognition model to determine the facial features of the target object in the target video frame as the target facial features; wherein, the facial recognition model is trained based on sample video frames and feature labels representing the facial features of sample objects in the sample video frames;

[0031] The facial expression feature vector determination module is used to determine the facial expression feature vector of the video to be processed based on the target facial features in each target video frame; wherein, the elements in the facial expression feature vector correspond one-to-one with each video frame in the video to be processed; each element represents the facial expression feature of the target object in the corresponding video frame; the facial expression feature vector represents the magnitude of the change in the facial expression feature of the target object in the video to be processed over time;

[0032] a target beat feature vector prediction module, configured to, for each to-be-processed audio, perform processing on the to-be-processed audio based on a pre-trained beat point prediction model to obtain a beat feature vector of the to-be-processed audio as a target beat feature vector; wherein elements in the target beat feature vector correspond to audio frames in the to-be-processed audio one by one, and one element represents whether the corresponding audio frame is a beat point; the target beat feature vector represents the amplitude of the beat of the to-be-processed audio changing over time; the beat point prediction model is obtained by training based on sample audios and sample beat labels representing whether each audio frame in the sample audios is a beat point;

[0033] a matching degree calculation module, configured to calculate the similarity between the expression feature vector and the target beat feature vector as the matching degree between the to-be-processed video and the to-be-processed audio;

[0034] a video generation module, configured to perform synthesis processing on the to-be-processed video and a target audio in each to-be-processed audio to obtain a target video; wherein the target audio has the maximum matching degree with the to-be-processed video.

[0035] Optionally, the target facial feature includes an organ state of at least one organ in the face of the target object.

[0036] The expression feature vector determination module is specifically configured to, for each target video frame, obtain an organ state value representing the organ state of at least one organ in the face of the target object in the target video frame; determine an expression feature value corresponding to the target video frame based on the obtained organ state value; and generate the expression feature vector of the to-be-processed video based on the expression feature values corresponding to the target video frames respectively; wherein the expression feature values corresponding to the video frames other than the target video frames in the expression feature vector are preset values.

[0037] Optionally, the expression feature vector determination module is specifically configured to, in the case where the organ state value is multiple, calculate a weighted sum of the organ state values as the expression feature value corresponding to the target video frame.

[0038] Optionally, the apparatus further comprises:

[0039] a first emotion label determination module, configured to, before the target beat feature vector prediction module performs, for each to-be-processed audio, processing on the to-be-processed audio based on a pre-trained beat point prediction model to obtain a beat feature vector of the to-be-processed audio as a target beat feature vector, perform, for each target video frame, determination of an emotion label corresponding to the organ state of at least one organ in the face of the target object in the target video frame in a preset correspondence between organ states and emotion labels as the emotion label corresponding to the target video frame.

[0040] a second emotion label determination module configured to determine an emotion label of the to-be-processed video based on respective emotion labels of respective target video frames;

[0041] a similarity calculation module configured to calculate, for each preset audio, a similarity between an emotion label of the preset audio and an emotion label of the to-be-processed video;

[0042] a to-be-processed audio determination module configured to determine a to-be-processed audio based on the preset audios, wherein an emotion label of the to-be-processed audio has a similarity to the emotion label of the to-be-processed video that is greater than a similarity threshold.

[0043] Optionally, the to-be-processed audio determination module is specifically configured to determine, from the preset audios, a preset audio corresponding to an emotion label that has a similarity to the emotion label of the to-be-processed video greater than the similarity threshold, as a candidate audio; for each candidate audio, if a time length of the candidate audio is the same as a time length of the to-be-processed video, the candidate audio is determined as the to-be-processed audio; if the time length of the candidate audio is greater than the time length of the to-be-processed video, an audio segment with the same time length as the to-be-processed video is extracted from the candidate audio to obtain the to-be-processed audio.

[0044] Optionally, the apparatus further comprises:

[0045] a target video frame determination module configured to, before the face feature prediction module performs inputting of a target video frame containing a face image of a target object in the to-be-processed video to the pre-trained face recognition model to determine a face feature of the target object in the target video frame as a target face feature, perform target detection on each video frame in the to-be-processed video to obtain a video frame containing the face image of the target object as the target video frame;

[0046] or,

[0047] performing target detection on each video frame in the to-be-processed video to obtain a video frame containing the face image of the target object; and performing sampling on the video frame containing the face image of the target object in the to-be-processed video to obtain the target video frame.

[0048] Optionally, the apparatus further comprises:

[0049] The first training module is configured to: acquire a sample video frame containing a face image of a sample object and a feature label representing a face feature of the sample object in the sample video frame; input the sample video frame into a face recognition model of an initial structure to obtain the face feature of the sample object in the sample video frame as a predicted face feature; calculate a first loss function value representing a difference between the feature label and the predicted face feature; and adjust model parameters of the face recognition model of the initial structure based on the first loss function value until a first preset convergence condition is reached to obtain a trained face recognition model.

[0050] Optionally, the device further comprises:

[0051] The second training module is configured to: acquire a sample audio and a sample beat label representing whether each audio frame in the sample audio is a beat point; process the sample audio based on a beat point prediction model of an initial structure to obtain a predicted beat label containing a probability that each audio frame in the sample audio is a beat point; calculate a second loss function value representing a difference between the sample beat label and the predicted beat label; and adjust model parameters of the beat point prediction model of the initial structure based on the second loss function value until a second preset convergence condition is reached to obtain a trained beat point prediction model.

[0052] In another aspect of the present application, an electronic device is provided, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are in communication with each other through the communication bus.

[0053] The memory is configured to store a computer program.

[0054] The processor is configured to execute the program stored in the memory to implement the steps of the video generation method.

[0055] In another aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the video generation method.

[0056] In another aspect of the present application, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the video generation method.

[0057] The video generation method provided in the embodiment of the application comprises the following steps: inputting a target video frame containing a face image of a target object in a to-be-processed video into a pre-trained face recognition model to determine a face feature of the target object in the target video frame as a target face feature; the face recognition model is obtained by training based on a sample video frame and a feature label representing a face feature of a sample object in the sample video frame; determining an expression feature vector of the to-be-processed video based on the target face feature in each target video frame; elements in the expression feature vector correspond to each video frame in the to-be-processed video one by one; one element represents an expression feature of the target object in the corresponding video frame; the expression feature vector represents a magnitude of a change of an expression feature of the target object in the to-be-processed video over time; for each to-be-processed audio, processing the to-be-processed audio based on a pre-trained beat point prediction model to obtain a beat feature vector of the to-be-processed audio as a target beat feature vector; elements in the target beat feature vector correspond to audio frames in the to-be-processed audio one by one, and one element represents whether the corresponding audio frame is a beat point; the target beat feature vector represents a magnitude of a change of a beat of the to-be-processed audio over time; the beat point prediction model is obtained by training based on a sample audio and a sample beat label representing whether each audio frame in the sample audio is a beat point; calculating a similarity between the expression feature vector and the target beat feature vector as a matching degree of the to-be-processed video and the to-be-processed audio; and performing synthesis processing on the to-be-processed video and the target audio in each to-be-processed audio to obtain a target video, wherein the matching degree of the target audio and the to-be-processed video is the largest.

[0058] Based on the above processing, the expression feature vector of the to-be-processed video can represent the magnitude of the change of the expression feature of the target object in the to-be-processed video over time, and the target beat feature vector of the to-be-processed audio represents the magnitude of the change of the beat point of the to-be-processed audio over time. Accordingly, the similarity between the expression feature vector and the target beat feature vector can represent the similarity between the magnitude of the change of the beat of the target audio and the magnitude of the change of the expression feature of the target object in the to-be-processed video. The matching degree of the target audio and the to-be-processed video is the largest, so the magnitude of the change of the beat of the target audio is more similar to the magnitude of the change of the expression feature of the target object in the to-be-processed video. Furthermore, the expression change of the target object in the generated target video is synchronized with the beat change of the target audio, that is, the target object does not need to adjust its expression multiple times to match the beat change of the audio, thereby improving the generation efficiency of the video. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description.

[0060] Figure 1 The flowchart of the video generation method provided in the embodiment of the application;

[0061] Figure 2 A flowchart of a face recognition model training method provided in an embodiment of the present application is shown in FIG. 1.

[0062] Figure 3 A flowchart of another video generation method provided in an embodiment of the present application is shown in FIG. 2.

[0063] Figure 4 A flowchart of a beat point prediction model training method provided in an embodiment of the present application is shown in FIG. 3.

[0064] Figure 5 A flowchart of another video generation method provided in an embodiment of the present application is shown in FIG. 4.

[0065] Figure 6 A flowchart of another video generation method provided in an embodiment of the present application is shown in FIG. 5.

[0066] Figure 7 A flowchart of another video generation method provided in an embodiment of the present application is shown in FIG. 6.

[0067] Figure 8 A structural diagram of a video generation device provided in an embodiment of the present application is shown in FIG. 7.

[0068] Figure 9 A structural diagram of an electronic device provided in an embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0070] Referring to Figure 1 , Figure 1 A flowchart of a video generation method provided in an embodiment of the present application is shown in FIG. 1, which can include the following steps.

[0071] S101: input a target video frame containing a face image of a target object in a to-be-processed video to a pre-trained face recognition model, and determine a face feature of the target object in the target video frame as a target face feature.

[0072] The face recognition model is obtained based on a sample video frame and a feature label representing a face feature of a sample object in the sample video frame.

[0073] S102: determine an expression feature vector of the to-be-processed video based on the target face features in each target video frame.

[0074] The elements in the expression feature vector correspond to each video frame in the to-be-processed video one by one; one element represents the expression feature of the target object in the corresponding video frame; and the expression feature vector represents the amplitude of the expression feature of the target object in the to-be-processed video changing over time.

[0075] S103: For each to-be-processed audio, the to-be-processed audio is processed based on a pre-trained beat point prediction model to obtain a beat feature vector of the to-be-processed audio as a target beat feature vector.

[0076] The elements in the target beat feature vector correspond to the audio frames in the to-be-processed audio one by one, one element represents whether the corresponding audio frame is a beat point, and the target beat feature vector represents the amplitude of the beat of the to-be-processed audio changing over time; and the beat point prediction model is obtained based on sample audio and sample beat labels representing whether each audio frame in the sample audio is a beat point.

[0077] S104: The similarity between the expression feature vector and the target beat feature vector is calculated as the matching degree of the to-be-processed video and the to-be-processed audio.

[0078] S105: The to-be-processed video and the target audio in each to-be-processed audio are synthesized to obtain a target video.

[0079] The matching degree of the target audio and the to-be-processed video is the largest.

[0080] According to the video generation method provided in the embodiment of the present application, the expression feature vector of the to-be-processed video can represent the amplitude of the expression feature of the target object in the to-be-processed video changing over time, and the target beat feature vector of the to-be-processed audio represents the amplitude of the beat point of the to-be-processed audio changing over time. Correspondingly, the similarity between the expression feature vector and the target beat feature vector can represent the similarity between the amplitude of the beat change of the target audio and the amplitude of the expression feature change of the target object in the to-be-processed video. The matching degree of the target audio and the to-be-processed video is the largest, so the amplitude of the beat change of the target audio and the amplitude of the expression feature change of the target object in the to-be-processed video are relatively similar. Further, the expression change of the target object in the generated target video is synchronized with the beat change of the target audio, that is, the target object does not need to adjust its expression multiple times to match the beat change of the audio, thereby improving the generation efficiency of the video.

[0081] For step S101, the to-be-processed video can be a video that currently needs to add audio. For example, an audio (for example, pure music) containing only accompaniment sound can be added to the to-be-processed video. Or an audio (for example, a song) containing human voice and accompaniment sound can be added to the to-be-processed video.

[0082] The target object can be a person, animal, etc. The facial features of the target object represent the state of at least one organ in the target object's face. For example, an open mouth, closed eyes, etc.

[0083] In one implementation, for each video frame in the video to be processed, the electronic device can input the video frame into a pre-trained facial recognition model to obtain the probability that the facial features of the target object in the video frame correspond to preset organ states. Then, the electronic device can determine the organ states whose probabilities are greater than a probability threshold from the preset organ states, and use these as the facial features of the target object in the video frame.

[0084] In another implementation, among the video frames in the video to be processed, there may be video frames that do not contain facial images of the target object. In such cases, it is not necessary to determine the facial features of the target object in these video frames. Accordingly, before step S101, the electronic device may determine the target video frames in the video to be processed that contain facial images of the target object in the following manner.

[0085] Method 1,

[0086] Perform object detection on each video frame in the video to be processed to obtain the video frame containing the facial image of the target object, which is then used as the target video frame.

[0087] For each video frame of the video to be processed, the electronic device can perform target detection on the video frame to determine whether the video frame contains a facial image of a target object. If the video frame contains a facial image of a target object, the electronic device determines that the video frame is a target video frame.

[0088] Based on the above processing, the electronic device can determine the target video frame containing the facial image of the target object in the video to be processed. Then, subsequent determinations only need to identify the facial features of the target object within the target video frame, reducing computational load and further improving video generation efficiency. Furthermore, by defining each video frame containing the facial image of the target object in the video to be processed as the target video frame, the facial features of the target object in all target video frames containing facial images of the target object in the video to be processed can be determined. Based on these facial features, the expression feature vector of the video to be processed can be determined. The determined expression feature vector accurately represents the magnitude of the change in the expression features of the target object in the video over time, thus improving the accuracy of the determined expression feature vector.

[0089] Method 2,

[0090] The electronic device can perform target detection on each video frame in the to-be-processed video to obtain a video frame containing a face image of a target object; and sample the video frame containing the face image of the target object in the to-be-processed video to obtain a target video frame.

[0091] The electronic device can perform target detection on each video frame in the to-be-processed video to determine a video frame containing a face image of a target object in the to-be-processed video (which can be referred to as a candidate video frame), and then sample the determined candidate video frame at a preset sampling interval, and take the sampling result as a target video frame.

[0092] For example, there are 20 candidate video frames containing a face image of a target object in the to-be-processed video, and the electronic device can select one video frame from each pair of adjacent candidate video frames, and obtain 10 target video frames.

[0093] Based on the above processing, the facial features of the target object in adjacent candidate video frames can be the same, that is, the facial features of the target object in the last candidate video frame to the next candidate video frame have not changed, and the electronic device can select a target video frame from the candidate video frames, so that only the facial features of the target object in the target video frame need to be determined in the subsequent process, which can reduce the amount of calculation and further improve the video generation efficiency.

[0094] For each target video frame, the electronic device can input the target video frame into a pre-trained face recognition model to obtain a probability that the facial features of the target object in the target video frame are in a preset organ state. Then, the electronic device can determine an organ state corresponding to a probability greater than a probability threshold from the preset organ states, and obtain the facial features of the target object in the target video frame as a target facial feature. The probability threshold can be set by a technician according to experience, for example, the probability threshold can be 0.6, or the probability threshold can also be 0.7, but is not limited thereto.

[0095] The plurality of organs of the face can include a nose, a mouth, an eye, a brow, an ear, a cheek, and the like. The preset organ states can include a custom organ state such as a mouth opening to a circle, a mouth opening to an ellipse, an eye closing, an eye staring, a brow raising, a brow frowning, and the like.

[0096] Alternatively, the preset organ states can also include a plurality of AUs (Action units) representing the motion states of the organs of the face included in the FACS (Facial Action Coding System). For example, AU1 represents the forward segment of the brow moving downward, AU2 represents the rear segment of the brow moving upward, AU4 represents the brow being pressed low and gathered toward the root of the nose, AU5 represents the lower eyelid being lifted, and the upper eyelid being pulled upward toward the eye socket.

[0097] The face recognition model can be a CNN (Convolutional Neural Networks) model, or can also be a VGG16 (Visual Geometry Group Network-16) model.

[0098] In addition, the electronic device can also train the face recognition model of the initial structure to obtain a trained face recognition model. Correspondingly, refer to Figure 2 , Figure 2 A flowchart of a face recognition model training method provided by an embodiment of the present application can include the following steps:

[0099] S201: Obtain a sample video frame containing a face image of a sample object, and a feature label representing a face feature of the sample object in the sample video frame.

[0100] S202: Input the sample video frame into a face recognition model of an initial structure to obtain a face feature of the sample object in the sample video frame as a predicted face feature.

[0101] S203: Calculate a first loss function value representing the difference between the feature label and the predicted face feature.

[0102] S204: Adjust the model parameters of the face recognition model of the initial structure based on the first loss function value until a first preset convergence condition is reached to obtain a trained face recognition model.

[0103] The sample object and the target object belong to the same type, for example, the target object is a person, and the sample object is also a person; or the target object is an animal, and the sample object is also an animal.

[0104] The electronic device can obtain a sample video frame containing a face image of a sample object, and a feature label representing a face feature of the sample object in the sample video frame. For example, the preset organ states include organ state 1, organ state 2, and organ state 3, and the face feature of the sample object in the sample video frame includes organ state 1 and organ state 3, and the feature label representing the face feature of the sample object in the sample video frame is [1, 0, 1].

[0105] The electronic device can also input the sample video frame into a face recognition model of an initial structure to obtain a predicted face feature of the sample object in the sample video frame, and the predicted face feature is the probability of the face feature of the sample object in the sample video frame being the preset organ state. For example, the preset organ states include organ state 1, organ state 2, and organ state 3, and the probability of the face feature of the sample object in the sample video frame being the preset organ state is [0.5, 0.3, 0.8].

[0106] Further, the electronic device can calculate a first loss function value representing a difference between the feature label of the sample video frame and the predicted facial feature, and adjust the model parameter of the initial structure facial recognition model based on the first loss function value until a first preset convergence condition is reached, to obtain a trained facial recognition model.

[0107] The first preset convergence condition can be that the number of training reaches a preset number, or the first preset convergence condition can also be that the loss function value calculated for a continuous preset number of times is less than a preset loss threshold.

[0108] For step S102, the elements in the expression feature vector of the to-be-processed video correspond one-to-one to each video frame in the to-be-processed video, and one element represents the expression feature of the target object in the corresponding video frame.

[0109] For each target video frame, the target facial feature of the target object in the target video frame includes an organ state of at least one organ in the face of the target object, and different organ states can be represented by different organ state values. The electronic device can determine the expression feature vector of the to-be-processed video according to the organ state value representing the organ state of at least one organ in the face of the target object in the target video frame.

[0110] For each organ state, the organ state value of the organ state can represent the degree of exaggeration of the organ state. For example, the organ state value of the mouth opening to a circle is 1, indicating that the degree of exaggeration of the mouth opening to a circle is large, the organ state value of the mouth opening to an ellipse is 0.6, indicating that the degree of exaggeration of the mouth opening to an ellipse is small, the organ state value of the eye is 0.9, indicating that the degree of exaggeration of the eye is also large, and the organ state value of the eye is 0.4, indicating that the degree of exaggeration of the eye is small.

[0111] In some embodiments, the target facial feature includes an organ state of at least one organ in the face of the target object, and accordingly, on the basis of Figure 1 , see Figure 3 , step S102 can include the following steps:

[0112] S1021: For each target video frame, obtain an organ state value representing an organ state of at least one organ in the face of the target object in the target video frame.

[0113] S1022: Based on the obtained organ state value, determine the expression feature value corresponding to the target video frame.

[0114] S1023: Based on the respective expression feature values corresponding to each target video frame, generate an expression feature vector of the to-be-processed video.

[0115] The expression feature value corresponding to the other video frame in the expression feature vector except the target video frame is a preset value.

[0116] For each target video frame, the electronic device obtains an organ state value representing an organ state of at least one organ in the face of the target object in the target video frame, and then calculates the expression feature value corresponding to the target video frame based on the obtained organ state value.

[0117] In an implementation, for each target video frame, if the target facial feature in the target video frame includes an organ state of an organ in the face of the target object, the electronic device can directly determine the organ state value of the organ state as the expression feature value corresponding to the target video frame.

[0118] In another implementation, step S1022 can include the following steps: in the case where there are multiple organ state values, calculating a weighted sum of the organ state values as the expression feature value corresponding to the target video frame.

[0119] For each target video frame, if the target facial feature in the target video frame includes organ states of multiple organs in the face of the target object, the electronic device obtains an organ state value of each organ to obtain multiple organ state values, and then the electronic device can calculate a weighted sum of the multiple organs to obtain the expression feature value corresponding to the target video frame.

[0120] Since the video frames (which can be referred to as non-target video frames) in the to-be-processed video except the target video frames do not contain the face image of the target object, the electronic device can directly determine the expression feature value corresponding to the non-target video frame as a preset value (for example, 0) for indicating that the video frame does not contain the face image of the target object.

[0121] Further, an expression feature vector representing the to-be-processed video can be obtained. Since the elements in the expression feature vector correspond one-to-one to the video frames in the to-be-processed video, one element represents the expression feature of the target object in the corresponding video frame, the expression feature vector can represent the amplitude of the expression feature of the target object in the to-be-processed video changing over time, for example, indicating that the expression of the target object in the to-be-processed video is becoming more and more exaggerated.

[0122] For step S103, the to-be-processed audio can only contain accompaniment sound, for example, pure music, or the to-be-processed audio can also contain human voice and accompaniment sound, for example, songs, etc. The electronic device can obtain multiple to-be-processed audios, and determine the target audio added to the to-be-processed video from the multiple to-be-processed audios according to the method provided in the embodiments of the present application.

[0123] The beat is a basic unit of time of accompaniment sound in the audio to be processed, and the beat point is an audio frame in which the beat of the audio to be processed is converted from a previous beat to a next beat.

[0124] An element in a target beat feature vector of an audio to be processed corresponds to an audio frame included in the audio to be processed, and the element represents a probability that the corresponding audio frame is a beat point. The beat feature vector of the audio to be processed can represent a magnitude of a change in the beat of the audio to be processed over time, for example, that the beat of the audio to be processed is getting faster.

[0125] For each audio to be processed, the electronic device can perform STFT (Short-Time Fourier Transform) processing on the audio to be processed to obtain a spectrogram of the audio to be processed. Then, the spectrogram of the audio to be processed is input to a pre-trained beat point prediction model to obtain a probability that each audio frame in the audio to be processed is a beat point, that is, to obtain a beat feature vector of the audio to be processed.

[0126] The beat point prediction model can be a YOLOv3 (You only look once-v3, a deep learning-based end-to-end target detection algorithm) model. Alternatively, the beat point prediction model can also be an R-CNN (Region Convolutional Neural Networks) model.

[0127] In addition, the electronic device can also train the beat point prediction model of the initial structure to obtain a trained beat point prediction model. Correspondingly, see Figure 4 , Figure 4 A flowchart of a beat point prediction model training method provided by an embodiment of the present application can include the following steps:

[0128] S401: Obtain a sample audio and sample beat labels indicating whether each audio frame in the sample audio is a beat point.

[0129] S402: Process the sample audio based on the beat point prediction model of the initial structure to obtain predicted beat labels indicating probabilities that each audio frame in the sample audio is a beat point.

[0130] S403: Calculate a second loss function value representing a difference between the sample beat labels and the predicted beat labels.

[0131] S404: Adjust model parameters of the beat point prediction model of the initial structure based on the second loss function value until a second preset convergence condition is reached to obtain a trained beat point prediction model.

[0132] The electronic device can obtain sample audio and sample beat labels indicating whether each audio frame in the sample audio is a beat point. For example, the sample audio includes audio frame 1, audio frame 2, audio frame 3, audio frame 4, and audio frame 5. Among them, audio frame 3 and audio frame 5 are beat points, and the sample beat labels are [0, 0, 1, 0, 1].

[0133] The electronic device can perform STFT processing on the sample audio to obtain a spectrogram of the sample audio. Then, the spectrogram of the sample audio is input into the beat point prediction model of the initial structure to obtain the probability that each audio frame in the sample audio is a beat point, that is, to obtain the predicted beat labels of the sample audio. For example, the sample audio includes audio frame 1, audio frame 2, audio frame 3, audio frame 4, and audio frame 5. The probability that each audio frame in the sample audio is a beat point (that is, the predicted beat labels) is [0.1, 0.3, 0.7, 0.2, 0.8].

[0134] Then, the electronic device can calculate a second loss function value representing the difference between the sample beat labels and the predicted beat labels. Further, the electronic device can adjust the model parameters of the beat point prediction model of the initial structure based on the second loss function value until a second preset convergence condition is reached to obtain a trained beat point prediction model.

[0135] The second preset convergence condition can be the same as the first preset convergence condition, or the second preset convergence condition can be different from the first preset convergence condition.

[0136] In an embodiment of the present application, based on Figure 3 , see Figure 5 Before step S103, the method can further include the following steps:

[0137] S106: For each target video frame, determine the emotion label corresponding to the organ state of at least one organ in the face of the target object in the target video frame in the preset correspondence between organ states and emotion labels as the emotion label corresponding to the target video frame.

[0138] S107: Determine the emotion label of the to-be-processed video based on the respective corresponding emotion labels of each target video frame.

[0139] S108: For each preset audio, calculate the similarity between the emotion label of the preset audio and the emotion label of the to-be-processed video.

[0140] S109: Determine the to-be-processed audio based on each preset audio.

[0141] Among them, the similarity between the emotion label of the to-be-processed audio and the emotion label of the to-be-processed video is greater than a similarity threshold.

[0142] The preset correspondence between organ states and emotion labels records emotion labels corresponding to each organ state. For example, AU1 corresponds to sadness and depression, AU2 corresponds to surprise and happiness, AU4 corresponds to anger and sadness, and AU5 corresponds to surprise.

[0143] For each target video frame, the electronic device can find, in the preset correspondence between organ states and emotion labels, an emotion label corresponding to an organ state of at least one organ included in a target facial feature in the target video frame, to obtain an emotion label corresponding to the target video frame. For example, if a target facial feature in a target video frame includes AU1, it can be determined that the emotion label corresponding to the target video frame is sadness and depression. If a target facial feature in a target video frame includes AU4, it can be determined that the emotion label corresponding to the target video frame is anger and sadness. If a target facial feature in a target video frame includes AU5, it can be determined that the emotion label corresponding to the target video frame is surprise.

[0144] In an implementation, for each emotion label, the electronic device can determine a number of target video frames corresponding to the emotion label (which can be referred to as a target number). Further, the electronic device can determine an emotion label with a maximum corresponding target number as an emotion label of the video to be processed.

[0145] In another implementation, the electronic device can calculate a ratio of the target number corresponding to the emotion label to a number of target video frames included in the video to be processed. If the calculated ratio reaches a preset ratio, the electronic device can determine that the emotion label is an emotion label of the video to be processed. The preset ratio can be set by a technician according to experience, for example, the preset ratio can be 0.5, or the preset ratio can also be 0.6, but is not limited thereto.

[0146] For example, the preset ratio is 0.5, and the target video frames included in the video to be processed are: video 1, video frame 2, and video frame 3. Video frame 1, video frame 1 corresponds to the emotion label of sadness and anger, video frame 2 corresponds to the emotion label of sadness and surprise, and video frame 3 corresponds to the emotion label of surprise. The ratio of the target number corresponding to sadness to the number of target video frames included in the video to be processed is 0.67, and the ratio of the target number corresponding to surprise to the number of target video frames included in the video to be processed is also 0.67. The electronic device can determine that the emotion label of the video to be processed is sadness and surprise.

[0147] The preset audio can include all audio that the electronic device can provide to the user. For each preset audio, an emotion label of the preset audio can be preset in advance.

[0148] In an implementation, for each preset audio, the electronic device can determine the number of emotion labels of the preset audio that are the same as the emotion label of the video to be processed as the similarity between the emotion label of the preset audio and the emotion label of the video to be processed.

[0149] In another implementation, for each preset audio, the electronic device can encode the emotion label of the preset audio in a preset encoding manner to obtain a feature vector representing the emotion label of the preset audio. The electronic device can also encode the emotion label of the video to be processed in the same encoding manner to obtain a feature vector representing the emotion label of the video to be processed. Then, the similarity between the feature vector representing the emotion label of the preset audio and the feature vector representing the emotion label of the video to be processed is calculated as the similarity between the emotion label of the preset audio and the emotion label of the video to be processed.

[0150] The preset encoding manner can be Onehot (one-hot) encoding, or the preset encoding manner can also be embedding (word embedding) encoding.

[0151] In an implementation, the electronic device can directly determine, from the preset audios, a preset audio whose corresponding emotion label has a similarity to the emotion label of the video to be processed that is greater than a preset similarity threshold, to obtain the audio to be processed. The similarity threshold can be set by a technician according to experience, and the preset similarity threshold can be 0.7 or 0.8, but is not limited thereto.

[0152] Based on the above processing, the audio to be processed whose emotion label has a greater similarity to the emotion label of the video to be processed can be determined, and then the emotion label of the target audio determined from the audio to be processed also has a greater similarity to the emotion label of the video to be processed, and thus the emotion label of the audio in the target video obtained can be matched with the emotion label of the character.

[0153] In another implementation, based on Figure 5 , see Figure 6 , step S109 can include the following steps:

[0154] S1091: From the preset audios, determine a preset audio whose corresponding emotion label has a similarity to the emotion label of the video to be processed that is greater than a similarity threshold, as an alternative audio.

[0155] S1092: For each alternative audio, if the duration of the alternative audio is the same as the duration of the video to be processed, determine the alternative audio as the audio to be processed.

[0156] S1093: If the length of the candidate audio is greater than the length of the to-be-processed video, an audio segment with the same length as the to-be-processed video is extracted from the candidate audio to obtain the to-be-processed audio.

[0157] The electronic device can determine, from the preset audios, a preset audio corresponding to an emotion label similar to the emotion label of the to-be-processed video by more than a similarity threshold, to obtain a candidate audio. Then, for each candidate audio, the electronic device can determine whether the length of the candidate audio is the same as the length of the to-be-processed video. If the length of the candidate audio is the same as the length of the to-be-processed video, the electronic device can directly determine the candidate audio as the to-be-processed audio.

[0158] If the length of the candidate audio is greater than the length of the to-be-processed video, the electronic device can extract an audio segment with the same length as the to-be-processed video from the candidate audio to obtain the to-be-processed audio.

[0159] For example, if the length of the candidate audio is an integer multiple of the length of the to-be-processed video, the electronic device can directly divide and cut the candidate audio according to the length of the to-be-processed video to obtain a plurality of audio segments with the same length as the to-be-processed video as the to-be-processed audio. If the length of the candidate audio is not an integer multiple of the length of the to-be-processed video, the electronic device can cut an audio segment with the same length as the to-be-processed video from the candidate audio starting from the beginning of the candidate audio according to the length of the to-be-processed video to obtain the to-be-processed audio.

[0160] For step S104, for each to-be-processed audio, the electronic device can calculate the similarity between the expression feature vector of the to-be-processed video and the target beat feature vector of the to-be-processed audio based on a preset similarity algorithm as the matching degree between the to-be-processed video and the to-be-processed audio. The matching degree represents the similarity between the amplitude of the beat change of the target audio and the amplitude of the expression feature change of the target object in the to-be-processed video.

[0161] The preset similarity algorithm can be a cosine similarity algorithm, or the preset similarity algorithm can be an Euclidean distance algorithm, etc.

[0162] For step S105, the electronic device can determine, from the to-be-processed audios, a target audio with the greatest matching degree with the to-be-processed video, so that the amplitude of the beat change of the target audio is more similar to the amplitude of the expression feature change of the target object in the to-be-processed video. Further, the electronic device can perform a synthesis process on the to-be-processed video and the target audio to obtain a target video, so that the expression change of the target object in the target video is synchronized with the beat change of the target audio.

[0163] Referring to Figure 7 , Figure 7 A flowchart of another video generation method provided by an embodiment of the present application.

[0164] The to-be-processed video is a short video, and the target object is a person. The electronic device performs face detection on each video frame in the to-be-processed video to obtain a target video frame containing a face image in the to-be-processed video. For each target video frame, the electronic device performs micro-expression recognition on the target video frame to obtain a target facial feature of the person in the target video frame. That is, the electronic device inputs each target video frame into a pre-trained face recognition model to obtain a target facial feature of the target object in the target video frame. The electronic device determines an expression feature vector of the to-be-processed video based on the target facial feature in each target video frame.

[0165] The music library contains a plurality of to-be-processed audios, and the electronic device can perform beat detection on the to-be-processed audios to obtain a target beat feature vector of the to-be-processed audios. That is, for each to-be-processed audio, the electronic device can process the to-be-processed audio based on a pre-trained beat point prediction model to obtain a target beat feature vector of the to-be-processed audio. Further, the electronic device can perform sequence matching on the to-be-processed video and the to-be-processed audios to obtain a target audio. That is, for each to-be-processed audio, the electronic device can calculate a similarity between the target beat feature vector of the to-be-processed audio and the expression feature vector of the to-be-processed video to obtain a matching degree of the to-be-processed audio and the to-be-processed video. The electronic device can determine, from the to-be-processed audios, a target audio having the largest matching degree with the to-be-processed video.

[0166] Further, the electronic device can perform automatic music matching, that is, the electronic device can perform synthesis processing on the to-be-processed video and the target audio to obtain a target video.

[0167] Based on the video generation method provided in the embodiment of the present application, the expression feature vector of the to-be-processed video can represent the amplitude of the expression feature of the target object in the to-be-processed video changing over time, and the target beat feature vector of the to-be-processed audio represents the amplitude of the beat point of the to-be-processed audio changing over time. Correspondingly, the similarity between the expression feature vector and the target beat feature vector can represent the similarity between the amplitude of the beat change of the target audio and the amplitude of the expression feature change of the target object in the to-be-processed video. The matching degree of the target audio and the to-be-processed video is the largest, and the amplitude of the beat change of the target audio is more similar to the amplitude of the expression feature change of the target object in the to-be-processed video. Further, the expression change of the target object in the generated target video is synchronized with the beat change of the target audio, that is, the target object does not need to adjust its expression multiple times to match the beat change of the audio, and thus the generation efficiency of the video can be improved.

[0168] With Figure 1 corresponding to the method embodiment of Figure 8 , Figure 8A structural diagram of a video generation apparatus provided by an embodiment of the present application, the apparatus comprising:

[0169] A face feature prediction module 801 is configured to input a target video frame containing a face image of a target object in a to-be-processed video into a pre-trained face recognition model, determine a face feature of the target object in the target video frame as a target face feature; wherein the face recognition model is obtained by training based on sample video frames and feature labels representing face features of sample objects in the sample video frames;

[0170] An expression feature vector determination module 802 is configured to determine an expression feature vector of the to-be-processed video based on the target face features in each target video frame; wherein elements in the expression feature vector correspond to each video frame in the to-be-processed video one by one; one element represents an expression feature of the target object in the corresponding video frame; and the expression feature vector represents a magnitude of a change in the expression feature of the target object in the to-be-processed video over time;

[0171] A target beat feature vector prediction module 803 is configured to, for each to-be-processed audio, process the to-be-processed audio based on a pre-trained beat point prediction model to obtain a beat feature vector of the to-be-processed audio as a target beat feature vector; wherein elements in the target beat feature vector correspond to audio frames in the to-be-processed audio one by one, and one element represents whether the corresponding audio frame is a beat point; the target beat feature vector represents a magnitude of a change in the beat of the to-be-processed audio over time; and the beat point prediction model is obtained by training based on sample audios and sample beat labels representing whether each audio frame in the sample audios is a beat point;

[0172] A matching degree calculation module 804 is configured to calculate a similarity between the expression feature vector and the target beat feature vector as a matching degree between the to-be-processed video and the to-be-processed audio;

[0173] A video generation module 805 is configured to perform a synthesis process on the to-be-processed video and a target audio in each to-be-processed audio to obtain a target video; wherein the target audio has the maximum matching degree with the to-be-processed video.

[0174] Optionally, the target face feature includes an organ state of at least one organ in the face of the target object;

[0175] The expression feature vector determination module 802 is specifically configured to, for each target video frame, obtain an organ state value representing an organ state of at least one organ in the face of the target object in the target video frame;

[0176] Determine the expression feature value corresponding to the target video frame based on the obtained organ state value;

[0177] Generate an expression feature vector of the to-be-processed video based on the expression feature values corresponding to each of the target video frames; wherein the expression feature values corresponding to the video frames other than the target video frames in the expression feature vector are preset values.

[0178] Optionally, the expression feature vector determination module 802 is specifically configured to, in the case where the organ state value is multiple, calculate a weighted sum of the organ state values as the expression feature value corresponding to the target video frame.

[0179] Optionally, the apparatus further comprises:

[0180] The first emotion label determination module is configured to, before the target beat feature vector prediction module 803 performs processing on each to-be-processed audio based on the pre-trained beat point prediction model to obtain the beat feature vector of the to-be-processed audio as the target beat feature vector, perform, for each target video frame, determination of an emotion label corresponding to the organ state of at least one organ in the face of the target object in the target video frame in a preset correspondence between organ states and emotion labels, as the emotion label corresponding to the target video frame.

[0181] The second emotion label determination module is configured to determine an emotion label of the to-be-processed video based on the emotion labels corresponding to each of the target video frames.

[0182] The similarity calculation module is configured to calculate, for each preset audio, a similarity between the emotion label of the preset audio and the emotion label of the to-be-processed video.

[0183] The to-be-processed audio determination module is configured to determine a to-be-processed audio based on the preset audios, wherein the similarity between the emotion label of the to-be-processed audio and the emotion label of the to-be-processed video is greater than a similarity threshold.

[0184] Optionally, the to-be-processed audio determination module is specifically configured to determine, from the preset audios, a preset audio corresponding to an emotion label whose similarity with the emotion label of the to-be-processed video is greater than the similarity threshold, as a candidate audio.

[0185] For each candidate audio, if the duration of the candidate audio is the same as the duration of the to-be-processed video, the candidate audio is determined as the to-be-processed audio; if the duration of the candidate audio is greater than the duration of the to-be-processed video, an audio segment with the same duration as the to-be-processed video is extracted from the candidate audio to obtain the to-be-processed audio.

[0186] Optionally, the apparatus further comprises:

[0187] a target video frame determination module, configured to, before the face feature prediction module 801 performs inputting of a target video frame containing a face image of a target object in a to-be-processed video into a pre-trained face recognition model to determine a face feature of the target object in the target video frame as a target face feature, perform target detection on each video frame in the to-be-processed video to obtain a video frame containing the face image of the target object as the target video frame;

[0188] Alternatively,

[0189] performing target detection on each video frame in the to-be-processed video to obtain a video frame containing the face image of the target object; and sampling the video frame containing the face image of the target object in the to-be-processed video to obtain the target video frame.

[0190] Optionally, the apparatus further includes:

[0191] a first training module, configured to obtain a sample video frame containing a face image of a sample object and a feature label representing a face feature of the sample object in the sample video frame;

[0192] inputting the sample video frame into an initial structure of a face recognition model to obtain the face feature of the sample object in the sample video frame as a predicted face feature;

[0193] calculating a first loss function value representing a difference between the feature label and the predicted face feature;

[0194] adjusting model parameters of the initial structure of the face recognition model based on the first loss function value until a first preset convergence condition is reached to obtain a trained face recognition model.

[0195] Optionally, the apparatus further includes:

[0196] a second training module, configured to obtain a sample audio and a sample beat label representing whether each audio frame in the sample audio is a beat point;

[0197] processing the sample audio based on an initial structure of a beat point prediction model to obtain a predicted beat label containing a probability that each audio frame in the sample audio is a beat point;

[0198] calculating a second loss function value representing a difference between the sample beat label and the predicted beat label;

[0199] adjusting model parameters of the initial structure of the beat point prediction model based on the second loss function value until a second preset convergence condition is reached to obtain a trained beat point prediction model.

[0200] Based on the video generation apparatus provided in the embodiments of the present application, the expression feature vector of the to-be-processed video can represent the amplitude of the expression feature of the target object in the to-be-processed video changing over time, and the target beat feature vector of the to-be-processed audio represents the amplitude of the beat point of the to-be-processed audio changing over time. Accordingly, the similarity between the expression feature vector and the target beat feature vector can represent the similarity between the amplitude of the beat change of the target audio and the amplitude of the expression feature change of the target object in the to-be-processed video. When the matching degree between the target audio and the to-be-processed video is the largest, the amplitude of the beat change of the target audio is more similar to the amplitude of the expression feature change of the target object in the to-be-processed video. Further, the expression change of the target object in the generated target video is synchronized with the beat change of the target audio, that is, the target object does not need to adjust its expression multiple times to match the beat change of the audio, and thus the generation efficiency of the video can be improved.

[0201] The embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, comprising a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902 and the memory 903 complete mutual communication through the communication bus 904, Figure 9

[0202] The memory 903 is used for storing a computer program.

[0203] The processor 901 is used for executing the program stored in the memory 903, and realizes the following steps:

[0204] The target video frame containing the facial image of the target object in the to-be-processed video is input into the pre-trained face recognition model, and the facial feature of the target object in the target video frame is determined as the target facial feature; wherein the face recognition model is obtained by training based on a sample video frame and a feature label representing the facial feature of a sample object in the sample video frame;

[0205] The expression feature vector of the to-be-processed video is determined based on the target facial feature in each target video frame; wherein the elements in the expression feature vector correspond to each video frame in the to-be-processed video one by one; one element represents the expression feature of the target object in the corresponding video frame; and the expression feature vector represents the amplitude of the expression feature of the target object in the to-be-processed video changing over time;

[0206] ​For each to-be-processed audio, a beat point prediction model that is pre-trained is used to process the to-be-processed audio, to obtain a beat feature vector of the to-be-processed audio as a target beat feature vector; wherein the elements in the target beat feature vector correspond to the audio frames in the to-be-processed audio one by one, and one element represents whether the corresponding audio frame is a beat point; the target beat feature vector represents the amplitude of the beat of the to-be-processed audio changing over time; the beat point prediction model is obtained by training based on sample audios and sample beat labels representing whether each audio frame in the sample audios is a beat point;

[0207] The similarity between the expression feature vector and the target beat feature vector is calculated as the matching degree between the to-be-processed video and the to-be-processed audio.

[0208] The to-be-processed video and the target audio in each to-be-processed audio are synthesized to obtain a target video; wherein the target audio has the maximum matching degree with the to-be-processed video.

[0209] The communication bus mentioned in the above electronic device can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0210] The communication interface is used for communication between the above electronic device and other devices.

[0211] The memory can include a random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0212] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0213] In yet another embodiment provided by the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the video generation method according to any one of the above embodiments.

[0214] In yet another embodiment provided by the present application, a computer program product is provided, and the computer program product includes instructions. When the computer program product is executed on a computer, the computer is caused to perform the video generation method according to any one of the above embodiments.

[0215] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the implementation produces the processes or functions according to the embodiments of the present application, entirely or partially. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), a semiconductor medium (such as a solid state disk (SSD)) or the like.

[0216] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0217] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0218] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A video generation method, characterized in that, The method includes: The target video frame containing the facial image of the target object in the video to be processed is input into a pre-trained facial recognition model to determine the facial features of the target object in the target video frame, which are then used as the target facial features; wherein, the facial recognition model is trained based on sample video frames and feature labels representing the facial features of sample objects in the sample video frames; Based on the facial features of the target in each target video frame, an expression feature vector is determined for the video to be processed; wherein, the elements in the expression feature vector correspond one-to-one with each video frame in the video to be processed; each element represents the expression feature of the target object in the corresponding video frame; the expression feature vector represents the magnitude of the change in the expression feature of the target object in the video to be processed over time. For each audio file to be processed, a pre-trained beat prediction model is used to process the audio file to obtain a beat feature vector, which serves as the target beat feature vector. Each element in the target beat feature vector corresponds one-to-one with an audio frame in the audio file to be processed, with each element indicating whether the corresponding audio frame is a beat point. The target beat feature vector represents the magnitude of the beat variation of the audio file over time. The beat prediction model is trained based on sample audio files and sample beat labels indicating whether each audio frame in the sample audio file is a beat point. Calculate the similarity between the facial expression feature vector and the target beat feature vector, and use it as the matching degree between the video to be processed and the audio to be processed; The target video is obtained by combining the video to be processed with the target audio from each audio source; wherein the target audio has the highest matching degree with the video to be processed.

2. The method according to claim 1, characterized in that, The target facial features include the organ state of at least one organ in the face of the target object; The step of determining the expression feature vector of the video to be processed based on the target facial features in each target video frame includes: For each target video frame, obtain an organ state value representing the organ state of at least one organ in the face of the target object in that target video frame; Based on the obtained organ state values, the facial expression feature values ​​corresponding to the target video frame are determined; Based on the facial expression feature values ​​corresponding to each target video frame, an facial expression feature vector of the video to be processed is generated.

3. The method according to claim 2, characterized in that, The step of determining the facial expression feature value corresponding to the target video frame based on the acquired organ state value includes: When there are multiple organ state values, the weighted sum of each organ state value is calculated and used as the facial expression feature value corresponding to the target video frame.

4. The method according to claim 2, characterized in that, Before processing each audio file based on a pre-trained beat prediction model to obtain its beat feature vector as the target beat feature vector, the method further includes: For each target video frame, in the preset correspondence between organ state and emotion label, the emotion label corresponding to the organ state of at least one organ in the face of the target object in the target video frame is determined as the emotion label corresponding to the target video frame. Based on the emotion tags corresponding to each target video frame, the emotion tag of the video to be processed is determined. For each preset audio, calculate the similarity between the emotion tag of the preset audio and the emotion tag of the video to be processed; The audio to be processed is determined based on each preset audio, wherein the similarity between the emotion tag of the audio to be processed and the emotion tag of the video to be processed is greater than a similarity threshold.

5. The method according to claim 4, characterized in that, The step of determining the audio to be processed based on each preset audio includes: From each preset audio, determine the preset audio whose corresponding emotion tag has a similarity to the emotion tag of the video to be processed greater than the similarity threshold, and use it as candidate audio; For each candidate audio, if the duration of the candidate audio is the same as the duration of the video to be processed, the candidate audio is determined to be the audio to be processed; if the duration of the candidate audio is longer than the duration of the video to be processed, an audio segment with the same duration as the video to be processed is extracted from the candidate audio to obtain the audio to be processed.

6. The method according to claim 1, characterized in that, Before inputting a target video frame containing a facial image of the target object into a pre-trained facial recognition model to determine the facial features of the target object in the target video frame as the target facial features, the method further includes: Target detection is performed on each video frame in the video to be processed to obtain a video frame containing the facial image of the target object, which is then used as the target video frame. or, Target detection is performed on each video frame in the video to be processed to obtain video frames containing the facial images of the target object; the video frames containing the facial images of the target object in the video to be processed are sampled to obtain target video frames.

7. The method according to claim 1, characterized in that, The training process of the facial recognition model includes: Obtain sample video frames containing facial images of the sample object, and feature labels representing the facial features of the sample object in the sample video frames; The sample video frames are input into the face recognition model of the initial structure to obtain the facial features of the sample object in the sample video frames, which are used as predicted facial features. Calculate the value of a first loss function representing the difference between the feature label and the predicted facial features; The model parameters of the face recognition model of the initial structure are adjusted based on the first loss function value until the first preset convergence condition is met, thereby obtaining a trained face recognition model.

8. The method according to claim 1, characterized in that, The training process of the beat point prediction model includes: Obtain sample audio and sample beat labels indicating whether each audio frame in the sample audio is a beat point; The beat prediction model based on the initial structure processes the sample audio to obtain a predicted beat label containing the probability of whether each audio frame in the sample audio is a beat point. Calculate the value of a second loss function representing the difference between the sample beat label and the predicted beat label; The model parameters of the beat point prediction model of the initial structure are adjusted based on the second loss function value until the second preset convergence condition is met, thereby obtaining the trained beat point prediction model.

9. A video generation apparatus, characterized in that, The device includes: The facial feature prediction module is used to input a target video frame containing a facial image of a target object in the video to be processed into a pre-trained facial recognition model to determine the facial features of the target object in the target video frame as the target facial features; wherein, the facial recognition model is trained based on sample video frames and feature labels representing the facial features of sample objects in the sample video frames; The facial expression feature vector determination module is used to determine the facial expression feature vector of the video to be processed based on the target facial features in each target video frame; wherein, the elements in the facial expression feature vector correspond one-to-one with each video frame in the video to be processed; each element represents the facial expression feature of the target object in the corresponding video frame; the facial expression feature vector represents the magnitude of the change in the facial expression feature of the target object in the video to be processed over time; The target beat feature vector prediction module is used to process each audio file based on a pre-trained beat prediction model to obtain a beat feature vector for that audio file, which serves as the target beat feature vector. Each element in the target beat feature vector corresponds one-to-one with an audio frame in the audio file, with each element indicating whether the corresponding audio frame is a beat point. The target beat feature vector represents the amplitude of the beat change of the audio file over time. The beat prediction model is trained based on sample audio files and sample beat labels indicating whether each audio frame in the sample audio file is a beat point. The matching degree calculation module is used to calculate the similarity between the facial expression feature vector and the target beat feature vector, which is used as the matching degree between the video to be processed and the audio to be processed. The video generation module is used to synthesize the video to be processed and the target audio in each audio to be processed to obtain the target video; wherein the target audio has the highest matching degree with the video to be processed.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-8.

Images