Video generation method and apparatus, electronic device, and medium

Abstract

This application discloses a video generation method, apparatus, electronic device, and medium, belonging to the field of artificial intelligence technology. The video generation method includes: extracting behavioral descriptive words and visual descriptive words from a first text; determining target video segments matching the behavioral descriptive words from N first videos, and determining target video frames matching the visual descriptive words from the N first videos; generating a target video based on the target video segments and the target video frames; wherein the N first videos are N videos in a video library similar to the first text; and N is an integer greater than 1.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a video generation method, apparatus, electronic device, and medium. Background Technology

[0002] With the rapid development of deep neural networks, algorithms related to video generation are becoming increasingly diverse, making it possible to directly generate videos that match the meaning of text descriptions.

[0003] In related technologies, the process of generating videos based on text typically involves inputting text into a network model, first using the text modality of the network model to extract the text features corresponding to the text, and then directly generating a video based on the text features using the video modality of the network model.

[0004] However, since the above scheme generates videos directly based on the extracted text features through a network model, it may fail to accurately extract features when there is an information gap between the text modality and the video modality of the network model, resulting in a mismatch between the generated video and the text description. Summary of the Invention

[0005] The purpose of this application is to provide a video generation method, apparatus, electronic device, and medium that can solve the problem of mismatch between the generated video and the text description.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a video generation method, the method comprising: extracting behavioral descriptors and visual descriptors from a first text; determining a target video segment matching the behavioral descriptors from N first videos, and determining a target video frame matching the visual descriptors from the N first videos; generating a target video based on the target video segment and the target video frame; wherein the N first videos are N videos in a video library that are similar to the first text; and N is an integer greater than 1.

[0008] Secondly, embodiments of this application provide a video generation apparatus, comprising: an extraction module, a determination module, and a generation module; wherein the extraction module is configured to extract behavioral descriptors and visual descriptors from a first text; the determination module is configured to determine, from N first videos, a target video segment matching the behavioral descriptors, and from the N first videos, a target video frame matching the visual descriptors; and the generation module is configured to generate a target video based on the target video segment and the target video frame determined by the determination module; wherein the N first videos are N videos in a video library that are similar to the first text; and N is an integer greater than 1.

[0009] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0010] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0011] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0012] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0013] In this embodiment, when generating video based on text, the electronic device can first extract behavioral descriptors and visual descriptors from the first text. Then, it determines target video segments matching the behavioral descriptors and target video frames matching the visual descriptors from N first videos similar to the first text in a video library. Finally, it generates a target video based on the target video segments and the target video frames; where N is an integer greater than 1. Thus, by using behavioral descriptors in the text to describe the subject's behavior, a video segment matching the subject's behavior is found in the first video; and by using visual descriptors in the text to describe the visual presentation, a video frame matching the visual presentation is found in the first video. By fusing the matched video segments and video frames, a more relevant and realistic target video can be obtained, ensuring the video quality of the final generated video. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of this application;

[0015] Figure 2 This is a schematic diagram of the structure of a multimodal feature extraction model provided in an embodiment of this application;

[0016] Figure 3 This is one of the processing flowcharts of a multimodal feature extraction model provided in the embodiments of this application;

[0017] Figure 4This is the second flowchart of a multimodal feature extraction model provided in this application embodiment;

[0018] Figure 5 This is the third flowchart of a multimodal feature extraction model provided in this application embodiment;

[0019] Figure 6 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application;

[0020] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0021] Figure 8 This is a hardware schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0023] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0024] The video generation method, apparatus, electronic device, and medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0025] In related technologies, the solutions used by electronic devices in the process of generating video based on text can be broadly divided into two types:

[0026] One approach is to employ a multimodal learning-based generative method. This involves using a separately trained text modality model to extract text features from the user's input text, and then feeding these text features into a generative adversarial network to generate a video. However, this method may use text and video modality models with different feature dimensions, leading to information gaps in the extracted features and making it difficult to match them effectively. Alternatively, if one modality model has limited effective information, it may fail to extract useful features, resulting in a mismatch between the generated video and the text description.

[0027] Another approach typically involves dividing a complete video into multiple frames. Then, when processing each frame, a fixed configuration is selected from a pool of text parameters and loaded onto a specific part of that image. Multiple configurations can be combined for a single frame. These processed frames are then stitched together to generate a custom video. However, this method results in a lack of coherence and realism in the generated video, and the loss of spatial information.

[0028] In the video generation method, apparatus, electronic device, and medium provided in the embodiments of this application, when generating video based on text, the electronic device can first extract behavioral descriptive words and visual descriptive words from the first text, then determine the target video segment matching the behavioral descriptive words and the target video frame matching the visual descriptive words from N first videos similar to the first text in the video library, and finally generate the target video based on the target video segment and the target video frame; where N is an integer greater than 1. Thus, by using behavioral descriptive words in the text to describe the subject's behavior to find video segments matching the subject's behavior in the first video, and using visual descriptive words in the text to describe the visual presentation to find video frames matching the visual presentation in the first video, the fusion of the matched video segments and video frames yields a more relevant and realistic target video, ensuring the video quality of the final generated video.

[0029] The execution entity of the video generation method provided in this embodiment can be a video generation device, which can be an electronic device, or a control module or processing module within the electronic device. The technical solution provided in this application embodiment will be described using an electronic device as an example.

[0030] This application provides a video generation method, such as... Figure 1 As shown, the video generation method may include the following steps 201 to 203:

[0031] Step 201: The electronic device extracts behavioral and visual descriptive words from the first text.

[0032] In this embodiment of the application, the aforementioned behavior descriptor can be a word in the first text used to describe the subject's behavior, such as running, high jump, flying, swimming, etc.

[0033] In this embodiment of the application, the aforementioned visual descriptive words can be words in the first text used to describe the visual presentation, such as a person wearing red clothes, a white puppy, or a blue sea.

[0034] In this embodiment of the application, after the electronic device obtains the first text, it can first perform word segmentation on the text, and then perform part-of-speech decomposition on the word segmentation to extract visual descriptive words and behavioral descriptive words respectively.

[0035] In this embodiment of the application, the first text may contain at least one behavior descriptive word.

[0036] In this embodiment of the application, the first text may contain at least one visual descriptive word.

[0037] Step 202: The electronic device determines the target video segment that matches the behavioral descriptor from the N first videos, and determines the target video frame that matches the visual descriptor from the N first videos.

[0038] Where N is an integer greater than 1.

[0039] In this embodiment of the application, the aforementioned N first videos are N videos in the video library that are similar to the first text.

[0040] In this embodiment of the application, the video library contains multiple videos, each video consisting of at least one video segment and / or at least two video frames.

[0041] In this embodiment of the application, when there are multiple video segments that match the behavior descriptor, the target video segment can be the video segment with the highest similarity to the behavior descriptor.

[0042] In this embodiment of the application, when the first text contains multiple behavioral descriptors, the electronic device can locate the video segment corresponding to each behavioral descriptor from the first video. For example, the target video segment may include the video segment corresponding to each of the multiple behavioral descriptors. Alternatively, the target video segment may include video segments corresponding to some of the behavioral descriptors.

[0043] In this embodiment of the application, when the first text contains multiple visual descriptors, the electronic device can locate the video frame corresponding to each visual descriptor from the first video. For example, the target video segment may include the video frame corresponding to each of the multiple visual descriptors. Alternatively, the target video frame may include video frames corresponding to some of the visual descriptors.

[0044] Step 203: The electronic device generates the target video based on the target video clip and the target video frame.

[0045] In this embodiment of the application, after determining the target video segment and the target video frame, the electronic device can fuse the target video segment and the target video frame to generate the target video.

[0046] Optionally, in this embodiment, the target video segment includes multiple video segments. Step 203, "the electronic device generates a target video based on the target video segment and the target video frame," may include steps 203a and 203b:

[0047] Step 203a: The electronic device sorts the multiple video clips according to the word order of the behavioral descriptions in the first text.

[0048] In this embodiment of the application, the electronic device can sort the determined video segments according to the original word order of the behavioral description words in the first text, so as to make them coherent and authentic in time sequence.

[0049] Step 203b: The electronic device merges the target video frame with the sorted video segments according to the word order of the visual descriptive words in the first text to generate the target video.

[0050] In this embodiment, the electronic device can input the target video frame into a general detection model to obtain subject information and background information. Then, according to the word order of visual descriptive words in the first text, it uses 3D rendering technology to fuse the subject information and the target video segment, making the subject in the target video frame move. Finally, it splices the subject information with the background information to generate the target video.

[0051] For example, the general detection model described above can be: a YOLOv3 detection model that performs multi-label classification on objects detected in an image and is trained and converged on the ImageNet dataset, which is used to extract subject feature information from text.

[0052] For example, the electronic device finds multiple first videos in a video library that are most similar to the text. Then, it searches for target video segments from these first videos that match the behavioral descriptors in the first text, extracts the motion information corresponding to the selected target video segments, and concatenates the motion information of the selected video segments sequentially according to the order of the behavioral descriptors in the first text. Simultaneously, it finds frames from the selected videos that are most similar to the visual descriptors, then uses a general detection model to obtain the subject and background. Finally, it uses 3D rendering technology to fuse the subject and motion information and concatenate it with the background to generate a video.

[0053] Thus, since the electronic device fuses the target video segments and target video frames according to the word order in the first text, the coherence of the target video segments and target video frames is ensured. In turn, while ensuring that the final generated video matches the text content of the first text, the accuracy of the spatial information of the video is guaranteed.

[0054] In the video generation method provided in this application embodiment, when generating a video based on text, the electronic device can first extract behavioral descriptive words and visual descriptive words from the first text, then determine the target video segment matching the behavioral descriptive words and the target video frame matching the visual descriptive words from N first videos similar to the first text in the video library, and finally generate the target video based on the target video segment and the target video frame; where N is an integer greater than 1. Thus, by using behavioral descriptive words in the text to describe the subject's behavior to find video segments matching the subject's behavior in the first video, and using visual descriptive words in the text to describe the visual presentation to find video frames matching the visual presentation in the first video, the fusion of the matched video segments and video frames yields a more relevant and realistic target video, ensuring the video quality of the final generated video.

[0055] Optionally, in this embodiment of the application, step 201, "the electronic device extracts behavioral descriptive words and visual descriptive words from the first text," may include steps 201a and 201b:

[0056] Step 201a: After the electronic device inputs the first text into the named entity recognition model, it performs word segmentation on the first text to obtain multiple words.

[0057] In this embodiment of the application, the aforementioned named entity recognition model can be: Named Entity Recognition (NER). Further, NER refers to a model capable of recognizing and extracting entity words from text.

[0058] Step 201b: The electronic device performs part-of-speech recognition on each of the multiple word segments based on the named entity recognition model to determine the behavioral descriptive words and visual descriptive words in the first text.

[0059] In this embodiment of the application, the electronic device can use the NER model to first segment the first text into multiple words, and then extract the visual descriptive words and behavioral descriptive words from the multiple words respectively.

[0060] In this way, by inputting the first text into the entity recognition model, the electronic device first segments the first text into words, and then determines the behavioral descriptive words and visual descriptive words in the multiple segments, which enables the electronic device to more accurately obtain the feature information contained in the first text.

[0061] Optionally, in this embodiment of the application, the video library includes multiple videos and video feature information of each of the multiple videos.

[0062] The following will describe the process of extracting video feature information for each video in the embodiments of this application:

[0063] Optionally, in this embodiment of the application, before step 202 above, the video generation method provided in this embodiment of the application may further include the following steps 301 to 302:

[0064] Step 301: The electronic device inputs multiple videos from the video library into the multimodal feature extraction model for feature extraction and outputs the video feature information of each video.

[0065] In this embodiment of the application, the video library may further include a list of video-related information. More specifically, the list of related information includes the video name of each video and the type of each video subject.

[0066] Step 302: The electronic device stores the video feature information of each video into the video library.

[0067] In this embodiment, the electronic device can use a multimodal feature extraction model to extract video feature information from all videos in the video library, then load all video feature information into the retrieval engine, and record the video name and the type of the video subject in the supplementary information list.

[0068] In this way, the video feature information of each video in the video library can be extracted directly using the multimodal feature extraction model, so that subsequent electronic devices can directly match the text features of the text to be used with the video feature information in the video library, thereby improving the matching efficiency.

[0069] Optionally, in this embodiment of the application, after the electronic device extracts the visual descriptive words of the first text, it can extract the subject in the visual descriptive words, then query the category mapping table to determine the subject type of the subject in the visual descriptive words of the first text, and finally, based on the subject type, filter out the video corresponding to the video type that matches the subject type from the video library, thereby forming a new video feature retrieval library.

[0070] For example, electronic devices can use named entity recognition models (e.g., NER pre-trained models) to extract the subject from visual descriptive words.

[0071] For example, the category mapping table above can be shown in Table 1 below. It should be noted that Table 1 only shows a portion of the detailed categories corresponding to some subject types and topic types. In practical applications, the category mapping table above can include more subject types and their corresponding detailed categories, which will not be elaborated upon here.

[0072] Table 1

[0073] category Detailed Categories people Men, women, the elderly, and children Four-legged animals Dogs, cats, bears, rabbits, ... Birds and animals Chickens, ducks, geese, parrots, ... plant Flowers, trees, ... airplane Passenger planes, cargo planes, airplanes, drones, ... vehicle Cars, trucks, buses, ... commodity Toys, household items, ...

[0074] This narrows down the range of videos that electronic devices can search for, and improves the search efficiency of electronic devices.

[0075] The following will explain how to implement the technical solution provided in this application based on this video library:

[0076] Optionally, in this embodiment of the application, before step 202 above, the video generation method provided in this embodiment of the application may further include the following steps A1 to A3:

[0077] Step A1: After the electronic device inputs the first text into the multimodal feature extraction model, it converts the first text into at least one text feature information.

[0078] In this embodiment, the multimodal feature extraction model can be a multi-level attention alignment model (MAAM). Further, the MAAM is an end-to-end multimodal video text retrieval model.

[0079] In this embodiment of the application, the at least one text feature information may include a text feature vector, such as a token corresponding to the text.

[0080] Step A2: Determine key text features from at least one text feature information.

[0081] In this embodiment of the application, the aforementioned key text feature information may be: text feature information in which the text feature vector satisfies a predetermined condition among the above at least one text feature information.

[0082] Step A3: Call the video library to cluster the key text feature information and the video feature information of each video to obtain the first video feature information, and take the video corresponding to the first video feature information as the first video.

[0083] In this embodiment of the application, the similarity between the first video feature information and the key text feature information satisfies the first condition.

[0084] In this embodiment, the first video feature information may include a video feature vector, such as a token corresponding to the video.

[0085] It should be noted that a token is used to represent the transformation of features into a fixed-dimensional sequence feature vector, where each atomic feature in the sequence is a token.

[0086] In this embodiment of the application, after the electronic device extracts the first text feature vector and obtains the video feature vector from the video library, it calculates the vector value to obtain the score corresponding to each video, thereby obtaining the similarity between the first text and each video.

[0087] In this embodiment of the application, the first condition can be: the video feature vector with the highest similarity value to the key text feature vector.

[0088] In this way, the electronic device improves the accuracy of identifying the first video by calculating the similarity between the extracted key text feature information and the video feature information in the video library, and identifies the video with the highest similarity value as the first video.

[0089] For example, let's take MAAM as an example to illustrate the extraction process of a multimodal feature extraction model.

[0090] For example, such as Figure 2 As shown, the MAAM model consists of two parts: a vision module and a text module. The vision module includes a Vision-transformer model (the visual model in the image-text pre-trained model) and a Guide-Study-Module (GSM). The text module includes a Text-transformer model (the text model in the image-text pre-trained model) and the GSM. The GSM module consists of an Attention module and a ClusterAlignment module, with the ClusterAlignment module being a shared module.

[0091] It should be noted that the Attention module mentioned above is used to guide the model to focus only on distinguishable features.

[0092] For example, the GSM described above aims to obtain distinctive feature information in each modality and to uniformly map the extracted salient feature vectors to a new feature space, so that the feature vectors of each modality are in the same semantic dimension in the new space, thereby eliminating the semantic information difference between modalities.

[0093] For example, such as Figure 3As shown, the Attention module in the GSM above is used to extract key information, reducing the number of output tokens in the penultimate layer of the Vision-transformer and Text-transformer from n to k, where k is the number of self-attention heads within the new model structure (transformer). L is the number of layers in the transformer structure, and a l Let a be the attention weight of the l-th layer of the transformer model, where l ranges from the first to the penultimate layer of the transformer structure. The last layer does not participate in the attention weight calculation. l The structure is shown in (Formula 1).

[0094]

[0095] in, This is the feature structure of a certain attention head in the l-th layer. Each attention head contains N tokens, and the structure is shown in (Formula 2):

[0096]

[0097] In the Attention module, the attention weights of all layers in (1, L-1) are multiplied by a matrix, as shown in (Equation 3):

[0098] a final =∏a l (Formula 3)

[0099] a final The second-to-last layer a of the transformer structure L-1 Perform matrix multiplication to obtain a select The attention vector is shown in (Equation 4):

[0100] a select =a final *a L-1 (Formula 4)

[0101] For a L-1 Each of the k components in the layer takes the maximum value. The structure is shown in (Formula 5):

[0102]

[0103] For ease of demonstration, the maximum component value selected by the kth attention head is denoted as the value. Let k selected attention tokens be concatenated with classification tokens to form a new sequence, replacing the original token sequence of layer L-1 and inputting it into the Lth layer of the transformer. The new sequence is in the form of (Equation 6):

[0104]

[0105] For example, such as Figure 4 As shown, the clustering alignment module is used to: cluster the k tokens output by the text Attention module and the k tokens output by the visual Attention module to obtain p shared centers {c1, c2, ..., c...} p Each token is re-represented using p shared centers. The tokens of the text model and the visual model are represented using the same set of vector bases, which further weakens the semantic difference between the two modalities.

[0106] In this model, each token in the text model and the visual model has a dot product with each component of the shared center. The product is then converted into a confidence score by the softmax function, representing the importance of the component of the current shared center to the specified token, thus indicating the data distribution of the token in the new feature space formed by the shared centers, as shown in (Equation 7).

[0107]

[0108] in, Let c represent the i-th token. j Let w represent the j-th shared center. ij This indicates the importance of the j-th shared center in representing the i-th token. The final representation of each token in the shared center features is shown in (Equation 8), where τ represents the total number of feature tokens in the text modality and the video modality, and there are k attention tokens in the output of the text modality and the visual modality, i.e., τ = 2k.

[0109]

[0110] For example, such as Figure 3 As shown, the feature vector extraction process of the above MAAM model includes the following steps S1 to S4:

[0111] Step S1: Obtain m tokens from the text using the Text-transformer model, where the category token (cls-token) is denoted as cls. g Then, these m tokens are passed through the Attention module in GSM to reduce the number of tokens to k+1.

[0112] Step S2: Sample 16 frames of the video evenly. For each frame, obtain n tokens through the Vision-transformer model. Take the cls-token corresponding to each frame and concatenate it with the newly added cls-token to form a new token sequence feature vector, where the newly added cls-token is denoted as cls. a Then, the feature vector of this new token sequence is input into the Attention module in GSM to extract key information, reducing the number of tokens to k+1.

[0113] Step S3: The text modality and video modality each output k attention tokens after removing the cls-token, and then pass them through the cluster alignment module to obtain a p-dimensional spatial representation.

[0114] Step S4: The overall training loss is shown in (Equation 9). After joint optimization of multiple losses, the model can learn the most representative attention feature vectors of video modality and text modality.

[0115] L = L g +L a +L c (Formula 9)

[0116] L g The global alignment loss is calculated using the global alignment module, as shown in Equation 10.

[0117]

[0118] To determine the number of all video-text pairs in the training set, Let i be the feature vector of the i-th video. Let i be the feature vector of the i-th text. The cosine similarity between video feature vectors and text feature vectors is calculated using the normalized cosine similarity. With normalization Calculate the dot product, as shown in (Formula 11):

[0119]

[0120] L a The attention alignment loss is shown in Equation 12, Sim(z) i ,z j) represents the cosine similarity between the attention feature vectors corresponding to samples i and j;

[0121]

[0122]

[0123] L c The cluster alignment loss is shown in Equation 13. Let i be the video feature vector after the i-th cluster. Let be the text feature vector after the i-th cluster.

[0124]

[0125] Optionally, in this embodiment of the application, step 202 above, "the electronic device determines the target video segment that matches the behavior descriptor from N first videos," may include steps B1 to B3:

[0126] Step B1: After the electronic device inputs the behavioral descriptor and N first videos into the multimodal feature extraction model, it converts the behavioral descriptor into at least one behavioral feature information and converts the N first videos into at least one video feature information.

[0127] In this embodiment of the application, the at least one behavioral feature information may include a behavioral feature vector, such as a token corresponding to a behavior.

[0128] Step B2: Determine key behavioral feature information from at least one behavioral feature information, and determine first key video feature information from at least one video feature information.

[0129] In this application embodiment, the aforementioned key behavioral feature information refers to the behavioral feature information that plays a decisive role among the above-mentioned at least one behavioral feature information.

[0130] Step B3: Based on the key behavioral feature information, determine the second video feature information from the first key video feature information, and take the video segment corresponding to the second video feature information as the target video segment.

[0131] In this embodiment of the application, the similarity between the second video feature information and the key behavioral feature information satisfies the second condition.

[0132] In this embodiment of the application, after the electronic device extracts the first key video feature vector and the key behavioral feature vector corresponding to the behavioral descriptor, it calculates the vector value to obtain the score corresponding to each video segment in the first video, thereby obtaining the similarity between the behavioral descriptor and each video segment.

[0133] In this embodiment of the application, the second condition can be: the first key video feature vector with the highest similarity value to the key behavior feature vector.

[0134] In this embodiment of the application, the aforementioned video segment is a video segment from the aforementioned first video. One first video corresponds to at least one video segment.

[0135] Example 1, taking five videos as the first example, the electronic device sends each behavior descriptor and the five videos into MAAM to extract behavior feature vectors. Then, it calculates the clustering loss L by combining each behavior descriptor with the feature vectors of the five selected videos. c Take out L c In the smallest video, the video frames corresponding to the k attention tokens in the visual module are arranged chronologically if the selected video frames are not adjacent. This forms a video segment. For example... Figure 5 As shown, the Attention tokens output by the Attention module for the text modality and video modality are labeled as 31 and 31 respectively. The original video frames corresponding to position 32 are used to form the video action segment most relevant to the current behavior description word. All video frames within the video segment are used to extract the main body keypoint position information using a keypoint retrieval model. In adjacent video frames, the position difference between the corresponding keypoints of the main body is calculated. For n video segments, n-1 sets of keypoint position differences are obtained and recorded as motion information.

[0136] In this way, the electronic device calculates the similarity between the extracted key behavioral feature information and the first key video feature information in the first video, and determines the video segment with the highest similarity value as the target video segment, thereby further improving the accuracy of the electronic device in determining the video segment.

[0137] Optionally, in this embodiment of the application, step 202 above, "the electronic device determines the target video frame that matches the visual descriptive word from N first videos," may include steps C1 to C3:

[0138] Step C1: After the electronic device inputs the visual descriptive words and N first videos into the multimodal feature extraction model, it converts the visual descriptive words into at least one visual feature information and converts the N first videos into at least one video feature information.

[0139] In this embodiment, the at least one visual feature information may include a visual feature vector, such as a visually corresponding token.

[0140] Step C2: Determine key visual feature information from at least one visual feature information, and determine second key video feature information from at least one video feature information.

[0141] In the embodiments of this application, the aforementioned key visual feature information refers to the visual feature information that plays a decisive role among the above-mentioned at least one visual feature information.

[0142] Step C3: Based on the key visual feature information, determine the third video feature information from the second key video feature information, and take the video frame corresponding to the third video feature information as the target video frame.

[0143] In this embodiment of the application, the similarity between the third video feature information and the key visual feature information satisfies the third condition.

[0144] In this embodiment of the application, after the electronic device extracts the second key video feature vector and the key visual feature vector corresponding to the visual descriptor, it calculates the vector value to obtain the score corresponding to each video frame in the first video, thereby obtaining the similarity between the visual descriptor and each video frame.

[0145] In this embodiment of the application, the third condition can be: the second key video feature vector with the highest similarity value to the key visual feature vector.

[0146] In this embodiment, the video frame refers to a video frame from the first video. One first video corresponds to at least two video frames.

[0147] Example 2, combined with Example 1, involves the electronic device sending each visual descriptor and the aforementioned 5 videos into MAAM to extract visual feature vectors. Then, it calculates the cosine similarity between each visual descriptor and all the attention tokens (5*k in total) in the 5 selected videos. The video frame corresponding to the attention token with the highest similarity is taken as the target video frame, which is the visual image that best matches the visual descriptor.

[0148] In this way, the electronic device calculates the similarity between the extracted key visual feature information and the second key video feature information in the first video, and determines the video frame with the highest similarity value as the target video frame, thereby further improving the accuracy of the electronic device in determining the video frame.

[0149] In related technologies, multimodal feature extraction model training schemes are generally divided into two types: synchronous training and asynchronous training. Among them:

[0150] Asynchronous training involves training each modality separately. After each modality is fully trained, features are extracted and similarity matching is performed directly. This training method results in visual modality models and text modality models with different feature dimensions. The features learned by each modality model have information gaps, making it difficult to match them well. Alternatively, when one modality has limited effective information, the model may not learn any useful information.

[0151] Simultaneous training involves training multiple modalities together. Each modal model focuses on information influenced by another modal model, and each modal model tends to learn features that are more relevant to the features of other modal models. However, co-training alone is not enough. Both video and text modalities contain a lot of redundant information, making it difficult for each modal model to learn key information. The models cannot capture the key content that the text and video want to express.

[0152] In this embodiment of the application, the design process of the multimodal feature extraction model (such as MAAM) is as follows:

[0153] First, the text modality model in MAAM uses an existing pre-trained text-transformer model structure as its base network structure, and samples 16 video frames at equal intervals using the video modality model in MAAM. Then, for each video frame, an existing pre-trained vision-transformer model structure is used as its base network structure. Furthermore, to remove redundant information, a Geometric Sense Model (GSM) is introduced after the two base network structures to guide each modality model to focus more on intermodal information. It should be noted that this GSM module consists of an Attention module and a Cluster Alignment module. Specifically, after the penultimate layer output of each modality in the GSM module, the Attention module is connected. It selects the k most discriminative tokens from each modality, concatenates them with the classification token to form a k+1 token sequence, replaces the penultimate layer output, and then passes through the final transformer layer to obtain k+1 attention feature tokens.

[0154] Secondly, to better align the features of the text modality model and the video modality model, the k attention feature tokens (excluding the classification token) can be fed into the Cluster Alignment module. Using the same basis vectors to represent the features of both the video and text modality models allows them to eliminate information gaps in the new feature space. Simultaneously, to improve the consistency of the two modal features, attention alignment loss and cluster alignment loss are introduced to maximize the consistency between the video and text modality models from local to global perspectives.

[0155] Thus, compared with the multimodal feature extraction model training schemes proposed in related technologies, the training scheme provided in this application adds a GSM module to the multimodal feature extraction model, enabling the trained multimodal feature extraction model to more accurately capture the key feature information contained in the text or video when extracting feature information.

[0156] The video generation method provided in this application can be executed by a video generation device. This application uses a video generation device executing the video generation method as an example to illustrate the video generation device provided in this application.

[0157] This application provides a video generation device, such as... Figure 6 As shown, the video generation device 400 includes: an extraction module 401, a determination module 402, and a generation module 403, wherein: the extraction module 401 is used to extract behavioral descriptive words and visual descriptive words from the first text; the determination module 402 is used to determine target video segments that match the behavioral descriptive words from N first videos, and to determine target video frames that match the visual descriptive words from the N first videos; the generation module 403 is used to generate a target video based on the target video segments and target video frames determined by the determination module 402; wherein the N first videos are N videos in a video library that are similar to the first text; N is an integer greater than 1.

[0158] Optionally, in this embodiment of the application, the extraction module 401 is specifically used to input the first text into the named entity recognition model, segment the first text to obtain multiple segments; based on the named entity recognition model, perform part-of-speech recognition on each of the multiple segments to determine the behavioral descriptive words and visual descriptive words in the first text.

[0159] Optionally, in this embodiment, the video library includes multiple videos and video feature information of each of the multiple videos; the determining module 402 is further configured to, before determining the target video segment matching the behavioral descriptor from the N first videos and the target video frame matching the visual descriptor from the N first videos, input the first text into a multimodal feature extraction model and convert the first text into at least one text feature information; determine key text feature information from the at least one text feature information; call the video library to cluster the key text feature information and the video feature information of each video to obtain first video feature information, and take the video corresponding to the first video feature information as the first video; wherein the similarity between the first video feature information and the key text feature information satisfies a first condition.

[0160] Optionally, in this embodiment of the application, the determining module 402 is specifically configured to input the behavior descriptor and the N first videos into a multimodal feature extraction model, convert the behavior descriptor into at least one behavior feature information, and convert the N first videos into at least one video feature information; determine key behavior feature information from the at least one behavior feature information, and determine first key video feature information from the at least one video feature information; determine second video feature information from the first key video feature information based on the key behavior feature information, and take the video segment corresponding to the second video feature information as the target video segment; wherein the similarity between the second video feature information and the key behavior feature information satisfies a second condition.

[0161] Optionally, in this embodiment, the determining module 402 is specifically configured to input the visual descriptor and the N first videos into a multimodal feature extraction model, convert the visual descriptor into at least one visual feature information, and convert the N first videos into at least one video feature information; determine key visual feature information from the at least one visual feature information, and determine second key video feature information from the at least one video feature information; determine third video feature information from the second key video feature information based on the key visual feature information, and take the video frame corresponding to the third video feature information as the target video frame; wherein the similarity between the third video feature information and the key visual feature information satisfies a third condition.

[0162] Optionally, in this embodiment of the application, the extraction module 401 is further configured to: input the plurality of videos in the video library into a multimodal feature extraction model for feature extraction, and output video feature information of each of the plurality of videos before the determining module 402 determines the target video segment that matches the behavior descriptor from the N first videos and the target video frame that matches the visual descriptor from the N first videos; and store the video feature information in the video library.

[0163] Optionally, in this embodiment of the application, the target video segment includes multiple video segments; the generation module 403 is specifically used to sort the multiple video segments according to the word order of the behavioral descriptive words in the first text; and to merge the target video frame with the sorted multiple video segments according to the word order of the visual descriptive words in the first text to generate a target video.

[0164] In the video generation apparatus provided in this application embodiment, when generating video based on text, the apparatus first extracts behavioral descriptive words and visual descriptive words from the first text. Then, it determines target video segments matching the behavioral descriptive words and target video frames matching the visual descriptive words from N first videos similar to the first text in a video library. Finally, it generates a target video based on the target video segments and the target video frames; where N is an integer greater than 1. Thus, by using behavioral descriptive words in the text to describe the subject's behavior to find video segments matching the subject's behavior in the first video, and using visual descriptive words in the text to describe the visual presentation to find video frames matching the visual presentation in the first video, the fusion of the matched video segments and video frames yields a more relevant and realistic target video, ensuring the quality of the final generated video.

[0165] The video generation device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0166] The video generation device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0167] The video generation device provided in this application embodiment can achieve... Figures 1 to 5 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0168] Optionally, such as Figure 7As shown, this application embodiment also provides an electronic device 600, including a processor 601 and a memory 602. The memory 602 stores a program or instructions that can run on the processor 601. When the program or instructions are executed by the processor 601, they implement the various steps of the above-described video generation method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0169] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0170] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0171] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0172] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0173] The processor 110 is configured to extract behavioral descriptors and visual descriptors from the first text; determine target video segments that match the behavioral descriptors from N first videos, and determine target video frames that match the visual descriptors from the N first videos; and generate a target video based on the target video segments and the target video frames; wherein the N first videos are N videos in a video library that are similar to the first text; and N is an integer greater than 1.

[0174] Optionally, in this embodiment of the application, the processor 110 is specifically used to input the first text into the named entity recognition model, perform word segmentation on the first text to obtain multiple word segments; and perform part-of-speech recognition on each of the multiple word segments based on the named entity recognition model to determine the behavioral descriptive words and visual descriptive words in the first text.

[0175] Optionally, in this embodiment, the video library includes multiple videos and video feature information of each of the multiple videos; the processor 110 is further configured to, before determining the target video segment matching the behavioral descriptor from the N first videos and the target video frame matching the visual descriptor from the N first videos, input the first text into a multimodal feature extraction model and convert the first text into at least one text feature information; determine key text feature information from the at least one text feature information; call the video library to cluster the key text feature information and the video feature information of each video to obtain first video feature information, and take the video corresponding to the first video feature information as the first video; wherein the similarity between the first video feature information and the key text feature information satisfies a first condition.

[0176] Optionally, in this embodiment of the application, the processor 110 is specifically configured to input the behavior descriptor and the N first videos into a multimodal feature extraction model, convert the behavior descriptor into at least one behavior feature information, and convert the N first videos into at least one video feature information; determine key behavior feature information from the at least one behavior feature information, and determine first key video feature information from the at least one video feature information; determine second video feature information from the first key video feature information based on the key behavior feature information, and take the video segment corresponding to the second video feature information as the target video segment; wherein the similarity between the second video feature information and the key behavior feature information satisfies a second condition.

[0177] Optionally, in this embodiment, the processor 110 is specifically configured to input the visual descriptor and the N first videos into a multimodal feature extraction model, convert the visual descriptor into at least one visual feature information, and convert the N first videos into at least one video feature information; determine key visual feature information from the at least one visual feature information, and determine second key video feature information from the at least one video feature information; determine third video feature information from the second key video feature information based on the key visual feature information, and take the video frame corresponding to the third video feature information as the target video frame; wherein the similarity between the third video feature information and the key visual feature information satisfies a third condition.

[0178] Optionally, in this embodiment of the application, the processor 110 is further configured to: input the plurality of videos in the video library into a multimodal feature extraction model for feature extraction, and output video feature information of each of the plurality of videos before determining the target video segment matching the behavior descriptor from the N first videos and the target video frame matching the visual descriptor from the N first videos; and store the video feature information in the video library.

[0179] Optionally, in this embodiment of the application, the target video segment includes multiple video segments; the processor 110 is specifically configured to sort the multiple video segments according to the word order of the behavioral descriptors in the first text; and to fuse the target video frame with the sorted multiple video segments according to the word order of the visual descriptors in the first text to generate a target video.

[0180] In the electronic device provided in this application embodiment, when generating video based on text, the electronic device can first extract behavioral descriptive words and visual descriptive words from the first text, then determine the target video segment matching the behavioral descriptive words and the target video frame matching the visual descriptive words from N first videos similar to the first text in the video library, and finally generate the target video based on the target video segment and the target video frame; where N is an integer greater than 1. Thus, by using behavioral descriptive words in the text to describe the subject's behavior to find video segments matching the subject's behavior in the first video, and using visual descriptive words in the text to describe the visual presentation to find video frames matching the visual presentation in the first video, the fusion of the matched video segments and video frames yields a more relevant and realistic target video, ensuring the video quality of the final generated video.

[0181] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0182] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0183] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0184] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described video generation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0185] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0186] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above video generation method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0187] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0188] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the video generation method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0189] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0191] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A video generation method, characterized in that, The method includes: Extract behavioral and visual descriptive words from the first text; From N first videos in the video library that are similar to the first text, determine the target video segment that matches the behavior descriptor, and from the N first videos, determine the target video frame that matches the visual descriptor; The target video segment and the target video frame are merged to generate the target video; The step of fusing the target video segment and the target video frame to generate the target video includes: The target video frame is input into a general detection model to obtain subject information and background information; According to the word order of the visual descriptive words in the first text, the main information and the target video segment are fused together, and the fused video segment is spliced ​​with the background information to generate the target video; Where N is an integer greater than 1.

2. The method according to claim 1, characterized in that, The extraction of behavioral and visual descriptive words from the first text includes: After inputting the first text into the named entity recognition model, the first text is segmented into multiple words. Based on the named entity recognition model, part-of-speech tagging is performed on each of the multiple word segments to determine the behavioral descriptive words and visual descriptive words in the first text.

3. The method according to claim 1, characterized in that, The video library includes multiple videos and video feature information for each of the multiple videos; Before determining the target video segment matching the behavioral descriptor from N first videos similar to the first text in the video library, and determining the target video frame matching the visual descriptor from the N first videos, the method further includes: After inputting the first text into the multimodal feature extraction model, the first text is converted into at least one text feature information; From the at least one text feature information, determine the key text feature information; The video library is invoked to cluster the key text feature information and the video feature information of each video to obtain the first video feature information, and the video corresponding to the first video feature information is taken as the first video; Wherein, the similarity between the first video feature information and the key text feature information satisfies the first condition.

4. The method according to claim 1, characterized in that, The step of determining the target video segment matching the behavior descriptor from N first videos similar to the first text in the video library includes: After inputting the behavior descriptor and the N first videos into the multimodal feature extraction model, the behavior descriptor is converted into at least one behavior feature information, and the N first videos are converted into at least one video feature information; From the at least one behavioral feature information, key behavioral feature information is determined, and from the at least one video feature information, first key video feature information is determined; Based on the key behavioral feature information, second video feature information is determined from the first key video feature information, and the video segment corresponding to the second video feature information is taken as the target video segment; Wherein, the similarity between the second video feature information and the key behavioral feature information satisfies the second condition.

5. The method according to claim 1 or 4, characterized in that, The step of determining the target video frame that matches the visual descriptive word from the N first videos includes: After inputting the visual descriptors and the N first videos into the multimodal feature extraction model, the visual descriptors are converted into at least one visual feature information, and the N first videos are converted into at least one video feature information; From the at least one visual feature information, determine key visual feature information, and from the at least one video feature information, determine second key video feature information; Based on the key visual feature information, the third video feature information is determined from the second key video feature information, and the video frame corresponding to the third video feature information is taken as the target video frame; Wherein, the similarity between the third video feature information and the key visual feature information satisfies the third condition.

6. The method according to any one of claims 3 to 5, characterized in that, Before determining the target video segment matching the behavioral descriptor from N first videos similar to the first text in the video library, and determining the target video frame matching the visual descriptor from the N first videos, the method further includes: The multiple videos in the video library are input into a multimodal feature extraction model for feature extraction, and the video feature information of each video is output. The video feature information is stored in the video library.

7. The method according to claim 1, characterized in that, The target video segment includes multiple video segments; The step of fusing the target video segment and the target video frame to generate the target video includes: The plurality of video segments are sorted according to the word order of the behavioral descriptors in the first text; According to the word order of the visual descriptive words in the first text, the target video frame is fused with the sorted plurality of video segments to generate the target video.

8. A video generation apparatus, characterized in that, The device includes: an extraction module, a determination module, and a generation module, wherein: The extraction module is used to extract behavioral descriptive words and visual descriptive words from the first text; The determining module is configured to determine a target video segment that matches the behavioral descriptor from N first videos similar to the first text in the video library, and to determine a target video frame that matches the visual descriptor from the N first videos; The generation module is used to fuse the target video segment and the target video frame determined by the determining module to generate a target video; The generation module is specifically used to input the target video frame into a general detection model to obtain subject information and background information; and to fuse the subject information and the target video segment according to the word order of the visual descriptive words in the first text, and to splice the fused video segment and the background information to generate the target video; Where N is an integer greater than 1.

9. The apparatus according to claim 8, characterized in that, The extraction module is specifically used for: After inputting the first text into the named entity recognition model, the first text is segmented into multiple words. Based on the named entity recognition model, part-of-speech tagging is performed on each of the multiple word segments to determine the behavioral descriptive words and visual descriptive words in the first text.

10. The apparatus according to claim 8, characterized in that, The video library includes multiple videos and video feature information for each of the multiple videos; The determining module is further configured to: Before determining the target video segment matching the behavioral descriptor from the N first videos similar to the first text in the video library, and the target video frame matching the visual descriptor from the N first videos, the first text is input into a multimodal feature extraction model and then converted into at least one text feature information. From the at least one text feature information, determine the key text feature information; The video library is invoked to cluster the key text feature information and the video feature information of each video to obtain the first video feature information, and the video corresponding to the first video feature information is taken as the first video; Wherein, the similarity between the first video feature information and the key text feature information satisfies the first condition.

11. The apparatus according to claim 8, characterized in that, The determining module is specifically used for: After inputting the behavior descriptor and the N first videos into the multimodal feature extraction model, the behavior descriptor is converted into at least one behavior feature information, and the N first videos are converted into at least one video feature information; From the at least one behavioral feature information, key behavioral feature information is determined, and from the at least one video feature information, first key video feature information is determined; Based on the key behavioral feature information, second video feature information is determined from the first key video feature information, and the video segment corresponding to the second video feature information is taken as the target video segment; Wherein, the similarity between the second video feature information and the key behavioral feature information satisfies the second condition.

12. The apparatus according to claim 8 or 11, characterized in that, The determining module is specifically used for: After inputting the visual descriptors and the N first videos into the multimodal feature extraction model, the visual descriptors are converted into at least one visual feature information, and the N first videos are converted into at least one video feature information; From the at least one visual feature information, determine key visual feature information, and from the at least one video feature information, determine second key video feature information; Based on the key visual feature information, the third video feature information is determined from the second key video feature information, and the video frame corresponding to the third video feature information is taken as the target video frame; Wherein, the similarity between the third video feature information and the key visual feature information satisfies the third condition.

13. The apparatus according to any one of claims 10 to 12, characterized in that, The extraction module is also used for: Before the determining module determines the target video segment that matches the behavior descriptor from the N first videos that are similar to the first text in the video library, and determines the target video frame that matches the visual descriptor from the N first videos, the multiple videos in the video library are input into a multimodal feature extraction model for feature extraction, and the video feature information of each video in the multiple videos is output. The video feature information is stored in the video library.

14. The apparatus according to claim 8, characterized in that, The target video segment includes multiple video segments; The generation module is specifically used for: The plurality of video segments are sorted according to the word order of the behavioral descriptors in the first text; According to the word order of the visual descriptive words in the first text, the target video frame is fused with the sorted plurality of video segments to generate the target video.

15. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the video generation method as described in any one of claims 1 to 7.

16. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the video generation method as described in any one of claims 1 to 7.

Images