Video commentary generation method and device, electronic equipment, medium and program product

Abstract

Embodiments of the present application disclose a video commentary generation method and device, electronic equipment, medium and program product; the method comprises: extracting a video feature of a target video; filtering at least one candidate audio feature matched with the video feature from a preset audio feature library to obtain a candidate audio feature set; for each candidate audio feature, filtering at least one candidate text feature matched with the candidate audio feature from a preset text feature library to obtain a candidate text feature set; determining a matching degree of each candidate feature combination composed of the video feature, the candidate audio feature in the candidate audio feature set and the candidate text feature in the candidate text feature set; based on the matching degree, selecting at least one target feature combination from each candidate feature combination, and generating a video commentary of the target video based on the target feature combination.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a video narration generation method, apparatus, electronic device, medium, and program product. Background Technology

[0002] Currently, when generating video narration, the process involves first analyzing the video content using a Natural Language Processing (NLP) model and generating structured narration scripts; then, matching popular copywriting templates with the video content and combining them with the structured narration scripts using semantic understanding to generate text material; finally, adding voice-over to the text material using fixed audio to generate a video narration that integrates video, text, and audio.

[0003] However, the method described above, which uses popular copywriting templates and fixed audio to generate video narrations, may produce video narrations with the same style for different video content, resulting in a low degree of matching between the generated video narrations and the video content. Summary of the Invention

[0004] This application provides a video narration generation method, apparatus, electronic device, medium, and program product to solve the problem in related technologies where the generated video narration has a low degree of matching with the video content when using popular text templates and fixed audio to generate video narration.

[0005] In a first aspect, embodiments of this application provide a video narration generation method, the method comprising:

[0006] Extract video features from the target video; From a pre-defined audio feature library, at least one candidate audio feature that matches the video features is selected to obtain a candidate audio feature set; for each candidate audio feature, at least one candidate text feature that matches the candidate audio feature is selected from a pre-defined text feature library to obtain a candidate text feature set. Determine the matching degree of each candidate feature combination consisting of video features, candidate audio features from the candidate audio feature set, and candidate text features from the candidate text feature set. Based on the matching degree, at least one target feature combination is selected from each candidate feature combination, and video narration of the target video is generated based on the target feature combination.

[0007] In some embodiments, after obtaining the candidate text feature set, the method provided by the first aspect further includes: The video features, candidate audio feature set, and candidate text feature set are stored as a doubly linked list. In the doubly linked list, the tail pointer of the video feature points to the head pointer of each candidate audio feature in the candidate audio feature set; the head pointer of each candidate audio feature in the candidate audio feature set points to the tail pointer of the video feature; the tail pointer of each candidate audio feature in the candidate audio feature set points to the head pointer of the candidate text feature set that matches the candidate audio feature; and the tail pointer of each candidate text feature in the candidate text feature set points to the head pointer of the candidate audio feature that matches the candidate text feature.

[0008] In some embodiments, the attention network is composed of at least one cascaded sub-attention network; Determine the matching degree of each candidate feature combination, which consists of video features, candidate audio features from the candidate audio feature set, and candidate text features from the candidate text feature set, including: Determine the candidate feature combinations corresponding to each link in the doubly linked list; wherein, the candidate feature combinations include the video features, candidate audio features, and candidate text features contained in the corresponding link; For each candidate feature combination, the video features, candidate audio features, and candidate text features contained in the candidate feature combination are converted into binary dot matrix data; The matching degree of the candidate feature combination is determined based on the binarized dot matrix data corresponding to the video features, candidate audio features, and candidate text features included in the candidate feature combination.

[0009] In some embodiments, based on the matching degree, at least one target feature combination is selected from each candidate feature combination, and a video narration of the target video is generated based on the target feature combination, including: Sort the links according to their matching degree; Based on the sorting results, select at least one target link from the links; Based on the combination of target features corresponding to at least one target link, generate at least one video narration for the target video.

[0010] In some embodiments, the matching degree of the candidate feature combination is determined based on the binarized bitmap data corresponding to the video features, candidate audio features, and candidate text features included in the candidate feature combination, including: Determine the first difference value between the binarized dot matrix data corresponding to the video features and the binarized dot matrix data corresponding to the candidate audio features; and the second difference value between the binarized dot matrix data corresponding to the candidate audio features and the binarized dot matrix data corresponding to the candidate text features; The matching degree of the candidate feature combination is determined based on the first difference value and the second difference value.

[0011] In some embodiments, the audio features in the audio feature library are audio features extracted from rap-style audio, and / or, the text features in the text feature library are text features extracted from abstract text.

[0012] Secondly, embodiments of this application provide a video narration generation apparatus, the apparatus comprising: The feature extraction module is used to extract video features from the target video; The candidate feature filtering module is used to filter at least one candidate audio feature that matches the video features from a preset audio feature library to obtain a candidate audio feature set; for each candidate audio feature, at least one candidate text feature that matches the candidate audio feature is filtered from a preset text feature library to obtain a candidate text feature set. The matching degree determination module is used to determine the matching degree of each candidate feature combination consisting of video features, candidate audio features in the candidate audio feature set, and candidate text features in the candidate text feature set. The video narration generation module is used to select at least one target feature combination from each candidate feature combination based on the matching degree, and generate video narration for the target video based on the target feature combination.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor, wherein, when the processor runs the computer program, it performs the method described in any embodiment of the first aspect.

[0014] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in any embodiment of the first aspect.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, causes the processor to perform the method described in any embodiment of the first aspect.

[0016] The video narration generation method provided in this application extracts video features from a target video, then selects at least one candidate audio feature from a preset audio feature library that matches the video features to obtain a candidate audio feature set. Next, for each candidate audio feature, at least one candidate text feature from a preset text feature library that matches the candidate audio feature is selected to obtain a candidate text feature set. The matching degree of each candidate feature combination, consisting of video features, candidate audio features from the candidate audio feature set, and candidate text features from the candidate text feature set, is then determined. Based on the matching degree, at least one target feature combination is selected from each candidate feature combination to generate the target video narration. The video narration technology transforms the final output from a rigid, fixed script template and audio. Instead, it starts with the video features of the target video itself, sequentially matching candidate audio and text features that are suitable for it. By filtering the matching degree, it selects the target feature combination that best matches the video content. This allows for the adaptive generation of video narrations with differentiated styles and high content matching for different video content. This effectively overcomes the problems of monotonous narration styles, stiff expression, and mismatch with video content in existing technologies. It also avoids the serious homogenization of video narration content caused by relying on popular script templates and fixed audio templates.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are merely embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort, and this application can be applied to other similar scenarios based on the provided drawings.

[0019] Figure 1 A flowchart illustrating a video narration generation method provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the principle of a doubly linked list provided in an embodiment of this application; Figure 3 A flowchart illustrating the method for determining the matching degree of candidate feature combinations provided in an embodiment of this application; Figure 4 A schematic diagram of the binarized dot matrix data provided in the embodiments of this application; Figure 5 A flowchart illustrating the method for generating video narration provided in an embodiment of this application; Figure 6This is a schematic diagram of the structure of a video narration generation device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. The described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the terms "system," "device," "unit," and / or "module" used in this application are methods of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they can be replaced by other expressions.

[0022] In the following text, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0023] Currently, the automatic generation method for video narration typically includes: first, analyzing the video content using an NLP model to generate structured narration scripts; then, matching preset popular text templates with the video content, and using semantic understanding technology to combine the popular text templates with the structured narration scripts to generate text material; finally, dubbing the text material with a fixed audio template to generate video narration content that includes video, text, and audio.

[0024] However, the above method has the following shortcomings: First, the narration style is monotonous and lacks adaptability. Due to the use of popular script templates and fixed audio templates, the generated video narrations lack personalized adjustments for different video content, resulting in stiff narrations and a low degree of matching with the actual content of the video; Second, the content is highly homogenized. After being processed using the methods described above, the resulting commentary is highly similar in both text and audio, making it difficult to generate distinctive commentary based on the differences in video content, thus reducing user viewing interest.

[0025] To address at least one of the aforementioned technical problems, embodiments of this application provide a video narration generation method, apparatus, electronic device, medium, and program product.

[0026] Figure 1 This is a flowchart illustrating a video narration generation method provided in an embodiment of this application.

[0027] like Figure 1 As shown, the video narration generation method provided in this application embodiment can be executed by an electronic device. Electronic devices include, but are not limited to, devices with data processing capabilities such as servers, personal computers, laptops, and tablets. The video narration generation method includes the following steps: Step 101: Extract video features from the target video; In some embodiments, the target video refers to the video for which video narration is to be generated.

[0028] In some embodiments, the target video may be a short video, movie clip, or animation shot by the user, or it may be all or part of a video clip extracted by the user from an existing video.

[0029] In some embodiments, a pre-trained feature extraction model can be used to extract video features from the target video; for example, the pre-trained feature extraction model can be a Convolutional Neural Network (CNN) model, a Transformer model, etc.

[0030] It should be noted that using a pre-trained feature extraction model to extract video features from a target video is a mature technology in this field, and will not be elaborated on here.

[0031] Step 102: Select at least one candidate audio feature that matches the video feature from the preset audio feature library to obtain a candidate audio feature set; for each candidate audio feature, select at least one candidate text feature that matches the candidate audio feature from the preset text feature library to obtain a candidate text feature set. In some embodiments, the pre-defined audio feature library contains at least one audio feature; wherein, to increase the appeal of the final generated video narration to a younger audience, the audio feature can be an audio feature extracted from rap-style audio. For example, an audio feature extraction model can be used to extract audio features from different rap audio recordings. It is understood that the audio feature extraction model can be a pre-trained audio feature extraction model, such as a CNN model, a Transformer model, etc.

[0032] In some embodiments, the pre-defined text feature library contains at least one text feature; wherein, in order to increase the appeal of the final generated video narration to a younger audience, the text feature can be a text feature extracted from an abstract form of copywriting. The abstract form of copywriting can be, for example, meme-worthy copywriting; for instance, a text feature extraction model can be used to extract text features from different meme-worthy copywriting. It is understood that the text feature extraction model can be a pre-trained text feature extraction model, such as a CNN model, a Transformer model, etc.

[0033] Understandably, when using a CNN model to extract audio features from rap audio or text features from parody videos, one can first encode the rap audio or parody video into a digital feature vector, and then use the digital feature vector as input to the CNN model to extract audio or text features.

[0034] In some embodiments, an electronic device performing the video narration generation method may have a pre-configured feature obfuscation space, and an audio feature library and a text feature library may be stored within the feature obfuscation space.

[0035] In some embodiments, when screening candidate audio features, the video features extracted in step 101 can be compared with each audio feature in the audio feature library to calculate the similarity between the video features and each audio feature, and then the audio features with the highest similarity ranking can be used as candidate audio features.

[0036] In some embodiments, similar to screening candidate audio features, when screening candidate text features, each of the aforementioned selected audio features can be compared with each of the text features in the text feature library, the similarity between the audio features and each text feature can be calculated, and then the text features with the highest similarity ranking can be used as candidate text features.

[0037] In some embodiments, it is understood that a video feature can filter out at least one audio feature, and an audio feature can filter out at least one text feature. For ease of description, the video feature of the target video is denoted by V, the candidate audio features are denoted by R1, R2, ..., Rn, and the candidate text features are denoted by T1, T2, ..., Tn. Then the candidate audio feature set is (R1, R2, ..., Rn), and the candidate text feature set is (T1, T2, ..., Tm).

[0038] Step 103: Determine the matching degree of each candidate feature combination consisting of video features, candidate audio features in the candidate audio feature set, and candidate text features in the candidate text feature set. In some embodiments, a candidate feature combination can be formed by video features, any candidate audio feature from the candidate audio feature set, and any candidate text feature from the candidate text feature set; for example, a candidate feature combination is (V, R1, T1); another example is (V, R2, T1); yet another example is (V, R1, T2), and so on. It should be noted that since candidate text features are selected based on candidate audio features, the preferred candidate feature combination should include video features, candidate audio features, and candidate text features matching the candidate audio features, rather than including arbitrary candidate text features.

[0039] In some embodiments, the matching degree of the candidate feature combination refers to the similarity between each feature (video feature, candidate audio feature, candidate text feature) in the candidate feature combination; wherein, the higher the matching degree, the higher the similarity between each feature in the candidate feature combination, the more matched the features in the candidate feature combination, and the more matched the generated video narration is with the target video.

[0040] Step 104: Based on the matching degree, select at least one target feature combination from each candidate feature combination, and generate video narration for the target video based on the target feature combination.

[0041] In some embodiments, candidate feature combinations can be sorted according to their matching degree, and the candidate feature combination with the highest matching degree can be selected as the target feature combination.

[0042] In some embodiments, a combination of target features corresponds to the generation of a video narration.

[0043] In some embodiments, a pre-trained video narration model can be used to generate video narration by taking the combination of target features as input.

[0044] In summary, the video narration generation method provided in this application extracts video features from the target video, then selects at least one candidate audio feature from a preset audio feature library that matches the video features to obtain a candidate audio feature set. Next, for each candidate audio feature, at least one candidate text feature from a preset text feature library that matches the candidate audio feature is selected to obtain a candidate text feature set. The matching degree of each candidate feature combination, consisting of video features, candidate audio features from the candidate audio feature set, and candidate text features from the candidate text feature set, is then determined. Based on the matching degree, at least one target feature combination is selected from each candidate feature combination to generate the target narration. The video narration technology transforms the final video narration from a rigid output based on fixed text templates and fixed audio. Instead, it starts with the video features of the target video itself, sequentially matching candidate audio features and candidate text features that are suitable for it. By filtering the matching degree, it selects the target feature combination that best matches the video content. This allows for the adaptive generation of video narrations with differentiated styles and high content matching for different video content. It effectively overcomes the problems of monotonous narration styles, stiff expression, and mismatch with video content in existing technologies. At the same time, it avoids the serious problem of content homogenization in video narrations caused by relying on popular text templates and fixed audio templates.

[0045] In some embodiments, after step 102, the video narration generation method provided in this application further includes: storing video features, candidate audio feature sets, and candidate text feature sets as doubly linked lists.

[0046] In the doubly linked list, the tail pointer of the video feature points to the head pointer of each candidate audio feature in the candidate audio feature set; the head pointer of each candidate audio feature in the candidate audio feature set points to the tail pointer of the video feature; the tail pointer of each candidate audio feature in the candidate audio feature set points to the head pointer of the candidate text feature in the candidate text feature set that matches the candidate audio feature; and the tail pointer of each candidate text feature in the candidate text feature set points to the head pointer of the candidate audio feature that matches the candidate text feature.

[0047] In some embodiments, a doubly linked list L1 with a head node can be generated and / or pre-configured in the feature confusion space; wherein, the head node means that L1 contains a fixed head node (head), which does not store any video features, candidate audio features, or candidate text features, but only serves as a unified entry point for L1. "Doubly linked" means that each data node in L1 contains two pointers (a head pointer (prev) and a tail pointer (next)) and a corresponding feature (a video feature, a candidate audio feature, or a candidate text feature). The head pointer is used to point to the preceding data node in L1 that matches the current data node, and the tail pointer is used to point to the following data node in L1 that matches the current data node, thereby supporting traversal of the entire L1 in either forward or backward directions starting from any data node.

[0048] Specifically, Figure 2 This is a schematic diagram illustrating the principle of a doubly linked list provided in an embodiment of this application. Figure 2 As shown, the data nodes in the doubly linked list are used to store video features, candidate audio features, or candidate text features, and the head and tail pointers of the data nodes record the multi-level association relationships between the features. Specifically, the video feature V of the target video is taken as the first data node of L1. The head pointer of V points to the head node of L1. Since each candidate audio feature matches V, the tail pointer of the first data node storing V points to the head pointer of each data node storing candidate audio features (R1, R2, ..., Rn); the head pointer of each data node storing candidate audio features (R1, R2, ..., Rn) also points to the tail pointer of the first data node storing V; the tail pointer of the data node storing candidate audio features (R1, R2, ..., Rn) points to the head pointer of each data node storing candidate audio features (R1, R2, ..., Rn). The head pointer of the data node storing the candidate text feature that matches the candidate audio feature is stored. For example, in step 102, if T1 is selected by R1 and matches R1, then the tail pointer of the data node storing R1 points to the head pointer of the data node storing T1. Similarly, the head pointer of the data node storing T1 points to the tail pointer of the data node storing R1. As another example, in step 102, if Tm is selected by Rn, then the tail pointer of the data node storing Rn points to the head pointer of the data node storing Tm. Similarly, the head pointer of the data node storing Tm points to the tail pointer of the data node storing Rn.

[0049] Figure 3 This is a flowchart illustrating a method for determining the matching degree of candidate feature combinations provided in an embodiment of this application. Figure 3 As shown, in some embodiments, step 103 includes: Step 301: Determine the candidate feature combinations corresponding to each link in the doubly linked list; In some embodiments, the candidate feature combination includes video features, candidate audio features, and candidate text features contained in the corresponding link.

[0050] It should be noted that the candidate audio features and candidate text features contained in the same candidate feature combination are matched. According to the aforementioned doubly linked list, the feature combination corresponding to each link in the doubly linked list is a candidate feature combination.

[0051] Step 302: For each candidate feature combination, convert the video features, candidate audio features, and candidate text features contained in the candidate feature combination into binary dot matrix data; In some embodiments, binarization can be used to convert video features, candidate audio features, and candidate text features into binary bitmap data. For example, the binary bitmap data includes only elements 0 and 1; as another example, Figure 4 This is a schematic diagram of the binarized dot matrix data provided in the embodiments of this application; as shown Figure 4 As shown, the binarized dot matrix data only includes black or white character boxes.

[0052] In some embodiments, the process of converting video features, candidate audio features, and candidate text features into binary bitmap data is as follows: As in step 102, video features, candidate audio features, and candidate text features can all be represented as digitized feature vectors. First, the digitized feature vectors of video features, candidate audio features, and candidate text features are mapped to the same feature space. For example, the digitized feature vectors of video features, candidate audio features, and candidate text features can be mapped to a feature space of dimension s×v, thereby generating feature matrices corresponding to video features, candidate audio features, and candidate text features, respectively, with each feature matrix having a size of s rows and v columns. Then, each element in each feature matrix is ​​binarized according to a preset binarization threshold: if the element value is greater than or equal to the binarization threshold, it is assigned a first logical value (e.g., 1); if the element value is less than the binarization threshold, it is assigned a second logical value (e.g., 0). After binarization, each feature matrix is ​​converted into a binary dot matrix data composed of the first logical value and the second logical value, where each logical value corresponds to a black and white character box in the binary dot matrix data. The first logical value can be represented as a black character box, and the second logical value can be represented as a white character box, and vice versa.

[0053] Step 303: Determine the matching degree of the candidate feature combination based on the binarized dot matrix data corresponding to the video features, candidate audio features, and candidate text features included in the candidate feature combination.

[0054] In some embodiments, step 103 includes: Determine the first difference value between the binarized dot matrix data corresponding to the video features and the binarized dot matrix data corresponding to the candidate audio features; and the second difference value between the binarized dot matrix data corresponding to the candidate audio features and the binarized dot matrix data corresponding to the candidate text features; The matching degree of the candidate feature combination is determined based on the first difference value and the second difference value.

[0055] In some embodiments, firstly, a first difference value is determined between the binary bitmap data corresponding to the video features and the binary bitmap data corresponding to the candidate audio features. Specifically, taking the elements in the binary bitmap data as represented by black and white character boxes as an example: the colors of the character boxes at the same position in the binary bitmap data corresponding to the video features and the binary bitmap data corresponding to the candidate audio features can be compared point by point: if the character boxes at the same position in the two bitmap data have the same color (e.g., both are black or both are white), the difference at that position is recorded as 0; if the character boxes at the same position in the two bitmap data have different colors, the difference at that position is recorded as 1. After traversing all positions, the differences at each position are accumulated to obtain the first difference value. Alternatively, the number of black character boxes and white character boxes in the two bitmap data can be directly counted, and the sum of the difference between the number of black character boxes and the difference between the number of white character boxes in the two bitmap data is taken as the first difference value.

[0056] Understandably, the smaller the first difference value, the higher the similarity between the video features and the candidate audio features.

[0057] Next, a second difference value is determined between the binary dot matrix data corresponding to the candidate audio features and the binary dot matrix data corresponding to the candidate text features. The calculation method for the second difference value is the same as that for the first difference value, and will not be repeated here.

[0058] Finally, the matching degree of the candidate feature combination is determined based on the first difference value and the second difference value.

[0059] In some embodiments, the sum of the first difference value and the second difference value can be used as the matching degree of the candidate feature combination. Specifically, the smaller the matching degree value, the higher the overall fit among the video features, candidate audio features, and candidate text features in the corresponding candidate feature combination; conversely, the larger the matching degree value, the lower the overall fit among the video features, candidate audio features, and candidate text features in the corresponding candidate feature combination.

[0060] In some embodiments, different weights may be assigned to the first difference value and the second difference value, for example, weight coefficients may be set according to the target video type or user preference, and the weighted sum of the first difference value and the second difference value may be used as the matching degree of the corresponding candidate feature combination, so as to enhance the flexibility and adaptability of the matching degree calculation.

[0061] Figure 5 This is a flowchart illustrating the method for generating video narration provided in an embodiment of this application. Figure 5 As shown, in some embodiments, step 104 includes the following steps: Step 501: Sort the links according to their matching degree; In some embodiments, leveraging the doubly circular nature of a doubly linked list, the link ordering is achieved by reversing the list: the data node corresponding to the link with the highest matching degree, along with its associated data nodes corresponding to candidate text features, are moved to the head of the doubly linked list (i.e., the position immediately adjacent to the head node). The data node corresponding to the link with the second highest matching degree is moved to the second position, and so on. This ensures that the order of data nodes in the doubly linked list directly reflects the link matching degree ranking result. This reordering operation only requires modifying the head and tail pointers of the relevant data nodes, without requiring additional storage space.

[0062] Step 502: Based on the sorting results, select at least one target link from the links; In some embodiments, the link that ranks first (i.e., has the highest matching degree) after sorting can be directly identified as the target link. The candidate feature combination corresponding to this target link is considered to be the target feature combination that best matches the video narration of the target video.

[0063] In some embodiments, the top N links after sorting can be designated as target links, where N is a preset positive integer (e.g., N=3). This allows for the generation of multiple video narrations in different styles for users to choose from, increasing the diversity of video narration content.

[0064] In some embodiments, target links can be filtered based on the comparison between the matching degree and a preset threshold; for example, links with a matching degree exceeding the preset threshold can be identified as target links. If the matching degree of all links is lower than the preset threshold, the preset threshold is lowered or the user is prompted to rebuild the bidirectional links.

[0065] In some embodiments, after the links in the doubly linked list are sorted by reversing the order, the N links starting from the head of the linked list are directly selected as the target links without any additional comparison operations.

[0066] Step 503: Generate at least one video narration for the target video based on the target feature combination corresponding to at least one target link.

[0067] In some embodiments, a pre-trained video narration generation model can be used to generate video narration for the target video. For example, video features, candidate audio features, and candidate text features from the target feature set can be input into the video narration generation model. The video narration generation model, based on an encoder-decoder architecture, automatically generates narration audio and subtitle timestamps aligned with the timeline of the target video frames, ultimately outputting a complete video narration. The video narration generation model can be a Transformer-based sequence-to-sequence model, or a diffusion-based audio generation model, etc.

[0068] It is understood that, in some embodiments, after generating video narration, the video narration generation method provided in this application further includes: arranging at least one video narration in the user interface in descending order of matching degree, setting the video narration with the highest matching degree as the default playback item, and highlighting or automatically previewing it. Users can switch to view other candidate video narrations by clicking or swiping.

[0069] In some embodiments, after generating video narration, the video narration generation method provided in this application further includes: destroying the doubly linked list, the candidate audio feature set and the candidate text feature set, releasing storage and computing resources, and retaining only the preset audio feature library and text feature library in the feature confusion space for use in the next video narration generation task.

[0070] To implement the video narration generation method provided in the embodiments of this application, the embodiments of this application also provide a video narration generation apparatus. For example... Figure 6 As shown, Figure 6 A schematic diagram of a video narration generation device provided in an embodiment of this application. The video narration generation device includes: Feature extraction module 601 is used to extract video features from the target video; The candidate feature filtering module 602 is used to filter at least one candidate audio feature that matches the video feature from a preset audio feature library to obtain a candidate audio feature set; and for each candidate audio feature, to filter at least one candidate text feature that matches the candidate audio feature from a preset text feature library to obtain a candidate text feature set. The matching degree determination module 603 is used to determine the matching degree of each candidate feature combination consisting of video features, candidate audio features in the candidate audio feature set, and candidate text features in the candidate text feature set. The video narration generation module 604 is used to select at least one target feature combination from each candidate feature combination based on the matching degree, and generate video narration for the target video based on the target feature combination.

[0071] The video narration generation device provided in this application embodiment further includes a doubly linked list construction module, which is used for: The video features, candidate audio feature set, and candidate text feature set are stored as a doubly linked list. In the doubly linked list, the tail pointer of the video feature points to the head pointer of each candidate audio feature in the candidate audio feature set; the head pointer of each candidate audio feature in the candidate audio feature set points to the tail pointer of the video feature; the tail pointer of each candidate audio feature in the candidate audio feature set points to the head pointer of the candidate text feature set that matches the candidate audio feature; and the tail pointer of each candidate text feature in the candidate text feature set points to the head pointer of the candidate audio feature that matches the candidate text feature.

[0072] In some embodiments, the matching degree determination module 603 is specifically used for: Determine the candidate feature combinations corresponding to each link in the doubly linked list; wherein, the candidate feature combinations include the video features, candidate audio features, and candidate text features contained in the corresponding link; For each candidate feature combination, the video features, candidate audio features, and candidate text features contained in the candidate feature combination are converted into binary dot matrix data; The matching degree of the candidate feature combination is determined based on the binarized dot matrix data corresponding to the video features, candidate audio features, and candidate text features included in the candidate feature combination.

[0073] In some embodiments, the video narration generation module 604 is specifically used for: Sort the links according to their matching degree; Based on the sorting results, select at least one target link from the links; Based on the combination of target features corresponding to at least one target link, generate at least one video narration for the target video.

[0074] In some embodiments, the matching degree determination module 603 is further specifically used for: Determine the first difference value between the binarized dot matrix data corresponding to the video features and the binarized dot matrix data corresponding to the candidate audio features; and the second difference value between the binarized dot matrix data corresponding to the candidate audio features and the binarized dot matrix data corresponding to the candidate text features; The matching degree of the candidate feature combination is determined based on the first difference value and the second difference value.

[0075] In some embodiments, the audio features in the audio feature library are audio features extracted from rap-style audio, and / or, the text features in the text feature library are text features extracted from abstract text.

[0076] The above embodiments of this application describe the apparatus provided in the embodiments of this application. To implement the functions of the methods provided in the above embodiments of this application, the video narration generation apparatus may include hardware structures and software modules, and may implement the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. One of the above functions may be executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules.

[0077] Figure 7 This is a block diagram illustrating an electronic device 700 for implementing the above-described video narration generation method, according to an exemplary embodiment. For example, the electronic device 700 may be a computer, a personal digital assistant, etc.

[0078] Reference Figure 7 The electronic device 700 may include a communication interface 701, capable of interacting with other devices; a processor 702, connected to the communication interface 701 to interact with other devices, used to execute the methods provided by one or more of the above-described technical solutions when running a computer program; and a memory 703, on which the computer program is stored. Specifically, the specific processing operations of the processor 702 can refer to the video narration generation method described in the above embodiments of this disclosure.

[0079] Of course, in practical applications, the various components in electronic device 700 are coupled together through bus system 704. It can be understood that bus system 704 is used to realize the connection and communication between these components. In addition to a data bus, bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 7 The general designated all buses as Bus System 704.

[0080] The memory 703 in this embodiment is used to store various types of data to support the operation of the electronic device 700. Examples of such data include any computer program used to operate on the electronic device 700.

[0081] The methods disclosed in the embodiments of this application can be applied to processor 702, or implemented by processor 702. Processor 702 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 702 or by instructions in software form. The processor 702 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 702 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 703. Processor 702 reads information from memory 703 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0082] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0083] Embodiments of this disclosure also propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the video narration generation method described in the above embodiments of this disclosure.

[0084] Embodiments of this disclosure also propose a computer program product, including a computer program that, when executed by a processor, implements the video narration generation method described in the above embodiments of this disclosure.

[0085] Embodiments of this disclosure also propose a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processor, cause the electronic device to perform the video narration generation method described in the above embodiments of this disclosure.

[0086] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0087] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0088] Any description of operation or method in the flowchart or otherwise described herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or operation, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0089] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0090] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0091] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by instructing related hardware through an operation sequence, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0092] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0093] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for generating video narration, characterized in that, The method includes: Extract video features from the target video; At least one candidate audio feature matching the video feature is selected from a preset audio feature library to obtain a candidate audio feature set; for each candidate audio feature, at least one candidate text feature matching the candidate audio feature is selected from a preset text feature library to obtain a candidate text feature set. Determine the matching degree of each candidate feature combination consisting of the video features, the candidate audio features in the candidate audio feature set, and the candidate text features in the candidate text feature set; Based on the matching degree, at least one target feature combination is selected from each of the candidate feature combinations, and a video narration of the target video is generated based on the target feature combination.

2. The method according to claim 1, characterized in that, After obtaining the candidate text feature set, the method further includes: The video features, the candidate audio feature set, and the candidate text feature set are stored as a doubly linked list; wherein, in the doubly linked list, the tail pointer of the video feature points to the head pointer of each candidate audio feature in the candidate audio feature set; the head pointer of each candidate audio feature in the candidate audio feature set points to the tail pointer of the video feature; the tail pointer of each candidate audio feature in the candidate audio feature set points to the head pointer of the candidate text feature set that matches the candidate audio feature; and the tail pointer of each candidate text feature in the candidate text feature set points to the head pointer of the candidate audio feature that matches the candidate text feature.

3. The method according to claim 2, characterized in that, Determining the matching degree of each candidate feature combination composed of the video features, candidate audio features from the candidate audio feature set, and candidate text features from the candidate text feature set includes: Determine the candidate feature combination corresponding to each link in the doubly linked list; wherein, the candidate feature combination includes video features, candidate audio features, and candidate text features contained in the corresponding link; For each of the candidate feature combinations, the video features, the candidate audio features, and the candidate text features contained in the candidate feature combination are converted into binary dot matrix data; The matching degree of the candidate feature combination is determined based on the binarized dot matrix data corresponding to the video features, the candidate audio features, and the candidate text features included in the candidate feature combination.

4. The method according to claim 3, characterized in that, The step of selecting at least one target feature combination from each of the candidate feature combinations based on the matching degree, and generating video narration for the target video based on the target feature combination, includes: The links are sorted according to their matching degree. Based on the sorting results, at least one target link is selected from the links; Based on the target feature combination corresponding to the at least one target link, at least one video narration is generated for the target video.

5. The method according to claim 3, characterized in that, The step of determining the matching degree of the candidate feature combination based on the binarized dot matrix data corresponding to the video features, the candidate audio features, and the candidate text features included in the candidate feature combination includes: Determine a first difference value between the binarized dot matrix data corresponding to the video feature and the binarized dot matrix data corresponding to the candidate audio feature; and a second difference value between the binarized dot matrix data corresponding to the candidate audio feature and the binarized dot matrix data corresponding to the candidate text feature; The matching degree of the candidate feature combination is determined based on the first difference value and the second difference value.

6. The method according to any one of claims 1 to 5, characterized in that, The audio features in the audio feature library are audio features extracted from rap-style audio, and / or the text features in the text feature library are text features extracted from abstract text.

7. A video narration generation device, characterized in that, The device includes: The feature extraction module is used to extract video features from the target video; The candidate feature filtering module is used to filter at least one candidate audio feature that matches the video feature from a preset audio feature library to obtain a candidate audio feature set; and for each candidate audio feature, to filter at least one candidate text feature that matches the candidate audio feature from a preset text feature library to obtain a candidate text feature set. The matching degree determination module is used to determine the matching degree of each candidate feature combination consisting of the video features, the candidate audio features in the candidate audio feature set, and the candidate text features in the candidate text feature set. The video narration generation module is used to select at least one target feature combination from each of the candidate feature combinations based on the matching degree, and generate a video narration of the target video based on the target feature combination.

8. An electronic device, characterized in that, include: A processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 6.

Images