Dynamic frame selection-based speed video undersievance method,

Abstract

The invention provides a split video understanding method and device based on dynamic frame selection, equipment and a medium, and relates to the field of video understanding. The method comprises the following steps: extracting a feature vector from each frame of an acquired split video by using a convolutional neural network; selecting two clustering algorithms to perform clustering analysis to obtain two clustering results; creating a corresponding consensus matrix according to the two clustering results, and adding element values to obtain a fused consensus matrix; classifying the fusion consensus matrix by adopting a preset threshold value, then calculating feature vector value differences between continuous frames in each category, and selecting two frames with the first two difference values as representative frames of the corresponding category; combining the representative frame with the first frame of the video, calculating a weight, and obtaining visual feature representation; the speech of the split video is converted into a text, visual feature representation is combined, feature extraction and fusion are performed through a deep learning algorithm, and an understanding document of the split video is obtained. According to the invention, the understanding accuracy of the split video can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of video understanding and artificial intelligence technology, and in particular to a method, device, equipment and medium for understanding storyboard videos based on dynamic frame selection. Background Art

[0002] Video understanding is a basic research problem in the field of computer vision, which aims to automatically identify and analyze the content in videos through intelligent technologies such as deep learning. By analyzing and learning videos, video understanding can identify information such as objects, actions, scenes, and emotions in videos, thereby improving the accuracy and efficiency of applications such as video retrieval, classification, labeling, and generation.

[0003] Most existing video understanding technology solutions use a method of randomly selecting 2 frames per second when selecting image frames for storyboard videos. As a result, when the feature extraction of the storyboard videos is subsequently performed, the image differences are not obvious, thus affecting the in-depth understanding of the video.

[0004] In view of this, the applicant filed this application after studying the existing technology. Summary of the invention

[0005] The present invention aims to provide a method, device, equipment and medium for understanding storyboard videos based on dynamic frame selection, so as to solve the problem that the image frames of storyboard videos selected in the existing methods are not representative enough, resulting in inaccurate subsequent feature extraction and affecting the deep understanding of the video.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0007] A storyboard video understanding method based on dynamic frame selection, comprising:

[0008] Obtain a storyboard video, and extract a feature vector from each frame of the storyboard video using a convolutional neural network;

[0009] Selecting two clustering algorithms to perform cluster analysis on the feature vectors respectively, thereby obtaining a first clustering result and a second clustering result represented by numerical values;

[0010] According to the first clustering result and the second clustering result, a corresponding consensus matrix is ​​created, and the element values ​​of the two matrices are added to obtain a fusion consensus matrix; wherein the consensus matrix is ​​an n×n real number matrix, and its dimension n is the same as the number of frames of the storyboard video;

[0011] Using a preset threshold, classifying the fusion consensus matrix;

[0012] According to the classified fusion consensus matrix, in each category, the difference in feature vector values ​​between consecutive frames is calculated and sorted in descending order, and the top two frames are selected as representative frames of the corresponding category;

[0013] According to the representative frame, combined with the first frame of the storyboard video, a weight is calculated, and all representative frames are fused to obtain a visual feature representation of the storyboard video;

[0014] The speech of the storyboard video is converted into text to obtain a language feature representation, which is combined with the visual feature representation, and feature extraction and fusion are performed through a deep learning algorithm to obtain an understanding copy of the storyboard video.

[0015] Preferably, the clustering algorithm includes K-means and hierarchical clustering algorithms.

[0016] Preferably, a weight is calculated based on the representative frame in combination with the first frame of the storyboard video, and the weight is expressed as:

[0017]

[0018] Wherein, W(x) represents the weight of frame x; a and b represent the selected representative frames respectively; D(x,a) represents the difference in feature vector values ​​between frame x and frame a; s represents the first frame of the storyboard video; and α represents the weight factor.

[0019] Preferably, when the fusion is performed by a deep learning algorithm, a weighted fusion method is adopted, and weights are set for the visual feature representation and the language feature representation respectively.

[0020] The present invention also provides a storyboard video understanding device based on dynamic frame selection, comprising:

[0021] A preliminary feature extraction unit, used to obtain a storyboard video and extract a feature vector from each frame of the storyboard video using a convolutional neural network;

[0022] A clustering integration unit, used for selecting two clustering algorithms, respectively performing clustering analysis on the feature vectors, thereby obtaining a first clustering result and a second clustering result represented by numerical values;

[0023] A fusion consensus matrix unit, used to create a corresponding consensus matrix according to the first clustering result and the second clustering result, and add the element values ​​of the two matrices to obtain a fusion consensus matrix; wherein the consensus matrix is ​​an n×n real number matrix, and its dimension n is the same as the number of frames of the storyboard video;

[0024] A threshold classification unit, used to classify the fusion consensus matrix using a preset threshold;

[0025] The representative frame unit is used to calculate the difference in feature vector values ​​between consecutive frames in each category according to the classified fusion consensus matrix, sort them in descending order, and select the top two frames as the representative frames of the corresponding category;

[0026] A visual feature representation unit, used to calculate a weight based on the representative frame and the first frame of the storyboard video, and fuse all the representative frames to obtain a visual feature representation of the storyboard video;

[0027] The output unit is used to convert the speech of the storyboard video into text to obtain a language feature representation, and combine it with the visual feature representation to perform feature extraction and fusion through a deep learning algorithm to obtain an understanding copy of the storyboard video.

[0028] The present invention also provides a storyboard video understanding device based on dynamic frame selection, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a storyboard video understanding method based on dynamic frame selection as described above.

[0029] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a storyboard video understanding method based on dynamic frame selection as described above is implemented.

[0030] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention combines the clustering results of two clustering algorithms, dynamically selects the two frames with the greatest difference in each category as the representative frames of the corresponding category, and combines the first frame of the storyboard video, and fuses all representative frames through an adaptive weight calculation method to obtain a visual feature representation. The visual feature representation obtained by this method can accurately select the most representative storyboard video image frame, thereby laying a foundation for subsequent further operations and improving accuracy.

[0032] At the same time, in the process of understanding the storyboard video, the present invention combines the voice text of the video and adopts a weighted fusion strategy to effectively improve the output effect of the storyboard video understanding solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of a storyboard video understanding method based on dynamic frame selection provided in Example 1.

[0035] Figure 2 A schematic diagram of a storyboard video understanding device based on dynamic frame selection provided in Example 2.

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] Embodiment 1

[0039] Embodiment 1 of the present invention provides a storyboard video understanding method based on dynamic frame selection, which can be implemented by a storyboard video understanding device based on dynamic frame selection (hereinafter referred to as storyboard video understanding device), and in particular, executed by one or more processors in the storyboard video understanding device.

[0040] In this embodiment, the storyboard video understanding device may be an electronic device equipped with a processor, the processor having a computer program of the storyboard video understanding method based on dynamic frame selection and the computer program can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., which is not limited here.

[0041] like Figure 1 As shown, a storyboard video understanding method based on dynamic frame selection includes steps S1 to S7.

[0042] S1, obtain a storyboard video, and use a convolutional neural network to extract a feature vector from each frame of the storyboard video.

[0043] In this embodiment, the convolutional neural network uses a pre-trained CNN model. The CNN model learns rich image feature representations and transfers them to the storyboard video processing task of this embodiment. By fine-tuning the model parameters until the model converges, a trained model is obtained to extract feature vectors of the storyboard video.

[0044] S2, selecting two clustering algorithms, and performing cluster analysis on the feature vectors respectively, so as to obtain a first clustering result and a second clustering result represented by numerical values;

[0045] In this embodiment, the clustering algorithm is a type of algorithm used to divide samples in a data set into multiple groups or clusters, so that the similarity of samples in the same cluster is high, while the similarity of samples between different clusters is low.

[0046] This embodiment selects K-means clustering and hierarchical clustering algorithms to perform cluster analysis on the feature vectors. Of course, other clustering algorithms, such as fuzzy C-means clustering, density peak clustering, etc., can also be selected according to actual needs, which is not limited here.

[0047] The principle of K-means clustering is: randomly select K center points as the initial cluster centers, then calculate the distance from each sample to the K center points, and divide the samples into clusters with the closest center points. Then, recalculate the center points of each cluster (that is, the mean of all points in the cluster), and iterate this process until the clusters no longer change or the preset number of iterations is reached.

[0048] The principle of hierarchical clustering is to cluster by building a hierarchical structure between data points. It can be a bottom-up agglomerative method (that is, each data point is initially regarded as a separate cluster, and then the nearest clusters are gradually merged), or a top-down divisive method (that is, all data points are initially regarded as one cluster, and then gradually split into smaller clusters).

[0049] S3, creating a corresponding consensus matrix according to the first clustering result and the second clustering result, and adding the element values ​​of the two matrices to obtain a fusion consensus matrix; wherein the consensus matrix is ​​an n×n real number matrix, and its dimension n is the same as the number of frames of the storyboard video;

[0050] In this step, an n×n consensus matrix is ​​created (n is the number of frames), where the elements C of the consensus matrix are ij Indicates whether frame i and frame j are in the same category, 0 means they are not in the same category, and 1 means they are in the same category. Then, for the results of K-means clustering and hierarchical clustering, create a consensus matrix respectively, and then add the element values ​​of these two matrices to get the fused consensus matrix. In this way, the element C′ of the fused consensus matrix ijThe value of will be between 0 and 2, where 0 means that both methods believe that frame i and frame j are not in the same category; 1 means that both methods believe that frame i and frame j are in the same category.

[0051] S4, using a preset threshold to classify the fusion consensus matrix.

[0052] In this step, a simple threshold method can be set to divide the fusion consensus matrix. For example, a fixed threshold is set, such as 1, and C′ ij Frames i and j with values ​​greater than 1 are classified into the same category.

[0053] S5, according to the classified fusion consensus matrix, in each category, the difference in feature vector values ​​between consecutive frames is calculated, and they are sorted in descending order, and the top two frames are selected as representative frames of the corresponding category.

[0054] In this step, the calculation expression of the eigenvector value difference is:

[0055] D(x,a)=F(x)-F(a);

[0056] Wherein, D(x,a) represents the difference in feature vector values ​​between frame x and frame a; F(x) represents the feature vector of frame x; and F(a) represents the feature vector of frame a.

[0057] In each category, the difference in feature vector values ​​between consecutive frames is calculated, and the two frames with the largest difference values ​​are selected as representative frames of the corresponding category, replacing the two randomly selected frames in the prior art, to improve the accuracy of subsequent feature extraction operations.

[0058] S6, based on the representative frame and the first frame of the storyboard video, calculate the weight, fuse all the representative frames, and obtain the visual feature representation of the storyboard video.

[0059] Furthermore, the weight expression is:

[0060]

[0061] Wherein, W(x) represents the weight of frame x; a and b represent the selected representative frames respectively; D(x,a) represents the difference in feature vector values ​​between frame x and frame a; s represents the first frame of the storyboard video; and α represents the weight factor.

[0062] All representative frames are puzzled to obtain a visual feature representation of the storyboard video.

[0063] S7, converting the speech of the storyboard video into text to obtain language feature representation, combining it with the visual feature representation, performing feature extraction and fusion through a deep learning algorithm, and obtaining an understanding copy of the storyboard video.

[0064] In this step, the speech in the video is converted into text to form a copy corresponding to the video content. This step helps to extract information from the video from another dimension, namely the language dimension.

[0065] The speech in the video can be converted into text with the help of existing speech recognition tools or algorithms, such as Hidden Markov Model (HMM), speech recognition algorithm based on neural network, etc.

[0066] In order to achieve a deep understanding of the storyboard video, the generated puzzle image (i.e., visual feature representation) and speech-to-text copy (i.e., language feature representation) are used as inputs for feature extraction and fusion through a deep learning algorithm. Specifically, in the feature fusion process, a weighted fusion strategy is adopted, and weights are set for the visual feature representation and the language feature representation respectively.

[0067] For example, according to actual needs, the weight ratio of visual feature representation and language feature representation is set to 85% and 15%. Although the information of speech-to-text conversion helps to understand the video content, the core of video understanding is still visual information, so the weight of visual feature representation should be greater. Of course, it can also be set to other ratios, which are not limited by the present invention.

[0068] Through a deep learning algorithm, after extracting key features from the visual feature representation and the language features, these key features are fused to obtain an understanding text of the corresponding storyboard video, thereby achieving a deep understanding of the storyboard video.

[0069] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0070] The present invention combines the clustering results of two clustering algorithms, dynamically selects the two frames with the largest difference in each category as the representative frames of the corresponding category, and combines the first frame of the storyboard video, and fuses all the representative frames through an adaptive weight calculation method to obtain a visual feature representation. The visual feature representation obtained by this method can accurately select the most representative storyboard video frame, thereby laying a foundation for subsequent further operations and improving accuracy.

[0071] At the same time, in the process of understanding the storyboard video, the present invention combines the voice text of the video and adopts a weighted fusion strategy to effectively improve the output effect of the storyboard video understanding solution.

[0072] Embodiment 2

[0073] like Figure 2 As shown, the second embodiment of the present invention further provides a storyboard video understanding device based on dynamic frame selection, comprising:

[0074] A preliminary feature extraction unit, used to obtain a storyboard video and extract a feature vector from each frame of the storyboard video using a convolutional neural network;

[0075] A clustering integration unit, used for selecting two clustering algorithms, respectively performing clustering analysis on the feature vectors, thereby obtaining a first clustering result and a second clustering result represented by numerical values;

[0076] A fusion consensus matrix unit, used to create a corresponding consensus matrix according to the first clustering result and the second clustering result, and add the element values ​​of the two matrices to obtain a fusion consensus matrix; wherein the consensus matrix is ​​an n×n real number matrix, and its dimension n is the same as the number of frames of the storyboard video;

[0077] A threshold classification unit, used to classify the fusion consensus matrix using a preset threshold;

[0078] The representative frame unit is used to calculate the difference in feature vector values ​​between consecutive frames in each category according to the classified fusion consensus matrix, sort them in descending order, and select the top two frames as the representative frames of the corresponding category;

[0079] A visual feature representation unit, used to calculate a weight based on the representative frame and the first frame of the storyboard video, and fuse all the representative frames to obtain a visual feature representation of the storyboard video;

[0080] The output unit is used to convert the speech of the storyboard video into text to obtain a language feature representation, and combine it with the visual feature representation to perform feature extraction and fusion through a deep learning algorithm to obtain an understanding copy of the storyboard video.

[0081] Embodiment 3

[0082] The third embodiment of the present invention also provides a storyboard video understanding device based on dynamic frame selection, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the storyboard video understanding method based on dynamic frame selection as described above.

[0083] Embodiment 4

[0084] The fourth embodiment of the present invention further provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the storyboard video understanding method based on dynamic frame selection as described above is implemented.

[0085] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0087] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. It should be noted that, in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0088] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0089] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0090] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0091] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A storyboard video understanding method based on dynamic frame selection, characterized in that: include: Obtain a storyboard video, and extract a feature vector from each frame of the storyboard video using a convolutional neural network; Selecting two clustering algorithms to perform cluster analysis on the feature vectors respectively, thereby obtaining a first clustering result and a second clustering result represented by numerical values; According to the first clustering result and the second clustering result, a corresponding consensus matrix is ​​created, and the element values ​​of the two matrices are added to obtain a fusion consensus matrix; wherein the consensus matrix is ​​an n×n real number matrix, and its dimension n is the same as the number of frames of the storyboard video; Using a preset threshold, classifying the fusion consensus matrix; According to the classified fusion consensus matrix, in each category, the difference in feature vector values ​​between consecutive frames is calculated and sorted in descending order, and the top two frames are selected as representative frames of the corresponding category; According to the representative frame, combined with the first frame of the storyboard video, a weight is calculated, and all representative frames are fused to obtain a visual feature representation of the storyboard video; The speech of the storyboard video is converted into text to obtain a language feature representation, which is combined with the visual feature representation, and feature extraction and fusion are performed through a deep learning algorithm to obtain an understanding copy of the storyboard video.

2. The method for understanding storyboard videos based on dynamic frame selection according to claim 1, characterized in that ,The clustering algorithms include K-means and hierarchical clustering algorithms.

3. The method for understanding storyboard videos based on dynamic frame selection according to claim 1, characterized in that ,According to the representative frame, combined with the first frame of the storyboard video, the weight is calculated, and the expression of the weight is: Wherein, W(x) represents the weight of frame x; a and b represent the selected representative frames respectively; D(x,a) represents the difference in feature vector values ​​between frame x and frame a; s represents the first frame of the storyboard video; and α represents the weight factor.

4. The method for understanding storyboard videos based on dynamic frame selection according to claim 1, characterized in that ,When fusion is performed through the deep learning algorithm, a weighted fusion method is adopted, and weights are set for the visual feature representation and the language feature representation respectively.

5. A storyboard video understanding device based on dynamic frame selection, characterized in that: include: A preliminary feature extraction unit, used to obtain a storyboard video and extract a feature vector from each frame of the storyboard video using a convolutional neural network; A clustering integration unit, used for selecting two clustering algorithms, respectively performing clustering analysis on the feature vectors, thereby obtaining a first clustering result and a second clustering result represented by numerical values; A fusion consensus matrix unit, used to create a corresponding consensus matrix according to the first clustering result and the second clustering result, and add the element values ​​of the two matrices to obtain a fusion consensus matrix; wherein the consensus matrix is ​​an n×n real number matrix, and its dimension n is the same as the number of frames of the storyboard video; A threshold classification unit, used to classify the fusion consensus matrix using a preset threshold; The representative frame unit is used to calculate the difference in feature vector values ​​between consecutive frames in each category according to the classified fusion consensus matrix, sort them in descending order, and select the top two frames as the representative frames of the corresponding category; A visual feature representation unit, used to calculate a weight based on the representative frame and the first frame of the storyboard video, and fuse all the representative frames to obtain a visual feature representation of the storyboard video; The output unit is used to convert the speech of the storyboard video into text to obtain a language feature representation, and combine it with the visual feature representation to perform feature extraction and fusion through a deep learning algorithm to obtain an understanding copy of the storyboard video.

6. A storyboard video understanding device based on dynamic frame selection, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a storyboard video understanding method based on dynamic frame selection as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a storyboard video understanding method based on dynamic frame selection as described in any one of claims 1 to 4 is implemented.

Images