Method and system for searching related videos according to video image materials
By preprocessing videos and feature extraction and matching of deep learning models, the problem of low video retrieval accuracy in the prior art is solved, and more accurate video image material source retrieval is achieved.
Patent Information
- Application Number
- CN202411478275.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-22
- Publication Date
- 2025-02-18
AI Technical Summary
When existing video search technology directly retrieves videos with images or video clips, the search accuracy has dropped significantly as the number of video creation increases.
By preprocessing the video, extracting keyframes and clustering, the keyframes and video image materials are featured extracted and matched based on the deep learning model, and videos containing similar features are selected.
The search accuracy of the source of video image materials is improved, and the problem of too few feature frames in the search results is avoided.
Smart Images
Figure CN119474458A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video retrieval, and in particular to a method and system for searching related videos based on video image materials. Background Art
[0002] In daily life, people may need to find the video source corresponding to an image or video based on an image or video for various reasons. For example, the prior art with publication number CN117251598A discloses a video retrieval method, which separates the original video into a video stream, an audio stream and a subtitle stream; extracts a text feature vector from the text data and the subtitle stream mapped by the audio stream; extracts an image feature vector from the shallow expression of the video stream in the key frame image; maps the text feature vector and the image feature vector to the same embedding domain and aligns them according to their similarity to obtain a text-image embedding vector; constructs a relational database based on the text-image embedding vector and the original video; retrieves a target video from multiple original videos based on the query text according to the relational database; and uses multimodal data to improve the efficiency and accuracy of video retrieval.
[0003] However, in the process of retrieving corresponding videos through images or video clips, existing retrieval methods retrieve videos containing similar images and texts through images and texts. However, with the explosive growth of video creation, a large number of images or video clips may be edited and spliced into various videos, resulting in a significant decrease in the accuracy of directly retrieving videos through images, texts or video clips. Summary of the invention
[0004] The object of the present invention is to provide a method and system for searching related videos based on video image materials, so as to solve the above-mentioned deficiencies in the prior art.
[0005] In order to achieve the above object, the present invention provides the following technical solution: a method for searching related videos according to video image materials, comprising the following steps:
[0006] S1, preprocessing the video to obtain the first key frame of the video;
[0007] S2, analyzing the first key frame of each video for clustering (such as K-means), obtaining multiple first key frame clusters, and setting the first key frame at the center of each first key frame cluster as the second key frame;
[0008] S3. Extract features from the second key frame based on a deep learning model to obtain a corresponding second key frame feature vector;
[0009] S4. Extracting features from the input video image material based on the deep learning model to obtain a corresponding material feature vector;
[0010] S5. Based on the deep learning model, output a second key frame corresponding to a second key frame feature vector whose similarity to the material feature vector is less than a set feature threshold, obtain a second target key frame, and obtain a video containing the second target key frame, which is marked as a preliminary target video; the present application does not limit the specific deep learning model, for example, the deep learning model may include ResNet, Inception, VGG, etc., and a pre-trained model may be used for feature extraction and feature vector similarity comparison;
[0011] S6, marking the first key frame in the first key frame cluster corresponding to the second target key frame as the first target key frame;
[0012] S7. Based on the similarity between the first key frame other than the first target key frame in each prepared target video and the first target key frame, determine whether the prepared target video meets the setting requirements; if so, set the prepared target video as the target video and output it.
[0013] Furthermore, the S1 comprises the following steps:
[0014] Calculate the similarity difference between two adjacent frames of the video;
[0015] A frame whose similarity difference with a previous frame is greater than a first similarity threshold is set as a first key frame.
[0016] Furthermore, the S1 further comprises the following steps:
[0017] Determine whether the number of frames between two first key frames is greater than or equal to two, and name the frames between the two first key frames as common frames;
[0018] If yes, then analyze the pairwise similarity difference between the first key frame and the common frame before the first key image frame, where the similarity difference = 1-similarity;
[0019] Update the first key frame to the first key frame or common frame with the smallest sum of similarity differences. For example, in a video, the first key frame is in the 7th frame, and the second key frame is in the 10th frame. Then, it is necessary to calculate the similarity differences between the 7th frame and the 8th frame, the 7th frame and the 9th frame, and the 8th frame and the 9th frame. If the similarity differences are 0.05, 0.04, and 0.07 respectively, the sum of the similarity differences between the 7th frame and the 8th frame, and the 7th frame and the 9th frame is the smallest, and the updated first key frame is still the 7th frame.
[0020] Further, the S7 comprises the following steps:
[0021] S7.1. Detect the target of the first key frame based on the target detection algorithm to obtain multiple detection targets. For example, a first key frame contains a person, a vehicle and a background. The target detection algorithm can detect and respectively divide the person, the vehicle and the background in the first key frame; wherein a pre-trained target detection algorithm can be used to implement target detection of the first key frame;
[0022] S7.2, retrieve the number of first key frames of non-first target key frames containing each number of detection targets in each preliminary target video. For example, if there are three detection targets, it is necessary to detect the number of first key frames containing three detection targets, the number of first key frames containing two detection targets, and the number of first key frames containing one detection target in the preliminary target video respectively. If one first key frame contains multiple identical detection targets, it is counted as one.
[0023] S7.3. Set weights for the first key frames of non-first target key frames containing each number of detection targets in the prepared target video, respectively. Based on the ratio of the number of first key frames containing each number of detection targets in each prepared target video to the number of first key frames and the corresponding weights, calculate the matching score of the prepared target video. For example, there are 3 detection targets. In the prepared target video, the number of first key frames containing 3 detection targets is 5, the number of first key frames containing 2 detection targets is 6, and the number of first key frames containing 1 detection target is 14. The number of first key frames is 40 in total. Then, the ratio of the number of first key frames containing each number of detection targets in each prepared target video to the number of first key frames is 5 / 40=1 / 8, 6 / 40=3 / 20, and 14 / 40=7 / 20, respectively. Then, the matching score = weight one * 1 / 8 + weight two * 3 / 20 + weight three * 7 / 20.
[0024] S7.4. Set the preliminary target video whose matching score is greater than the set matching score threshold as the target video.
[0025] Further, the step S7.2 includes the following steps:
[0026] Acquire the number of first target key frames of each prepared target video, and sort the prepared target videos in descending order according to the number of first target key frames to obtain a prepared target video sequence, wherein the more the number of first target key frames in the prepared target video is, the more frames similar to the input video image material the prepared target video contains, the greater the relevance between the prepared target video and the input video image material is, and the more likely it is that the video needs to be retrieved;
[0027] According to the sequence order of the prepared target videos, the number of first key frames containing non-first target key frames of each number of detected targets in each prepared target video is retrieved.
[0028] A system for searching related videos based on video image materials, comprising a video preprocessing module, a video image material input module, a feature extraction module, a feature matching module, and a video matching module;
[0029] The video preprocessing module is used to preprocess the video;
[0030] The video image material input module is used for allowing the user to input video image materials;
[0031] The feature extraction module is used to extract features from the video and the input video image material to obtain a second key frame feature vector and a material feature vector respectively;
[0032] The feature matching module is used to match the video image material with the video by comparing the similarity between the second key frame feature vector and the material feature vector to obtain a preliminary target video;
[0033] The video matching module is used to select a prepared target video that meets the setting requirements based on the similarity between the first key frame that is not the first target key frame in each prepared target video and the first target key frame, and set it as the target video output.
[0034] Compared with the prior art, the present invention provides a method and system for searching related videos based on video image materials. By setting a video preprocessing module, a video image material input module, a feature extraction module, a feature matching module, and a video matching module, the selection of key frames is optimized during the preprocessing of the video, and the key frames are clustered to obtain second key frames that can represent each key frame cluster. When feature matching is performed on the input video image material, a video containing a similar second key frame can be matched first, and then based on the similarity between other key frames in the video and the second key frame, the video containing more features of the input video image material as a whole is screened out, so as to avoid too few frames of the features of the video image material contained in the retrieved video, thereby improving the accuracy of retrieving the source of the video image material. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0036] Figure 1 A method step diagram provided for an embodiment of the present invention;
[0037] Figure 2 An S7 step diagram provided for an embodiment of the present invention;
[0038] Figure 3 Another S7 step diagram is provided for an embodiment of the present invention;
[0039] Figure 4 A system structure block diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION
[0040] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.
[0041] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.
[0042] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0043] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.
[0044] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.
[0045] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0046] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.
[0047] The embodiments described herein may be described with reference to plan views and / or cross-sectional views by means of idealized schematic diagrams of the present disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings have schematic properties, and the shapes of the regions shown in the figures illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.
[0048] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.
[0049] See also Figure 1-Figure 4 , a method for searching related videos according to video image materials, comprising the following steps:
[0050] S1, preprocessing the video to obtain the first key frame of the video, including the following steps:
[0051] S1.1, calculate the similarity difference between two adjacent frames of the video;
[0052] S1.2. Set a frame whose similarity difference with the previous frame is greater than a first similarity threshold as the first key frame.
[0053] S1.3, determining whether the number of frames between two first key frames is greater than or equal to two, and naming the frames between the two first key frames as ordinary frames;
[0054] S1.4, if yes, then analyze the pairwise similarity difference between the first key frame and the common frame before the first key image frame, where the similarity difference = 1-similarity;
[0055] S1.5. Update the first key frame to the first key frame or common frame with the smallest sum of similarity differences. For example, in a video, the first key frame is in the 7th frame, and the second key frame is in the 10th frame. Then, it is necessary to calculate the similarity differences between the 7th frame and the 8th frame, the 7th frame and the 9th frame, and the 8th frame and the 9th frame respectively. If the similarity differences are 0.05, 0.04, and 0.07 respectively, the sum of the similarity differences between the 7th frame and the 8th frame, and the 7th frame and the 9th frame is the smallest, and the updated first key frame is still the 7th frame.
[0056] S2, analyzing the first key frame of each video for clustering (such as K-means), obtaining multiple first key frame clusters, and setting the first key frame at the center of each first key frame cluster as the second key frame;
[0057] S3. Extract features of the second key frame based on the deep learning model to obtain a corresponding feature vector of the second key frame;
[0058] S4, extracting features from the input video image material based on the deep learning model to obtain a corresponding material feature vector;
[0059] S5. Based on the deep learning model, output a second key frame corresponding to the second key frame feature vector whose similarity with the material feature vector is less than the set feature threshold, obtain a second target key frame, and obtain a video containing the second target key frame, which is marked as a preliminary target video; the present application does not limit the specific deep learning model, for example, the deep learning model may include ResNet, Inception, VGG, etc., and a pre-trained model may be used for feature extraction and feature vector similarity comparison;
[0060] S6, marking the first key frame in the first key frame cluster corresponding to the second target key frame as the first target key frame;
[0061] S7, based on the similarity between the first key frame of the non-first target key frame in each prepared target video and the first target key frame, judging whether the prepared target video meets the setting requirements, if so, setting the prepared target video as the target video and outputting it, specifically, comprising the following steps:
[0062] S7.1. Detecting a target in the first key frame based on a target detection algorithm to obtain a plurality of detected targets. For example, a first key frame includes a person, a vehicle, and a background. The target detection algorithm can detect and respectively separate the person, the vehicle, and the background in the first key frame. A pre-trained target detection algorithm can be used to implement target detection in the first key frame.
[0063] S7.2.1. Obtain the number of first target key frames of each prepared target video, and sort the prepared target videos in descending order according to the number of first target key frames to obtain a prepared target video sequence, wherein the more the number of first target key frames in the prepared target video is, the more frames the prepared target video contains that are similar to the input video image material, the greater the relevance between the prepared target video and the input video image material is, and the more likely it is that the video needs to be retrieved;
[0064] S7.2.2. Retrieve the number of first key frames of non-first target key frames containing each number of detected targets in each of the prepared target videos in the order of the prepared target video sequence.
[0065] For example, if there are three detection targets, then the number of first key frames containing three detection targets, the number of first key frames containing two detection targets, and the number of first key frames containing one detection target in the preparation target video need to be detected respectively, and one first key frame containing multiple identical detection targets is counted as one;
[0066] S7.3. Set weights for the first key frames of non-first target key frames containing each number of detection targets in the prepared target video, respectively. Based on the ratio of the number of first key frames containing each number of detection targets in each prepared target video to the number of first key frames and the corresponding weights, calculate the matching score of the prepared target video. For example, there are 3 detection targets. In the prepared target video, the number of first key frames containing 3 detection targets is 5, the number of first key frames containing 2 detection targets is 6, and the number of first key frames containing 1 detection target is 14. The number of first key frames is 40 in total. Then, the ratio of the number of first key frames containing each number of detection targets in each prepared target video to the number of first key frames is 5 / 40=1 / 8, 6 / 40=3 / 20, and 14 / 40=7 / 20, respectively. Then, the matching score = weight one * 1 / 8 + weight two * 3 / 20 + weight three * 7 / 20.
[0067] S7.4. Set the preliminary target video whose matching score is greater than the set matching score threshold as the target video.
[0068] A system for searching related videos based on video image materials, comprising a video preprocessing module, a video image material input module, a feature extraction module, a feature matching module, and a video matching module;
[0069] The video preprocessing module is used to preprocess the video;
[0070] The video image material input module is used for users to input video image materials;
[0071] The feature extraction module is used to extract features from the video and the input video image material, and obtain a second key frame feature vector and a material feature vector respectively;
[0072] The feature matching module is used to match the video image material with the video by comparing the similarity between the feature vector of the second key frame and the feature vector of the material, and obtain a preliminary target video;
[0073] The video matching module is used to select a prepared target video that meets the setting requirements based on the similarity between the first key frame that is not the first target key frame in each prepared target video and the first target key frame, and set it as the target video output.
[0074] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.
Claims
1. A method for searching related videos based on video image materials, characterized in that: The following steps are involved: S1, preprocessing the video to obtain the first key frame of the video; S2, analyzing the first key frame of each video for clustering to obtain multiple first key frame clusters, and setting the first key frame at the center of each first key frame cluster as the second key frame; S3. Extract features from the second key frame based on a deep learning model to obtain a corresponding second key frame feature vector; S4. Extracting features from the input video image material based on the deep learning model to obtain a corresponding material feature vector; S5. Based on the deep learning model, output a second key frame corresponding to a second key frame feature vector whose similarity to the material feature vector is less than a set feature threshold, obtain a second target key frame, and obtain a video containing the second target key frame, and mark it as a preliminary target video; S6, marking the first key frame in the first key frame cluster corresponding to the second target key frame as the first target key frame; S7. Based on the similarity between the first key frame other than the first target key frame in each prepared target video and the first target key frame, determine whether the prepared target video meets the setting requirements; if so, set the prepared target video as the target video and output it.
2. A method for searching related videos according to video image material according to claim 1, characterized in that: The S1 comprises the following steps: Calculate the similarity difference between two adjacent frames of the video; A frame whose similarity difference with a previous frame is greater than a first similarity threshold is set as a first key frame.
3. A method for searching related videos according to video image material according to claim 2, characterized in that: The S1 further comprises the following steps: Determine whether the number of frames between two first key frames is greater than or equal to two, and name the frames between the two first key frames as common frames; If yes, then analyze the pairwise similarity difference between the first key frame and the common frame before the first key image frame, where the similarity difference = 1-similarity; The first key frame is updated to be the first key frame or the common frame with the smallest sum of similarity differences.
4. The method for searching related videos according to video image material according to claim 1, characterized in that: The S7 comprises the following steps: S7.
1. Detecting the target of the first key frame based on a target detection algorithm to obtain multiple detected targets; S7.2, retrieving the number of first key frames containing non-first target key frames of each number of detection targets in each prepared target video; S7.3, setting weights for the first key frames of non-first target key frames containing each number of detection targets in the preliminary target video, respectively, and calculating the matching score of the preliminary target video based on the ratio of the number of first key frames containing non-first target key frames containing each number of detection targets in each preliminary target video to the number of first key frames and the corresponding weights; S7.
4. Set the preliminary target video whose matching score is greater than the set matching score threshold as the target video.
5. A method for searching related videos according to video image material according to claim 4, characterized in that: The step S7.2 comprises the following steps: Obtaining the number of first target key frames of each prepared target video, and sorting the prepared target videos in descending order according to the number of first target key frames, to obtain a prepared target video sequence; According to the sequence order of the prepared target videos, the number of first key frames containing non-first target key frames of each number of detected targets in each prepared target video is retrieved.
6. A system for searching related videos based on video image materials, applicable to the method for searching related videos based on video image materials according to any one of claims 1 to 5, characterized in that: It includes a video preprocessing module, a video image material input module, a feature extraction module, a feature matching module, and a video matching module; The video preprocessing module is used to preprocess the video; The video image material input module is used for allowing the user to input video image materials; The feature extraction module is used to extract features from the video and the input video image material to obtain a second key frame feature vector and a material feature vector respectively; The feature matching module is used to match the video image material with the video by comparing the similarity between the second key frame feature vector and the material feature vector to obtain a preliminary target video; The video matching module is used to select a prepared target video that meets the setting requirements based on the similarity between the first key frame that is not the first target key frame in each prepared target video and the first target key frame, and set it as the target video output.
Citation Information
Patent Citations
Video retrieval method
CN117251598A