A video retrieval method based on multi-label relationship of association analysis

By employing a multi-label relationship method based on association analysis, video segmentation, and key feature extraction, the problems of low efficiency and insufficient accuracy in existing video retrieval technologies are solved, achieving efficient and accurate video retrieval results.

CN113361313BActive Publication Date: 2024-11-15WENZHOU UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110193192.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-02-20
Publication Date
2024-11-15
Estimated Expiration
2041-02-20

AI Technical Summary

Technical Problem

Current video retrieval technologies mainly rely on manual or simple single-tag retrieval, resulting in high resource consumption and low retrieval efficiency, and are unable to effectively address the issues of missed and false detections of violent and terrorist videos.

Method used

We employ a multi-label relationship method based on association analysis. Through video segmentation, key feature extraction, and label association analysis, we accurately identify video shots and key frames. We use multi-label datasets and feature information for video matching, and combine inter-frame difference and deduplication processing to improve retrieval accuracy and efficiency.

Benefits of technology

It achieves high accuracy and efficiency in video retrieval, reduces resource consumption, minimizes missed and false detections, and improves the overall effectiveness of video retrieval.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113361313B_ABST
    Figure CN113361313B_ABST
Patent Text Reader

Abstract

The present invention discloses a video retrieval method based on multi-label relationship of association analysis, which comprises the following steps: obtaining a video to be retrieved; obtaining multiple video shots of the video to be retrieved; processing each video shot to obtain a key video frame of the video to be retrieved; processing a sample video to obtain a video tag of the sample video; extracting feature information from the sample video tag to obtain a key video frame of the sample video tag; searching in the video to be retrieved based on the key video frame of the sample video, if the number of key video frames in a certain video shot found in the video to be retrieved is greater than or equal to a preset threshold of the number of sample video frames, it indicates that a video corresponding to the sample video is retrieved, otherwise a video corresponding to the sample video is not retrieved; the present invention has the beneficial effect of being able to realize accurate retrieval of videos and improving retrieval efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a retrieval method, and in particular to a video retrieval method based on multi-label relations of association analysis. Background Art

[0002] With the rapid development of network technology, various multimedia videos are presented to people, and violent and terrorist videos are also quietly spread and diffused. The spread and diffusion of videos affects people's physical and mental health, so it is necessary to retrieve the videos contained in the videos and give people warnings. However, the current retrieval of videos is mainly through manual retrieval or simple single-label retrieval, which consumes a lot of financial and material resources.

[0003] Therefore, it is urgent to propose a video retrieval method based on multi-label relationships based on association analysis to solve the problems of missed detection and false detection of videos. Summary of the invention

[0004] In view of this, the present invention provides a video retrieval method based on multi-label relationship of association analysis to solve the above technical problems.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A multi-label video retrieval method based on association analysis comprises the following steps:

[0007] Step 1: Obtain the video to be retrieved, wherein the video information to be retrieved includes video information and non-video information;

[0008] Step 2: Segment the video to be retrieved into a plurality of video shots of the video to be retrieved according to the video segmentation technology;

[0009] Step 3: Process each video shot according to the video playback order, obtain several key video frames of the video to be retrieved, and extract key features;

[0010] Step 4: Process the sample video to obtain a sample video label of the sample video; the sample video information includes a multi-label data set such as basic information, background knowledge, and behavioral knowledge;

[0011] Step 5: extracting feature information from the sample video label to obtain a number of key video frames of the sample video label;

[0012] Step 6: Search the video to be retrieved based on the key video frames of the sample video. If the number of key video frames found in the video to be retrieved is greater than or equal to a preset threshold of the number of key video frames of the sample video, it means that the video corresponding to the sample video is retrieved. Otherwise, the video corresponding to the sample video is not retrieved.

[0013] Furthermore, obtaining multiple tags of the sample video specifically includes the following steps:

[0014] Step 4.1: Obtain feature information of the sample video;

[0015] Step 4.2: Input the acquired feature information of the sample video into different models to obtain multiple video labels;

[0016] Step 4.3: Select a sample video label of the video from the multiple video labels according to the probability values ​​of the multiple video labels.

[0017] Furthermore, processing each shot in the video to be retrieved and multiple labels of the sample video of the sample video specifically includes the following steps:

[0018] Step 5.1: generating multiple key video frames based on the inter-frame differences between each shot in the video to be retrieved and multiple labels of the sample video;

[0019] Step 5.2: Determine a limit value from the key video frame, and use the limit value as a key video point of the key video frame of the video to be retrieved and the sample video.

[0020] Furthermore, the specific steps of obtaining the key video frame from each video shot of the video to be retrieved and the sample video include:

[0021] The first frame of the video to be retrieved and the sample video is taken as a key video frame, and the subsequent video frames are compared with the key video frames in sequence. When the frame difference between the Nth frame and the first frame exceeds a set threshold, the Nth frame becomes a new key video frame, and then the subsequent video frames are compared with the new key video frame of the Nth frame until the last frame, to obtain multiple new key video frames, and the same key video frames in the multiple new key video frames are de-overlapped to determine the final key video frames of all the videos to be retrieved and the sample video.

[0022] Furthermore, the de-overlapping method of the key video frames is as follows: Step 1: Obtain all key video frames;

[0023] In the second step, the quality scores of all key video frames are determined;

[0024] Step 3: removing key video frames whose quality scores are less than a first preset value in the key video;

[0025] The fourth step is to calculate the similarity between every two key video frames in all key video frames;

[0026] In the fifth step, when the calculated similarity is greater than a second preset value, the key video frames with lower quality scores are removed from the key video frames to obtain new key video frames.

[0027] Furthermore, the limit values ​​are the maximum and minimum values ​​of the key video frames.

[0028] Further, the preset threshold is set to 95%, and the first preset value and the second preset value are both set to 60%.

[0029] Furthermore, the feature information type includes at least one type of texture feature information, color feature information, shape feature information and edge feature information.

[0030] Furthermore, when there are more than two types of feature information, the two types of feature vectors are weighted and fused, and the key video frames are determined through association analysis.

[0031] It can be seen from the above technical solution that the advantages of the present invention are:

[0032] 1. Through multi-label association analysis, the videos in the search video can be accurately searched, which can improve the speed of video search and the accuracy of the video;

[0033] 2. This video retrieval method based on multi-label relationships of association analysis overcomes the shortcomings of traditional video retrieval, reduces resource consumption, and improves the retrieval efficiency of video retrieval.

[0034] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In the attached picture:

[0036] Figure 1 This is a step diagram of a video retrieval method based on multi-label relationships of association analysis according to the present invention.

[0037] Figure 2 The present invention is a flowchart of a video retrieval method based on multi-label relationships of association analysis.

[0038] Figure 3 A diagram of the steps for obtaining sample video labels in a video retrieval method of a multi-label relationship based on association analysis of the present invention.

[0039] Figure 4 A diagram of the steps for deduplication of key video frames in a video retrieval method of a multi-label relationship based on association analysis according to the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] like Figure 1 and Figure 2 A video retrieval method based on a multi-label relationship of association analysis is shown, and the video retrieval method based on a multi-label relationship of association analysis comprises the following steps:

[0042] Step 1: Obtain the video to be retrieved, wherein the video information to be retrieved includes video information and non-video information;

[0043] Step 2: Segment the video to be retrieved into a plurality of video shots of the video to be retrieved according to the video segmentation technology;

[0044] Specifically, the segmentation of the video to be retrieved in the present invention is achieved through video segmentation technology, and the specific steps include: based on the acquisition of multiple video frames, judging the continuity between the multiple video frames, and comparing the current video frame with the previous video frame; according to the judgment result, the video to be retrieved is divided into several video shots, if they are continuous, they are the same video shot, if they are not continuous, they are written as one video shot.

[0045] The present invention obtains video frames in time sequence, judges the video to be retrieved, and divides all the video to be retrieved according to the judgment result to obtain a plurality of video shots.

[0046] Step 3: Process each video shot according to the video playback order, obtain several key video frames of the video to be retrieved, and extract key features;

[0047] Specifically, the key features may be extracted by means of feature dimensionality reduction.

[0048] Step 4: Process the sample video to obtain a sample video label of the sample video; the sample video information includes a multi-label data set such as basic information, background knowledge, and behavioral knowledge;

[0049] Specifically, Figure 3 As shown, the acquisition of multiple labels of the sample video includes the following process:

[0050] Step 4.1: Obtain feature information of the sample video;

[0051] Step 4.2: Inputting the acquired feature information of the video into different classification models to obtain multiple video labels;

[0052] Step 4.3: Select a sample video label of the video from the multiple video labels according to the probability values ​​of the multiple video labels.

[0053] Step 5: extracting feature information from the sample video label to obtain a number of key video frames of the sample video label;

[0054] The types of feature information here can be texture feature information, color feature information, shape feature information, edge feature information, etc. These feature information can be represented by vectors. If there are more than two types of features, vector weighted fusion is used for association analysis to determine the key video frames of the sample video label.

[0055] The key video frames and key points of the key videos of each shot in the video to be retrieved and multiple sample video labels of the sample video are obtained as follows:

[0056] Step 5.1: Process each shot in the video to be retrieved and multiple tags of the sample video to generate multiple key video frames;

[0057] Specifically, the processing method may adopt either an inter-frame difference method or a video key frame extraction model method. The processing method adopted in the present invention is the inter-frame difference method to determine the key video frame.

[0058] Step 5.2: Determine limit values ​​from the key video frame, and use the limit values ​​as key video points of the key video frame; the limit values ​​are maximum values ​​and minimum values, and the limit values ​​can be determined using a Gaussian scale space method.

[0059] Furthermore, the step of obtaining key video frames of multiple labels for each shot of the video to be retrieved and the sample video is as follows: the first frame of the video to be retrieved is used as the key video frame, and the subsequent video frames are compared with the key video frames in chronological order; when the inter-frame difference between the Nth frame and the first frame exceeds a set threshold, the Nth frame becomes a new key video frame, and then the subsequent video frames are compared with the new key video frame of the Nth frame until the last frame, to obtain multiple new key video frames, and the same key video frames in the multiple new key video frames are de-overlapped, that is, repeated similar video key frames are removed, and finally new key video frames of all the videos to be retrieved and the sample videos are determined.

[0060] The most representative key video frames are obtained by deduplicating the key video frames and removing the overlapping information. The acquisition of key video frames mainly involves extracting the most representative image information of each shot in the video to be retrieved. Often, the representative key video frames have the greatest image difference from the adjacent key video frames. Deduplication of key video frames can reduce the workload of retrieval and save retrieval time to a certain extent. Generally, there are often key video frames with high repetitiveness among the multiple key video frames obtained. Therefore, it is necessary to deduplicate all key video frames, retain any key video frame or specific key video frame among all key video frames with high repetitiveness and representativeness, and remove other key video frames among the key video frames with high repetitiveness, thereby obtaining new key video frames.

[0061] For example, the extracted multiple key video frames include key video frame 1, key video frame 2, key video frame 3, key video frame 4 to key video frame 10, among which key video frame 3 and key video frame 4 have a high repetitiveness, so key video frame 3 is retained and key video frame 4 is removed, and the new key video frames obtained are key video frame 1, key video frame 2, key video frame 3 to key video frame 10 in sequence.

[0062] In the present invention, by performing deduplication processing on the key video frames of the video to be retrieved and the sample video, it is ensured that the key video frames represent more representative content, thereby improving the probability of retrieving videos.

[0063] Specifically, deduplication of all key video frames obtained to obtain new key video frames includes the following steps:

[0064] The first step is to obtain all key video frames.

[0065] Specifically, all key video frames are obtained from the video to be retrieved and the sample video.

[0066] In the second step, the quality scores of all key video frames are determined.

[0067] Specifically, each key video frame is analyzed to determine a quality score of each key video frame, and the quality score is used to evaluate the quality of the key video frame. The quality score of the key video frame can be achieved through dimensional data.

[0068] The third step is to remove the key video frames whose quality scores are less than the first preset value from all the key video frames.

[0069] Specifically, after determining the quality score of the key video frame, the quality score of each key video frame is compared with a first preset value, and the key video frames with quality scores less than the first preset value are removed from all key video frames, and the key video frames with quality scores greater than or equal to the first preset value are retained.

[0070] For example, the first preset value is 0.9, and the quality scores of all key videos are between 0 and 1, then the key video frames with a quality score less than 0.9 in all key videos are removed.

[0071] The fourth step is to calculate the similarity between every two key video frames in all key video frames.

[0072] Specifically, the method for calculating the two key video frames can be implemented by using the minimum mean square method.

[0073] In the fifth step, when the calculated similarity is greater than the second preset value, the key video frames with lower quality scores are removed from all the key video frames to obtain new key video frames.

[0074] Specifically, when the similarity is greater than the second preset value, it is determined that two key video frames among all key video frames are likely to be the same, indicating that the two key video frames are repetitive. Therefore, it is necessary to remove the key video frame with a lower quality score and retain the key video frame with a higher quality score to obtain a new key video frame.

[0075] Step 6: Search the video to be retrieved based on the key video frames of the sample video. If the number of key video frames found in the video to be retrieved is greater than or equal to a preset threshold of the number of key video frames of the sample video, it means that the video corresponding to the sample video is retrieved. Otherwise, the video corresponding to the sample video is not retrieved.

[0076] Specifically, the larger the preset threshold is, the higher the matching accuracy is. The larger the first preset value and the second preset value are, the greater the repetitiveness of the two key video frames in all key videos is. The first preset value and the second preset value can both be set to 60%, indicating that the greater the repetitiveness of the key video frames is. The preset threshold of the present invention can be set to 95%, so that the accuracy of matching the retrieved video with the sample video is higher, the retrieval is more accurate, the probability of misdetection and missed detection of videos is reduced, and video dissemination is purified.

[0077] The present invention retrieves the video to be retrieved in a manner corresponding to the sample video. First, the key video frame of the video to be retrieved is determined, and then the key video frame of the sample video is determined by label association analysis. The key video frame of the sample video is retrieved with the key video frame of the video to be retrieved. When the matching rate is greater than 95%, the video exists in the video to be retrieved, thereby achieving high-precision retrieval of the video to be retrieved, while reducing costs and improving the actual user experience.

[0078] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video retrieval method based on multi-label relationship of association analysis, characterized in that: The following steps are involved: Step 1: Obtain the video to be retrieved, wherein the video information to be retrieved includes video information and non-video information; Step 2: Segment the video to be retrieved into a plurality of video shots of the video to be retrieved according to the video segmentation technology; Step 3: Process each video shot according to the video playback order, obtain several key video frames of the video to be retrieved, and extract key features; Step 4: Process the sample video to obtain a sample video label of the sample video; the sample video information includes a multi-label data set of basic information, background knowledge and behavior knowledge; Step 5: extracting feature information from the sample video label to obtain a key video frame of the sample video label; Step 6: searching the video to be retrieved based on the key video frames of the sample video. If the number of key video frames found in the video to be retrieved is greater than or equal to a preset threshold of the number of key video frames of the sample video, it means that the video corresponding to the sample video is retrieved. Otherwise, the video corresponding to the sample video is not retrieved. Acquiring multiple tags of the sample video specifically includes the following steps: Step 4.1: Obtain feature information of the sample video; Step 4.2: Input the acquired feature information of the sample video into different models to obtain multiple video labels; Step 4.3: Selecting a sample video label of the video from the multiple video labels according to the probability values ​​of the multiple video labels; Processing each shot of the video to be retrieved and multiple labels of the sample video of the sample video specifically includes the following steps: Step 5.1: Generate multiple key video frames based on the inter-frame differences between each shot of the video to be retrieved and multiple labels of the sample video; Step 5.2: determining a limit value from a key video frame, and using the limit value as a key video point of the key video frame of the video to be retrieved and the sample video; The preset threshold is set to 95%, and the first preset value and the second preset value are set to 60%. The preset threshold is used to determine whether the retrieved video is a video corresponding to the sample video, and the first preset value and the second preset value are used to perform de-overlapping processing on key video frames; The feature information type includes at least one of texture feature information, color feature information, shape feature information and edge feature information; When the feature information types are more than two, the two types of feature vectors are weighted and fused, and the key video frames are determined through association analysis.

2. The video retrieval method based on multi-label relationship of association analysis as claimed in claim 1, characterized in that: The specific steps of obtaining the key video frame from each video shot of the video to be retrieved and the sample video include: The first frame of the video to be retrieved and the sample video is taken as a key video frame, and the subsequent video frames are compared with the key video frames in sequence. When the frame difference value between the Nth frame and the first frame exceeds a set threshold, the Nth frame becomes a new key video frame, and then the subsequent video frames are compared with the new key video frame of the Nth frame until the last frame, to obtain multiple new key video frames, and the same key video frames in the multiple new key video frames are de-overlapped to determine the final key video frames of all the videos to be retrieved and the sample video.

3. The video retrieval method based on multi-label relationship of association analysis as claimed in claim 2, characterized in that: The de-overlapping method of the key video frames is as follows: Step 1: Obtain all key video frames; In the second step, the quality scores of all key video frames are determined; Step 3: removing key video frames whose quality scores are less than a first preset value in the key video; The fourth step is to calculate the similarity between every two key video frames in all key video frames; In the fifth step, when the calculated similarity is greater than a second preset value, the key video frames with lower quality scores are removed from the key video frames to obtain new key video frames.

4. The video retrieval method based on multi-label relationship of association analysis as claimed in claim 3, characterized in that: The limit values ​​are the maximum and minimum values ​​of the key video frame.