A video similar segment search method based on straight line detection

Abstract

The application discloses a video similar segment searching method based on straight line detection, which comprises the following steps: extracting key frames of a query video to obtain a key frame set; extracting image features of each key frame in the key frame set of the video to obtain an image feature set of the video; comparing each image feature in the image feature set of the query video, finding out K image features most similar to the image feature value in the image feature set of a reference video, obtaining a feature candidate set and saving the feature candidate set; drawing the feature candidate set into a two-dimensional image, performing line segment detection on the two-dimensional image by a line segment detection method to obtain candidate line segments, and filtering the detected candidate line segments to obtain a candidate line segment set; wherein the straight line detection method in the key frame matching can detect complex segmented video similar segments, realizes accurate positioning of the similar segments, and is more suitable for application in engineering practice environment.

Description

Technical Field

[0001] This invention relates to the field of multimedia technology, and more specifically to a method for searching similar video segments based on line detection. Background Technology

[0002] With the rapid development and application of multimedia technology, video-related application scenarios are increasing, especially with the rise of self-media, video has entered people's daily lives. Video production has become simpler; for example, editing and processing one or more videos can generate new video content. Therefore, video service platforms need to perform duplicate detection on video content to determine whether the video is similar to other content and reduce the risk of copyright issues.

[0003] The patent "CN202010245319" entitled "A Method for Searching for Similar Video Segments" uses deep learning features based on image content to calculate video similarity and complete the search. A pair of similar frames needs to have a similarity score higher than a preset threshold. Subsequent processing is then performed on the similar frame pairs to obtain similar segments, achieving accurate location of similar segments. However, this invention also has certain shortcomings. The required preset threshold is difficult to determine in advance. If it is set too high, the similarity score may fall below the threshold after video editing such as cropping, scaling, or compression, thus failing to detect the similarity. If it is set too low, irrelevant videos may exceed the threshold, resulting in incorrect matches. Furthermore, when searching for matching points, dynamic programming or conditional randomization methods are required, resulting in a time complexity of at least O(n^2). 2 This will increase processing time and reduce efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a video similarity segment search method based on line detection. The line detection method in keyframe matching can detect complex segmented video similar segments and achieve accurate positioning of similar segments. At the same time, it avoids the problems caused by using predefined similarity thresholds, takes into account the search positioning speed, and is more adaptable to engineering practice environments.

[0005] To solve the above technical problems, the present invention adopts the following technical solution: a video similar segment search method based on line detection, comprising the following steps:

[0006] S110. Extract keyframes from the query video to obtain a keyframe set of the query video. Extract keyframes from the reference video in the video library to obtain a keyframe set of the reference video.

[0007] S120. Extract the image features of each keyframe in the keyframe set of the query video to obtain the image feature set of the query video. Extract the image features of each keyframe in the keyframe set of the reference video to obtain the image feature set of the reference video.

[0008] S130. Compare each image feature in the query video image feature set, find the K image features in the reference video image feature set that are most similar to the feature value of this image, obtain the feature candidate set and save it;

[0009] S140. Draw the feature candidate set into a two-dimensional image, perform line segment detection on the two-dimensional image using a line segment detection method to obtain candidate line segments, and then filter the candidate line segments that pass the detection to obtain a candidate line segment set.

[0010] S150. Handling of anomalies in candidate line segments within the candidate line segment set;

[0011] S160. Based on the candidate line segment set after handling the abnormal situation, locate similar segments in the query video and the reference video.

[0012] In the aforementioned video similarity segment search method based on line detection, step S110 uses a uniform sampling method to extract keyframes from the query video and the reference video. Both the query video and the reference video are composed of multiple consecutive frames. A fixed number of frames is preset, and a keyframe is extracted every preset number of frames to obtain the keyframe sets of the query video and the reference video. The length of the keyframe set of the query video is M, and the length of the keyframe set of the reference video is N. Each keyframe in the keyframe set corresponds to a video segment with a preset fixed frame interval. Extracting keyframes can also extract audio segments within the corresponding time slice to form corresponding audio-video information pairs (keyframe, audio segment). The combination of the two serves as the description information of the time slice.

[0013] In the aforementioned video similarity segment search method based on line detection, step S120 uses a traditional method to extract image features of keyframes. This involves extracting image features from each keyframe in the query video keyframe set to obtain an image feature set for the query video; and extracting image features from each keyframe in the reference video keyframe set to obtain an image feature set for the reference video. Specifically, the traditional method can be SIFT features or a deep learning-based method to obtain embedding vectors. Keyframe feature extraction may also include extracting corresponding audio feature information, combining the feature information of the keyframe images and audio feature information through methods such as weighting and concatenation to form a feature description of the corresponding time slice.

[0014] The aforementioned video similar segment search method based on line detection, wherein step S130 includes:

[0015] S131. For each keyframe Q in the query video keyframe set... i , keyframe Q i The corresponding image features are compared with the image features in the reference video image feature set to find the keyframe Q in the reference video image feature set. i The K most similar image features are identified, and the reference video keyframes corresponding to the K most similar image features are used as feature candidates. After the search is completed, a set of feature candidates is obtained.

[0016] S132. The method for obtaining similarity values ​​corresponds to the method for extracting image features of keyframes, and SIFT can be used for calculation; when extracting image features of keyframes using deep learning-based embedding vector methods, cosine similarity can be calculated.

[0017] S133. Save the feature candidate set and similarity values ​​as a list of binary pairs Q, Q{(r i1 s i1 ), , ..., (r ij s ij ), ..., (r iK s ，K )}, its r ij Indicates with Q i The index of the j-th similar reference video keyframe, s ij Indicates with Q i The similarity value of the j-th similar reference video keyframe.

[0018] The aforementioned video similar segment search method based on line detection, wherein step S140 includes:

[0019] S141. Draw a two-dimensional image I according to the list Q. The horizontal axis of the two-dimensional image I is the sequence number of the query video key frame, and the vertical axis of the two-dimensional image I is the sequence number of the reference video key frame. Each query video key frame corresponds to K reference video key frames. There are K points on each query video key frame sequence number on the two-dimensional image, where each point corresponds to the sequence number of a reference video key frame.

[0020] S142. Line Segment Detection: Weighted Hough line detection is used to find all candidate line segments that meet the conditions based on the points in the 2D image I. The candidate line segments are composed of points in the 2D image I. The weight of each point is taken as the similarity value between the query keyframe and the reference keyframe corresponding to that point. Points with higher similarity values ​​have higher weights. Points with higher similarity values ​​can form candidate line segments in priority over points with lower similarity values, resulting in a candidate line segment set CandSet1. When using Hough line detection, similarity weights can also be omitted, that is, each point has an equal weight.

[0021] S143. Each candidate line segment in the candidate line segment set CandSet1 also contains a set of matching points. Each matching point in the set of matching points is a tuple (i,j), where i is the index number of the query video keyframe and j is the index number of the reference video keyframe.

[0022] S144. Perform coarse filtering on the above candidate line segment set CandSet1, and retain the candidate line segments under the following conditions: the number of query video keyframes of the candidate line segment is not less than A, and the slope of the candidate line segment is between [1 / k, k], to obtain the candidate line segment set CandSet2.

[0023] S145. Further filter the candidate line segment set CandSet2. When the span between the start and end matching points of the candidate line segments in CandSet2 is large, further filtering is required:

[0024] Based on the matching point set of each candidate line segment in the candidate line segment set CandSet2, calculate its span D and number of points X. According to the Helmholtz criterion, the lower the probability of a randomly generated event, the stronger the significance of the event. The reference video keyframes in one type of matching point set are approximated as an increasing subsequence. The probability P of an increasing subsequence of length x not less than X in a random sequence of length D is used as the filtering condition. The reference video keyframes in the other type of matching point set are approximated as a decreasing subsequence. The probability P of a decreasing subsequence of length x not less than X in a random sequence of length D is used as the filtering condition. The probability P is less than a certain preset threshold p.

[0025] The method for calculating probability P(x>=X):

[0026] P(x>=X)=1-NormCDF(X,mean,var);

[0027] Where NormCDF is the standard normal cumulative distribution function, mean is its mean, and var is its variance. var = 0.819D 1 / 3 ;

[0028] After filtering, the candidate line segment set CandSet3 is obtained.

[0029] The aforementioned video similarity segment search method based on line detection includes two abnormal cases in step S150: one is that there is a discontinuity within the candidate line segment, and the other is that the candidate line segments overlap. The candidate line segment set CandSet3 is further processed to address these two abnormal cases:

[0030] S151. Calculate the similarity of each candidate line segment in the candidate line segment set CandSet3 as the score, and sort all candidate line segments in the candidate line segment set CandSet3 according to the score from high to low to obtain the list CandList1.

[0031] S152. Handling Discontinuities within Candidate Line Segments: A preset maximum interval n is used. Each candidate in CandList1 undergoes a truncation check. If the distance between two adjacent points in the candidate line segment's matching point set exceeds the preset maximum interval n, the line segment is considered broken. From these two adjacent points, the candidate line segment is divided into two sub-candidate line segments. The first sub-candidate line segment undergoes coarse and further filtering. If it meets the conditions, it is added to the result list CandList2; otherwise, it is discarded. The second sub-candidate line segment undergoes coarse and further filtering. If it meets the conditions, it is added to the result list CandList2; otherwise, it is discarded. This process continues until all candidates in CandList1 have undergone truncation checks.

[0032] S153. Handling overlapping candidate line segments: Calculate the similarity of each candidate line segment in the result list CandList2 as a score. Sort all candidate line segments in the result list CandList2 according to the score from high to low. Take one candidate line segment from the result list CandList2 in turn and check its overlap with the line segments in the result list CandList2 that are sorted after it. If there is an overlap, cut off the corresponding overlapping part of the candidate line segment. Continue until every candidate line segment in the result list CandList2 has been taken out and judged. Finally, obtain the result list ResultList. The candidate line segments in the result list ResultList are then truncated. The order of handling discontinuities within candidate line segments and handling overlapping candidate line segments can be reversed.

[0033] The aforementioned video similarity segment search method based on line detection involves locating similar segments in the query video and the reference video based on each candidate line segment in the result list ResultList.

[0034] Compared with existing technologies, the innovative technical points of this invention are as follows: based on the image feature set, the feature values ​​are drawn into a two-dimensional image, and the Hough line detection method is used to detect the two-dimensional image. The Hough line detection method has linear complexity, which effectively improves the detection rate. The keyframe matching problem is transformed into a line segment detection problem, and the Helmholtz criterion is used for filtering. After post-processing, the matching segments are obtained, avoiding problems caused by using predefined similarity thresholds. The designed line segment filtering method and post-processing method both have linear computational complexity, thus enabling efficient operation of the detection task. Attached Figure Description

[0035] Figure 1 This is a flowchart of the present invention;

[0036] Figure 2 This is a schematic diagram of the line segment detection results described in this invention;

[0037] Figure 3 This is a schematic diagram illustrating a case where the span of a candidate line segment is relatively large, as described in this invention.

[0038] Figure 4 This is a schematic diagram illustrating the discontinuous line segment detection process described in this invention.

[0039] Figure 5 This is a schematic diagram illustrating the overlapping of line segments in the line segment detection described in this invention;

[0040] Figure 6 This is a schematic diagram of the truncation operation when line segments overlap in the line segment detection described in this invention;

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0042] Embodiment 1 of the present invention: A video similar segment search method based on line detection, comprising the following steps:

[0043] S110. Extract keyframes from the query video to obtain a keyframe set for the query video. Extract keyframes from the reference video in the video library to obtain a keyframe set for the reference video. Use a uniform sampling method to extract keyframes from the query video and the reference video. Both the query video and the reference video are composed of multiple consecutive frames. Preset a fixed number of frames. Extract one keyframe every preset number of frames to obtain the keyframe sets for the query video and the reference video. The length of the keyframe set for the query video is M, and the length of the keyframe set for the reference video is N. Each keyframe in the keyframe set corresponds to a video segment with a preset fixed number of frame intervals.

[0044] S120. Extract image features from each keyframe in the keyframe set of the query video to obtain the image feature set of the query video. Extract image features from each keyframe in the keyframe set of the reference video to obtain the image feature set of the reference video. The traditional method can specifically be SIFT features or a deep learning-based method to obtain the embedding vector.

[0045] S130. Compare each image feature in the query video image feature set, find the K image features in the reference video image feature set that are most similar to this image feature value, obtain the feature candidate set and save it; the steps are as follows:

[0046] S131. For each keyframe Q in the query video keyframe set... i , keyframe Q i The corresponding image features are compared with the image features in the reference video image feature set to find the keyframe Q in the reference video image feature set. i The K most similar image features are identified, and the reference video keyframes corresponding to the K most similar image features are used as feature candidates. After the search is completed, a set of feature candidates is obtained.

[0047] S132. The method for obtaining similarity values ​​corresponds to the method for extracting image features of keyframes, and SIFT can be used for calculation; when extracting image features of keyframes using deep learning-based embedding vector methods, cosine similarity can be calculated.

[0048] S133. Save the feature candidate set and similarity values ​​as a list of binary pairs Q, Q{(r i1 s i1 ), , ..., (r ij s ij ), ..., (r iK s iK )} its r ij Indicates with Q i The index of the j-th similar reference video keyframe, s ij Indicates with Q i Similarity value of the j-th similar reference video keyframe

[0049] S140. Draw the feature candidate set into a two-dimensional image, perform line segment detection on the two-dimensional image using a line segment detection method to obtain candidate line segments, and then filter the candidate line segments that pass the detection to obtain a candidate line segment set; the steps are as follows:

[0050] S141. Draw a two-dimensional image I according to the list Q. The horizontal axis of the two-dimensional image I is the sequence number of the query video key frame, and the vertical axis of the two-dimensional image I is the sequence number of the reference video key frame. Each query video key frame corresponds to K reference video key frames. There are K points on each query video key frame sequence number on the two-dimensional image, where each point corresponds to the sequence number of a reference video key frame.

[0051] S142, Line Segment Detection: Weighted Hough is used to detect lines. Based on the points in the two-dimensional image I, all candidate line segments that meet the conditions are found. The candidate line segments are composed of points in the two-dimensional image I. The weight of each point is taken as the similarity value between the query keyframe and the reference keyframe corresponding to that point. Points with higher similarity values ​​have higher weights. Points with higher similarity values ​​can form candidate line segments in priority over points with lower similarity values, thus obtaining the candidate line segment set CandSet1.

[0052] S143. Each candidate line segment in the candidate line segment set CandSet1 also contains a set of matching points. Each matching point in the set of matching points is a tuple (i,j), where i is the index number of the query video keyframe and j is the index number of the reference video keyframe.

[0053] S144. Perform coarse filtering on the above candidate line segment set CandSet1, and retain the candidate line segments under the following conditions: the number of query video keyframes of the candidate line segment is not less than A, and the slope of the candidate line segment is between [1 / k, k], to obtain the candidate line segment set CandSet2.

[0054] S145. Further filter the candidate line segment set CandSet2. When the span between the start and end matching points of the candidate line segments in CandSet2 is large, such as... Figure 3 Further filtering is needed;

[0055] Based on the matching point set of each candidate line segment in the candidate line segment set CandSet2, calculate its span D and number of points X. According to the Helmholtz criterion, the lower the probability of a randomly generated event, the stronger the significance of the event. The reference video keyframes in one type of matching point set are approximated as an increasing subsequence. The probability P of an increasing subsequence of length x not less than X in a random sequence of length D is used as the filtering condition. The reference video keyframes in the other type of matching point set are approximated as a decreasing subsequence. The probability P of a decreasing subsequence of length x not less than X in a random sequence of length D is used as the filtering condition. The probability P is less than a certain preset threshold p.

[0056] The method for calculating probability P(x>=X):

[0057] P(x>=X)=1-NormCDF(X,mean,var);

[0058] Where NormCDF is the standard normal cumulative distribution function, mean is its mean, and var is its variance. var = 0.819D 1 / 3 ;

[0059] After filtering, the candidate line segment set CandSet3 is obtained;

[0060] S150. Handling of anomalies in the candidate line segment set; there are two types of anomalies, one of which is the appearance of discontinuities within the candidate line segment (see details). Figure 4 Another scenario involves overlapping candidate line segments (see details). Figure 5 To address these two abnormal situations, the candidate line segment set CandSet3 is further processed:

[0061] S151. Calculate the similarity of each candidate line segment in the candidate line segment set CandSet3 as the score, and sort all candidate line segments in the candidate line segment set CandSet3 according to the score from high to low to obtain the list CandList1.

[0062] S152. Handling Discontinuities within Candidate Line Segments: A preset maximum interval n is used. Each candidate in CandList1 undergoes a truncation check. If the distance between two adjacent points in the candidate line segment's matching point set exceeds the preset maximum interval n, the line segment is considered broken. From these two adjacent points, the candidate line segment is divided into two sub-candidate line segments. The first sub-candidate line segment undergoes coarse and further filtering. If it meets the conditions, it is added to the result list CandList2; otherwise, it is discarded. The second sub-candidate line segment undergoes coarse and further filtering. If it meets the conditions, it is added to the result list CandList2; otherwise, it is discarded. This process continues until all candidates in CandList1 have undergone truncation checks.

[0063] S153. Handling overlapping candidate line segments: For each candidate line segment in the result list CandList2, calculate the sum of similarities of each point in the matching point set of the candidate line segment as a score. Sort all candidate line segments in the result list CandList2 according to the score from high to low. Take one candidate line segment from the result list CandList2 in turn and check its overlap with the line segments in the result list CandList2 that are ranked after it. If there is an overlap, perform a truncation operation to remove the corresponding overlapping part of the candidate line segment. Continue until every candidate line segment in the result list CandList2 has been taken out and judged. Finally, obtain the result list ResultList. The candidate line segments in the result list ResultList are then truncated and judged.

[0064] S160. Based on the candidate segment set after handling the abnormal situation, locate similar segments in the query video and the reference video; each candidate segment in the ResultList contains the correspondence between the query video and the reference video. Based on this correspondence, it can be converted into the correspondence of time segments, that is, locate similar segments in the query video and the reference video.

[0065] Embodiment 2 of the present invention: A video similar segment search method based on line detection, comprising the following steps:

[0066] S110. Extract keyframes from the query video to obtain a keyframe set for the query video. Extract keyframes from the reference video in the video library to obtain a keyframe set for the reference video. Use a uniform sampling method to extract keyframes from the query video and the reference video. Both the query video and the reference video are composed of multiple consecutive frames. Preset a fixed number of frames. Extract one keyframe every preset number of frames to obtain the keyframe sets for the query video and the reference video. The length of the keyframe set for the query video is M, and the length of the keyframe set for the reference video is N. Each keyframe in the keyframe set corresponds to a video segment with a preset fixed number of frame intervals.

[0067] S120. Extract image features from each keyframe in the keyframe set of the query video to obtain the image feature set of the query video. Extract image features from each keyframe in the keyframe set of the reference video to obtain the image feature set of the reference video. The traditional method can specifically be SIFT features or a deep learning-based method to obtain the embedding vector.

[0068] S130. Compare each image feature in the query video image feature set, find the K image features in the reference video image feature set that are most similar to this image feature value, obtain the feature candidate set and save it; the steps are as follows:

[0069] S131. For each keyframe Q in the query video keyframe set... i , keyframe Q i The corresponding image features are compared with the image features in the reference video image feature set to find the keyframe Q in the reference video image feature set. i The K most similar image features are identified, and the reference video keyframes corresponding to the K most similar image features are used as feature candidates. After the search is completed, a set of feature candidates is obtained.

[0070] S132. The method for obtaining similarity values ​​corresponds to the method for extracting image features of keyframes, and SIFT can be used for calculation; when extracting image features of keyframes using deep learning-based embedding vector methods, cosine similarity can be calculated.

[0071] S133. Save the feature candidate set and similarity values ​​as a list of binary pairs Q, Q{(r i1 S i1 ),,...,(r ij s ij ), ..., (r iK s iK )} its r ij Indicates with Q i The index of the j-th similar reference video keyframe, s ij Indicates with Q i Similarity value of the j-th similar reference video keyframe

[0072] S140. Draw the feature candidate set into a two-dimensional image, perform line segment detection on the two-dimensional image using a line segment detection method to obtain candidate line segments, and then filter the candidate line segments that pass the detection to obtain a candidate line segment set; the steps are as follows:

[0073] S141. Draw a two-dimensional image I based on the list Q. The horizontal axis of the two-dimensional image I represents the sequence number of the query video keyframe, and the vertical axis represents the sequence number of the reference video keyframe. Each query video keyframe corresponds to K reference video keyframes. Each query video keyframe sequence number on the two-dimensional image has K points, where each point corresponds to a reference video keyframe sequence number. For details on the two-dimensional image drawn when K is 3, see [link to image]. Figure 2 ;

[0074] S142, Line Segment Detection: Weighted Hough is used to detect lines. Based on the points in the two-dimensional image I, all candidate line segments that meet the conditions are found. The candidate line segments are composed of points in the two-dimensional image I. The weight of each point is taken as the similarity value between the query keyframe and the reference keyframe corresponding to that point. Points with higher similarity values ​​have higher weights. Points with higher similarity values ​​can form candidate line segments in priority over points with lower similarity values, thus obtaining the candidate line segment set CandSet1.

[0075] S143. Each candidate line segment in the candidate line segment set CandSet1 also contains a set of matching points. Each matching point in the set of matching points is a tuple (i,j), where i is the index number of the query video keyframe and j is the index number of the reference video keyframe.

[0076] S144. Perform coarse filtering on the above candidate line segment set CandSet1, and retain the candidate line segments under the following conditions: the number of query video keyframes of the candidate line segment is not less than A, and the slope of the candidate line segment is between [1 / k, k], to obtain the candidate line segment set CandSet2; where A can be 10 and k can be 8.

[0077] S145. Further filter the candidate line segment set CandSet2. When the span between the start and end matching points of the candidate line segments in CandSet2 is large, further filtering is required:

[0078] Based on the matching point set of each candidate line segment in the candidate line segment set CandSet2, calculate its span D and number of points X. According to the Helmholtz criterion, the lower the probability of a randomly generated event, the stronger the significance of the event. The reference video keyframes in one type of matching point set are approximated as an increasing subsequence. The probability P of an increasing subsequence of length x not less than X appearing in a random sequence of length D is used as the filtering condition. The reference video keyframes in the other type of matching point set are approximated as a decreasing subsequence. The probability P of a decreasing subsequence of length x not less than X appearing in a random sequence of length D is used as the filtering condition. The probability P is less than a certain preset threshold p; p can be 1%.

[0079] The method for calculating probability P(x>=X):

[0080] P(x>=X)=1-NormCDF(X,mean,var);

[0081] Where NormCDF is the standard normal cumulative distribution function, mean is its mean, and var is its variance. var = 0.819D 1 / 3 ;

[0082] After filtering, the candidate line segment set CandSet3 is obtained;

[0083] S150. Handling of anomalies in the candidate line segment set; There are two types of anomalies: one is that there is a discontinuity within the candidate line segment, and the other is that there is an overlap between candidate line segments. Further processing is performed on the candidate line segment set CandSet3 for these two anomalies:

[0084] S151. Calculate the similarity of each candidate line segment in the candidate line segment set CandSet3 as the score, and sort all candidate line segments in the candidate line segment set CandSet3 according to the score from high to low to obtain the list CandList1.

[0085] S152. Handling Discontinuities within Candidate Line Segments: A preset maximum interval n is used. Each candidate in CandList1 undergoes a truncation check. If the distance between two adjacent points in the candidate line segment's matching point set exceeds the preset maximum interval n, the line segment is considered broken. From these two adjacent points, the candidate line segment is divided into two sub-candidate line segments. The first sub-candidate line segment undergoes coarse and further filtering. If it meets the conditions, it is added to the result list CandList2; otherwise, it is discarded. The second sub-candidate line segment undergoes coarse and further filtering. If it meets the conditions, it is added to the result list CandList2; otherwise, it is discarded. This process continues until all candidates in CandList1 have undergone truncation checks.

[0086] S153. Handling Candidate Line Segment Overlap: For each candidate line segment in the result list CandList2, calculate the sum of similarities of each point in the matching point set of the candidate line segment as a score. Sort all candidate line segments in the result list CandList2 according to the score from high to low. Take one candidate line segment from the result list CandList2 in turn and perform overlap detection with the line segments in the result list CandList2 that are ranked after it. If overlap exists, perform a truncation operation to remove the corresponding overlapping part of the candidate line segment. See details. Figure 6 The process continues until every candidate line segment in the result list CandList2 has been extracted and judged, finally resulting in the result list ResultList. The candidate line segments in the result list ResultList are then truncated and judged S160. Based on the set of candidate line segments after handling abnormal situations, similar segments in the query video and the reference video are located. Each candidate line segment in the result list ResultList contains the correspondence between the query video and the reference video. Based on this correspondence, it can be converted into the correspondence of time segments, that is, the similar segments in the query video and the reference video are located.

[0087] The working principle of one embodiment of the present invention is as follows: Keyframes of the query video are extracted to obtain a keyframe set; keyframes of a reference video in the video library are extracted to obtain a keyframe set; image features of each keyframe in the keyframe set of the query video are extracted to obtain an image feature set; image features of each keyframe in the keyframe set of the reference video are extracted to obtain an image feature set; for each image feature in the image feature set of the query video, the K most similar image features in the image feature set of the reference video are found to obtain a candidate feature set and saved; the candidate feature set is drawn as a two-dimensional image, and line segment detection is performed on the two-dimensional image to obtain candidate line segments; the candidate line segments that pass the detection are then filtered to obtain a candidate line segment set; anomalies in the candidate line segment set are processed; based on the candidate line segment set after processing anomalies, similar segments in the query video and the reference video are located.

Claims

1. A video similarity segment search method based on line detection, characterized in that, Includes the following steps: S110. Extract keyframes from the query video to obtain a keyframe set of the query video. Extract keyframes from the reference video in the video library to obtain a keyframe set of the reference video. S120. Extract the image features of each keyframe in the keyframe set of the query video to obtain the image feature set of the query video. Extract the image features of each keyframe in the keyframe set of the reference video to obtain the image feature set of the reference video. S130. Compare each image feature in the query video image feature set, find the K image features in the reference video image feature set that are most similar to the feature value of this image, obtain the feature candidate set and save it; S140. The feature candidate set is drawn into a two-dimensional image. Line segment detection is performed on the two-dimensional image to obtain candidate line segments. The candidate line segments that pass the detection are then filtered to obtain a candidate line segment set. The line segment detection uses weighted Hough line detection. The weight of each point in the two-dimensional image is taken as the similarity value between the query keyframe and the reference keyframe corresponding to that point. Points with higher similarity values ​​are given priority over points with lower similarity values ​​to form candidate line segments, resulting in a candidate line segment set CandSet1. The candidate line segment set CandSet1 is coarsely filtered to obtain a candidate line segment set CandSet2. The candidate line segment set CandSet2 is further filtered according to the Helmholtz criterion to obtain a candidate line segment set CandSet3. S150. Process the anomalies of candidate line segments in the candidate line segment set CandSet3; the anomalies include discontinuities within candidate line segments and overlapping candidate line segments. S160. Based on the candidate line segment set after handling the abnormal situation, locate similar segments in the query video and the reference video.

2. The video similarity segment search method based on line detection according to claim 1, characterized in that, Step S110 uses a uniform sampling method to extract keyframes from the query video and the reference video to obtain a set of keyframes for the query video and the reference video.

3. The video similarity segment search method based on line detection according to claim 1, characterized in that, Step S120 uses SIFT or deep learning to extract image features of key frames, extracts image features of each key frame in the key frame set of the query video, and obtains the image feature set of the query video. Extract the image features of each keyframe from the keyframe set of the reference video to obtain the image feature set of the reference video.

4. The video similarity segment search method based on line detection according to claim 1, characterized in that, There are two abnormal situations in step S150: one is that there is a discontinuity within the candidate line segment, and the other is that the candidate line segments overlap. The candidate line segment set CandSet3 is further processed to address these two abnormal situations: S151. Calculate the similarity of each candidate line segment in the candidate line segment set CandSet3 as the score, and sort all candidate line segments in the candidate line segment set CandSet3 according to the score from high to low to obtain the list CandList1. S152. Handling Discontinuities within Candidate Line Segments: A preset maximum interval n is used. Each candidate in CandList1 undergoes a truncation check. If the distance between two adjacent points in the candidate line segment's matching point set exceeds the preset maximum interval n, the line segment is considered broken. From these two adjacent points, the candidate line segment is divided into two sub-candidate line segments. The first sub-candidate line segment undergoes coarse and further filtering. If it meets the conditions, it is added to the result list CandList2; otherwise, it is discarded. The second sub-candidate line segment undergoes coarse and further filtering. If it meets the conditions, it is added to the result list CandList2; otherwise, it is discarded. This process continues until all candidates in CandList1 have undergone truncation checks. S153. Handling overlapping candidate line segments: Calculate the similarity of each candidate line segment in the result list CandList2 as a score. Sort all candidate line segments in the result list CandList2 according to the score from high to low. Take one candidate line segment from the result list CandList2 in turn and perform overlap detection with the line segments in the result list CandList2 that are sorted after it. If there is an overlap, cut off the corresponding overlapping part of the candidate line segment. Continue until every candidate line segment in the result list CandList2 has been taken out and judged. Finally, obtain the result list ResultList. The candidate line segments in the result list ResultList are then truncated and judged.

5. The video similarity segment search method based on line detection according to claim 4, characterized in that, Each candidate line segment in the ResultList contains a correspondence between the query video and the reference video. Based on this correspondence, similar segments in the query video and the reference video are located.

Images