Search method and search device for video material segments, electronic device

Abstract

This application relates to the field of multimedia data analysis technology, and discloses a method, retrieval device, and electronic device for retrieving video clips. The retrieval method includes: extracting a first semantic feature sequence with semantic invariance from the query video; obtaining a second semantic feature sequence of candidate video clips to be retrieved; matching the first and second semantic feature sequences to determine multiple candidate matching frame pairs between the query video and the candidate video clips; performing temporal alignment and deduplication processing on the multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video clips. This application can improve the accuracy of retrieving video clips.

Description

Technical Field

[0001] This application relates to the field of multimedia data analysis technology, such as a method and device for retrieving video clips, and an electronic device. Background Technology

[0002] Currently, with the development of short video platforms, video footage is being used to create new videos. Therefore, in practical applications, such as copyright infringement monitoring and tracing the source of user-generated content, it is necessary to retrieve and identify the video footage segments being used in the final query video.

[0003] In related technologies, visual feature matching (e.g., perceptual hashing, feature point matching) is typically used to calculate the similarity between the visual features of the query video and the video material, and to determine the matching result between the query video and the video material.

[0004] In the process of implementing the embodiments of this disclosure, it was found that the related technology has at least the following problems: When searching for videos based on secondary creation of video materials, spatial and temporal editing is performed on the video materials. Spatial editing includes: scaling, cropping, rotating, adjusting color filters, modifying brightness / contrast, and adding various forms of visual occlusion (such as subtitles, logos, stickers, picture-in-picture, etc.). Temporal editing includes: non-linear processing such as video acceleration, slow motion, and frame skipping. Spatial and temporal editing can fundamentally change visual characteristics, making it easy for the matched results between the determined query video and the video material to have misjudgments and omissions, resulting in low accuracy. Therefore, how to improve the accuracy of retrieving video material segments has become an urgent technical problem to be solved.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0007] This disclosure provides a method, device, and electronic device for retrieving video clips, which can improve the accuracy of retrieving video clips.

[0008] In some embodiments, a method for retrieving video clips includes: extracting a first semantic feature sequence with semantic invariance from a query video; obtaining a second semantic feature sequence of candidate video clips to be retrieved; matching the first semantic feature sequence and the second semantic feature sequence to determine multiple candidate matching frame pairs between the query video and the candidate video clips; performing temporal alignment and deduplication processing on the multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video clips.

[0009] Optionally, extracting a first semantic feature sequence with semantic invariance from the query video includes: sampling keyframes of the query video to determine multiple key images; extracting high-dimensional feature vectors of the multiple key images using a pre-built feature extraction model; and normalizing the high-dimensional feature vectors to obtain the first semantic feature sequence.

[0010] Optionally, matching the first semantic feature sequence and the second semantic feature sequence to determine multiple candidate matching frame pairs between the query video and the candidate video material includes: calculating the cosine similarity between each semantic feature in the first semantic feature sequence and the second semantic feature sequence; and determining feature pairs with a cosine similarity greater than a preset threshold as candidate matching frame pairs.

[0011] Optionally, the candidate matching frame pairs include the query frame timestamp, the source frame timestamp, and similarity. Temporal alignment and deduplication are performed on multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video source. This includes: calculating the time difference between the query frame and the source frame in each candidate matching frame pair based on the query frame timestamp and the source frame timestamp; statistically analyzing the dispersion of the time differences of all candidate matching frame pairs to determine whether a preset constant distribution condition is met; if the constant distribution condition is met, the time difference is divided into corresponding time bins, and the matching result is determined based on the valid matching frame pairs within each time bin; wherein, for multiple candidate matching frame pairs within the same time bin and corresponding to the same query frame, only the one with the highest similarity is retained as the valid matching frame pair for that query frame within that time bin; if the constant distribution condition is not met, a dynamic time warping algorithm or the longest common subsequence algorithm is used to perform non-linear temporal alignment on the first semantic feature sequence and the second semantic feature sequence to determine the matching result.

[0012] Optionally, the matching result is determined based on the valid matching frame pairs within each time bin, including: summing the similarities of all valid matching frame pairs within each time bin and using them as the scores for each time bin; determining the time difference corresponding to the time bin with the highest score as the temporal alignment offset of the query video relative to the candidate video material, and using the temporal alignment offset as the matching result between the query video and the candidate video material.

[0013] Optionally, the matching result includes the temporal alignment offset of the query video relative to the candidate video material; after using the temporal alignment offset as the matching result between the query video and the candidate video material, the retrieval method further includes: filtering out a subset of matching frame pairs that meet the temporal alignment offset from multiple candidate matching frame pairs; calculating the sum of similarities of all matching frame pairs in the subset of matching frame pairs; normalizing the sum of similarities with the time span or total number of frames occupied by the subset of matching frame pairs in the query video to obtain the confidence score of the matching result.

[0014] Optionally, after obtaining the confidence level of the matching result, the retrieval method further includes: performing secondary temporal verification on the first semantic feature sequence and the second semantic feature sequence when the confidence level of the matching result is within a set verification interval; and updating the matching result based on the result of the secondary temporal verification.

[0015] In some embodiments, a retrieval device for video clips includes: an extraction module for extracting a first semantic feature sequence with semantic invariance from a query video; an acquisition module for acquiring a second semantic feature sequence of candidate video clips to be retrieved; a matching module for matching the first semantic feature sequence and the second semantic feature sequence to determine multiple candidate matching frame pairs between the query video and the candidate video clips; and a determination module for performing temporal alignment and deduplication processing on the multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video clips.

[0016] In some embodiments, a device for retrieving video clips includes a processor and a memory storing program instructions, the processor being configured to execute the video clip retrieval method described above when running the program instructions.

[0017] In some embodiments, the electronic device includes: a device body; and the aforementioned retrieval device for video clips, which is mounted on the device body.

[0018] The method, apparatus, and electronic device for retrieving video clips provided in this disclosure can achieve the following technical effects: In this embodiment, when detecting video footage used in a query video, a first semantic feature sequence with semantic invariance is first extracted from the query video. Then, the first semantic feature sequence is matched with a second semantic feature sequence of the candidate video footage to determine multiple candidate matching frame pairs between the query video and the candidate video footage. This reduces the impact of spatial domain editing operations on the detection and matching process compared to matching video footage based on visual features. After determining multiple candidate matching frame pairs, temporal alignment and deduplication are performed on these pairs to determine the final matching result between the query video and the candidate video footage. This reduces the impact of temporal domain editing operations on the detection and matching process. Therefore, this embodiment can improve the accuracy of retrieving video footage segments.

[0019] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a schematic diagram of an electronic device provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of a method for retrieving video clips provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of another method for retrieving video clips provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of another method for retrieving video clips provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of a device for retrieving video clips provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of a device for retrieving video clips provided in an embodiment of this disclosure. Detailed Implementation

[0021] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0022] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0023] Unless otherwise stated, the term "multiple" means two or more.

[0024] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0025] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0026] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in the present disclosure can be combined with each other.

[0028] like Figure 1 As shown, the electronic device 100 provided in this embodiment includes a device body 110 and a retrieval device 500 (600) for video clips.

[0029] Specifically, a retrieval device 500 (600) for video clips is provided on the device body 110.

[0030] Optionally, the retrieval device 600 for video clips includes a processor. When detecting video clips used in a query video, the processor can extract a first semantic feature sequence with semantic invariance from the query video and match the first semantic feature sequence with a second semantic feature sequence of candidate video clips to be retrieved in a base database to determine multiple candidate matching frame pairs between the query video and the candidate video clips. After determining multiple candidate matching frame pairs, the processor can perform temporal alignment and deduplication processing on the multiple candidate matching frame pairs to determine the final matching result between the query video and the candidate video clips.

[0031] In conjunction with the aforementioned electronic device, this disclosure provides a method for retrieving video clips, wherein the execution entity of this retrieval method can be a processor of a device for a method of retrieving video clips. Figure 2 As shown, the retrieval method includes: S201, The processor extracts the first semantic feature sequence with semantic invariance from the query video.

[0032] Specifically, semantic features with semantic invariance can still stably express the semantic information of a video even after spatial domain editing operations such as image scaling, cropping, rotation, color filter adjustment, and brightness / contrast modification.

[0033] Specifically, the query video can first be processed into a continuous sequence of image frames, and then the sequence of image frames can be sequentially input into a pre-built feature extraction model to extract semantic features in order to determine the first semantic feature sequence of the query video.

[0034] S202, the processor obtains the second semantic feature sequence of the candidate video material to be retrieved.

[0035] Specifically, the video material database is a database that stores a large number of video materials, and the candidate video materials to be searched are one or more video materials in the database.

[0036] Specifically, the second semantic feature sequence is a semantic feature sequence extracted from candidate video materials using the same extraction method as the first semantic feature sequence. The second semantic feature sequence is associated with the video materials and stored in the video material database. S203, the processor matches the first semantic feature sequence and the second semantic feature sequence to determine multiple candidate matching frame pairs between the query video and the candidate video material.

[0037] Specifically, by comparing the first semantic feature sequence of the query video and the second semantic feature sequence of the candidate video footage, it is possible to identify semantically similar frame pairs between the query video and the candidate video footage. These frame pairs correspond to similar scenes or content in the videos. Therefore, by matching the first semantic feature sequence and the second semantic feature sequence, multiple candidate matching frame pairs can be determined.

[0038] Specifically, a method for calculating semantic feature similarity (e.g., calculating cosine similarity or Euclidean distance) can be used to determine the similarity between each feature vector in the first semantic feature sequence and each feature vector in the second semantic feature sequence, and then frame pairs with similarity exceeding a set threshold are identified as candidate matching frame pairs.

[0039] Optionally, matching the first semantic feature sequence and the second semantic feature sequence to determine multiple candidate matching frame pairs between the query video and the candidate video material includes: calculating the cosine similarity between each semantic feature in the first semantic feature sequence and the second semantic feature sequence; and determining feature pairs with a cosine similarity greater than a preset threshold as candidate matching frame pairs.

[0040] S204, the processor performs time alignment and deduplication processing on multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video materials.

[0041] Specifically, due to the temporal nature of videos, the identified candidate matching frame pairs may exhibit temporal misalignment and duplication. Temporal alignment and deduplication processes can arrange these candidate matching frame pairs in the correct chronological order, reducing the impact of temporal editing operations on the query video's matching results and removing duplicate matches. Therefore, after identifying multiple candidate matching frame pairs, temporal alignment and deduplication processes are necessary to determine the final matching result between the query video and the candidate video materials.

[0042] Specifically, the matching frame pairs can be sorted according to the timestamp information of the frames in their respective videos, so that the candidate matching frame pairs are arranged in the order of video playback time, and multiple candidate matching frame pairs are time-aligned.

[0043] Specifically, by setting similarity thresholds and matching interval conditions, duplicate matching frame pairs in multiple candidate matching frame pairs can be removed, and the most representative and unique matching results can be retained, thus achieving deduplication of multiple candidate matching frame pairs.

[0044] Specifically, after time alignment and deduplication, the remaining candidate matching frame pairs constitute the final matching result between the query video and the candidate video materials.

[0045] In this embodiment, when detecting video footage used in a query video, a first semantic feature sequence with semantic invariance is first extracted from the query video. Then, the first semantic feature sequence is matched with a second semantic feature sequence of the candidate video footage to determine multiple candidate matching frame pairs between the query video and the candidate video footage. This reduces the impact of spatial domain editing operations on the detection and matching process compared to matching video footage based on visual features. After determining multiple candidate matching frame pairs, temporal alignment and deduplication are performed on these pairs to determine the final matching result between the query video and the candidate video footage. This reduces the impact of temporal domain editing operations on the detection and matching process. Therefore, this embodiment can improve the accuracy of retrieving video footage segments.

[0046] In some embodiments, extracting a first semantic feature sequence with semantic invariance from a query video includes: sampling keyframes of the query video to determine multiple key images; extracting high-dimensional feature vectors of the multiple key images using a pre-built feature extraction model; and normalizing the high-dimensional feature vectors to obtain the first semantic feature sequence.

[0047] Specifically, selecting representative and informative key frames from the query video to determine multiple key images can reduce the computational load of subsequent semantic feature extraction while retaining the main content information of the query video, which is beneficial to improving the efficiency of feature extraction.

[0048] Specifically, inter-frame differencing or color histogram calculation methods can be used to determine key images across multiple frames in the query video. For example, the pixel differences between adjacent frames in the query video can be calculated; if the difference exceeds a set threshold, the current frame is identified as a key frame, and the image corresponding to that frame is designated as the key frame image. Another example is calculating the color histogram of each frame in the query video; by comparing the differences in the color histograms of adjacent frames, key frames and key frame images can be determined.

[0049] Specifically, the feature extraction model is a deep learning model trained on a large-scale image dataset, such as a convolutional neural network (CNN). The CNN model can be ResNet, VGG, Inception, etc. Because deep learning models have powerful feature extraction capabilities and can learn high-level semantic features in images, they can be used to construct feature extraction models for extracting semantically invariant high-dimensional feature vectors.

[0050] Specifically, by normalizing the extracted high-dimensional feature vectors, the dimensional feature vectors of different keyframes can be made comparable, and the dimensional differences that may exist in the high-dimensional feature vectors can be eliminated.

[0051] In this embodiment, when extracting the first semantic feature sequence with semantic invariance from the query video, representative and informative key frames are first selected from the query video to determine multiple key images. Then, a feature extraction model is used to extract features from these multiple key images. This reduces the computational load when extracting semantic features and improves the efficiency of extracting the first semantic feature sequence.

[0052] In some embodiments, candidate matching frame pairs include a query frame timestamp, a source frame timestamp, and a similarity score. Temporal alignment and deduplication are performed on multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video source. This includes: calculating the time difference between the query frame and the source frame in each candidate matching frame pair based on the query frame timestamp and the source frame timestamp; statistically analyzing the dispersion of the time differences across all candidate matching frame pairs to determine if a preset constant distribution condition is met; if the constant distribution condition is met, dividing the time difference into corresponding time bins and determining the matching result based on the valid matching frame pairs within each time bin; wherein, for multiple candidate matching frame pairs within the same time bin and corresponding to the same query frame, only the one with the highest similarity is retained as the valid matching frame pair for that query frame within that time bin; if the constant distribution condition is not met, a dynamic time warping algorithm or a longest common subsequence algorithm is used to perform non-linear temporal alignment on the first semantic feature sequence and the second semantic feature sequence to determine the matching result.

[0053] Specifically, the query frame timestamp records the exact position of a query frame on the video timeline. The source frame timestamp records the position of the source frame matching the query frame on the source video timeline. Similarity reflects the degree of semantic similarity between the query frame and the source frame.

[0054] Specifically, the time difference between the query frame and the source frame in a candidate matching frame pair is an important indicator for measuring the relative positional relationship of the candidate matching frame pair on the time axis. By calculating the time difference, we can understand the distribution of different candidate matching frame pairs in time, so as to facilitate subsequent time alignment and deduplication of multiple candidate matching frame pairs.

[0055] Specifically, for each pair of candidate matching frames, the time difference between the query frame and the source frame in the candidate matching frame pair can be determined by subtracting the query frame timestamp from the source frame timestamp.

[0056] Specifically, the dispersion of time differences can be measured by calculating statistics such as variance and standard deviation. A smaller variance or standard deviation indicates a more concentrated distribution of time differences and less dispersion; a larger variance or standard deviation indicates a more dispersed distribution and greater dispersion. A preset constant distribution condition corresponds to a threshold range used to determine whether the distribution of time differences is relatively stable. If the dispersion of time differences is small, and most time differences fall within this threshold range, the constant distribution condition is considered met; otherwise, it is considered not to meet the constant distribution condition.

[0057] Specifically, if the time differences corresponding to multiple candidate matching frame pairs satisfy a constant distribution condition, it indicates that no temporal editing operation was performed when the query video was generated. Therefore, in this case, the time differences are divided into corresponding time bins, and the matching results are determined based on the valid matching frame pairs within each time bin.

[0058] Specifically, within a bin at the same time, there may be multiple source frames that match the same query frame. These source frames may be duplicates or have low similarity. By retaining the matching frame pair with the highest similarity, duplicate and low-quality matching results can be removed, improving the accuracy and effectiveness of matching.

[0059] Specifically, if the time differences corresponding to multiple candidate matching frame pairs do not satisfy a constant distribution condition, it indicates that temporal editing was performed when the query video was generated. Therefore, in this case, a dynamic time warping algorithm or the longest common subsequence algorithm is needed to perform non-linear temporal alignment on the first and second semantic feature sequences to determine the matching result. This can reduce the impact of temporal editing on the detection and matching.

[0060] Optionally, a dynamic time warping algorithm or a longest common subsequence algorithm is used to perform nonlinear temporal alignment on the first semantic feature sequence and the second semantic feature sequence to determine the matching result. This includes: calculating the time offset for each candidate matching frame pair under multiple different speed assumptions; performing binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the candidate video material; and outputting the matching result including the optimal playback speed and alignment position of the query video relative to the candidate video material.

[0061] Optionally, when using the dynamic time warping algorithm or the longest common subsequence algorithm for nonlinear time alignment, if the computational cost is too high, a restricted window strategy can be adopted to improve computational efficiency.

[0062] In this embodiment, the time difference between the query frame and the source frame is calculated to determine whether temporal editing was performed when the query video was generated. If temporal editing was performed, a dynamic time warping algorithm or the longest common subsequence algorithm is used to non-linearly align the first and second semantic feature sequences to determine the matching result. This reduces the impact of temporal editing on the detection and matching, improving the accuracy of the matching results.

[0063] In some embodiments, determining the matching result based on the valid matching frame pairs within each time bin includes: summing the similarities of all valid matching frame pairs within each time bin and using them as the scores for each time bin; determining the time difference corresponding to the time bin with the highest score as the temporal alignment offset of the query video relative to the candidate video material, and using the temporal alignment offset as the matching result between the query video and the candidate video material.

[0064] Specifically, by summing the similarities of valid matching frame pairs within each time bin and using this sum as the score for that time bin, the quality and strength of the matching within each time bin can be quantified. A higher score indicates a better matching situation within that time bin, and a higher semantic consistency between the query video and the candidate video material within that time range.

[0065] Specifically, the highest-scoring time bin indicates the best match between the query video and candidate video clips within that time frame. Determining the time difference corresponding to this time bin as the temporal alignment offset accurately reflects the relative positions of the query video and candidate video clips on the timeline, achieving temporal alignment. Therefore, the temporal alignment offset can be used as the matching result between the query video and candidate video clips.

[0066] This disclosure provides another method for retrieving video clips, such as... Figure 3 As shown, the retrieval method includes: S301, The processor extracts the first semantic feature sequence with semantic invariance from the query video.

[0067] S302, the processor obtains the second semantic feature sequence of the candidate video material to be retrieved.

[0068] S303, the processor matches the first semantic feature sequence and the second semantic feature sequence to determine multiple candidate matching frame pairs between the query video and the candidate video material.

[0069] S304, the processor performs temporal alignment and deduplication processing on multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video material; wherein, the matching result includes the temporal alignment offset of the query video relative to the candidate video material.

[0070] S305, the processor selects a subset of matching frame pairs that meet the timing alignment offset from multiple candidate matching frame pairs.

[0071] Specifically, filtering frame pairs that match the temporal alignment offset from multiple candidate matching frame pairs can more accurately reflect the matching relationship between the query video and the candidate video materials.

[0072] S306, the processor calculates the sum of similarities of all matching frame pairs in the subset of matching frame pairs, and normalizes the sum of similarities with the time span or total number of frames occupied by the subset of matching frame pairs in the query video to obtain the confidence of the matching result.

[0073] Specifically, similarity reflects the degree of semantic similarity between query frames and source video frames. Calculating the sum of similarities for all matching frame pairs in a subset of matching frames provides a comprehensive measure of the overall matching degree between the query video and candidate video source video within the time frame represented by that subset. A higher sum of similarities indicates a better match between the two within that time frame.

[0074] Specifically, there is a lack of a unified standard for directly using the sum of similarities to measure the reliability of matching results. Therefore, it is necessary to normalize the sum of similarities with the time span or total number of frames occupied by the subset of matched frame pairs in the query video, so as to map the confidence of the matching results to a unified range (e.g., [0, 1]), making the matching results of different query videos comparable.

[0075] Optionally, the confidence score of the matching result can be calculated according to the following expression: ; in, The confidence level of the matching results. The sum of similarities. To match the time span or total number of frames of a subset of frame pairs in the query video.

[0076] In this embodiment, after determining the matching result, the confidence level of the matching result is also calculated. This makes the matching results of different query videos comparable, allowing users to judge the quality of the search matching based on the confidence level and evaluate the matching results.

[0077] This disclosure provides another method for retrieving video clips, such as... Figure 4 As shown, the retrieval method includes: S401, The processor extracts the first semantic feature sequence with semantic invariance from the query video.

[0078] S402, the processor obtains the second semantic feature sequence of the candidate video material to be retrieved.

[0079] S403, the processor matches the first semantic feature sequence and the second semantic feature sequence to determine multiple candidate matching frame pairs between the query video and the candidate video material.

[0080] S404, the processor performs temporal alignment and deduplication processing on multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video material; wherein, the matching result includes the temporal alignment offset of the query video relative to the candidate video material.

[0081] S405, the processor selects a subset of matching frame pairs that meet the timing alignment offset from multiple candidate matching frame pairs.

[0082] S406, the processor calculates the sum of similarities of all matching frame pairs in the subset of matching frame pairs, and normalizes the sum of similarities with the time span or total number of frames occupied by the subset of matching frame pairs in the query video to obtain the confidence of the matching result.

[0083] S407, when the confidence level of the matching result is within the set verification range, the processor performs secondary temporal verification on the first semantic feature sequence and the second semantic feature sequence.

[0084] Specifically, the validation interval is set based on actual application scenarios and experience. For example, it can be set to [0.6, 0.8]. After obtaining the confidence level of the matching result, the confidence level of the matching result is compared with the set validation interval. If the confidence level falls within the set validation interval, it indicates that the current matching result has a certain degree of uncertainty; it is neither very reliable nor completely unreliable. Therefore, in this case, a second temporal validation is required for the first semantic feature sequence and the second semantic feature sequence.

[0085] Specifically, secondary temporal verification will re-examine the temporal correspondence between the two sequences at the semantic level. For example, a temporal alignment model in a deep learning network can be used to re-determine the temporal matching degree of the first semantic feature sequence and the second semantic feature sequence.

[0086] S408, the processor updates the matching result based on the result of the second timing verification.

[0087] Specifically, the reliability of the original matching results can be further determined based on the results of the second temporal verification. If the second temporal verification shows that the temporal match between the two semantic feature sequences is good, the confidence of the matching result can be increased. If the second temporal verification shows that the temporal match between the two semantic feature sequences is not ideal, the temporal alignment offset needs to be corrected or the confidence level needs to be decreased. Therefore, the matching results can be updated based on the results of the second temporal verification.

[0088] In this embodiment, after determining the confidence level of the matching result, it is further confirmed whether the confidence level of the matching result falls within a set verification interval. If it does, a second temporal verification is performed on the first and second semantic feature sequences, and the matching result is updated based on the result of the second temporal verification. This helps reduce misjudgments and makes the final determined matching result more reliable.

[0089] Combination Figure 5 As shown in the figure, this disclosure provides a retrieval device 500 for video clips, including: an extraction module 501, an acquisition module 502, a matching module 503, and a determination module 504. The extraction module 501 extracts a first semantic feature sequence with semantic invariance from the query video. The acquisition module 502 acquires a second semantic feature sequence of candidate video clips to be retrieved. The matching module 503 matches the first and second semantic feature sequences to determine multiple candidate matching frame pairs between the query video and the candidate video clips. The determination module 504 performs temporal alignment and deduplication processing on the multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video clips.

[0090] Combination Figure 6 As shown, this disclosure provides a video clip retrieval device 600, comprising a processor 601 and a memory 602. Optionally, the device may further include a communication interface 603 and a bus 604. The processor 601, communication interface 603, and memory 602 can communicate with each other via the bus 604. The communication interface 603 can be used for information transmission. The processor 601 can invoke logical instructions stored in the memory 602 to execute the video clip retrieval method described in the above embodiment.

[0091] Furthermore, the logic instructions in the aforementioned memory 602 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0092] The memory 602, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 601 executes functional applications and data processing by running the program instructions / modules stored in the memory 602, that is, it implements the video clip retrieval method in the above embodiments.

[0093] The memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 602 may include high-speed random access memory and may also include non-volatile memory.

[0094] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described method for retrieving video clips.

[0095] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0096] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0097] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0098] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for retrieving video clips, characterized in that, include: Extract the first semantic feature sequence with semantic invariance from the query video; Obtain the second semantic feature sequence of the candidate video material to be retrieved; The first semantic feature sequence and the second semantic feature sequence are matched to determine multiple candidate matching frame pairs between the query video and the candidate video material; Perform temporal alignment and deduplication on multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video materials.

2. The retrieval method according to claim 1, characterized in that, Extract the first semantic feature sequence with semantic invariance from the query video, including: Keyframe sampling is performed on the query video to determine multiple key images; High-dimensional feature vectors of multiple key images are extracted using a pre-built feature extraction model. The high-dimensional feature vectors are normalized to obtain the first semantic feature sequence.

3. The retrieval method according to claim 1, characterized in that, The first semantic feature sequence and the second semantic feature sequence are matched to determine multiple candidate matching frame pairs between the query video and the candidate video material, including: Calculate the cosine similarity between each semantic feature in the first semantic feature sequence and the second semantic feature sequence; Feature pairs with a cosine similarity greater than a preset threshold are identified as candidate matching frame pairs.

4. The retrieval method according to claim 1, characterized in that, Candidate matching frame pairs include query frame timestamp, source frame timestamp, and similarity. Perform temporal alignment and deduplication on multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video materials, including: Calculate the time difference between the query frame and the source frame in each candidate matching frame pair based on the query frame timestamp and the source frame timestamp. The dispersion of the time difference of all candidate matching frame pairs is statistically analyzed to determine whether the preset constant distribution condition is met. Under the condition of constant distribution, the time difference is divided into corresponding time bins, and the matching result is determined according to the valid matching frame pairs in each time bin; wherein, for multiple candidate matching frame pairs in the same time bin and corresponding to the same query frame, only the one with the highest similarity is retained as the valid matching frame pair of the query frame in that time bin. In cases where the constant distribution condition is not met, a dynamic time warping algorithm or the longest common subsequence algorithm is used to perform nonlinear temporal alignment on the first and second semantic feature sequences to determine the matching result.

5. The retrieval method according to claim 4, characterized in that, The matching results are determined based on the valid matching frame pairs within each time bin, including: The sum of similarities of all valid matching frame pairs within each time bin is calculated and used as the score for each time bin. The time difference corresponding to the time bin with the highest score is determined as the temporal alignment offset of the query video relative to the candidate video material, and the temporal alignment offset is used as the matching result between the query video and the candidate video material.

6. The retrieval method according to any one of claims 1 to 5, characterized in that, The matching results include the temporal alignment offset of the query video relative to the candidate video clips; After using the temporal alignment offset as the matching result between the query video and the candidate video clips, the retrieval method also includes: Select a subset of matching frame pairs that match the temporal alignment offset from multiple candidate matching frame pairs; Calculate the sum of similarities for all matching frame pairs in the subset of matching frame pairs; The confidence score of the matching result is obtained by normalizing the sum of similarities with the time span or total number of frames occupied by the subset of matching frame pairs in the query video.

7. The retrieval method according to claim 6, characterized in that, After obtaining the confidence level of the matching results, the retrieval method also includes: If the confidence level of the matching result is within the set verification range, perform secondary temporal verification on the first semantic feature sequence and the second semantic feature sequence. Update the matching results based on the results of the second time-series verification.

8. A device for retrieving video clips, characterized in that, include: The extraction module is used to extract the first semantic feature sequence with semantic invariance from the query video; The acquisition module is used to acquire the second semantic feature sequence of the candidate video material to be retrieved; The matching module is used to match the first semantic feature sequence and the second semantic feature sequence to determine multiple candidate matching frame pairs between the query video and the candidate video material; The determination module is used to perform temporal alignment and deduplication on multiple candidate matching frame pairs to determine the matching result between the query video and the candidate video materials.

9. A device for retrieving video clips, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to, when running the program instructions, execute the method for retrieving video clips as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Equipment body; The retrieval device for video clips as described in claim 8 or 9 is installed on the device body.

Images