Search method and search device for video material, and electronic device

Abstract

This application relates to the field of multimedia data analysis technology, and discloses a method, device, and electronic device for retrieving video materials. The retrieval method includes: identifying multiple candidate matching frame pairs between the query video and source videos based on the semantic feature sequences of the query video and source videos in a database; calculating the time offset of each candidate matching frame pair under multiple different speed assumptions; performing binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source videos; and outputting retrieval results including the source video's ID in the database, the optimal playback speed, and the alignment position. This application can improve the accuracy of retrieving video materials.

Description

Technical Field

[0001] This application relates to the field of multimedia data analysis technology, such as a method and device for retrieving video materials, and an electronic device. Background Technology

[0002] Currently, with the development of short video platforms, video footage is being used to create new videos. Therefore, in practical applications, such as copyright infringement monitoring and tracing the source of user-generated content, it is necessary to retrieve and identify the video footage being used in the final query video.

[0003] In related technologies, visual feature matching is usually used to calculate the similarity between the visual features of the query video (e.g., color distribution features) and the video material, and to determine the matching result between the query video and the video material.

[0004] In the process of implementing the embodiments of this disclosure, it was found that the related technology has at least the following problems: When searching for videos based on secondary creation of video materials, non-linear processing such as speeding up, slowing down, and frame extraction may be applied to the video materials. This can easily lead to misjudgments and missed detections in the search results, resulting in low accuracy. Therefore, improving the accuracy of video material retrieval has become an urgent technical problem to be solved.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0007] This disclosure provides a method, device, and electronic device for retrieving video materials, which can improve the accuracy of video material retrieval.

[0008] In some embodiments, a method for retrieving video footage includes: identifying multiple candidate matching frame pairs between a query video and footage videos in a database based on semantic feature sequences of the query video and footage videos in a database; calculating the time offset for each candidate matching frame pair under multiple different speed assumptions; performing binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the footage videos; and outputting retrieval results including the footage video's ID in the database, the optimal playback speed, and the alignment position.

[0009] Optionally, based on the semantic feature sequences of the query video and the source videos in the database, multiple candidate matching frame pairs between the query video and the source videos are identified, including: obtaining the semantic feature sequences of the query video and the source videos; and determining multiple candidate matching frame pairs between the query video and the source videos based on the similarity of the semantic feature sequences of the query video and the source videos.

[0010] Optionally, obtaining the semantic feature sequences of the query video and the source video includes: sampling keyframes of the query video and the source video respectively to determine multiple key images; extracting high-dimensional feature vectors of the multiple key images through a pre-built feature extraction model; and normalizing the high-dimensional feature vectors to obtain the semantic feature sequences of the query video and the source video.

[0011] Optionally, the candidate matching frame pairs include the query frame timestamp and the source frame timestamp; the time offset under multiple different speed assumptions is calculated for each candidate matching frame pair according to the following expression: × ; in, Assuming the current speed, Let r be the time offset under the current speed assumption r. For the timestamp of the source frame, To query the frame timestamp.

[0012] Optionally, a binning statistical voting process is performed on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source video. This includes: establishing a voting bin for each speed assumption and mapping the time offset to the corresponding time interval bin in each voting bin; wherein, for multiple candidate matching frame pairs mapped to the same time interval bin, if multiple candidate matching frame pairs originate from the same query frame, only the candidate matching frame pair with the highest similarity is selected as the voting score contributed by that time interval bin; the cumulative voting score of each time interval bin in all voting bins is calculated, and the total voting score of each voting bin is calculated; the speed assumption corresponding to the target voting bin with the highest total voting score is determined as the optimal playback speed, and the time offset corresponding to the time interval bin with the highest cumulative voting score in the target voting bin is determined as the time alignment position.

[0013] Optionally, before outputting search results including the source video's ID in the database, optimal playback speed, and alignment position, the search method further includes: calculating the sum of similarities or cumulative coverage duration of all candidate matching frame pairs that satisfy the optimal playback speed and time alignment position; calculating the ratio of the sum of similarities or cumulative coverage duration to the total duration of the query video as the matching confidence between the query video and the source video; and determining to output search results if the matching confidence is greater than a set threshold.

[0014] Optionally, before calculating the ratio of the sum of similarities or cumulative coverage time to the total duration of the query video, the retrieval method may further include: normalizing the sum of similarities or cumulative coverage time to the total duration of the query video.

[0015] In some embodiments, a retrieval device for video footage includes: an identification module for identifying multiple candidate matching frame pairs between a query video and a source video based on semantic feature sequences of the query video and source videos in a database; a calculation module for calculating the time offset for each candidate matching frame pair under multiple different speed assumptions; a voting module for performing binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source video; and an output module for outputting retrieval results including the source video's number in the database, the optimal playback speed, and the alignment position.

[0016] In some embodiments, a video material retrieval apparatus includes a processor and a memory storing program instructions, the processor being configured to execute the video material retrieval method described above when the program instructions are executed.

[0017] In some embodiments, the electronic device includes: a device body; and the aforementioned retrieval device for video materials, which is mounted on the device body.

[0018] The method, apparatus, and electronic device for retrieving video materials provided in this disclosure can achieve the following technical effects: In this embodiment, when detecting video footage used in a query video, multiple speed assumptions are pre-constructed. After identifying multiple candidate matching frame pairs between the query video and the source videos based on the semantic feature sequences of the query video and source videos in the database, the time offset under multiple different speed assumptions is calculated for each candidate matching frame pair. Then, binning statistical voting is performed on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source videos. Finally, the retrieval results, including the source video's database ID, optimal playback speed, and alignment position, are output. This approach, compared to matching video footage based on visual features, reduces the impact of non-linear editing processing in the time dimension on the detection and matching, thus reducing the risk of false positives and false negatives. Therefore, this embodiment can improve the accuracy of retrieving video footage.

[0019] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a schematic diagram of an electronic device provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of a method for retrieving video materials provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of another method for retrieving video materials provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of another method for retrieving video materials provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of a video material retrieval device provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of a video material retrieval device provided in an embodiment of this disclosure. Detailed Implementation

[0021] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0022] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0023] Unless otherwise stated, the term "multiple" means two or more.

[0024] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0025] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0026] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in the present disclosure can be combined with each other.

[0028] like Figure 1 As shown, the electronic device 100 provided in this embodiment includes a device body 110 and a retrieval device 500 (600) for video materials.

[0029] Specifically, a retrieval device 500 (600) for video material is provided on the device body 110.

[0030] Optionally, the retrieval device 600 for video footage includes a processor. When detecting video footage used in a query video, the processor can identify multiple candidate matching frame pairs between the query video and the footage videos in the database based on the semantic feature sequences of the query video and the footage videos in the database. For each candidate matching frame pair, the processor calculates the time offset under multiple different speed assumptions. After determining the time offset, the processor can perform binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the footage videos, thereby outputting retrieval results including the footage video's database number, optimal playback speed, and alignment position.

[0031] In conjunction with the aforementioned electronic device, this disclosure provides a method for retrieving video footage, wherein the executing entity of this retrieval method can be a processor of a device for retrieving video footage segments. For example... Figure 2 As shown, the retrieval method includes: S201, the processor identifies multiple candidate matching frame pairs between the query video and the source videos based on the semantic feature sequences of the query video and the source videos in the database.

[0032] Specifically, the base library is a database that stores a large amount of source video footage.

[0033] Specifically, based on the semantic feature sequences of the query video and the source videos in the database, multiple candidate matching frame pairs between the query video and the source videos in the database are identified, that is, the combination of frames that may have similar content between the query video and the source videos is searched.

[0034] Specifically, the similarity (e.g., Euclidean distance or cosine similarity) between the semantic features of each frame of the query video and the semantic features of each frame of the source video can be calculated. Then, based on the similarity calculation results, frame pairs with similarity higher than a set threshold are identified as candidate matching frame pairs.

[0035] Specifically, each candidate matching frame pair includes the query frame timestamp, the source frame timestamp, and the similarity score.

[0036] S202, the processor calculates the time offset for each candidate matching frame pair under multiple different speed assumptions.

[0037] Specifically, several different speed assumptions are preset, such as 0.5x, 0.75x, 1x, 1.5x, and 2x. These speed assumptions need to cover common video playback speed variations.

[0038] Specifically, when generating query videos, non-linear processing in the temporal dimension may be applied to the video footage, such as speeding up, slowing down, frame extraction, and reordering of segments. By calculating the temporal offset between the query video and the candidate matching frame pairs of the source video under different speed assumptions, the matching situation under different speed assumptions can be analyzed, thereby reducing the impact of non-linear editing in the temporal dimension on the detection and matching.

[0039] Specifically, for each candidate matching frame pair, it is assumed that the query video is played at a certain speed. Based on this speed assumption, the time offset is obtained by calculating the corresponding time position of the query frame in the source video. For example, if it is assumed that the query video is played at twice the speed of the source video, then the time position of the query frame in the source video should be half of the time position of that frame in the query video, and the time offset can be calculated from this.

[0040] Optionally, the candidate matching frame pairs include the query frame timestamp and the source frame timestamp; the time offset under multiple different speed assumptions is calculated for each candidate matching frame pair according to the following expression: × ; in, Assuming the current speed, Let r be the time offset under the current speed assumption r. For the timestamp of the source frame, To query the frame timestamp.

[0041] S203, the processor performs binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source video.

[0042] Specifically, the optimal playback speed of the query video relative to the source video represents how many times the source video was sped up to obtain the query video. The alignment position of the query video relative to the source video represents the original start and end times of the query video within the source video.

[0043] Specifically, by performing binning and statistical voting on the time offset under each speed assumption, it is possible to determine how the query video adjusts the playback speed and start / end time of the source video. Therefore, the optimal playback speed and alignment position of the query video relative to the source video can be determined.

[0044] Specifically, the range of time offsets can be divided into multiple intervals to determine multiple time interval bins. For each velocity hypothesis, the number of candidate matching frame pairs within each time interval bin is counted as votes for that velocity hypothesis and that time interval bin. The more votes, the more likely the time offset and velocity hypothesis corresponding to that time interval bin are correct. Thus, based on the voting results, the time offset interval with the most votes and its corresponding velocity hypothesis can be identified. This velocity hypothesis represents the optimal playback speed of the query video relative to the source video, and the time offset represented by this time offset interval represents the alignment position of the query video within the source video.

[0045] S204, the processor outputs search results including the source video's ID in the database, optimal playback speed, and alignment position.

[0046] Specifically, each video clip is assigned a unique number. Therefore, when outputting search results, the search results can include the video clip's number in the database, the optimal playback speed, and the alignment position.

[0047] Specifically, the output search results can be in the form of text reports, graphical interfaces, etc., so that users can intuitively view and use the search results. For example, in a graphical interface, the video source number can be displayed, and the matching between the query video and the source video at the optimal playback speed and alignment position can be shown in a visual way, such as overlapping some frames.

[0048] In this embodiment, when detecting video footage used in a query video, multiple speed assumptions are pre-constructed. After identifying multiple candidate matching frame pairs between the query video and the source videos based on the semantic feature sequences of the query video and source videos in the database, the time offset under multiple different speed assumptions is calculated for each candidate matching frame pair. Then, binning statistical voting is performed on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source videos. Finally, the retrieval results, including the source video's database ID, optimal playback speed, and alignment position, are output. This approach, compared to matching video footage based on visual features, reduces the impact of non-linear editing processing in the time dimension on the detection and matching, thus reducing the risk of false positives and false negatives. Therefore, this embodiment can improve the accuracy of retrieving video footage.

[0049] In some embodiments, based on the semantic feature sequences of the query video and source videos in the database, multiple candidate matching frame pairs between the query video and source videos are identified, including: obtaining the semantic feature sequences of the query video and source videos; and determining multiple candidate matching frame pairs between the query video and source videos based on the similarity of the semantic feature sequences of the query video and source videos.

[0050] Specifically, by comparing the similarity between the semantic feature sequences of the query video and the source video, semantically similar frame pairs can be identified between the two videos. These frame pairs correspond to similar scenes or content within the videos. Therefore, multiple candidate matching frame pairs between the query video and the source video can be determined based on the similarity between their semantic feature sequences.

[0051] Specifically, methods for calculating semantic feature similarity (e.g., calculating cosine similarity or Euclidean distance) can be used to determine the similarity between the feature vectors in the semantic feature sequence of the query video and the semantic feature sequence of the source video, and then frame pairs with similarity exceeding a set threshold are identified as candidate matching frame pairs.

[0052] In some embodiments, obtaining the semantic feature sequence of the query video and the source video includes: sampling keyframes of the query video and the source video respectively to determine multiple key images; extracting high-dimensional feature vectors of the multiple key images through a pre-built feature extraction model; and normalizing the high-dimensional feature vectors to obtain the semantic feature sequence of the query video and the source video.

[0053] Specifically, selecting representative and informative key frames from both the query video and the source video to determine multiple key images can reduce the computational load of subsequent semantic feature extraction, while retaining the main content information of both the query video and the source video, which is beneficial to improving the efficiency of feature extraction.

[0054] Specifically, keyframe images can be determined using methods such as inter-frame differencing or color histogram calculation. For example, the pixel differences between adjacent frames in a video can be calculated, and when the difference exceeds a set threshold, the current frame is identified as a keyframe, and the image corresponding to that frame is designated as the keyframe image. Alternatively, the color histogram of each frame in the video can be calculated, and keyframes and keyframe images can be determined by comparing the differences between the color histograms of adjacent frames.

[0055] Specifically, the feature extraction model is a deep learning model trained on a large-scale image dataset, such as a convolutional neural network (CNN). The CNN model can be ResNet, VGG, Inception, etc. Because deep learning models have powerful feature extraction capabilities and can learn high-level semantic features in images, they can be used to construct feature extraction models for extracting semantically invariant high-dimensional feature vectors.

[0056] Specifically, by normalizing the extracted high-dimensional feature vectors, the dimensional feature vectors of different keyframes can be made comparable, and the dimensional differences that may exist in the high-dimensional feature vectors can be eliminated.

[0057] In this embodiment, when extracting semantically invariant semantic feature sequences from the query video and the source video, representative and informative keyframes are first selected from the query video to determine multiple key images. Then, a feature extraction model is used to extract features from these key images. This reduces the computational load during semantic feature extraction and improves the efficiency of semantic feature sequence extraction.

[0058] In some embodiments, binning statistical voting is performed on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source video. This includes: establishing a voting bin for each speed assumption and mapping the time offset to the corresponding time interval bin in each voting bin; wherein, for multiple candidate matching frame pairs mapped to the same time interval bin, if the multiple candidate matching frame pairs originate from the same query frame, only the candidate matching frame pair with the highest similarity is selected as the voting score contributed by that time interval bin; the cumulative voting score of each time interval bin in all voting bins is calculated, and the total voting score of each voting bin is calculated; the speed assumption corresponding to the target voting bin with the highest total voting score is determined as the optimal playback speed, and the time offset corresponding to the time interval bin with the highest cumulative voting score in the target voting bin is determined as the time alignment position.

[0059] Specifically, by establishing a separate voting box for each velocity hypothesis, an independent statistical space can be created for different velocity hypotheses, so as to accurately count the matching situation of each time offset interval under the velocity hypothesis.

[0060] Specifically, within each ballot box, the possible range of time offset is divided into several consecutive, non-overlapping time interval bins. The size of the bins is set according to actual needs; for example, they can be divided according to fixed time intervals (such as 0.1 seconds, 0.5 seconds, etc.).

[0061] Specifically, for each candidate matching frame pair, based on its corresponding velocity assumption, its calculated time offset is mapped to the corresponding time interval bin of the voting box corresponding to the velocity assumption. For example, if the candidate matching frame pair belongs to the 1x velocity assumption and its time offset is 0.3 seconds, and the time interval bins in the 1x velocity voting box include the 0-0.2 second bin, the 0.2-0.4 second bin, and the 0.4-0.6 second bin, then this candidate matching frame will be mapped to the 0.2-0.4 second bin.

[0062] Specifically, for multiple candidate matching frame pairs mapped to the same time interval bin, if multiple candidate matching frame pairs originate from the same query frame, only the candidate matching frame pair with the highest similarity is selected as the voting score contributed by that time interval bin. This can ensure that each query frame contributes at most one vote in each time interval bin, avoiding statistical bias caused by duplicate counting.

[0063] Specifically, for each time interval bin, the number of all valid candidate matching frame pairs (i.e., the filtered candidate matching frame pairs) is counted, and this number is used as the cumulative voting score for that time interval bin. In this way, the voting score of each time interval bin within each voting box can be obtained.

[0064] Specifically, if the total score of the voting box is the highest, it indicates that the speed assumption corresponding to that voting box is a better match for the playback speed of the source video. Therefore, the speed assumption corresponding to the target voting box with the highest total cumulative voting score can be determined as the optimal playback speed.

[0065] Specifically, if the time interval binning is the time interval binning with the highest cumulative vote score within the target voting bin, it means that the query video, after being adjusted according to the time offset corresponding to this time interval binning, is a better match for the source video. Therefore, the time offset corresponding to the time interval binning with the highest cumulative vote score within the target voting binning bin can be determined as the time alignment position.

[0066] In this embodiment, when determining the optimal playback speed and alignment position of the query video relative to the source video, independent voting boxes are established for different speed assumptions, and time intervals are divided into boxes. This comprehensively and accurately captures the matching situation of the query video and the source video under different speed assumptions. This effectively avoids matching omissions or errors caused by speed differences, improving the accuracy of the determined optimal playback speed and alignment position.

[0067] This disclosure provides another method for retrieving video materials, such as... Figure 3 As shown, the retrieval method includes: S301, the processor identifies multiple candidate matching frame pairs between the query video and the source videos based on the semantic feature sequences of the query video and the source videos in the database.

[0068] S302, the processor calculates the time offset for each candidate matching frame pair under multiple different speed assumptions.

[0069] S303, the processor performs binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source video.

[0070] S304, the processor calculates the sum of similarities or cumulative coverage duration of all candidate matching frame pairs that satisfy the optimal playback speed and time alignment position.

[0071] Specifically, by calculating the sum of similarities among all candidate matching frame pairs that satisfy the optimal playback speed and time alignment, the similarity of all candidate matching frame pairs that meet this condition can be quantified, measuring the overall similarity between the query video and the source video. The larger the sum of similarities, the more similar these candidate frame pairs are, and the stronger the correlation between the query video and the source video under this matching method.

[0072] Specifically, by calculating the cumulative coverage duration of all candidate matching frame pairs that satisfy the optimal playback speed and time alignment position, the coverage of these candidate matching frame pairs in the query video can be measured in the time dimension, thus measuring the degree of temporal matching between the query video and the source video. The longer the cumulative coverage duration, the more time these candidate frame pairs cover in the query video, and the stronger the temporal correlation between the query video and the source video.

[0073] S305, the processor calculates the ratio of the sum of similarities or cumulative coverage time to the total duration of the query video, as the matching confidence of the query video and the source video.

[0074] Specifically, by calculating the ratio of the sum of similarities or the cumulative coverage time to the total duration of the query video, the matching confidence of the query video and the source video can be used to quantify the credibility of the search results.

[0075] S306, when the matching confidence level is greater than a set threshold, the processor outputs search results including the video source number in the database, the optimal playback speed, and the alignment position.

[0076] Specifically, if the match confidence score is greater than a set threshold, it indicates that the search results are highly reliable. Therefore, in this case, search results can be output including the video source's ID in the database, the optimal playback speed, and the alignment position.

[0077] In this embodiment, before outputting the search results, the matching confidence score between the query video and the source video is calculated based on the optimal playback speed and time alignment. Search results are only output when the matching confidence score exceeds a set threshold. This improves the accuracy and reliability of the output search results and helps avoid producing low-quality results.

[0078] This disclosure provides another method for retrieving video materials, such as... Figure 4 As shown, the retrieval method includes: S401, the processor identifies multiple candidate matching frame pairs between the query video and the source videos based on the semantic feature sequences of the query video and the source videos in the database.

[0079] S402, the processor calculates the time offset for each candidate matching frame pair under multiple different speed assumptions.

[0080] S403, the processor performs binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source video.

[0081] S404, the processor calculates the sum of similarities or cumulative coverage duration of all candidate matching frame pairs that satisfy the optimal playback speed and time alignment position.

[0082] S405, the processor normalizes the sum or cumulative coverage time of similarity with the total duration of the queried video.

[0083] Specifically, by normalizing the sum of similarities or cumulative coverage time with the total duration of the query video, the difference in dimensions between different parameters can be eliminated, making it easier to map the matching confidence to a unified range (e.g., [0, 1]), so that the matching results of different query videos are comparable.

[0084] S406, the processor calculates the ratio of the sum of similarities or cumulative coverage time to the total duration of the query video, which is used as the matching confidence of the query video and the source video.

[0085] S407: When the matching confidence level is greater than a set threshold, the processor outputs search results including the video source number in the database, the optimal playback speed, and the alignment position.

[0086] In this embodiment, when calculating the ratio of the sum of similarities or cumulative coverage time to the total duration of the query video as the matching confidence score between the query video and the source video, the sum of similarities or cumulative coverage time and the total duration of the query video are first normalized. This eliminates the difference in dimensions between different parameters, maps the matching confidence score to a uniform range, and makes the retrieval results of different query videos comparable.

[0087] Combination Figure 5As shown in the figure, this disclosure provides a video material retrieval device 500, including: an identification module 501, a calculation module 502, a voting module 503, and an output module 504. The identification module 501 is used to identify multiple candidate matching frame pairs between the query video and the source videos based on the semantic feature sequences of the query video and source videos in the database. The calculation module 502 is used to calculate the time offset for each candidate matching frame pair under multiple different speed assumptions. The voting module 503 is used to perform binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source videos. The output module 504 is used to output retrieval results including the source video's number in the database, the optimal playback speed, and the alignment position.

[0088] Combination Figure 6 As shown, this embodiment of the disclosure provides a video material retrieval device 600, comprising a processor 601 and a memory 602. Optionally, the device may further include a communication interface 603 and a bus 604. The processor 601, communication interface 603, and memory 602 can communicate with each other via the bus 604. The communication interface 603 can be used for information transmission. The processor 601 can invoke logical instructions stored in the memory 602 to execute the video material retrieval method described in the above embodiment.

[0089] Furthermore, the logic instructions in the aforementioned memory 602 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0090] The memory 602, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 601 executes functional applications and data processing by running the program instructions / modules stored in the memory 602, thereby implementing the video material retrieval method described in the above embodiments.

[0091] The memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 602 may include high-speed random access memory and may also include non-volatile memory.

[0092] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described method for retrieving video materials.

[0093] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0094] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0095] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0096] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for retrieving video materials, characterized in that, include: Based on the semantic feature sequences of the query video and the source videos in the database, multiple candidate matching frame pairs between the query video and the source videos are identified; For each candidate matching frame pair, calculate the time offset under multiple different velocity assumptions; For each speed assumption, the time offset is binned and statistically voted to determine the optimal playback speed and alignment position of the query video relative to the source video. The output includes the video source's ID in the database, the optimal playback speed, and the alignment position as search results.

2. The retrieval method according to claim 1, characterized in that, Based on the semantic feature sequences of the query video and source videos in the database, multiple candidate matching frame pairs between the query video and source videos are identified, including: Obtain the semantic feature sequences of the query video and the source video; Based on the similarity of the semantic feature sequences of the query video and the source video, multiple candidate matching frame pairs are determined between the query video and the source video.

3. The retrieval method according to claim 2, characterized in that, Obtain the semantic feature sequences of the query video and the source videos, including: Keyframe sampling was performed on both the query video and the source video to determine multiple key images; High-dimensional feature vectors of multiple key images are extracted using a pre-built feature extraction model. Normalize the high-dimensional feature vectors to obtain the semantic feature sequences of the query video and the source video.

4. The retrieval method according to claim 1, characterized in that, Candidate matching frame pairs include query frame timestamps and source frame timestamps; the time offset under multiple different speed assumptions is calculated for each candidate matching frame pair according to the following expression: × ； in, Assuming the current speed, Let r be the time offset under the current speed assumption r. For the timestamp of the source frame, To query the frame timestamp.

5. The retrieval method according to claim 1, characterized in that, For each speed assumption, binning and statistical voting are performed on the time offset to determine the optimal playback speed and alignment position of the query video relative to the source video, including: A separate voting box is established for each velocity assumption, and the time offset is mapped to the corresponding time interval bin in each voting box. Among them, for multiple candidate matching frame pairs mapped to the same time interval bin, if multiple candidate matching frame pairs originate from the same query frame, only the candidate matching frame pair with the highest similarity is selected as the voting score contributed by that time interval bin. Calculate the cumulative voting score for each time interval in all ballot boxes, and then calculate the total voting score for each ballot box; The optimal playback speed is assumed to be the speed corresponding to the target voting box with the highest total voting score. The time offset corresponding to the time interval with the highest cumulative voting score within the target voting box is determined as the time alignment position.

6. The retrieval method according to any one of claims 1 to 5, characterized in that, Before outputting search results including the source video's ID in the database, optimal playback speed, and alignment position, the search method also includes: Calculate the sum of similarities or cumulative coverage duration of all candidate matching frame pairs that satisfy the optimal playback speed and time alignment position; The ratio of the sum of similarities or cumulative coverage time to the total duration of the query video is used as the confidence score for the match between the query video and the source video. If the matching confidence level is greater than a set threshold, the search results are determined and output.

7. The retrieval method according to claim 6, characterized in that, Before calculating the ratio of the sum of similarities or cumulative coverage time to the total duration of the queried videos, the retrieval method also includes: The sum or cumulative similarity coverage time is normalized to the total duration of the queried video.

8. A device for retrieving video materials, characterized in that, include: The recognition module is used to identify multiple candidate matching frame pairs between the query video and the source videos based on the semantic feature sequences of the query video and the source videos in the database; The calculation module is used to calculate the time offset for each candidate matching frame pair under multiple different velocity assumptions; The voting module is used to perform binning statistical voting on the time offset under each speed assumption to determine the optimal playback speed and alignment position of the query video relative to the source video. The output module is used to output search results, including the video source's ID in the database, optimal playback speed, and alignment position.

9. A device for retrieving video materials, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to, when running the program instructions, execute the method for retrieving video material as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Equipment body; The retrieval device for video materials as described in claim 8 or 9 is installed on the device body.

Images