Video retrieval method and device, computer readable storage medium, and electronic device

Abstract

The embodiments of the present disclosure relate to the field of computer technology, and more specifically, the embodiments of the present disclosure relate to a video retrieval method and apparatus, a computer-readable storage medium, and an electronic device. The method comprises: obtaining a video to be retrieved, extracting key frames included in the video to be retrieved, and obtaining a plurality of key frame images based on the key frames; inputting the plurality of key frame images into a video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, and obtaining a video fingerprint of the video to be retrieved through the video-level features and video segment-level features; and performing a search using the video fingerprint to obtain a video similar to the video to be retrieved. The present method can reduce the storage cost of video fingerprints and speed up the retrieval of videos to be retrieved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more specifically, relate to a video retrieval method and apparatus, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no description herein is admitted to be prior art by inclusion in this section.

[0003] In recent years, Internet-related industries have developed rapidly. As a more vivid and intuitive way of expressing information, video has been favored by more and more people. There are more and more video application scenarios on the Internet. However, the prosperous development of the video industry has also brought huge challenges to the protection of original videos. With the gradual expansion of the scale of video resources, it is difficult to manually find duplicate videos or infringing videos from massive videos. In current technology, video retrieval can include: extracting video key frame images, using the features of each key frame image as a video fingerprint, and searching through video fingerprints. During the retrieval process, the video fingerprint of the video to be queried is compared with the video fingerprint of each video in the video library to determine videos similar to the video to be retrieved. Summary of the invention

[0004] However, in the current video retrieval method, on the one hand, the features of all key frame images of the video to be retrieved are used as video fingerprints, which makes the video fingerprint relatively monotonous and can only be compared with the features of the videos in the video database through the features of the key frame images; on the other hand, during retrieval, the features of all key frame images of the video to be retrieved need to be compared with all the features of all videos in the video database, which reduces the retrieval speed.

[0005] Therefore, there is a great need for an improved video retrieval method and apparatus, a computer-readable storage medium and an electronic device, so as to provide a video retrieval method that can reduce the video fingerprint storage cost and speed up the retrieval speed of the video to be retrieved.

[0006] In this context, embodiments of the present disclosure are intended to provide a video retrieval method and apparatus, a computer-readable storage medium, and an electronic device.

[0007] According to one aspect of the present disclosure, a video retrieval method is provided, comprising:

[0008] Acquire a video to be retrieved, extract key frames included in the video to be retrieved, and obtain multiple key frame images according to the key frames;

[0009] Inputting the multiple key frame images into a video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, and obtaining the video fingerprint of the video to be retrieved through the video-level features and video segment-level features;

[0010] The video fingerprint is used to perform a search to obtain videos similar to the video to be searched.

[0011] In an exemplary embodiment of the present disclosure, the video fingerprint model at least includes a shot boundary prediction model.

[0012] In an exemplary embodiment of the present disclosure, the video fingerprint extraction model also includes a key frame image feature extraction model, a timing model and a video feature extraction model.

[0013] In an exemplary embodiment of the present disclosure, the key frame image is input into the video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, including:

[0014] Inputting the plurality of key frame images into the key frame image feature extraction model, and obtaining key frame image features corresponding to each of the key frame images through a deep feature network in the key frame image feature extraction model;

[0015] Inputting the key frame image features into the time series model, learning the key frame image features, and obtaining time series features corresponding to the key frame image features;

[0016] Inputting the temporal features into the video feature extraction model to obtain video-level features corresponding to the video to be retrieved;

[0017] The temporal features are input into the shot boundary prediction model to obtain segment sub-features, and the segment sub-features are input into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved.

[0018] In an exemplary embodiment of the present disclosure, the video retrieval method further includes:

[0019] Calculating color histograms of the plurality of key frame images to obtain color histogram features corresponding to each of the key frame images;

[0020] Splicing the color histogram features and the key frame image features of each key frame image to obtain a splicing feature corresponding to each key frame image;

[0021] Inputting the splicing feature into the time series model, learning the splicing feature, and obtaining a first time series feature corresponding to the splicing feature;

[0022] Inputting the first temporal feature into the video feature extraction model to obtain a video-level feature corresponding to the video to be retrieved;

[0023] The first temporal feature is input into the shot boundary prediction model to obtain a first segment sub-feature, and the first segment sub-feature is input into the video feature extraction model to obtain a video segment-level feature corresponding to the video to be retrieved.

[0024] In an exemplary embodiment of the present disclosure, the key frame image features are input into the time series model, the key frame image features are learned, and the time series features corresponding to the key frame image features are obtained, including:

[0025] Multiplying each of the key frame image features by the initialization matrix of the time series model to obtain a query vector, a key vector and a value vector corresponding to each of the key frame image features; wherein the initialization matrix includes three matrices;

[0026] Calculating the correlation between each query vector corresponding to the key frame image feature and the key vector by a similarity function;

[0027] According to the correlation and the value vector corresponding to each of the key frame image features, a temporal feature corresponding to the key frame image feature is obtained.

[0028] In an exemplary embodiment of the present disclosure, the temporal features are input into the video feature extraction model to obtain video-level features corresponding to the video to be retrieved, including:

[0029] Splicing the time series features corresponding to the key frame images to obtain a first spliced ​​time series feature;

[0030] The first splicing time series feature is input into the feature extraction model, and the first splicing time series feature is pooled and normalized to obtain a video-level feature corresponding to the video to be retrieved.

[0031] In an exemplary embodiment of the present disclosure, the temporal features are input into the shot boundary prediction model to obtain segment sub-features, and the segment sub-features are input into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved, including:

[0032] The plurality of temporal features are converted into a plurality of first probability values ​​by using an excitation function in the shot boundary prediction model, and when it is determined that any of the first probability values ​​is greater than a preset probability value, a key frame image corresponding to any of the first probability values ​​is used as a shot boundary of the video to be retrieved;

[0033] According to the shot boundaries of the video to be retrieved, the temporal features are converted into segment sub-features;

[0034] The segment sub-features are input into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved.

[0035] In an exemplary embodiment of the present disclosure, converting the temporal features into segment sub-features according to the shot boundaries of the video to be retrieved includes:

[0036] The time series features corresponding to the key frame images constitute a feature sequence;

[0037] Acquire a shot boundary of the video to be retrieved included in a key frame image of the video to be retrieved, and a first position of a temporal feature corresponding to the shot boundary in the feature sequence;

[0038] The timing features of the non-shot boundaries before the first position are obtained, and the timing features of the non-shot boundaries and the timing features corresponding to the shot boundaries are spliced ​​to obtain the segment sub-features.

[0039] In an exemplary embodiment of the present disclosure, the video retrieval method further includes:

[0040] Parsing the video to be retrieved through a codec protocol to obtain key frames included in the video to be retrieved;

[0041] According to the key frame, the key frame image included in the video to be retrieved is extracted.

[0042] In an exemplary embodiment of the present disclosure, using the video fingerprint to perform a search to obtain a video similar to the video to be searched includes:

[0043] Acquire video-level features corresponding to videos included in a video database, and construct a search tree according to the video-level features included in the video database;

[0044] Searching in the search tree using the video-level features included in the video fingerprint set to obtain a plurality of candidate videos similar to the video to be retrieved;

[0045] The video segment level features of the candidate video are obtained, and the video and / or video segment most similar to the video to be retrieved is obtained through the video segment level features corresponding to the video to be retrieved and the video segment level features of the candidate video.

[0046] According to one aspect of the present disclosure, a video retrieval device is provided, comprising:

[0047] A key frame image acquisition module is used to acquire a video to be retrieved, extract key frames included in the video to be retrieved, and obtain multiple key frame images according to the key frames;

[0048] A video fingerprint acquisition module, used to input the multiple key frame images into a video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, and obtain the video fingerprint of the video to be retrieved through the video-level features and video segment-level features;

[0049] The video retrieval module is used to use the video fingerprint to perform retrieval to obtain videos similar to the video to be retrieved.

[0050] In an exemplary embodiment of the present disclosure, the video fingerprint model at least includes a shot boundary prediction model.

[0051] In an exemplary embodiment of the present disclosure, the video fingerprint extraction model also includes a key frame image feature extraction model, a timing model and a video feature extraction model.

[0052] In an exemplary embodiment of the present disclosure, the video fingerprint acquisition module includes:

[0053] A key frame image feature acquisition module, used to input the multiple key frame images into the key frame image feature extraction model, and obtain key frame image features corresponding to each of the key frame images through a deep feature network in the key frame image feature extraction model;

[0054] A time series feature acquisition module, used for inputting the key frame image features into the time series model, learning the key frame image features, and obtaining time series features corresponding to the key frame image features;

[0055] A video-level feature acquisition module, used for inputting the temporal features into the video feature extraction model to obtain video-level features corresponding to the video to be retrieved;

[0056] The video segment level feature acquisition module is used to input the temporal features into the shot boundary prediction model to obtain segment sub-features, and input the segment sub-features into the video feature extraction model to obtain video segment level features corresponding to the video to be retrieved.

[0057] In an exemplary embodiment of the present disclosure, the video retrieval device further includes:

[0058] A color histogram feature extraction module, used to calculate the color histograms of the multiple key frame images to obtain color histogram features corresponding to each of the key frame images;

[0059] A splicing feature acquisition module, used for splicing the color histogram features of each of the key frame images and the key frame image features to obtain a splicing feature corresponding to each of the key frame images;

[0060] A first time series feature acquisition module, configured to input the splicing feature into the time series model, learn the splicing feature, and obtain a first time series feature corresponding to the splicing feature;

[0061] A video-level feature acquisition module, used for inputting the first temporal feature into the video feature extraction model to obtain a video-level feature corresponding to the video to be retrieved;

[0062] The video segment level feature acquisition module is used to input the first temporal feature into the shot boundary prediction model to obtain the first segment sub-feature, and input the first segment sub-feature into the video feature extraction model to obtain the video segment level feature corresponding to the video to be retrieved.

[0063] In an exemplary embodiment of the present disclosure, the time series feature acquisition module includes:

[0064] A key frame image feature initial processing module, used for multiplying each of the key frame image features with the initialization matrix of the time series model to obtain a query vector, a key vector and a value vector corresponding to each of the key frame image features; wherein the initialization matrix includes three matrices;

[0065] A correlation calculation module, used for calculating the correlation between each query vector corresponding to the key frame image feature and the key vector through a similarity function;

[0066] The time series feature acquisition module is used to obtain the time series feature corresponding to the key frame image feature according to the correlation and the value vector corresponding to each key frame image feature.

[0067] In an exemplary embodiment of the present disclosure, the video-level feature acquisition module includes:

[0068] A first splicing time series feature determination module, used for splicing the time series features corresponding to the key frame images to obtain a first splicing time series feature;

[0069] The video-level feature acquisition module is used to input the first splicing time series feature into the feature extraction model, pool and normalize the first splicing time series feature, and obtain the video-level feature corresponding to the video to be retrieved.

[0070] In an exemplary embodiment of the present disclosure, the video segment level feature acquisition module includes:

[0071] a shot boundary determination module, configured to convert the plurality of temporal features into a plurality of first probability values ​​through an activation function in the shot boundary prediction model, and when it is determined that any of the first probability values ​​is greater than a preset probability value, use a key frame image corresponding to any of the first probability values ​​as a shot boundary of the video to be retrieved;

[0072] A segment sub-feature acquisition module, used for converting the temporal features into segment sub-features according to the shot boundaries of the video to be retrieved;

[0073] The video segment level feature determination module is used to input the segment sub-features into the video feature extraction model to obtain the video segment level features corresponding to the video to be retrieved.

[0074] In an exemplary embodiment of the present disclosure, the fragment sub-feature acquisition module includes:

[0075] A feature sequence construction module, used for forming a feature sequence from the time series features corresponding to the key frame images;

[0076] A first position determination module is used to obtain a shot boundary of the video to be retrieved included in a key frame image of the video to be retrieved, and a first position of a temporal feature corresponding to the shot boundary in the feature sequence;

[0077] The fragment sub-feature acquisition module is used to acquire the temporal features of the non-shot boundary included before the first position, and to splice the temporal features of the non-shot boundary and the temporal features corresponding to the shot boundary to obtain the fragment sub-features.

[0078] In an exemplary embodiment of the present disclosure, the video retrieval device further includes:

[0079] A key frame determination module, used to parse the video to be retrieved through a codec protocol to obtain key frames included in the video to be retrieved;

[0080] The key frame image extraction module is used to extract the key frame image included in the video to be retrieved according to the key frame.

[0081] In an exemplary embodiment of the present disclosure, the video retrieval module includes:

[0082] A search tree construction module, used to obtain video-level features corresponding to videos included in the video database, and to construct a search tree according to the video-level features included in the video database;

[0083] A candidate video determination module, configured to search the search tree using the video-level features included in the video fingerprint set to obtain a plurality of candidate videos similar to the video to be retrieved;

[0084] The video retrieval module is used to obtain the video segment level features of the candidate video, and obtain the video and / or video segment most similar to the video to be retrieved through the video segment level features corresponding to the video to be retrieved and the video segment level features of the candidate video.

[0085] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned video retrieval method is implemented.

[0086] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0087] Processor; and

[0088] A memory, configured to store executable instructions of the processor;

[0089] The processor is configured to execute any one of the above-mentioned video retrieval methods by executing the executable instructions.

[0090] According to the video retrieval method of the disclosed embodiment, after obtaining the key frame image of the video to be retrieved, the key frame image is input into the video fingerprint extraction model to obtain the video-level features and video segment-level features of the video to be retrieved, and the video-level features and video segment-level features of the video to be retrieved are used to retrieve videos similar to the video to be retrieved. The video-level features and video segment-level features of the video to be retrieved are used as the video fingerprint of the video to be retrieved, which increases the diversity of the video fingerprint. At the same time, the retrieval is performed only based on the video-level features and video segment-level features of the video to be retrieved, which avoids comparing the features of all key image frames of the video to be retrieved with the features of all videos in the video fingerprint library, thereby improving the retrieval speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0092] Figure 1 A flowchart of a video retrieval method according to an embodiment of the present disclosure is schematically shown;

[0093] Figure 2 A flowchart of a method for obtaining multiple key frame images according to a key frame according to an embodiment of the present disclosure is schematically shown;

[0094] Figure 3 A schematic diagram of a video fingerprint extraction model according to an embodiment of the present disclosure is schematically shown;

[0095] Figure 4 A flowchart of a method for inputting a key frame image into a video fingerprint extraction model to obtain video-level features and video segments and features corresponding to a video to be retrieved according to an embodiment of the present disclosure is schematically shown;

[0096] Figure 5 A flowchart of a video retrieval method according to an embodiment of the present disclosure is schematically shown;

[0097] Figure 6 A flowchart of a method for inputting key frame image features into a time series model to obtain time series features corresponding to the key frame image features according to an embodiment of the present disclosure is schematically shown;

[0098] Figure 7 A flowchart of a method for inputting temporal features into a video feature extraction model to obtain video-level features corresponding to a video to be retrieved according to an embodiment of the present disclosure is schematically shown;

[0099] Figure 8 A flowchart of a method for inputting segment sub-features into a video feature extraction model to obtain video segment-level features corresponding to a video to be retrieved according to an embodiment of the present disclosure is schematically shown;

[0100] Fig. 9 A flowchart of a method for converting temporal features into segment sub-features according to shot boundaries of a video to be retrieved according to an embodiment of the present disclosure is schematically shown;

[0101] Fig.10 A flowchart of a method for retrieving a video similar to a video to be retrieved by using a video fingerprint according to an embodiment of the present disclosure is schematically shown;

[0102] Fig.11 A block diagram of a video retrieval device according to an embodiment of the present disclosure is schematically shown;

[0103] Fig.12 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown;

[0104] Fig.13 A block diagram of an electronic device according to a disclosed embodiment is schematically shown.

[0105] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0106] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0107] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0108] According to an embodiment of the present disclosure, a video retrieval method, a video retrieval device, a computer-readable storage medium, and an electronic device are provided.

[0109] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.

[0110] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0112] In the related art, the features of key frame images can be roughly divided into two types: based on traditional image algorithms and based on deep learning algorithms; the features extracted based on traditional image algorithms can be divided into methods based on global features and methods based on local features. In the method based on global features, the global features of the video to be retrieved are obtained by compressing the global features of the image; in the method based on local features, the local features of the video to be retrieved are obtained by extracting the features of special points of interest and boundary areas in the video to be retrieved, wherein the features of the boundary areas may include Harris corner points, scale-invariant features, and fast robustness features; based on deep learning algorithms, the features of key frame images are extracted based on neural networks. After the features of the key frame images are extracted, the features of the key frame images of the video to be retrieved are compared with the video fingerprints of the videos in the video library, and then a video similar to the video to be retrieved is obtained. However, compared with the deep learning algorithm, the above-mentioned method of extracting the features of key frame images has poor robustness and versatility of the traditional image algorithm. At the same time, the features extracted by the deep learning algorithm lose the temporal features of the key frame images, resulting in insufficient representation capabilities of the extracted features; the above-mentioned retrieval method uses frame-by-frame feature comparison, resulting in a very slow retrieval speed.

[0113] In view of the above content, the basic idea of ​​the present disclosure is: according to the video retrieval method and video retrieval device of the implementation mode of the present disclosure, after obtaining the key frame image of the video to be retrieved, the key frame image is input into the video fingerprint extraction model to obtain the video-level features and video clip-level features corresponding to the video to be retrieved, and retrieval is performed through the video-level features and video clip-level features to obtain a video similar to the video to be retrieved.

[0114] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0115] Exemplary Methods

[0116] Combine the following Figure 1 A video retrieval method according to an exemplary embodiment of the present disclosure is described below.

[0117] refer to Figure 1 , the video retrieval method may include the following steps:

[0118] S1, obtaining a video to be retrieved, extracting key frames included in the video to be retrieved, and obtaining a plurality of key frame images according to the key frames;

[0119] S2, inputting the multiple key frame images into a video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, and obtaining the video fingerprint of the video to be retrieved through the video-level features and video segment-level features;

[0120] S3. Use the video fingerprint to perform a search to obtain videos similar to the video to be searched.

[0121] In the video retrieval method of the disclosed embodiment, after the key frame image of the video to be retrieved is obtained, the key frame image is input into the video fingerprint extraction model to obtain the video-level features and video-segment-level features of the video to be retrieved, and the video-level features and video-segment-level features of the video to be retrieved are used to retrieve a video similar to the video to be retrieved. The video to be retrieved is retrieved by using the video-level features and video-segment-level features of the video to be retrieved, which reduces the number of features included in the video fingerprint and reduces the storage cost of the video fingerprint. At the same time, the retrieval is performed only based on the video-level features and video-segment-level features of the video to be retrieved, which avoids comparing the features of all key image frames of the video to be retrieved with the features of all key frame images in the video fingerprint library, thereby improving the retrieval speed.

[0122] In step S1, a video to be retrieved is obtained, key frames included in the video to be retrieved are extracted, and a plurality of key frame images are obtained according to the key frames.

[0123] In an exemplary embodiment of the present disclosure, the above-mentioned video retrieval method can be applied to a server for performing video retrieval. The video to be retrieved can be a variety show video clip, an entertainment video, or a self-media video. In this exemplary embodiment, the video to be retrieved is not specifically limited. The key frame included in the video to be retrieved, namely the I (Iframe) frame, includes a large amount of information and is the most important frame for compression encoding between video frames. The key frame may include the image background in the image corresponding to the frame and the entity included in the image. When the image background in the frame image changes compared with the image in the previous frame image, the frame may be a key frame; when the entity included in the frame image changes compared with the entity included in the previous frame image, the frame may also be a key frame. The entity included in the image may be a still life, an animal, or a game character. In this example embodiment, there is no specific limitation on the entity. The change in the entity included in the frame image may be a change in the number of entities included in the frame image compared with the previous frame, a change in the position of the entity included in the frame image compared with the previous frame, or a change in the type of the entity included in the frame image compared with the previous frame. In this example embodiment, there is no specific limitation on the change in the entity. When obtaining key frames, the key frames of the video to be retrieved can be extracted through an open source video codec tool, wherein the open source video codec tool can be FFmpeg or OpenCV. In this exemplary embodiment, the open source video codec tool is not specifically limited. Figure 2 As shown, obtaining multiple key frame images according to the key frames may include step S21 and step S22:

[0124] Step S21. Parse the video to be retrieved through a codec protocol to obtain key frames included in the video to be retrieved;

[0125] Step S22: extract the key frame images included in the video to be retrieved according to the key frame.

[0126] In the following, step S21 and step S22 are explained and illustrated. Specifically, first, the video to be retrieved is parsed through the codec protocol to obtain the key frame included in the video to be retrieved and the time point at which the key frame appears in the video to be retrieved, and then, according to the key frame and the time point at which the key frame appears in the video to be retrieved, the key frame image included in the video to be retrieved is extracted.

[0127] For example, when the video to be retrieved is a motion video, when the motion video is parsed through the codec protocol, the key frame obtained can be that the motion posture of the moving subject in any frame in the video to be retrieved changes compared with the previous frame, or the relative position of the moving subject in the current frame changes compared with the previous frame, or the background of any frame changes compared with the previous frame. After the key frame is parsed and obtained, the time point when the key frame appears in the video is determined, and finally, the key frame image is extracted according to the time point.

[0128] In step S2, the multiple key frame images are input into a video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, and the video fingerprint of the video to be retrieved is obtained through the video-level features and video segment-level features.

[0129] In an exemplary embodiment of the present disclosure, the video fingerprint extraction model at least includes a shot boundary prediction model. The shot boundary prediction model is used to perform shot boundary detection on the video to be retrieved, and obtain a key frame image belonging to the shot boundary of the video to be retrieved in the key frame image input into the video fingerprint extraction model, wherein when the background image of the current frame in the video to be retrieved is transformed compared with the previous frame, the image corresponding to the current frame can be a shot boundary, or when the subject in the image corresponding to the current frame changes compared with the previous frame, the image corresponding to the current frame can also be a shot boundary, wherein the background image of the current frame can be a game scene or other scenes; the moving subject can be a game character included in the game scene, or can be a subject included in other scenes, etc., which is not specifically limited in this exemplary embodiment. Shot boundary detection can divide the video to be retrieved into multiple sub-segments. Through the shot boundary detection model, multiple features input into the shot boundary prediction model can be re-joined to obtain segment sub-features. Through the shot boundary prediction model, the number of features is reduced and the storage cost of the features is reduced.

[0130] Based on this, in the exemplary embodiment of the present disclosure, reference Figure 3As shown, the video fingerprint extraction model may also include a key frame image feature extraction model 301, a timing model 302 and a video feature extraction model 303; wherein the key frame image feature extraction model 301 is a deep feature network, which is composed of a multi-layer convolutional neural network, wherein the feature extraction network may be ResNet or VGG, and in this exemplary embodiment, the feature extraction network is not specifically limited, and is used to obtain the underlying features and high-level semantic features of the key frame image input to the video fingerprint extraction model, wherein the input data is the key frame image, and the output data is the key frame image feature; the timing model 302 is a self-attention model, whose input data is key frame image features and output data is time series features, which is used to model key frame image features in time sequence, thereby improving the robustness of features; video feature extraction model 303 is used to obtain the video fingerprint of the video to be retrieved, that is, video-level features and video segment-level features. When the features output by the video feature extraction model are video-level features, the corresponding input features are: the time series features output by the time series model; when the features output by the video feature extraction model are video segment-level features, the corresponding input features are: the segment sub-features output by the shot boundary prediction model.

[0131] In the exemplary implementation method of the present disclosure, reference is made to Figure 4 As shown, the key frame image is input into the video fingerprint extraction model to obtain the video-level features and video segment-level features corresponding to the video to be retrieved, which may include steps S41 to S44:

[0132] Step S41. Input the plurality of key frame images into the key frame image feature extraction model, and obtain key frame image features corresponding to each of the key frame images through a deep feature network in the key frame image feature extraction model;

[0133] Step S42: inputting the key frame image features into the time series model, learning the key frame image features, and obtaining time series features corresponding to the key frame image features;

[0134] Step S43. Inputting the temporal features into the video feature extraction model to obtain video-level features corresponding to the video to be retrieved;

[0135] Step S44: Input the temporal features into the shot boundary prediction model to obtain segment sub-features, and input the segment sub-features into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved.

[0136] Steps S41 to S44 are explained and illustrated below. Specifically, first, the key frame image is input into the key frame image feature extraction model in the video fingerprint extraction model, and the key frame image features of the key frame image are extracted through the deep feature network included in the key frame image feature extraction model; then, the key frame image features are input into the timing model, and the key frame image features are learned through the timing model to obtain the timing features corresponding to the key frame image; finally, multiple timing features are spliced ​​into one feature and input into the video feature extraction model to obtain the video-level features of the video to be retrieved; and, the timing features are input into the shot boundary prediction model to obtain the segment sub-features of the video to be retrieved, and the obtained segment sub-features are input into the video feature extraction model to obtain the video segment-level features of the video to be retrieved. After the key frame image features are extracted, the key frame image features are modeled in time sequence through the timing model, which improves the robustness of the features. The timing features are spliced ​​into one feature and the video-level features that can represent the entire video to be retrieved are obtained through the video feature extraction model, thereby improving the diversity of video fingerprints.

[0137] In an exemplary embodiment of the present disclosure, the underlying features and high-level features of the key frame image can be obtained through the key frame image feature extraction model, wherein the underlying features and high-level features of the key frame image are the semantic features of the key frame image mentioned by the deep feature network, and the underlying features may include: features such as contour, edge, color, texture and shape. The semantic information included in the underlying features is relatively small, but the target position is accurate. The semantic information in the high-level features is relatively rich, but the target position is relatively rough, and has a stronger resolution ability. In order to improve the robustness of the video fingerprint representation, before learning the temporal features, the global features of the key frame image, that is, the color histogram features, can also be obtained. For details, refer to Figure 5 As shown, the video retrieval method may further include steps S51 to S55:

[0138] Step S51. Calculate the color histograms of the multiple key frame images to obtain color histogram features corresponding to each of the key frame images;

[0139] Step S52: splicing the color histogram features of each key frame image and the key frame image features to obtain splicing features corresponding to each key frame image;

[0140] Step S53: inputting the splicing feature into the time series model, learning the splicing feature, and obtaining a first time series feature corresponding to the splicing feature;

[0141] Step S54. Inputting the first temporal feature into the video feature extraction model to obtain a video-level feature corresponding to the video to be retrieved;

[0142] Step S55. Input the first temporal feature into the shot boundary prediction model to obtain a first segment sub-feature, and input the first segment sub-feature into the video feature extraction model to obtain a video segment-level feature corresponding to the video to be retrieved.

[0143] Steps S51 to S55 are explained and illustrated below. Specifically, first, for each key frame image, its color histogram is calculated to obtain the color histogram features of each key frame image; then, the color histogram features of each key frame image and the key frame image features are spliced ​​to obtain the splicing features corresponding to each key frame image; then, the splicing features are input into the timing model, and the splicing features are learned to obtain the first timing features corresponding to the splicing features; finally, the first timing features are input into the feature extraction model to obtain the video-level features corresponding to the video to be retrieved; and, the first timing features are input into the shot boundary prediction model to obtain the first segment sub-features, and the first segment sub-features are input into the feature extraction model to obtain the video segment-level features of the video to be retrieved. By splicing the color histogram features with the key frame image features to obtain the splicing features, the scale invariance of the key frame image features is further improved.

[0144] In the exemplary embodiment of the present disclosure, referring to Figure 6 As shown, the key frame image features are input into the time series model, the key frame image features are learned, and the time series features corresponding to the key frame image features are obtained, which may include steps S61 to S63:

[0145] Step S61. Multiply each of the key frame image features by the initialization matrix of the time series model to obtain a query vector, a key vector and a value vector corresponding to each of the key frame image features; wherein the initialization matrix includes three matrices;

[0146] Step S62. Calculate the correlation between each query vector corresponding to the key frame image feature and the key vector by using a similarity function;

[0147] Step S63: Obtain the time series features corresponding to the key frame image features according to the correlation and the value vector corresponding to each key frame image feature.

[0148] Hereinafter, step S61 to step S63 will be explained and illustrated. Specifically, after obtaining the key frame image features of the video to be retrieved, in order to make each key frame image produce a more distinguishable feature representation and improve the robustness of the video fingerprint representation, it is also necessary to learn the key frame image features through the self-attention model to obtain the time series features of the key frame images; the calculation process of the self-attention model may include: calculating the weight coefficient according to the query vector and the key vector, and performing weighted summation of the value vector according to the weight coefficient to obtain the time series features; specifically, first, by multiplying the key frame image features with the three initialization matrices of the time series model, the query vector, the key vector and the value vector are obtained; then, the query vector and the key vector are calculated through the similarity function to obtain the similarity between the query vector and the key vector, and after obtaining the similarity, the similarity can also be normalized by the softmax function; wherein the similarity function can be a dot product, a concatenation, or a perceptron, and the similarity function is not specifically limited in this exemplary embodiment; finally, the similarity corresponding to each key frame image feature and the value vector are weighted summed to obtain the time series features corresponding to each key frame image feature. Through this temporal model, the underlying features of the key frame images, the high-level semantic features and the temporal correlation between the key frame images are combined, thereby improving the robustness of the video fingerprint representation.

[0149] After obtaining the temporal features of the key frame images, the temporal features corresponding to each key frame image can be input into the video feature extraction model to obtain the video-level features of the video to be retrieved. Figure 7 As shown, inputting the temporal features into the video feature extraction model to obtain the video-level features corresponding to the video to be retrieved may include step S71 and step S72:

[0150] Step S71. Splicing the time series features corresponding to the key frame images to obtain a first spliced ​​time series feature;

[0151] Step S72: Input the first splicing time series feature into the feature extraction model, perform pooling and normalization on the first splicing time series feature, and obtain a video-level feature corresponding to the video to be retrieved.

[0152] Step S71 and step S72 are explained and illustrated below. Specifically, after obtaining the time series features corresponding to the key frame image through the time series model, the obtained multiple time series features can be spliced ​​into one feature, that is, the time series features are spliced ​​to obtain a first spliced ​​time series feature; then, the first spliced ​​time series feature is input into the feature extraction model, and the first spliced ​​time series feature is pooled and normalized to obtain the video-level feature of the video to be retrieved.

[0153] After the temporal features are obtained, the temporal features can also be input into the shot boundary prediction model to obtain the video segment level features of the video to be retrieved. In the exemplary embodiment of the present disclosure, refer to Figure 8 As shown, the temporal features are input into the shot boundary prediction model to obtain segment sub-features, and the segment sub-features are input into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved, which may include steps S81 to S83:

[0154] Step S81. converting the plurality of temporal features into a plurality of first probability values ​​through an activation function in the shot boundary prediction model, and when it is determined that any of the first probability values ​​is greater than a preset probability value, taking a key frame image corresponding to any of the first probability values ​​as a shot boundary of the video to be retrieved;

[0155] Step S82: converting the temporal features into segment sub-features according to the shot boundaries of the video to be retrieved;

[0156] Step S83: Input the segment sub-features into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved.

[0157] Steps S81 to S83 are explained and illustrated below. Specifically, the shot boundary prediction model is composed of multiple fully connected networks. After the temporal features are input into the shot boundary detection model, first, the temporal features are converted into 1*N dimensional features through the fully connected network, where N is the number of key frame images input into the video fingerprint extraction model; then, the multiple 1*N dimensional features are converted into multiple first probability values ​​using an activation function. When any of the first probability values ​​is greater than a preset probability value, the key frame image corresponding to the feature can be considered to be the shot boundary of the video to be retrieved; then, according to the shot boundary of the video to be retrieved, the temporal features are converted into multiple fragment sub-features; finally, the multiple fragment sub-features are input into the feature extraction model to obtain the video fragment-level features of the video to be retrieved. The shot boundary prediction model is used to predict the shot boundary of the input temporal features, and the temporal features are converted into fragment sub-features according to the predicted shot boundary. Compared with the number of features input into the shot boundary prediction model, the shot boundary prediction model reduces the number of output features.

[0158] Further, in the exemplary embodiment of the present disclosure, reference is made to Fig. 9 As shown, according to the shot boundary of the video to be retrieved, converting the temporal features into segment sub-features may include steps S91 to S93:

[0159] Step S91. Constructing a feature sequence from the time series features corresponding to the key frame image;

[0160] Step S92. Acquire the shot boundary of the video to be retrieved included in the key frame image of the video to be retrieved, and the first position of the temporal feature corresponding to the shot boundary in the feature sequence;

[0161] Step S93: Acquire the timing features of the non-shot boundaries before the first position, and concatenate the timing features of the non-shot boundaries and the timing features corresponding to the shot boundaries to obtain the segment sub-features.

[0162] Steps S91 to S93 are explained and illustrated below. Specifically, first, the temporal features of the key frame images are formed into a feature sequence, which can be recorded as {X1, X2, X3, ..., Xn}, where n is a positive integer, and then the shot boundary of the video to be retrieved, the temporal features corresponding to the shot boundary, and the first position of the temporal features in the feature sequence are obtained; then, the temporal features of the non-shot boundaries included before the first position are obtained, and the temporal features of the non-shot boundaries are spliced ​​with the temporal features of the shot boundaries to obtain the fragment sub-features.

[0163] For example, when the number of temporal features input into the video feature extraction model is 5, the temporal features corresponding to the 5 key frame images can be represented by the feature sequence {X1, X2, X3, X4, X5}. If key frame image 2 and key frame image 5 are the shot boundaries of the video to be retrieved, first, the position of the temporal features of the shot boundaries in the feature sequence is determined. It can be obtained that in the feature sequence, the position of the temporal feature X2 in the feature sequence is 2, and the position of the temporal feature X5 in the feature sequence is 5; then the non-shot boundary included before position 2 is obtained. The timing feature X1 of position 2 and the timing features X3 and X4 of the non-shot boundary before position 5 are obtained; finally, the timing feature X2 of position 2 is spliced ​​with the non-shot boundary timing feature X1 before position 2 to obtain the first segment sub-feature, and the timing feature X5 of position 5 is spliced ​​with the non-shot boundary timing features X3 and X4 before position 5 to obtain the second segment sub-feature. Finally, the first segment sub-feature and the second segment sub-feature are input into the video feature extraction model to obtain two video segment-level features of the video to be retrieved. Compared with the number of features input into the shot boundary prediction model, the shot boundary prediction model reduces the number of output features, and the segment sub-features obtained in the shot boundary prediction model are input into the video feature extraction model to obtain the video segment-level features corresponding to the video to be retrieved. Compared with the related art that uses the features of all key frame images of the video to be retrieved as the video fingerprint of the video to be retrieved, the present disclosure uses the obtained video-level features and video segment-level features as the video fingerprint of the video to be retrieved, which reduces the number of features included in the video fingerprint and reduces the storage cost of the video fingerprint.

[0164] In step S3, the video fingerprint is used to perform a search to obtain videos similar to the video to be searched.

[0165] In an exemplary embodiment of the present disclosure, after obtaining the video fingerprint of the video to be retrieved, that is, the video-level features and the video segment-level features, a two-stage search can be performed based on the video fingerprint to obtain videos and / or video segments similar to the video to be retrieved. Fig.10 As shown, using the video fingerprint to search and obtain a video similar to the video to be searched may include steps S1001 to S1003:

[0166] Step S1001. Obtain video-level features corresponding to videos included in a video database, and construct a search tree according to the video-level features included in the video database;

[0167] Step S1002. Searching in the search tree using the video-level features included in the video fingerprint set to obtain a plurality of candidate videos similar to the video to be retrieved;

[0168] Step S1003: Obtain video segment-level features of the candidate video, and obtain the video and / or video segment most similar to the video to be retrieved through the video segment-level features corresponding to the video to be retrieved and the video segment-level features of the candidate video.

[0169] In the following, step S1001 to step S1003 will be explained and illustrated. Specifically, when performing a search, first, the search is performed through the video-level features of the video to be searched, that is, first, a search is performed in the video database based on the video-level features of the video to be searched, and multiple candidate videos that are most similar to the video to be searched are determined. Since the magnitude of the video fingerprints included in the video database is very large, in order to improve the search efficiency, a search tree can be constructed based on the similarity of the video-level features included in the video library, wherein the video-level features can be constructed as a KD (K-Dimensional tree) tree, or as other search trees. In this exemplary implementation method, the search tree is not specifically limited. When searching in a video database through the video-level features of the video to be retrieved, in order to improve the retrieval efficiency, NNS (Nearest Neighbor Search) can be used. NNS is also called nearest point search, which is an optimization problem of finding the nearest point in the scale space. Other search methods can also be used, which are not specifically limited in this exemplary implementation method. After obtaining multiple candidate videos similar to the video to be retrieved, the video segment-level features of the video to be retrieved can be compared one by one with the video segment-level features of the multiple candidate videos to obtain videos and / or video segments similar to the video to be retrieved.

[0170] Through two-stage retrieval, it is avoided to compare all key frame image features of the video to be retrieved with the key frame image features of all videos in the video library, thereby improving the retrieval speed; in the related art, if the video fingerprint of the video to be retrieved includes y key frame image features, the number of videos included in the video database is m, and each video contains y features, it is necessary to compare m*y*y times when retrieving the video to be retrieved; in the present disclosure, if the video fingerprint of the video to be retrieved includes n video clip-level features, the number of candidate videos similar to the video to be retrieved is x, and each candidate video includes n features, then when using two-stage retrieval in the present disclosure, the number of comparisons is x+n*n, wherein the number n of video clip-level features included in the video fingerprint of the video to be retrieved in the present disclosure is less than the number y of features included in the video fingerprint of the video to be retrieved in the related art, and the retrieval speed of the present disclosure is improved by compared with the retrieval speed of the related art. times.

[0171] In summary, the method provided by the present disclosure can be applied to retrieve videos and / or video clips similar to the video to be retrieved from a massive video database. When retrieving a video to be retrieved, first obtain a key frame image of the video to be retrieved, input the key frame image into a key frame image feature extraction model of a video fingerprint extraction model, and obtain key frame image features; then calculate the color histogram features of the key frame image, splice the key frame image features with the color histogram features to obtain splicing features, and input the splicing features into a timing model, learn the input splicing features through the timing model, and obtain timing features; then input the timing features into a video feature extraction model to obtain video-level features of the video to be retrieved; and, input the timing features into a shot boundary prediction model, convert the timing features into multiple fragment sub-features, and input the multiple fragment sub-features into a video feature extraction model to obtain multiple fragment sub-features of the video to be retrieved; finally, retrieve a video database according to the video-level features of the video to be retrieved to obtain multiple candidate videos similar to the video to be retrieved, and compare the multiple video fragment-level features of the video to be retrieved with the video fragment-level features of the multiple candidate videos to obtain videos and / or video fragments similar to the video to be retrieved. On the one hand, the underlying features, high-level semantic features, color histogram features and timing features of the key frame images are integrated during feature extraction, which improves the robustness of the video fingerprint representation; on the other hand, the number of video fingerprints is reduced, reducing the storage cost of video fingerprints; on the third hand, a two-stage retrieval is adopted to reduce the number of comparisons of the video fingerprints of the video to be retrieved in the video database, thereby improving the retrieval speed.

[0172] Exemplary Devices

[0173] After introducing the video retrieval method according to the exemplary embodiment of the present disclosure, Fig.11 A video retrieval device according to an exemplary embodiment of the present disclosure is described.

[0174] refer to Fig.11 As shown, the video retrieval device 11 of the exemplary embodiment of the present disclosure may include: a key frame image acquisition module 1101, a video fingerprint acquisition module 1102 and a video retrieval module 1103; wherein:

[0175] The key frame image acquisition module 1101 can be used to acquire a video to be retrieved, extract key frames included in the video to be retrieved, and obtain multiple key frame images according to the key frames.

[0176] The video fingerprint acquisition module 1102 can be used to input the multiple key frame images into the video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, and obtain the video fingerprint of the video to be retrieved through the video-level features and video segment-level features.

[0177] The video retrieval module 1103 can be used to perform retrieval using the video fingerprint to obtain videos similar to the video to be retrieved.

[0178] According to an exemplary embodiment of the present disclosure, the video fingerprint model at least includes a shot boundary prediction model.

[0179] According to an exemplary embodiment of the present disclosure, the video fingerprint extraction model also includes a key frame image feature extraction model, a timing model and a video feature extraction model.

[0180] According to an exemplary embodiment of the present disclosure, the video fingerprint acquisition module includes:

[0181] A key frame image feature acquisition module, used to input the multiple key frame images into the key frame image feature extraction model, and obtain key frame image features corresponding to each of the key frame images through a deep feature network in the key frame image feature extraction model;

[0182] A time series feature acquisition module, used for inputting the key frame image features into the time series model, learning the key frame image features, and obtaining time series features corresponding to the key frame image features;

[0183] A video-level feature acquisition module, used for inputting the temporal features into the video feature extraction model to obtain video-level features corresponding to the video to be retrieved;

[0184] The video segment level feature acquisition module is used to input the temporal features into the shot boundary prediction model to obtain segment sub-features, and input the segment sub-features into the video feature extraction model to obtain video segment level features corresponding to the video to be retrieved.

[0185] According to an exemplary embodiment of the present disclosure, the video retrieval device further includes:

[0186] A color histogram feature extraction module, used to calculate the color histograms of the multiple key frame images to obtain color histogram features corresponding to each of the key frame images;

[0187] A splicing feature acquisition module, used for splicing the color histogram features of each of the key frame images and the key frame image features to obtain a splicing feature corresponding to each of the key frame images;

[0188] A first time series feature acquisition module, configured to input the splicing feature into the time series model, learn the splicing feature, and obtain a first time series feature corresponding to the splicing feature;

[0189] A video-level feature acquisition module, used for inputting the first temporal feature into the video feature extraction model to obtain a video-level feature corresponding to the video to be retrieved;

[0190] The video segment level feature acquisition module is used to input the first temporal feature into the shot boundary prediction model to obtain the first segment sub-feature, and input the first segment sub-feature into the video feature extraction model to obtain the video segment level feature corresponding to the video to be retrieved.

[0191] According to an exemplary embodiment of the present disclosure, the time series feature acquisition module includes:

[0192] A key frame image feature initial processing module, used for multiplying each of the key frame image features with the initialization matrix of the time series model to obtain a query vector, a key vector and a value vector corresponding to each of the key frame image features; wherein the initialization matrix includes three matrices;

[0193] A correlation calculation module, used for calculating the correlation between each query vector corresponding to the key frame image feature and the key vector through a similarity function;

[0194] The time series feature acquisition module is used to obtain the time series feature corresponding to the key frame image feature according to the correlation and the value vector corresponding to each key frame image feature.

[0195] According to an exemplary embodiment of the present disclosure, the video-level feature acquisition module includes:

[0196] A first splicing time series feature determination module, used for splicing the time series features corresponding to the key frame images to obtain a first splicing time series feature;

[0197] The video-level feature acquisition module is used to input the first splicing time series feature into the feature extraction model, pool and normalize the first splicing time series feature, and obtain the video-level feature corresponding to the video to be retrieved.

[0198] According to an exemplary embodiment of the present disclosure, the video segment level feature acquisition module includes:

[0199] a shot boundary determination module, configured to convert the plurality of temporal features into a plurality of first probability values ​​through an activation function in the shot boundary prediction model, and when it is determined that any of the first probability values ​​is greater than a preset probability value, use a key frame image corresponding to any of the first probability values ​​as a shot boundary of the video to be retrieved;

[0200] A segment sub-feature acquisition module, used for converting the temporal features into segment sub-features according to the shot boundaries of the video to be retrieved;

[0201] The video segment level feature determination module is used to input the segment sub-features into the video feature extraction model to obtain the video segment level features corresponding to the video to be retrieved.

[0202] According to an exemplary embodiment of the present disclosure, the fragment sub-feature acquisition module includes:

[0203] A feature sequence construction module, used for forming a feature sequence from the time series features corresponding to the key frame images;

[0204] A first position determination module is used to obtain a shot boundary of the video to be retrieved included in a key frame image of the video to be retrieved, and a first position of a temporal feature corresponding to the shot boundary in the feature sequence;

[0205] The fragment sub-feature acquisition module is used to acquire the temporal features of the non-shot boundary included before the first position, and to splice the temporal features of the non-shot boundary and the temporal features corresponding to the shot boundary to obtain the fragment sub-features.

[0206] According to an exemplary embodiment of the present disclosure, the video retrieval device further includes:

[0207] A key frame determination module, used to parse the video to be retrieved through a codec protocol to obtain key frames included in the video to be retrieved;

[0208] The key frame image extraction module is used to extract the key frame image included in the video to be retrieved according to the key frame.

[0209] According to an exemplary embodiment of the present disclosure, the video retrieval module includes:

[0210] A search tree construction module, used to obtain video-level features corresponding to videos included in the video database, and to construct a search tree according to the video-level features included in the video database;

[0211] A candidate video determination module, configured to search the search tree using the video-level features included in the video fingerprint set to obtain a plurality of candidate videos similar to the video to be retrieved;

[0212] The video retrieval module is used to obtain the video segment level features of the candidate video, and obtain the video and / or video segment most similar to the video to be retrieved through the video segment level features corresponding to the video to be retrieved and the video segment level features of the candidate video.

[0213] Since the functional modules of the video device in the disclosed embodiment are the same as those in the disclosed embodiment of the video retrieval method, they will not be described in detail here.

[0214] Exemplary Storage Media

[0215] After introducing the video retrieval method and device according to the exemplary embodiment of the present disclosure, Fig.12 A computer-readable storage medium according to an exemplary embodiment of the present disclosure is described.

[0216] refer to Fig.12 As shown, a program product 1200 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0217] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0218] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0219] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0220] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0221] Exemplary Electronic Devices

[0222] After introducing the storage medium of the exemplary embodiment of the present disclosure, next, reference is made to Fig.13 An electronic device according to an exemplary embodiment of the present disclosure is described.

[0223] Fig.13The electronic device 1300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0224] like Fig.13 As shown, the electronic device 1300 is in the form of a general computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310), and a display unit 1340.

[0225] The storage unit stores a program code, which can be executed by the processing unit 1310, so that the processing unit 1310 performs the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification. For example, the processing unit 1310 can perform the following steps: Figure 1 Steps S1 to S3 shown in FIG.

[0226] The storage unit 1320 may include a volatile storage unit, such as a random access storage unit (RAM) 13201 and / or a cache storage unit 13202 , and may further include a read-only storage unit (ROM) 13203 .

[0227] The storage unit 1320 may also include a program / utility 13204 having a set (at least one) of program modules 13205, such program modules 13205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0228] The bus 1330 may include a data bus, an address bus, and a control bus.

[0229] The electronic device 1300 may also communicate with one or more external devices 1400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and such communication may be performed via an input / output (I / O) interface 1350. The electronic device 1300 also includes a display unit 1340, which is connected to the input / output (I / O) interface 1350 for display. In addition, the electronic device 1300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1360. As shown, the network adapter 1360 communicates with other modules of the electronic device 1300 via a bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0230] It should be noted that although several modules or sub-modules of the audio playback device and the audio sharing device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided to be embodied by multiple units / modules.

[0231] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0232] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A video retrieval method, characterized in that: include: Acquire a video to be retrieved, extract key frames included in the video to be retrieved, and obtain multiple key frame images according to the key frames; Input the multiple key frame images into the video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, and obtain the video fingerprint of the video to be retrieved through the video-level features and video segment-level features; wherein the video fingerprint extraction model at least includes a shot boundary prediction model, a key frame image feature extraction model, a timing model and a video feature extraction model; input the key frame images into the video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, including: input the multiple key frame images into the key frame image feature extraction model, and obtain the video-level features and video segment-level features corresponding to the video to be retrieved through the video-level features and video segment-level features. The deep feature network in the key frame image feature extraction model obtains key frame image features corresponding to each of the key frame images; the key frame image features are input into the timing model, the key frame image features are learned, and the timing features corresponding to the key frame image features are obtained; the timing features are input into the video feature extraction model to obtain video-level features corresponding to the video to be retrieved; the timing features are input into the shot boundary prediction model to obtain segment sub-features, and the segment sub-features are input into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved; Acquire video-level features corresponding to videos included in a video database, and construct a search tree based on the video-level features included in the video database; use the video-level features of the video to be retrieved to search in the search tree to obtain multiple candidate videos similar to the video to be retrieved; acquire video-segment-level features of the candidate videos, and obtain videos and / or video segments most similar to the video to be retrieved through the video-segment-level features corresponding to the video to be retrieved and the video-segment-level features of the candidate videos.

2. The video retrieval method according to claim 1, characterized in that: The video retrieval method further comprises: Calculating color histograms of the plurality of key frame images to obtain color histogram features corresponding to each of the key frame images; Splicing the color histogram features and the key frame image features of each key frame image to obtain a splicing feature corresponding to each key frame image; Inputting the splicing feature into the time series model, learning the splicing feature, and obtaining a first time series feature corresponding to the splicing feature; Inputting the first temporal feature into the video feature extraction model to obtain a video-level feature corresponding to the video to be retrieved; The first temporal feature is input into the shot boundary prediction model to obtain a first segment sub-feature, and the first segment sub-feature is input into the video feature extraction model to obtain a video segment-level feature corresponding to the video to be retrieved.

3. The video retrieval method according to claim 1, characterized in that: Inputting the key frame image features into the time series model, learning the key frame image features, and obtaining time series features corresponding to the key frame image features, including: Multiplying each of the key frame image features with the initialization matrix of the time series model to obtain a query vector, a key vector and a value vector corresponding to each of the key frame image features; wherein the initialization matrix includes three matrices; Calculating the correlation between each query vector corresponding to the key frame image feature and the key vector by a similarity function; According to the correlation and the value vector corresponding to each of the key frame image features, a temporal feature corresponding to the key frame image feature is obtained.

4. The video retrieval method according to claim 1, characterized in that: Inputting the temporal features into the video feature extraction model to obtain video-level features corresponding to the video to be retrieved includes: Splicing the time series features corresponding to the key frame images to obtain a first spliced ​​time series feature; The first splicing time series feature is input into the feature extraction model, and the first splicing time series feature is pooled and normalized to obtain a video-level feature corresponding to the video to be retrieved.

5. The video retrieval method according to claim 4, characterized in that: Inputting the temporal features into the shot boundary prediction model to obtain segment sub-features, and inputting the segment sub-features into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved, including: The plurality of temporal features are converted into a plurality of first probability values ​​by using an activation function in the shot boundary prediction model, and when it is determined that any of the first probability values ​​is greater than a preset probability value, a key frame image corresponding to any of the first probability values ​​is used as a shot boundary of the video to be retrieved; According to the shot boundaries of the video to be retrieved, the temporal features are converted into segment sub-features; The segment sub-features are input into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved.

6. The video retrieval method according to claim 5, characterized in that: According to the shot boundary of the video to be retrieved, the temporal features are converted into segment sub-features, including: The time series features corresponding to the key frame images constitute a feature sequence; Acquire a shot boundary of the video to be retrieved included in a key frame image of the video to be retrieved, and a first position of a temporal feature corresponding to the shot boundary in the feature sequence; The timing features of the non-shot boundaries before the first position are obtained, and the timing features of the non-shot boundaries and the timing features corresponding to the shot boundaries are spliced ​​to obtain the segment sub-features.

7. The video retrieval method according to any one of claims 1 to 6, characterized in that: The video retrieval method further comprises: Parsing the video to be retrieved through a codec protocol to obtain key frames included in the video to be retrieved; According to the key frame, the key frame image included in the video to be retrieved is extracted.

8. A video retrieval device, characterized in that: include: A key frame image acquisition module is used to acquire a video to be retrieved, extract key frames included in the video to be retrieved, and obtain multiple key frame images according to the key frames; The video fingerprint acquisition module is used to input the multiple key frame images into the video fingerprint extraction model to obtain video-level features and video segment-level features corresponding to the video to be retrieved, and obtain the video fingerprint of the video to be retrieved through the video-level features and video segment-level features; wherein the video fingerprint extraction model at least includes a shot boundary prediction model, a key frame image feature extraction model, a timing model and a video feature extraction model; the video fingerprint acquisition module includes: a key frame image feature acquisition module, used to input the multiple key frame images into the key frame image feature extraction model, and obtain the video fingerprint of the video to be retrieved through the deep feature network in the key frame image feature extraction model Key frame image features corresponding to each of the key frame images; a temporal feature acquisition module, used to input the key frame image features into the temporal model, learn the key frame image features, and obtain temporal features corresponding to the key frame image features; a video-level feature acquisition module, used to input the temporal features into the video feature extraction model to obtain video-level features corresponding to the video to be retrieved; a video segment-level feature acquisition module, used to input the temporal features into the shot boundary prediction model to obtain segment sub-features, and input the segment sub-features into the video feature extraction model to obtain video segment-level features corresponding to the video to be retrieved; The video retrieval module is used to use the video fingerprint to perform retrieval to obtain videos similar to the video to be retrieved, including: a search tree construction module, used to obtain video-level features corresponding to the videos included in the video database, and to construct a search tree based on the video-level features included in the video database; a candidate video determination module, used to use the video-level features of the video to be retrieved to search in the search tree to obtain multiple candidate videos similar to the video to be retrieved; a video retrieval module, used to obtain video segment-level features of the candidate videos, and obtain the video and / or video segment most similar to the video to be retrieved through the video segment-level features corresponding to the video to be retrieved and the video segment-level features of the candidate videos.

9. The video retrieval device according to claim 8, characterized in that: The video retrieval device also includes: A color histogram feature extraction module, used to calculate the color histograms of the multiple key frame images to obtain color histogram features corresponding to each of the key frame images; A splicing feature acquisition module, used for splicing the color histogram features of each of the key frame images and the key frame image features to obtain a splicing feature corresponding to each of the key frame images; A first time series feature acquisition module, used for inputting the splicing feature into the time series model, learning the splicing feature, and obtaining a first time series feature corresponding to the splicing feature; A video-level feature acquisition module, used for inputting the first temporal feature into the video feature extraction model to obtain a video-level feature corresponding to the video to be retrieved; The video segment level feature acquisition module is used to input the first temporal feature into the shot boundary prediction model to obtain the first segment sub-feature, and input the first segment sub-feature into the video feature extraction model to obtain the video segment level feature corresponding to the video to be retrieved.

10. The video retrieval device according to claim 8, characterized in that: The time series feature acquisition module includes: A key frame image feature initial processing module, used for multiplying each of the key frame image features with the initialization matrix of the time series model to obtain a query vector, a key vector and a value vector corresponding to each of the key frame image features; wherein the initialization matrix includes three matrices; A correlation calculation module, used for calculating the correlation between each query vector corresponding to the key frame image feature and the key vector through a similarity function; The time series feature acquisition module is used to obtain the time series feature corresponding to the key frame image feature according to the correlation and the value vector corresponding to each key frame image feature.

11. The video retrieval device according to claim 8, characterized in that: The video-level feature acquisition module includes: A first splicing time series feature determination module, used for splicing the time series features corresponding to the key frame images to obtain a first splicing time series feature; The video-level feature acquisition module is used to input the first splicing time series feature into the feature extraction model, pool and normalize the first splicing time series feature, and obtain the video-level feature corresponding to the video to be retrieved.

12. The video retrieval device according to claim 11, characterized in that: The video segment level feature acquisition module includes: a shot boundary determination module, configured to convert the plurality of temporal features into a plurality of first probability values ​​through an activation function in the shot boundary prediction model, and when it is determined that any of the first probability values ​​is greater than a preset probability value, use a key frame image corresponding to any of the first probability values ​​as a shot boundary of the video to be retrieved; A segment sub-feature acquisition module, used for converting the temporal features into segment sub-features according to the shot boundaries of the video to be retrieved; The video segment level feature determination module is used to input the segment sub-features into the video feature extraction model to obtain the video segment level features corresponding to the video to be retrieved.

13. The video retrieval device according to claim 12, characterized in that: The fragment sub-feature acquisition module includes: A feature sequence construction module, used for forming a feature sequence from the time series features corresponding to the key frame images; A first position determination module is used to obtain a shot boundary of the video to be retrieved included in a key frame image of the video to be retrieved, and a first position of a temporal feature corresponding to the shot boundary in the feature sequence; The fragment sub-feature acquisition module is used to acquire the temporal features of the non-shot boundary included before the first position, and to splice the temporal features of the non-shot boundary and the temporal features corresponding to the shot boundary to obtain the fragment sub-features.

14. The video retrieval device according to any one of claims 8 to 13, characterized in that: The video retrieval device also includes: A key frame determination module, used to parse the video to be retrieved through a codec protocol to obtain key frames included in the video to be retrieved; The key frame image extraction module is used to extract the key frame image included in the video to be retrieved according to the key frame.

15. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video retrieval method according to any one of claims 1 to 7 is implemented.

16. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the video retrieval method according to any one of claims 1 to 7 by executing the executable instructions.

Images