Video retrieval method and system, electronic equipment and storage medium

Abstract

The invention provides a video retrieval method and system, electronic equipment and a storage medium, and belongs to the technical field of video retrieval. The video retrieval method comprises the steps that an identification feature database is generated, and each identification feature is associated with a unique video; a retrieval picture input by a user is obtained, and retrieval features of the retrieval picture are extracted; according to the retrieval features, the matched recognition features are retrieved; and according to the matched identification feature, obtaining the video associated with the matched identification feature. According to the technical scheme, the recognition features of the videos are extracted in advance, so that accurate association between the recognition features and the videos is ensured; according to the method, the retrieval features of the retrieval picture input by the user are extracted, the content of the retrieval picture is considered, the comprehensiveness of the retrieval features is ensured, and video retrieval is performed by matching the retrieval features and the recognition features, so that the accuracy of video retrieval is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video retrieval technology, and in particular to a video retrieval method, system, electronic device and storage medium. Background Art

[0002] With the rapid growth and widespread use of video content, users are faced with the challenge of how to retrieve the original video corresponding to a specific video frame image from a massive amount of videos. For example, when a user watches a video clip, he or she may become interested in the video clip and then hope to find the original video to which the video clip belongs by using video screenshots. Traditional video retrieval methods are mainly based on text metadata, such as titles and descriptions, while ignoring the content of the video frame itself. Such traditional methods have many limitations, such as titles and descriptions may be inaccurate or incomplete, and cannot provide accurate retrieval results for videos that are not clearly identified, such as specific scenes or images. Summary of the invention

[0003] The present disclosure provides a video retrieval method, system, electronic device and storage medium to solve the technical problem of inaccurate retrieval results caused by video retrieval based on text metadata in the prior art.

[0004] The present disclosure provides this summary of the invention to introduce concepts in a brief form, which will be described in detail in the following detailed description. This summary of the invention is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In order to solve the above technical problems, the present disclosure provides a video retrieval method, comprising: generating a database of identification features, wherein each identification feature is associated with a unique video; Obtaining a search image input by a user, and extracting search features from the search image; According to the search feature, searching for the matching identification feature; According to the matched identification feature, the video associated with the matched identification feature is obtained.

[0006] In order to solve the above technical problems, the present disclosure also provides a video retrieval system, which adopts the following technical solutions, including: A storage unit, used to store a database of identification features, wherein each identification feature is associated with a unique video; An acquisition unit, used to acquire a search image input by a user and extract search features from the search image; An identification unit, configured to retrieve the matching identification feature according to the search feature; A feedback unit is used to obtain the video associated with the matched identification feature according to the matched identification feature, and feed back the video to the user.

[0007] In order to solve the above technical problems, the present disclosure also provides an electronic device, which adopts the following technical solution, including: a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0008] In order to solve the above technical problems, the present disclosure also provides a computer-readable storage medium, which adopts the following technical solution, including: a computer program is stored on the computer-readable storage medium, and the computer program implements the above method when executed by a processor.

[0009] The positive and progressive effects of this disclosure: The technical solution disclosed in the present invention extracts the recognition features of each video in advance, ensures the accurate association between the recognition features and the video, and stores the recognition features in a recognition feature database, which facilitates subsequent video retrieval and is beneficial to improving the efficiency of video retrieval; by extracting the retrieval features of the retrieval picture input by the user, the content of the retrieval picture itself is taken into consideration, ensuring the comprehensiveness of the retrieval features, and using the matching of the retrieval features and the recognition features to retrieve the video, it is beneficial to improve the accuracy of video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a video retrieval method provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of a video retrieval system provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of an electronic device provided according to an embodiment of the present disclosure.

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale. DETAILED DESCRIPTION

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure; the terms "including" and "having" and any variations thereof in the specification and claims of the present disclosure and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present disclosure or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0013] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present disclosure. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0014] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0015] The first aspect of the embodiment of the present disclosure is introduced below: a video retrieval method.

[0016] Figure 1 Flow chart of a video retrieval method provided by an embodiment of the present disclosure. Figure 1 As shown, the video retrieval method includes steps S101-S104.

[0017] S101, generating an identification feature database, wherein each identification feature is associated with a unique video.

[0018] It should be noted here that the embodiment of the present disclosure refers to a sign or symbol extracted based on the content and other information of the video to which it is associated. The identification feature is unique and can point to a specific video, thereby ensuring the accuracy of video retrieval. Among them, the identification feature database is constantly updated and can continuously supplement the identification features of newly added videos to ensure that the scope of video retrieval is large enough. In addition, the identification features in the identification feature database can be simultaneously associated with the index, plot description and other information of the corresponding video (for example, film and television works, etc.) in order to identify and match the search images (for example, screenshots or photos, etc.) provided by the user, thereby improving the accuracy and efficiency of video retrieval.

[0019] S102, obtaining a search image input by a user, and extracting search features from the search image.

[0020] It should be noted here that the embodiments of the present disclosure can determine the retrieval features according to the characters, scenes, places, atmospheres, etc. in the retrieval pictures input by the user. Compared with the traditional retrieval based on text metadata (e.g., title and description), the video retrieval based on pictures is more accurate and comprehensive. When extracting the retrieval features of the retrieval pictures, for example, a deep learning model can be used to learn the high-level expression of the retrieval pictures, and the retrieval pictures can be converted into vectors to represent the corresponding retrieval features. Among them, the deep learning model is a machine learning model based on artificial neural networks (ANN), and its core idea is to learn the feature representation and pattern of data through a multi-level neural network structure. These models are usually composed of neural networks with multiple levels (depth), each layer contains multiple neurons, and the layers are connected by weights. The characteristics of deep learning models include multi-level structures, feature representation learning, end-to-end learning, training using back propagation algorithms, and the need for large-scale data and computing resources. Deep learning models are particularly applicable in fields such as image recognition. For example, convolutional neural networks (CNN) are used to extract the recognition features of retrieval pictures, and vector space models (Vector Space Model) are used to convert the recognition features into vector representations. When building a video search engine, these vectors will be associated with the unique identifier of the video and stored in the database. The retrieval images uploaded by users generate vectors through the same feature extraction model, and the most similar vectors are found in the recognition feature database through the vector retrieval algorithm. Once the most similar vector is found, the video information matching the retrieval image can be retrieved and returned to the user. In addition, computer vision can be used to extract retrieval features from the retrieval images. Among them, computer vision is to use various imaging systems instead of visual organs as input sensitive means, and computers replace the brain to complete processing and interpretation, so that computers can observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously. Therefore, the recognition features of the retrieval images can be analyzed conveniently, quickly and accurately through computer vision.

[0021] S103: Retrieve the matching identification feature according to the search feature.

[0022] The embodiments of the present disclosure need to explain that the higher the matching degree between the search feature and the identification feature, the higher the accuracy of the retrieved video. For example, when the search feature and the identification feature are both represented by vectors, the similarity of the two vectors can be calculated as the matching degree, and the video corresponding to the identification feature whose similarity meets the preset threshold is selected as the search result for output.

[0023] S104: Acquire the video associated with the matched identification feature according to the matched identification feature.

[0024] It should be noted that, since the identification feature is only associated with a unique video, the unique video can be accurately matched according to the identification feature. Of course, there can be multiple matching identification features, and correspondingly multiple videos can also be obtained. For example, when the search picture input by the user corresponds to multiple videos, all the corresponding multiple videos can be retrieved to ensure the comprehensiveness of the search results.

[0025] The embodiments of the present disclosure have a variety of application scenarios in life. For example, when a user is interested in a certain video clip being watched and hopes to find the original video or film and television work to which the video clip belongs, the user can take a screenshot of a certain video frame of the video clip to obtain a search image, and perform video search on the search image to obtain the original video to understand the plot before and after the video clip, etc., to meet the user's viewing psychology and improve the user's experience. For another example, when a user sees a film and television drama clip played on the screen in a public place and becomes interested in the plot, but it is inconvenient to ask others, the user can perform video search by taking a plot picture (i.e., a search photo) to obtain information such as the original video of the film and television drama clip.

[0026] Further, in some embodiments of the present disclosure, when executing the step of generating an identification feature database, wherein each identification feature is associated with a unique video, it can be achieved by but not limited to the following method, including the following steps: Extracting key frame images of each of the videos; Extracting the identification features from the key frame images; The identification feature is associated with the corresponding video, and the identification feature database is generated.

[0027] The embodiment of the present disclosure needs to be explained here that the selection of key frame images of each video can be determined according to the content of the video frame, and video frames with information such as specific characters or specific scenes can be selected. Video frames can also be selected according to fixed time intervals. The smaller the time interval, the higher the accuracy of subsequent video retrieval. The specific time interval setting can be determined according to the user's accuracy requirements for video retrieval. When extracting identification features of each key frame image, the retrieval features can be determined according to the characters, scenes, locations, atmospheres, etc. in the key frame image to improve the accuracy of the retrieval. When extracting identification features of key frame images, for example, a deep learning model can be used to learn the high-level expression of key frame images, and the key frame images can be converted into vectors to represent their corresponding identification features. The identification features are associated with the corresponding video, for example, the identification features are associated with the unique identifier of the video, and this association is stored in the identification feature database. In addition, when the video is decomposed into multiple video frames, the obtained video frames can be packaged and uploaded to a file storage platform (such as OSS) so that the video frames can be screened to obtain key frames, and then key frame images can be obtained.

[0028] In one or more embodiments, for example, key objects, scenes, and actions in video frames are encoded into high-dimensional vectors and a unique "fingerprint" is created for each video, including: Use pre-trained deep learning models such as YOLO (You Only Look Once) or Faster R-CNN is used to detect objects in video frames, pose estimation algorithms are applied to identify human actions, and scene recognition algorithms are used to identify the scene environment of video frames; for each identified object, action, and scene, its feature vector is extracted, which is completed through the internal layers of the model, usually the second to last layer (the layer before the fully connected layer). The feature vector is a point in a high-dimensional space that can represent the key information of the original data; the extracted feature vector is encoded, for example, an autoencoder or a hash function can be used. In order to create a unique "fingerprint" of the video, multiple feature vectors can be combined and an aggregation strategy such as vector averaging or principal component analysis (PCA) can be applied; the encoded feature vector is stored in a database, each vector is associated with the metadata of its original video (such as video ID, timestamp, etc.), here, the database can be a traditional relational database or a database specially designed for high-dimensional data; when a user uploads a retrieval picture, the system extracts the feature vector of the picture (i.e., the retrieval vector) and compares it with the feature vector stored in the database (i.e., the recognition vector), and uses an approximate nearest neighbor (ANN) search algorithm, such as the algorithm provided by Faiss, to quickly find the most similar video "fingerprint".

[0029] Further, in some embodiments of the present disclosure, when the step of associating the identification feature with the corresponding video and generating the identification feature database is performed, it can be implemented by but not limited to the following method, including the following steps: Obtaining a unique identifier for each of the videos; Associating the identification feature with the unique identifier of the corresponding video to obtain a first association relationship; The first association relationship is stored in the identification feature database.

[0030] The embodiment of the present disclosure needs to explain that the unique identifier of the video is the identity symbol of the video, which can uniquely identify the video, associate the identification feature with the unique identifier of the corresponding video, and uniquely identify the video based on the association, i.e., the first association relationship, to ensure the accuracy of video retrieval. The first association relationship is stored in the identification feature database, which facilitates the subsequent rapid matching of the corresponding video based on the identification feature, and is conducive to improving the efficiency of video retrieval.

[0031] Further, in some embodiments of the present disclosure, when the step of associating the identification feature with the corresponding video and generating the identification feature database is performed, it can be implemented by but not limited to the following method, including the following steps: Obtaining the time when the key frame image appears in the corresponding video as the positioning time; Associating the identification feature with the positioning time of the corresponding key frame image to obtain a second association relationship; The second association relationship is stored in the identification feature database.

[0032] It should be noted that the positioning time can reflect the time when the key frame image appears in the corresponding video, that is, when the video is played to the positioning time, the video frame image content appears in the video screen. The identification feature is associated with the corresponding positioning time, that is, the time when the key frame image appears in the video is recorded, so that when the user retrieves the video later, it can quickly locate the positioning time, and then confirm the content of the video frame image, and evaluate the accuracy of the video retrieval result. The second association relationship is stored in the identification feature database, which is convenient for quickly checking the video frame image, and then checking the accuracy of the video retrieval. At the same time, it is beneficial for users to understand the content of the video before and after the video frame appears. For example, when the retrieved image is from a TV series, the determination of the positioning time is beneficial for users to quickly locate the time when the retrieved image appears in the TV series, and then facilitate understanding of the plot before and after the video frame corresponding to the retrieved image. In addition, 3 milliseconds of information can be added after the positioning moment to ensure uniqueness and improve compatibility.

[0033] Furthermore, in some embodiments of the present disclosure, the video retrieval method further includes the following steps: Converting the key frame image into a recognition vector, and using the recognition vector as the recognition feature; Converting the search image into a search vector, and using the search vector as the search feature; Calculating the similarity between the search vector and each of the recognition vectors; The video corresponding to the recognition vector whose similarity meets a preset threshold is selected as the retrieval result.

[0034] It should be noted here that the embodiment of the present disclosure is that the image vectorization is to extract the image feature vector and use the feature vector to represent the image. The image feature vector is a mathematical representation used to describe the content of the image, which can reflect the color, texture, shape and other information of the image. The image feature vector can be used for image retrieval, classification, recognition, etc. There are many methods to extract image feature vectors, for example, SIFT algorithm and SURF algorithm. Among them, SIFT (Scale Invariant Feature Transform) algorithm is a scale-invariant feature extraction method, which can detect stable key points in different scale spaces and generate descriptors with uniqueness and invariance; and SURF (Speeded Up Robust Features) algorithm is a fast and robust feature extraction method, which is based on Harris corner detection and scale-invariant feature transform (SIFT) algorithm. Of course, the above method is only an example, and the method of image vectorization in this embodiment is not limited to this.

[0035] In the embodiment of the present disclosure, the key frame image and the search image are vectorized respectively to obtain the recognition vector and the search vector, which can accurately reflect the content of the key frame image and the search image and improve the accuracy of the matching between the two. When calculating the similarity between the search vector and each recognition vector, many methods can be used, for example, BFMatcher and FlannBasedMatcher, wherein BFMatcher is a brute force matching method, and its principle is that for each feature point in the first image, all feature points in the second image are traversed to find the closest one or more feature points as the matching result; FlannBasedMatcher is an approximate nearest neighbor matching method, and its principle is to use a fast index structure to speed up the search of feature points, thereby reducing time complexity. Then calculate the similarity score, for example, first calculate the distance of each matching pair, and then average all the distances to obtain a similarity score, the smaller the score, the more similar. Among them, the distance between the two vectors can be represented by the cosine distance. Specifically, in some embodiments, a threshold value can be set for the similarity score or the cosine distance to screen the search results to obtain one or more videos that meet the search accuracy requirements.

[0036] Furthermore, in some embodiments of the present disclosure, the video retrieval method further includes the following steps: According to the needs of the user, the acquired video is processed to obtain the target frame or video clip required by the user.

[0037] It should be noted here that, after the user obtains the unique identifier of the video through the backend server, the retrieved video can be processed according to different business and product requirements, for example, the original video frame corresponding to the retrieved image can be output, a certain segment of the video can be output, etc. The identified positioning time and the unique identifier of the video can also be returned to the client at the same time, so that the user can play the video from the positioning time.

[0038] Furthermore, in some embodiments of the present disclosure, the video retrieval method further includes the following steps: Calculating the positioning time difference between the searched image and the key frame image corresponding to the retrieved identification feature in the video; A positive and negative time window is set to determine whether the positioning time difference falls within the range of the positive and negative time window. If so, the retrieval result is a positive example, otherwise, the retrieval result is a negative example.

[0039] It should be noted here that the embodiment of the present disclosure has a unique correct time location in the original video, but the key frame image corresponding to the retrieved identification feature may be a picture before or after that time. It is only because the content of the picture does not change much in a short period of time that it is output as a retrieval result. The difference between the location time of the retrieved image and the location time of the key frame image is the location time difference. The positive and negative time windows are a time range that meets the user's accuracy requirements. The positive example indicates that the location time difference meets the accuracy requirements, and the negative example indicates that the location time difference does not meet the accuracy requirements. For example, the retrieval results within the positive and negative time windows set within plus or minus 3 seconds are positive examples, and other cases are negative examples. This method is used to reflect the accuracy of video retrieval. In order to evaluate the effect of image retrieval, the mAP (mean Average Precision, the average accuracy of all categories) indicator can be used to evaluate the retrieval results.

[0040] A more specific video retrieval method according to an embodiment of the present disclosure is given below, which includes at least the following steps: first, preprocessing: when a newly released video album needs to be included in the image search function, first use the offline service to extract frames from the video, divide the video into multiple key frame images, and then package the extracted key frame images and upload them to a file storage platform (such as OSS) so that the key frame images can be screened and modified in the future; second, identification feature extraction: download the key frame images of the album videos that need to be stored in the library, and perform reasoning through the AI ​​model to obtain the identification vector (i.e., identification feature) corresponding to each key frame image; third, storage of identification features: The recognition vector and the ID and time information of the relevant video are stored in a search engine (such as the Elasticsearch search engine). The latest official version of Elasticsearch (8.0) supports the KNN vector retrieval function, which can be used as a vector database here. Considering that the number of new album videos released is usually small, incremental insertion has little effect on real-time retrieval; Fourth, online recall, users upload retrieval pictures such as stills through the "camera" button of the APP, and the backend search server receives the retrieval pictures sent by the mobile phone, and performs format verification, compression and unified standard scaling, and then sends the processed retrieval picture request to the AI ​​model inference service. The recognition vector of the retrieval picture is obtained through AI model inference, and then the recognition vector is queried through Elasticsearch to obtain the top K results with the closest cosine distance to the retrieval picture and the corresponding scores, and then selects to output K videos or one or several of them according to user needs. Among them, KNN is the K-Nearest Neighbor (kNN) algorithm, which is a learning algorithm for supervised learning. The working mechanism of KNN is: given a test sample, find the K training samples closest to it in the training sample based on a certain distance metric, and then make predictions based on the information of these K neighbors. KNN actually uses the training data set to divide the feature space vector and use it as its classification model. The model has three basic elements: distance metric, selection of K value, and classification decision rule. Distance is a measure of the similarity between two features. There are many ways to measure distance, including Euclidean distance, cosine distance, Manhattan distance, etc.; different choices of K value will result in different final results. The setting of specific values ​​can be determined according to user needs. The second aspect of the embodiment of the present disclosure is introduced below: video retrieval system.

[0041] like Figure 2 As shown, the video retrieval system 200 includes: A storage unit 201 is used to store a database of identification features, wherein each identification feature is associated with a unique video; The acquisition unit 202 is used to acquire a search image input by a user and extract search features from the search image; The identification unit 203 is used to retrieve the matching identification feature according to the search feature; The feedback unit 204 is used to obtain the video associated with the matched identification feature according to the matched identification feature, and feed back the video to the user.

[0042] The video retrieval system 200 of the present invention extracts the recognition features of each video in advance, ensures the accurate association between the recognition features and the video, and stores the recognition features in the recognition feature database of the storage unit 201, which facilitates subsequent video retrieval and is beneficial to improving the efficiency of video retrieval; extracts the retrieval features of the retrieval picture input by the user through the acquisition unit 202, takes into account the content of the retrieval picture itself, and ensures the comprehensiveness of the retrieval features; matches the retrieval features and the recognition features through the recognition unit 203, which is beneficial to improving the accuracy of video retrieval; and feeds back the video to the user through the feedback unit 204 to complete the video retrieval.

[0043] The third and fourth aspects of the embodiments of the present disclosure are introduced below: an electronic device and a computer-readable storage medium.

[0044] The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the video retrieval method described above when executing the computer program.

[0045] A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the video retrieval method described above when executed by a processor.

[0046] like Figure 3 As shown, it shows a schematic diagram of an electronic device suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure is only an example and should not bring any limitation to the function and scope of use of the embodiment of the present disclosure.

[0047] Continue to refer to Figure 3 The electronic device includes a processing device 301, a storage device 302, a sensor device 303, a bus 304, an input / output (I / O) interface 305, a multimedia device 306, a power supply device 307, and a communication device 308.

[0048] The processing device 301 (such as a central processing unit, a graphics processing unit, etc.) is used to control the overall operation of the electronic device. The processing device 301 may include one or more processors to execute instructions to complete all or part of the steps of the above method. In addition, the processing device 301 may also include one or more modules for processing and interacting with other devices.

[0049] The storage device 302 is used to store various types of data. The storage device 302 may include various types of computer-readable storage media or a combination thereof, such as an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above.

[0050] Among them, more specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0051] The sensor device 303 is used to sense the specified measured information and convert it into a usable output signal according to a certain rule, and may include one or more sensors. For example, it may include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor, etc., for detecting changes in the opening / closing state, relative positioning, acceleration / deceleration, temperature, humidity and light of the electronic device.

[0052] The processing device 301 , the storage device 302 , and the sensor device 303 are connected to each other via a bus 304 . An input / output (I / O) interface 305 is also connected to the bus 304 .

[0053] The multimedia device 306 may include input devices such as a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, etc. for receiving input signals from a user. The various input devices may cooperate with various sensors of the above-mentioned sensor devices to complete, for example, gesture operation input, image recognition input, distance detection input, etc. The multimedia device may also include output devices such as a liquid crystal display (LCD), a speaker, a vibrator, etc.

[0054] The power supply device 307 is used to provide power to various devices in the electronic device, and may include a power management system, one or more power supplies, and components for distributing power to other devices.

[0055] The communication device 308 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data.

[0056] The above-mentioned devices can also be connected to the I / O interface 305 to realize the application of the electronic device.

[0057] Although the electronic device with various devices is shown in the figure, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0058] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0059] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0060] It should be noted that the above-mentioned computer-readable medium of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. In the present disclosure, the computer-readable signal medium may include a data signal propagated in the baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in combination with an instruction execution system, device or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0061] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0062] Further, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Python, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0063] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0064] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0065] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0066] Below, the features of the above-mentioned embodiments according to the present disclosure are briefly summarized and listed.

[0067] According to one or more embodiments of the present disclosure, a video retrieval method is provided, which adopts the following technical solution, including: generating a database of identification features, wherein each identification feature is associated with a unique video; Obtaining a search image input by a user, and extracting search features from the search image; According to the search feature, searching for the matching identification feature; According to the matched identification feature, the video associated with the matched identification feature is obtained.

[0068] According to one or more embodiments of the present disclosure, a video retrieval method is provided, which adopts the following technical solution, wherein generating an identification feature database, wherein each identification feature is associated with a unique video, includes: Extracting key frame images of each of the videos; Extracting the identification features from the key frame images; The identification feature is associated with the corresponding video, and the identification feature database is generated.

[0069] According to one or more embodiments of the present disclosure, a video retrieval method is provided, which adopts the following technical solution, wherein the identification feature is associated with the corresponding video and the identification feature database is generated, including: Obtaining a unique identifier for each of the videos; Associating the identification feature with the unique identifier of the corresponding video to obtain a first association relationship; The first association relationship is stored in the identification feature database.

[0070] According to one or more embodiments of the present disclosure, a video retrieval method is provided, which adopts the following technical solution, wherein the identification feature is associated with the corresponding video and the identification feature database is generated, and further includes: Obtaining the time when the key frame image appears in the corresponding video as the positioning time; Associating the identification feature with the positioning time of the corresponding key frame image to obtain a second association relationship; The second association relationship is stored in the identification feature database.

[0071] According to one or more embodiments of the present disclosure, a video retrieval method is provided, which adopts the following technical solution, and the video retrieval method further includes: Converting the key frame image into a recognition vector, and using the recognition vector as the recognition feature; Converting the search image into a search vector, and using the search vector as the search feature; Calculating the similarity between the search vector and each of the recognition vectors; The video corresponding to the recognition vector whose similarity meets a preset threshold is selected as the retrieval result.

[0072] According to one or more embodiments of the present disclosure, a video retrieval method is provided, which adopts the following technical solution, and the video retrieval method further includes: According to the needs of the user, the acquired video is processed to obtain the target frame or video clip required by the user.

[0073] According to one or more embodiments of the present disclosure, a video retrieval method is provided, which adopts the following technical solution, and the video retrieval method further includes: Calculating the positioning time difference between the searched image and the key frame image corresponding to the retrieved identification feature in the video; A positive and negative time window is set to determine whether the positioning time difference falls within the range of the positive and negative time window. If so, the retrieval result is a positive example, otherwise, the retrieval result is a negative example.

[0074] According to one or more embodiments of the present disclosure, a video retrieval system is provided, which adopts the following technical solution, and the video retrieval system includes: A storage unit, used to store a database of identification features, wherein each identification feature is associated with a unique video; An acquisition unit, used to acquire a search image input by a user and extract search features from the search image; An identification unit, configured to retrieve the matching identification feature according to the search feature; A feedback unit is used to obtain the video associated with the matched identification feature according to the matched identification feature, and feed back the video to the user.

[0075] According to one or more embodiments of the present disclosure, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the aforementioned method when executing the computer program.

[0076] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0077] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0078] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0079] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video retrieval method, characterized in that: include: generating a database of identification features, wherein each identification feature is associated with a unique video; Obtaining a search image input by a user, and extracting search features from the search image; According to the search feature, searching for the matching identification feature; According to the matched identification feature, the video associated with the matched identification feature is obtained.

2. The method according to claim 1, characterized in that The generating of the identification feature database, wherein each identification feature is associated with a unique video, includes: Extracting key frame images of each of the videos; Extracting the identification features from the key frame images; The identification feature is associated with the corresponding video, and the identification feature database is generated.

3. The method according to claim 2, characterized in that The step of associating the identification feature with the corresponding video and generating the identification feature database includes: Obtaining a unique identifier for each of the videos; Associating the identification feature with the unique identifier of the corresponding video to obtain a first association relationship; The first association relationship is stored in the identification feature database.

4. The method according to claim 3, characterized in that The step of associating the identification feature with the corresponding video and generating the identification feature database further includes: Obtaining the time when the key frame image appears in the corresponding video as the positioning time; Associating the identification feature with the positioning time of the corresponding key frame image to obtain a second association relationship; The second association relationship is stored in the identification feature database.

5. The method according to claim 2, characterized in that: The video retrieval method further comprises: Converting the key frame image into a recognition vector, and using the recognition vector as the recognition feature; Converting the search image into a search vector, and using the search vector as the search feature; Calculating the similarity between the search vector and each of the recognition vectors; The video corresponding to the recognition vector whose similarity meets a preset threshold is selected as the retrieval result.

6. The method according to claim 1, characterized in that The video retrieval method further comprises: According to the needs of the user, the acquired video is processed to obtain the target frame or video clip required by the user.

7. The method according to claim 2, characterized in that The video retrieval method further comprises: Calculating the positioning time difference between the searched image and the key frame image corresponding to the retrieved identification feature in the video; A positive and negative time window is set to determine whether the positioning time difference falls within the range of the positive and negative time window. If so, the retrieval result is a positive example, otherwise, the retrieval result is a negative example.

8. A video retrieval system, used to implement the method according to any one of claims 1 to 7, characterized in that: include: A storage unit, used to store a database of identification features, wherein each identification feature is associated with a unique video; An acquisition unit, used to acquire a search image input by a user and extract search features from the search image; An identification unit, configured to retrieve the matching identification feature according to the search feature; A feedback unit is used to obtain the video associated with the matched identification feature according to the matched identification feature, and feed back the video to the user.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images