Video retrieval method, retrieval device and electronic device

Abstract

The application provides a video retrieval method, a retrieval device and an electronic equipment. The retrieval method comprises the following steps: obtaining a target frame, wherein the target frame is obtained by detecting a predetermined object through a camera; performing feature extraction on the predetermined object in the target frame to obtain a predetermined feature vector; and searching whether a target object exists in the target frame according to the predetermined feature vector to obtain a search result, wherein the target object is one of the predetermined objects. In the scheme, the target frame is obtained by detecting the predetermined object through the camera, and then it is only necessary to search whether the target object exists in the target frame, without searching all frames in the video. In this way, the number of searches can be reduced, the search efficiency is guaranteed to be high, and the problem of low video search efficiency in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of video retrieval, and more specifically, to a video retrieval method, retrieval device, and electronic device. Background Technology

[0002] With the rapid development of video surveillance technology, the monitoring process generates massive amounts of data. Users often need to find specific objects from these large amounts of video footage. Manual searching is extremely time-consuming, labor-intensive, and inefficient, and users are prone to missing objects when fatigued. Therefore, automated methods for retrieving specific objects from surveillance videos are highly valuable.

[0003] The inventors understood that existing video retrieval methods typically detect salient targets such as people or vehicles in video frames, then use feature extractors to abstract the segmented target images into high-dimensional vector features, and match the high-dimensional vector features to be retrieved with high-dimensional vector features in the database to complete the retrieval. However, processing all frames of the video results in low efficiency.

[0004] Therefore, there is an urgent need for a solution that can improve the efficiency of video retrieval.

[0005] The information disclosed above in the background section is only intended to enhance the understanding of the background art of the art described herein. Therefore, the background art may contain certain information that does not constitute prior art known to those skilled in the art in this country. Summary of the Invention

[0006] The main objective of this application is to provide a video retrieval method, retrieval device, and electronic device to solve the problem of low efficiency in video retrieval in the prior art.

[0007] According to one aspect of the present invention, a video retrieval method is provided, comprising: acquiring a target frame, the target frame being obtained by detecting a predetermined object through a camera; extracting features from the predetermined object in the target frame to obtain a predetermined feature vector; and retrieving whether a target object exists in the target frame based on the predetermined feature vector to obtain a retrieval result, wherein the target object is one of the predetermined objects.

[0008] Optionally, when there are Q consecutive target frames, retrieve whether there is a target object in the target frames according to the predetermined feature vector, and obtain a retrieval result, including: starting from the first target frame, retrieve whether there is the target object in the target frame according to the predetermined feature vector; in the case of the target frame where the target object is retrieved, and determine the corresponding frame number as the starting frame number; perform trajectory tracking on multiple target frames after the starting frame number, determine the frame number of the target frame corresponding to the disappearance of the target object, obtain the ending frame number, and determine that the target object is detected in all target frames from the starting frame number to the ending frame number.

[0009] Optionally, perform trajectory tracking on multiple target frames after the starting frame number, determine the frame number of the target frame corresponding to the disappearance of the target object, obtain the ending frame number, and determine that the target object is detected in all target frames from the starting frame number to the ending frame number, including: retrieve P target frames after the starting frame number and calculate the retrieval success ratio, where the retrieval success ratio is the ratio of the target frames in which the target object is retrieved among the P target frames, and P + 1 < Q; in the case where the retrieval success ratio is greater than a predetermined value, determine that the target object is retrieved in all P target frames; start tracking the trajectory from the (P + 2)-th target frame, determine the frame number of the target frame corresponding to the disappearance of the target object, obtain the ending frame number, and determine that the target object is detected in all target frames from the starting frame number to the ending frame number.

[0010] Optionally, retrieve whether there is the target object in the target frame according to the predetermined feature vector, including: obtain the feature vector of the target object to get the target feature vector; determine the similarity between the predetermined feature vector and the target feature vector; according to the similarity, determine whether there is the target object in the target frame.

[0011] Optionally, obtain the feature vector of the target object, including: obtain a predetermined image, where the predetermined image includes the target object; based on a feature extractor, perform feature extraction on the predetermined image to obtain the feature vector of the target object.

[0012] Optionally, there are N obtained target feature vectors and any two of the target feature vectors are different, where N is a positive integer greater than 1. Determining the similarity between the predetermined feature vector and the target feature vector includes: determining the similarity between each of the predetermined feature vectors and the target feature vector to obtain N similarities; determining whether the target object exists in the target frame according to the similarity includes: obtaining the first M similarities in descending order among the multiple similarities, where 0 < M < N; determining that the target object is retrieved in the target frame when all M similarities are less than a predetermined distance threshold.

[0013] Optionally, performing feature extraction on the predetermined object in the target frame to obtain a predetermined feature vector includes: based on a feature extractor, performing feature extraction on the predetermined object in the target frame to obtain the predetermined feature vector, and the feature extractor is obtained by training through a neural network.

[0014] Optionally, after obtaining a retrieval result by retrieving whether a target object exists in the target frame according to the predetermined feature vector, the retrieval method further includes: optimizing the retrieval result based on reordering and outputting the optimized retrieval result.

[0015] According to another aspect of the embodiments of the present invention, there is also provided a video retrieval device, including: an acquisition unit configured to acquire a target frame, where the target frame is obtained by detecting a predetermined object through a camera; a feature extraction unit configured to perform feature extraction on the predetermined object in the target frame to obtain a predetermined feature vector; a retrieval unit configured to retrieve whether a target object exists in the target frame according to the predetermined feature vector to obtain a retrieval result, and the target object is one of the predetermined objects.

[0016] According to yet another aspect of the embodiments of the present invention, there is also provided an electronic device, including: a camera, one or more processors, a memory, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include those for executing any one of the video retrieval methods.

[0017] In this embodiment of the invention, the video retrieval method first obtains a target frame by detecting a predetermined object using a camera. Then, features of the predetermined object in the target frame are extracted to obtain a predetermined feature vector. Finally, based on the predetermined feature vector, the existence of a target object in the target frame is retrieved to obtain a retrieval result. In this scheme, the predetermined object is detected by a camera to obtain a target frame. Then, only the existence of a target object in the target frame needs to be retrieved; it is not necessary to retrieve all frames in the video. This reduces the number of retrievals, ensuring high retrieval efficiency and thus solving the problem of low efficiency in video retrieval in the prior art. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 A schematic diagram of a video retrieval method according to an embodiment of this application is shown;

[0020] Figure 2 A schematic diagram of a video retrieval device according to an embodiment of this application is shown. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] As mentioned in the background section, the efficiency of video retrieval in the prior art is relatively low. In order to solve the above problems, a typical embodiment of this application provides a video retrieval method, retrieval device and electronic device.

[0025] According to embodiments of this application, a video retrieval method is provided.

[0026] Figure 1 This is a flowchart of a video retrieval method according to an embodiment of this application. For example... Figure 1 As shown, the method includes the following steps:

[0027] Step S101: Obtain the target frame, which is obtained by detecting the predetermined object through the camera;

[0028] Step S102: Extract features from the predetermined object in the target frame to obtain a predetermined feature vector;

[0029] Step S103: Based on the predetermined feature vector, search for whether there is a target object in the target frame and obtain the search result. The target object is one of the predetermined objects.

[0030] In the aforementioned video retrieval method, firstly, a target frame is obtained by detecting a predetermined object using a camera. Then, features of the predetermined object in the target frame are extracted to obtain a predetermined feature vector. Finally, based on the predetermined feature vector, the presence of a target object in the target frame is checked to obtain the retrieval result. In this scheme, the predetermined object is detected by a camera to obtain a target frame, and then only the presence of a target object in the target frame needs to be checked, eliminating the need to search all frames in the video. This reduces the number of searches, ensuring high retrieval efficiency and thus solving the problem of low efficiency in existing video retrieval technologies.

[0031] Of course, in practical applications, the above-mentioned cameras can be network cameras, but are not limited to network cameras; they can also be other types of cameras.

[0032] In one specific embodiment of this application, motion detection, human detection, vehicle detection, or detection of other objects of interest are performed on a network camera, and information such as frames containing specific targets, target locations, and target types are stored. The target location can be the specific position of the predetermined object within a target frame, and the target type can be the type of the predetermined object. For example, when the detected target object is a person, the predetermined object is a person, and the target frame is a frame containing a person. After detection by the network camera, the information such as frames containing targets, target locations, and target types can be stored on a server, but is not limited to a server; it can also be stored in other locations, such as a cloud platform.

[0033] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0034] Specifically, in order to further ensure high retrieval efficiency and reduce the number of retrievals, in one embodiment of this application, when there are Q consecutive target frames, the retrieval of whether a target object exists in the target frames is based on the predetermined feature vector to obtain retrieval results, including: starting from the first target frame, the retrieval of whether a target object exists in the target frames based on the predetermined feature vector; if a target frame containing the target object is found, the corresponding frame number is determined as the starting frame number; trajectory tracking is performed on multiple target frames after the starting frame number to determine the frame number of the target frame where the target object disappears, obtaining the ending frame number, and determining that the target object has been detected in all target frames from the starting frame number to the ending frame number.

[0035] In another embodiment of the present application, trajectory tracking is performed on multiple of the above-mentioned target frames after the above-mentioned starting frame number, the frame number of the above-mentioned target frame corresponding to the disappearance of the above-mentioned target object is determined to obtain an end frame number, and it is determined that the above-mentioned target object is detected in all the above-mentioned target frames from the above-mentioned starting frame number to the above-mentioned end frame number, including: retrieving P of the above-mentioned target frames after the above-mentioned starting frame number and calculating a retrieval success ratio, where the retrieval success ratio is the ratio of the target object retrieved in the P target frames, and P + 1 < Q; in the case where the retrieval success ratio is greater than a predetermined value, it is determined that the target object is retrieved in all P of the above-mentioned target frames; starting from the (P + 2)-th of the above-mentioned target frames, tracking trajectories are performed, and the frame number of the above-mentioned target frame corresponding to the disappearance of the above-mentioned target object is determined to obtain an end frame number, and it is determined that the above-mentioned target object is detected in all the above-mentioned target frames from the above-mentioned starting frame number to the above-mentioned end frame number. In this solution, in the case where the retrieval success ratio is greater than a predetermined value, it is determined that the target object is retrieved in all P of the above-mentioned target frames, which ensures that the retrieval result is relatively accurate. Subsequently, in the case where the target object is determined to be retrieved, the target frames are not retrieved anymore, but the trajectories of the above-mentioned target frames are tracked, which can further reduce the number of retrievals and further ensure a relatively high retrieval efficiency.

[0036] Specifically, in the actual application process, the present application does not limit the above-mentioned predetermined value, and only needs to be able to determine that the target object is retrieved in all P target frames according to the actual situation.

[0037] In another embodiment of the present application to ensure that the determined result is relatively accurate, according to the above-mentioned predetermined feature vector, it is retrieved whether the above-mentioned target object exists in the above-mentioned target frame, including: obtaining the feature vector of the above-mentioned target object to obtain a target feature vector; determining the similarity between the above-mentioned predetermined feature vector and the above-mentioned target feature vector; and determining whether the above-mentioned target object exists in the above-mentioned target frame according to the above-mentioned similarity.

[0038] In a specific embodiment of the present application, the above-mentioned similarity includes, but is not limited to, cosine distance, Jaccard distance, Levenshtein distance, Hamming distance, etc.

[0039] In still another embodiment of the present application, obtaining the feature vector of the above-mentioned target object includes: obtaining a predetermined image, where the above-mentioned predetermined image includes the above-mentioned target object; and performing feature extraction on the above-mentioned predetermined image based on a feature extractor to obtain the above-mentioned feature vector of the above-mentioned target object. Specifically, the feature extraction of the predetermined image by the feature extractor ensures that the feature vector of the target object can be obtained relatively efficiently.

[0040] In the actual application process, the above-mentioned predetermined image can be an image obtained by the camera of a mobile phone, or an image obtained by a video camera, or an image pre-stored in a database. In this application, the acquisition method of the above-mentioned predetermined image is not limited to the above. In addition, the above-mentioned predetermined image can be one or multiple. In this application, the number of the above-mentioned predetermined images is not limited to the above. During the process of feature extraction of the above-mentioned predetermined image, the above-mentioned feature extractor automatically extracts the above-mentioned feature vector of the above-mentioned target object and saves the extracted above-mentioned feature vector to form a database for subsequent retrieval.

[0041] In one embodiment of this application, there are N target feature vectors obtained and any two of the above-mentioned target feature vectors are different, where N is a positive integer greater than 1. Determining the similarity between the above-mentioned predetermined feature vector and the above-mentioned target feature vector includes: determining the similarity between each of the above-mentioned predetermined feature vectors and the above-mentioned target feature vector to obtain N such similarities; according to the above-mentioned similarities, determining whether the above-mentioned target object exists in the above-mentioned target frame includes: obtaining the first M of the above-mentioned multiple similarities in descending order, where 0 < M < N; in the case where all M of the above-mentioned similarities are less than a predetermined distance threshold, it is determined that the above-mentioned target object is retrieved in the above-mentioned target frame. In this solution, in the case where it is determined that all M similarities are less than the predetermined distance threshold, it is determined that the target object is retrieved in the target frame, which ensures the accuracy of the retrieval result.

[0042] In the actual application process, this application does not limit the above-mentioned predetermined distance threshold, and only needs to be able to determine that the above-mentioned target object is retrieved in the above-mentioned target frame in the actual application process.

[0043] In a specific embodiment of this application, for a target in a video frame, calculate the distances between the mapping vector of this target and the mapping vectors of all query targets in the database, and select the M data with the closest distances. If a certain proportion of the same target exists among the M data and the distance is less than the predetermined distance threshold, it indicates that the target is retrieved; otherwise, it indicates that the target does not exist in the database.

[0044] In order to ensure efficient feature extraction of a predetermined object, in another embodiment of this application, feature extraction of the above-mentioned predetermined object in the above-mentioned target frame to obtain a predetermined feature vector includes: based on a feature extractor, performing feature extraction on the above-mentioned predetermined object in the above-mentioned target frame to obtain the above-mentioned predetermined feature vector, and the above-mentioned feature extractor is obtained by training through a neural network.

[0045] In one specific embodiment of this application, the feature extractor can be trained using a neural network. This neural network can be ResNet or a convolutional neural network, but is not limited to ResNet and convolutional neural networks; other neural networks are also possible. During the training of the feature extractor, the training dataset uses a dataset of the target objects of interest, including but not limited to adults, children, cars, and pets. The ratio of the training dataset to the test dataset can be 8:2 or 7:3, but is not limited to these two ratios; other suitable ratios are also possible.

[0046] In actual feature extraction, training loss functions include, but are not limited to, cross-entropy loss, center loss, triplet loss, etc. Furthermore, in this application, the feature extraction process of the target frame can be executed on the server side, but is not limited to the server side mentioned above; it can also be executed on a computer, cloud platform, etc.

[0047] In another embodiment of this application, after searching for the existence of a target object in the target frame based on the predetermined feature vector and obtaining the search result, the search method further includes: optimizing the search result based on reordering and outputting the optimized search result, which further ensures higher search accuracy and more accurate search results.

[0048] In practical applications, the above reordering methods can be Average Query Expansion (AQE) or K-reciprocal encoding, but are not limited to these two methods.

[0049] In one specific embodiment of this application, re-ranking is achieved by optimizing the initial search results to obtain secondary search results, thereby improving search accuracy. A common method involves averaging the M feature vectors retrieved in the first search, and then using this average feature vector to search for the M nearest feature vectors in the query set, which are then used as the secondary search results.

[0050] This application also provides a video retrieval device. It should be noted that the video retrieval device of this application can be used to execute the video retrieval method provided in this application. The video retrieval device provided in this application is described below.

[0051] Figure 2 This is a schematic diagram of a video retrieval device according to an embodiment of this application. For example... Figure 2 As shown, the device includes:

[0052] The acquisition unit 10 is used to acquire a target frame, which is obtained by detecting a predetermined object through a camera;

[0053] Feature extraction unit 20 is used to extract features from the predetermined object in the target frame to obtain a predetermined feature vector;

[0054] The retrieval unit 30 is used to retrieve whether a target object exists in the target frame based on the predetermined feature vector, and obtain the retrieval result, wherein the target object is one of the predetermined objects.

[0055] In the aforementioned video retrieval device, the acquisition unit acquires a target frame, which is obtained by detecting a predetermined object through a camera; the feature extraction unit extracts features from the predetermined object in the target frame to obtain a predetermined feature vector; and the retrieval unit searches for the existence of a target object in the target frame based on the predetermined feature vector, obtaining a retrieval result, where the target object is one of the predetermined objects. In this scheme, the predetermined object is detected by a camera to obtain the target frame, and then only the existence of the target object in the target frame needs to be searched, eliminating the need to search all frames in the video. This reduces the number of searches, ensuring high retrieval efficiency and thus solving the problem of low efficiency in video retrieval in the prior art.

[0056] Of course, in practical applications, the above-mentioned cameras can be network cameras, but are not limited to network cameras; they can also be other types of cameras.

[0057] In one specific embodiment of this application, motion detection, human detection, vehicle detection, or detection of other objects of interest are performed on a network camera, and information such as frames containing specific targets, target locations, and target types are stored. The target location can be the specific position of the predetermined object within a target frame, and the target type can be the type of the predetermined object. For example, when the detected target object is a person, the predetermined object is a person, and the target frame is a frame containing a person. After detection by the network camera, the information such as frames containing targets, target locations, and target types can be stored on a server, but is not limited to a server; it can also be stored in other locations, such as a cloud platform.

[0058] Specifically, in order to further ensure a relatively high retrieval efficiency and reduce the number of retrievals, in an embodiment of the present application, the above-mentioned retrieval unit further includes a first retrieval module, a first determination module, and a second determination module. Among them, the first retrieval module is configured to, when there are Q consecutive target frames, starting from the first target frame, retrieve whether the target object exists in the target frames according to the above-mentioned predetermined feature vector; the first determination module is configured to, when the target object is retrieved in the target frame, and determine the corresponding frame number as the starting frame number; the second determination module is configured to perform trajectory tracking on multiple target frames after the starting frame number, determine the frame number of the target frame corresponding to the disappearance of the target object, obtain the ending frame number, and determine that the target object is detected in all the target frames from the starting frame number to the ending frame number.

[0059] In another embodiment of the present application, the second determination module further includes a calculation sub-module, a first determination sub-module, and a second determination sub-module. Among them, the calculation sub-module is configured to retrieve P target frames after the starting frame number and calculate the retrieval success ratio. The retrieval success ratio is the ratio of the target frames in which the target object is retrieved among the P target frames, where P + 1 < Q; the first determination sub-module is configured to determine that the target object is retrieved in all P target frames when the retrieval success ratio is greater than a predetermined value; the second determination sub-module is configured to start tracking the trajectory from the (P + 2)-th target frame, determine the frame number of the target frame corresponding to the disappearance of the target object, obtain the ending frame number, and determine that the target object is detected in all the target frames from the starting frame number to the ending frame number. In this solution, when the retrieval success ratio is greater than the predetermined value, it is determined that the target object is retrieved in all the P target frames, which ensures that the retrieval result is relatively accurate. Subsequently, when it is determined that the target object is retrieved, the target frames are not retrieved anymore, but the trajectory of the target frames is tracked, which can further reduce the number of retrievals and further ensure a relatively high retrieval efficiency.

[0060] Specifically, in the actual application process, the present application does not limit the above-mentioned predetermined value, and only needs to be able to determine that the target object is retrieved in all P target frames according to the actual situation.

[0061] In another embodiment of the present application, to ensure that the determined result is relatively accurate, the above first retrieval module further includes a first acquisition sub-module, a third determination sub-module, and a fourth determination sub-module. Among them, the first acquisition sub-module is used to acquire the feature vector of the above target object to obtain a target feature vector; the third determination sub-module is used to determine the similarity between the above predetermined feature vector and the above target feature vector; the fourth determination sub-module is used to determine whether the above target object exists in the above target frame according to the above similarity.

[0062] In a specific embodiment of the present application, the above similarity includes, but is not limited to, cosine distance, Jaccard distance, Levenshtein distance, Hamming distance, etc.

[0063] In another embodiment of the present application, the first acquisition sub-module further includes a second acquisition sub-module and a first feature extraction sub-module. Among them, the second acquisition sub-module is used to acquire a predetermined image, and the predetermined image includes the above target object; the first feature extraction sub-module is used to perform feature extraction on the predetermined image based on the above feature extractor to obtain the above feature vector of the above target object.

[0064] In the actual application process, the above predetermined image can be an image obtained by the camera of a mobile phone, or an image obtained by a camera, or an image stored in a database in advance. In the present application, it is not limited to the acquisition method of the above predetermined image. In addition, the above predetermined image can be one or multiple. In the present application, it is not limited to the number of the above predetermined images. In the process of performing feature extraction on the above predetermined image, the above feature extractor automatically extracts the above feature vector of the above target object and saves the extracted above feature vector to form a database for subsequent retrieval.

[0065] In an embodiment of the present application, the above third determination sub-module includes a fifth determination sub-module, which is used to determine the similarity between each of the above predetermined feature vectors and the above target feature vector to obtain N similarities when there are N target feature vectors obtained and any two of the above target feature vectors are different, and N is a positive integer greater than 1; the fourth determination sub-module includes a third acquisition sub-module and a sixth determination sub-module. Among them, the third acquisition sub-module is used to acquire the first M similarities in the order from largest to smallest among the multiple above similarities, where 0 < M < N; the sixth determination sub-module is used to determine that the above target object is retrieved in the above target frame when all M of the above similarities are less than a predetermined distance threshold. In this solution, when it is determined that all M similarities are less than the predetermined distance threshold, it is determined that the target object is retrieved in the target frame, which ensures that the retrieval result is relatively accurate.

[0066] In practical applications, this application does not limit the aforementioned predetermined distance threshold; it only requires that the target object be retrieved in the target frame in practical applications.

[0067] In one specific embodiment of this application, for a target in a video frame, the distance between the target's mapping vector and the mapping vectors of all query targets in the database is calculated, and the M closest data are selected. If a certain proportion of the M data contain the same target and the distance is less than a predetermined distance threshold, it indicates that the target has been retrieved; otherwise, it indicates that the target does not exist in the database.

[0068] In order to ensure that the feature extraction of the predetermined object can be performed more efficiently, in another embodiment of this application, the feature extraction unit further includes a feature extraction module, which is used to extract features of the predetermined object in the target frame based on the feature extractor to obtain the predetermined feature vector. The feature extractor is obtained by training a neural network.

[0069] In one specific embodiment of this application, the feature extractor can be trained using a neural network. This neural network can be ResNet or a convolutional neural network, but is not limited to ResNet and convolutional neural networks; other neural networks are also possible. During the training of the feature extractor, the training dataset uses a dataset of the target objects of interest, including but not limited to adults, children, cars, and pets. The ratio of the training dataset to the test dataset can be 8:2 or 7:3, but is not limited to these two ratios; other suitable ratios are also possible.

[0070] In actual feature extraction, training loss functions include, but are not limited to, cross-entropy loss, center loss, triplet loss, etc. Furthermore, in this application, the feature extraction process of the target frame can be executed on the server side, but is not limited to the server side mentioned above; it can also be executed on a computer, cloud platform, etc.

[0071] In another embodiment of this application, the retrieval device further includes an optimization unit, which is used to optimize the retrieval results based on reordering after retrieving whether a target object exists in the target frame according to the predetermined feature vector and obtaining the retrieval results, and output the optimized retrieval results, thereby further ensuring higher retrieval accuracy and more accurate retrieval results.

[0072] In practical applications, the above reordering methods can be Average Query Expansion (AQE) or K-reciprocal encoding, but are not limited to these two methods.

[0073] In one specific embodiment of this application, re-ranking is achieved by optimizing the initial search results to obtain secondary search results, thereby improving search accuracy. A common method involves averaging the M feature vectors retrieved in the first search, and then using this average feature vector to search for the M nearest feature vectors in the query set, which are then used as the secondary search results.

[0074] The aforementioned video retrieval device includes a processor and a memory. The aforementioned acquisition unit, feature extraction unit, and retrieval unit are all stored as program units in the memory, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0075] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the low efficiency of video retrieval in existing technologies.

[0076] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0077] This invention provides an electronic device, including: a camera, one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for performing any of the above-described video retrieval methods.

[0078] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0079] Step S101: Obtain the target frame, which is obtained by detecting the predetermined object through the camera;

[0080] Step S102: Extract features from the predetermined object in the target frame to obtain a predetermined feature vector;

[0081] Step S103: Based on the predetermined feature vector, search for whether there is a target object in the target frame and obtain the search result. The target object is one of the predetermined objects.

[0082] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0083] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0084] Step S101: Obtain the target frame, which is obtained by detecting the predetermined object through the camera;

[0085] Step S102: Extract features from the predetermined object in the target frame to obtain a predetermined feature vector;

[0086] Step S103: Based on the predetermined feature vector, search for whether there is a target object in the target frame and obtain the search result. The target object is one of the predetermined objects.

[0087] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units described above can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0089] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0091] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0092] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0093] 1) In the video retrieval method of this application, firstly, a target frame is obtained by detecting a predetermined object using a camera; then, features of the predetermined object in the target frame are extracted to obtain a predetermined feature vector; finally, based on the predetermined feature vector, the existence of a target object in the target frame is retrieved to obtain the retrieval result. In this scheme, the predetermined object is detected by a camera to obtain a target frame, and then it is only necessary to retrieve whether a target object exists in the target frame, without having to retrieve all frames in the video. This reduces the number of retrievals and ensures high retrieval efficiency, thereby solving the problem of low efficiency in video retrieval in the prior art.

[0094] 2) In the video retrieval device of this application, the acquisition unit is used to acquire a target frame, which is obtained by detecting a predetermined object through a camera; the feature extraction unit is used to extract features from the predetermined object in the target frame to obtain a predetermined feature vector; the retrieval unit is used to retrieve whether a target object exists in the target frame based on the predetermined feature vector, and obtain a retrieval result, wherein the target object is one of the predetermined objects. In this scheme, the predetermined object is detected by a camera to obtain a target frame, and then it is only necessary to retrieve whether a target object exists in the target frame, without having to retrieve all frames in the video. This reduces the number of retrievals and ensures high retrieval efficiency, thereby solving the problem of low efficiency in video retrieval in the prior art.

[0095] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A retrieval method of a video, characterized by, The method comprises the following steps: obtaining a target frame, the target frame being obtained by detecting a predetermined object through a camera; extracting features of the predetermined object in the target frame to obtain a predetermined feature vector; searching whether a target object exists in the target frame according to the predetermined feature vector to obtain a search result, the target object being one of the predetermined objects; when there are Q consecutive target frames, searching whether a target object exists in the target frame according to the predetermined feature vector to obtain a search result, comprising: starting from the first target frame, searching whether the target object exists in the target frame according to the predetermined feature vector; when the target object is searched in the target frame, determining the corresponding frame number as a starting frame number; performing trajectory tracking on a plurality of target frames after the starting frame number to determine the frame number of the target frame corresponding to the disappearance of the target object, obtaining an ending frame number, and determining that all target frames from the starting frame number to the ending frame number detect the target object, searching whether the target object exists in the target frame according to the predetermined feature vector, comprising: obtaining a feature vector of the target object to obtain a target feature vector; determining the similarity between the predetermined feature vector and the target feature vector; and determining whether the target object exists in the target frame according to the similarity, obtaining the feature vector of the target object, comprising: obtaining a predetermined image, the predetermined image comprising the target object; and performing feature extraction on the predetermined image based on a feature extractor to obtain the feature vector of the target object, the target feature vector obtained has N target feature vectors and any two target feature vectors are different, N being a positive integer greater than 1, and the similarity between the predetermined feature vector and the target feature vector is determined, comprising: determining the similarity between each predetermined feature vector and the target feature vector to obtain N similarities; and determining whether the target object exists in the target frame according to the similarity, comprising: obtaining the first M similarities in the M similarities in descending order, 0 < M < N; and when the M similarities are all less than a predetermined distance threshold, determining that the target object is searched in the target frame.

2. The method of claim 1, wherein, performing trajectory tracking on a plurality of target frames after the starting frame number to determine the frame number of the target frame corresponding to the disappearance of the target object, obtaining an ending frame number, and determining that all target frames from the starting frame number to the ending frame number detect the target object, comprising: searching P target frames after the starting frame number and calculating a search success ratio, the search success ratio being the ratio of the target objects searched in the P target frames, wherein P+1 < Q; when the search success ratio is greater than a predetermined value, determining that the target object is searched in the P target frames; Tracking trajectories from a P+2th target frame, and determining a frame number of the target frame corresponding to disappearance of the target object, to obtain an end frame number, and determining that all the target frames from the start frame number to the end frame number detect the target object.

3. The search method according to any one of claims 1 or 2, characterized in that, The predetermined object in the target frame is subjected to feature extraction to obtain a predetermined feature vector, including: Based on a feature extractor, the predetermined object in the target frame is subjected to feature extraction to obtain the predetermined feature vector, and the feature extractor is obtained by training a neural network.

4. The search method according to any one of claims 1 or 2, wherein, After retrieving whether there is a target object in the target frame based on the predetermined feature vector to obtain a retrieval result, the retrieval method further includes: Based on reordering, the retrieval result is optimized to output the optimized retrieval result.

5. A retrieval apparatus of a video, characterized by comprising: Including: An acquisition unit is configured to acquire a target frame, the target frame being obtained by detecting a predetermined object through a camera; A feature extraction unit is configured to extract features of the predetermined object in the target frame to obtain a predetermined feature vector; A retrieval unit is configured to retrieve whether there is a target object in the target frame based on the predetermined feature vector to obtain a retrieval result, the target object being one of the predetermined objects; The retrieval unit further includes a first retrieval module, a first determination module and a second determination module, wherein the first retrieval module is configured to, when there are continuous Q target frames, start from a first target frame, retrieve whether there is the target object in the target frame based on the predetermined feature vector; the first determination module is configured to, when the target frame in which the target object exists is retrieved, determine that a corresponding frame number is a start frame number; and the second determination module is configured to track trajectories of a plurality of target frames after the start frame number, determine a frame number of the target frame corresponding to disappearance of the target object, obtain an end frame number, and determine that all the target frames from the start frame number to the end frame number detect the target object. The first retrieval module further includes a first acquisition submodule, a third determination submodule and a fourth determination submodule, wherein the first acquisition submodule is configured to acquire a feature vector of the target object to obtain a target feature vector; the third determination submodule is configured to determine a similarity between the predetermined feature vector and the target feature vector; and the fourth determination submodule is configured to determine whether there is the target object in the target frame based on the similarity. The first acquisition submodule further includes a second acquisition submodule and a first feature extraction submodule, wherein the second acquisition submodule is configured to acquire a predetermined image, the predetermined image including the target object; and the first feature extraction submodule is configured to extract features of the predetermined image based on a feature extractor to obtain the feature vector of the target object. The fourth determination submodule comprises a third acquisition submodule and a sixth determination submodule. The third acquisition submodule is configured to acquire M largest similarities in the plurality of similarities in descending order, 0 < M < N. The sixth determination submodule is configured to determine that the target object is retrieved in the target frame when the M similarities are all less than a predetermined distance threshold.

6. An electronic device, comprising: Comprise: A camera, one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise a program for executing the video retrieval method in any one of claims 1 to 4.

Images