Video retrieval method, device and storage medium

Abstract

The application relates to the technical field of video processing, and discloses a video retrieval method, a device and a storage medium. The method comprises the following steps: acquiring a retrieval request carrying retrieval information; retrieving a target video frame from a database based on the retrieval information; and outputting the target video frame. In the manner, the application can quickly and accurately retrieve a video frame meeting requirements from a video.

Description

Technical Field

[0001] This application relates to the field of video processing technology, specifically to a video retrieval method, device, and storage medium. Background Technology

[0002] With the development of computer and mobile internet technologies, video cameras are widely used in various security monitoring systems. In the security monitoring field, the proliferation of video cameras and their continuous 24 / 7 recording have led to a rapid increase in video data volume. When emergencies occur or specific targets need to be traced, users often need to extract relevant key video frames from massive amounts of video data. Relying on manual frame-by-frame review to filter eligible video frames is not only time-consuming, labor-intensive, and inefficient, but also prone to causing crucial clues to be missed due to visual fatigue. How to achieve automated retrieval to quickly and accurately obtain target video frames is a problem that needs to be solved. Summary of the Invention

[0003] In view of the above problems, this application provides a video retrieval method, device and storage medium to solve the problems of low efficiency and accuracy of video retrieval in the prior art.

[0004] According to one aspect of the embodiments of this application, a video retrieval method is provided, the method comprising: acquiring a retrieval request carrying retrieval information; retrieving a target video frame from a database based on the retrieval information; and outputting the target video frame.

[0005] In one optional approach, the retrieval information includes a target retrieval object, and the database includes multiple video frames and multiple sets of tag information, with each video frame corresponding to one of the multiple sets of tag information; retrieving the target video frame from the database based on the retrieval information includes: retrieving candidate video frames from the database whose tag information matches the retrieval information; and retrieving the target video frame from the candidate video frames based on the target retrieval object.

[0006] In one optional approach, the tag information includes category information of objects in the video frames corresponding to the tag information; retrieving candidate video frames from the database that match the tag information and the retrieval information includes: determining the category attribute of the target retrieval object in the category belonging relationship based on a preset category belonging relationship, wherein the category attribute includes subordinate class objects and superior class objects, the category belonging relationship is the belonging relationship between subordinate class objects and superior class objects, and the category attribute of the category information in the tag information is a superior class object; if the target retrieval object is a subordinate class object in the category belonging relationship, then the target retrieval object is mapped to its subordinate target superior class object, and video frames whose category information in the tag information contains the category of the target superior class object are retrieved from the database to obtain the candidate video frames; if the target retrieval object is a superior class object in the category belonging relationship, then video frames whose category information in the tag information contains the category of the target retrieval object are retrieved from the database to obtain the candidate video frames.

[0007] In one optional approach, the retrieval information further includes a target time period, and the tag information further includes the acquisition time of the video frame corresponding to the tag information; the step of mapping the target retrieval object to its corresponding target superordinate class object if the target retrieval object is a subordinate class object in the category attribution relationship, and retrieving video frames from the database whose category information in the tag information contains the category of the target superordinate class object to obtain the candidate video frames, includes: if the target retrieval object is a subordinate class object in the category attribution relationship, mapping the target retrieval object to its corresponding target superordinate class object, and retrieving video frames from the database whose category information in the tag information contains the category of the target superordinate class object to obtain the candidate video frames, including .... The process of obtaining candidate video frames by retrieving video frames whose category information includes the category of the target superordinate object and whose acquisition time belongs to the target time period, and then retrieving video frames whose category information includes the category of the target retrieval object from the tag information in the database, if the target retrieval object is the superordinate object in the category attribution relationship, includes: retrieving video frames whose category information includes the category of the target retrieval object from the tag information in the database and whose acquisition time belongs to the target time period, if the target retrieval object is the superordinate object in the category attribution relationship, and then retrieving video frames whose category information includes the category of the target retrieval object and whose acquisition time belongs to the target time period, from the tag information in the database, and then retrieving candidate video frames.

[0008] In one optional approach, the retrieval information further includes a target identifier. The multiple video frames are acquired by multiple camera devices, and each camera device corresponds one-to-one with a multiple identifier. The tag information also includes the identifier of the camera device for the video frame corresponding to the tag information. If the target retrieval object is a subordinate class object in the category attribution relationship, then the target retrieval object is mapped to its subordinate target superordinate class object, and video frames whose category information in the tag information contains the category of the subordinate target class object are retrieved from the database to obtain the candidate video frames. This includes: if the target retrieval object is a subordinate class object in the category attribution relationship, then the target retrieval object is mapped to its subordinate target superordinate class object. The process involves retrieving video frames from the database whose category information in the tag information contains the category of the target parent object and whose camera device identifier is the target identifier, to obtain candidate video frames. The step of retrieving video frames from the database whose category information in the tag information contains the category of the target retrieval object and whose camera device identifier is the target identifier, if the target retrieval object is a parent object in the category attribution relationship, includes: retrieving video frames from the database whose category information in the tag information contains the category of the target retrieval object and whose camera device identifier is the target identifier, to obtain candidate video frames.

[0009] In one optional approach, the retrieval information includes a target retrieval object, and the database includes multiple video frames and multiple sets of tag information, with each video frame corresponding one-to-one with the multiple sets of tag information; retrieving the target video frame from the database based on the retrieval information includes: determining a first verification object based on the target retrieval object; and retrieving the target video frame from the database based on the target retrieval object and the first verification object.

[0010] In one optional approach, retrieving the target video frame from the database based on the target retrieval object and the first verification object includes: determining the similarity between the video frame in the database and the target retrieval object to obtain a first similarity corresponding to the video frame; determining the similarity between the video frame in the database and the first verification object to obtain a second similarity corresponding to the video frame; and retrieving the target video frame from the database based on the first similarity and the second similarity.

[0011] In one optional approach, retrieving the target video frame from the database based on the first similarity and the second similarity includes: determining the similarity between the video frame in the database and a second verification object, obtaining a third similarity corresponding to the video frame, wherein the second verification object includes the target retrieval object and the first verification object; determining a video frame that meets the conditions as the target video frame, wherein the video frame that meets the conditions is one where the maximum value among the first similarity, the second similarity, and the third similarity corresponding to the video frame is either the first similarity or the third similarity, and the first similarity is greater than a first similarity threshold.

[0012] In one optional approach, the retrieval information includes a target retrieval object; retrieving target video frames from the database based on the retrieval information includes: determining the similarity between video frames in the database and the target retrieval object, obtaining a first similarity corresponding to the video frame; identifying video frames with the first similarity greater than a first similarity threshold as the target video frames; and outputting the target video frames includes: displaying target video frames with the first similarity greater than a second similarity threshold in a first display format; and displaying target video frames with the first similarity less than or equal to the second similarity threshold in a second display format, wherein the second similarity threshold is higher than the first similarity threshold; wherein the first similarity threshold and the second similarity threshold are predetermined through the following steps: A sample set is obtained, comprising multiple sets of positive sample image-text pairs and multiple sets of negative sample image-text pairs; the similarity of each sample image-text pair in the sample set is calculated to obtain a positive sample similarity set and a negative sample similarity set; in the same coordinate system, multiple positive sample similarities in the positive sample similarity set are fitted into a positive sample similarity distribution curve, and multiple negative sample similarities in the negative sample similarity set are fitted into a negative sample similarity distribution curve; the similarity corresponding to the intersection point of the positive sample similarity distribution curve and the negative sample similarity distribution curve is determined as the first similarity threshold; a similarity interval that includes only the positive sample similarity and excludes the negative sample similarity is determined in the coordinate system, and one similarity in the similarity interval is determined as the second similarity threshold.

[0013] In one alternative approach, the database is pre-established through the following steps: acquiring a video stream; performing target detection on video frames in the video stream; if a video frame in the video stream is detected to include a preset target object, then extracting a video frame from the video stream at a first frame extraction rate as a first database-building video frame; if no video frame in the video stream is detected to include the preset target object, then extracting a video frame from the video stream at a second frame extraction rate as a first database-building video frame, wherein the second frame extraction rate is lower than the first frame extraction rate; and establishing the database based on the first database-building video frame.

[0014] In one optional embodiment, the video stream is a video stream captured at a first frame rate. The step of performing target detection on the video frames in the video stream includes: performing target detection on the video frames in the video stream at a preset second frame rate, wherein the detected video frames form a first video frame set, and the second frame rate is lower than the first frame rate; the step of extracting video frames from the video stream at a first frame extraction rate as first database-building video frames if a video frame in the video stream is detected to include the preset target object, and then extracting video frames from the first video frame set at the first frame extraction rate as first database-building video frames; and the step of extracting video frames from the video stream at a second frame extraction rate as first database-building video frames if no video frame in the video stream is detected to include the preset target object, and then extracting video frames from the first video frame set at the second frame extraction rate as first database-building video frames.

[0015] In one optional approach, the first video frame set includes M video frames ordered chronologically by acquisition time, the first frame extraction rate is N1 / M, and the second frame extraction rate is N2 / M, where N1, N2, and M are all positive integers, and N2 < N1 < M; the step of extracting video frames from the first video frame set at the first frame extraction rate if the detected video frame in the video stream includes the preset target object, and using the first frame extraction rate as the first database video frame, includes: if there are K first video frames in the first video frame set that include the preset target object, and K < N1, then obtain K first video frames, and extract N1-K video frames from the MK video frames in the first video frame set excluding the first video frames, as the first database video frames, where K is a positive integer, and the extraction of N1-K video frames is full. The following steps are taken: K first video frames and N1-K video frames are discontinuous video frames in the first video frame set; if K first video frames exist in the first video frame set, and K = N1, then K first video frames are acquired as the first database construction video frames; if K first video frames exist in the first video frame set, and K > N1, then N1 discontinuous video frames are extracted from the K first video frames as the first database construction video frames; if no video frame in the video stream is detected to include the preset target object, then video frames are extracted from the first video frame set at the second frame extraction rate as the first database construction video frames, including: if no first video frame exists in the first video frame set, then N2 discontinuous video frames are extracted from the first video frame set as the first database construction video frames.

[0016] In one optional approach, the database is pre-established through the following steps: acquiring a video stream; determining whether the degree of image change of adjacent video frames in the video stream meets a preset condition; if the degree of image change of adjacent video frames in the video stream meets the preset condition, then extracting video frames from the video stream at a third frame extraction rate as second database-building video frames; if the degree of image change of adjacent video frames in the video stream does not meet the preset condition, then extracting video frames from the video stream at a fourth frame extraction rate as second database-building video frames, wherein the fourth frame extraction rate is lower than the third frame extraction rate; and establishing the database based on the second database-building video frames.

[0017] In one optional approach, determining whether the degree of image change of adjacent video frames in the video stream meets a preset condition includes: calculating the average value of the pixel difference between each group of adjacent video frames in the video stream to obtain multiple average values ​​corresponding one-to-one with the multiple groups of adjacent video frames; determining whether the average value is greater than a preset threshold; if the average value is greater than the preset threshold, then determining that the degree of image change of the group of adjacent video frames corresponding to the average value meets the preset condition.

[0018] In one optional approach, the video stream comprises L video frames ordered chronologically by acquisition time, the third frame extraction rate is N3 / L, the fourth frame extraction rate is N4 / L, where L, N3, and N4 are all positive integers, and N4 < N3 < L; after determining whether the average value is greater than a preset threshold, the method further comprises: determining the total number P of second video frames in the L video frames, wherein the second video frames are video frames in each group of adjacent video frames where the average value is greater than the preset threshold, P is an integer, and P ≥ 0; if the degree of image change of adjacent video frames in the video stream meets the preset condition, then video frames are extracted from the video stream at the third frame extraction rate as second library video frames, comprising: if 0 < P < N3, then obtaining P second video frames, and extracting from the L video frames the second video frames after removing the first P second video frames from the group of adjacent video frames. N3-P video frames are extracted from LP video frames other than the second video frame as the second database video frames. The extraction of the N3-P video frames satisfies the following conditions: P second video frames and the N3-P video frames are non-contiguous video frames in the L video frames. If P=N3, then P second video frames are obtained as the second database video frames. If P>N3, then N3 non-contiguous video frames are extracted from P second video frames sorted according to the acquisition time order as the second database video frames. If the degree of image change of adjacent video frames in the video stream does not meet the preset condition, then video frames are extracted from the video stream at a fourth extraction rate as the second database video frames, including: if P=0, then N4 non-contiguous video frames are extracted from the L video frames as the second database video frames.

[0019] According to another aspect of the embodiments of this application, a video retrieval device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the video retrieval method as described above.

[0020] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the video retrieval method as described above.

[0021] In this embodiment of the application, by obtaining retrieval information, video frames that match the retrieval information are automatically retrieved from the database. The entire process does not require manual intervention, thereby effectively improving retrieval efficiency and accuracy.

[0022] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0023] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This illustration shows an application scenario diagram of the database creation method provided in the embodiments of this application; Figure 2 A flowchart illustrating the database creation method provided in an embodiment of this application is shown; Figure 3 A schematic flowchart of a database creation method provided in another embodiment of this application is shown; Figure 4 A schematic flowchart of a database creation method according to another embodiment of this application is shown; Figure 5 A flowchart illustrating the video retrieval method provided in an embodiment of this application is shown; Figure 6 A flowchart illustrating a video retrieval method according to another embodiment of this application is shown; Figure 7 A flowchart illustrating a video retrieval method according to another embodiment of this application is shown; Figure 8 A flowchart illustrating a video retrieval method according to another embodiment of this application is shown; Figure 9 A flowchart illustrating a video retrieval method according to another embodiment of this application is shown; Figure 10 A schematic diagram of the positive sample similarity distribution curve and the negative sample similarity distribution curve provided in the embodiments of this application is shown; Figure 11 A schematic diagram of the structure of the video retrieval device provided in an embodiment of this application is shown. Detailed Implementation

[0024] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0025] In the field of security monitoring, cameras are typically used to record the monitored area continuously around the clock, resulting in a large amount of video data. When responding to emergencies or tracing specific targets, it is necessary to sift through this massive amount of video data to extract relevant video frames. However, the traditional method of manually reviewing frame by frame is not only labor-intensive and inefficient, but also prone to causing visual fatigue due to prolonged work, leading to the omission of critical video frames. For example, when a user needs to determine the movement trajectory of a target (such as a person or a cat), it is often necessary to sift through massive amounts of video data to find video frames containing that target. If this is done entirely manually frame by frame, it is not only time-consuming and laborious, but also extremely inefficient.

[0026] To automate the retrieval of target video frames (i.e., video frames that meet the requirements), this application proposes a video retrieval method. This method involves obtaining a retrieval request carrying search information, and then retrieving video frames matching the search information from a database based on that request, thereby obtaining the target video frame. This approach effectively replaces manual review, enabling automated retrieval of massive amounts of video data, improving the efficiency and accuracy of target video frame retrieval, and meeting the needs for rapid location and verification in scenarios such as security monitoring.

[0027] In practical applications, for ease of unified management and operation, video data collected by camera devices is typically stored entirely in a database, and then retrieved when a target video frame needs to be retrieved. However, this full-scale storage method has significant limitations: the more video frames included in the database, the larger the amount of data required for comparison when retrieving the target video frame, leading to a decrease in retrieval efficiency. Furthermore, in the large amount of video data recorded by camera devices around the clock, if no target enters the monitored area, the captured video frames are often just meaningless static background images; while the target video frames that users actually need to retrieve are usually images containing a specific target.

[0028] Based on this, if some valid video frames can be initially screened from the original video data, and a database can be built using only these valid video frames, the storage size and redundancy of the database can be reduced. Then, when retrieving target video frames from the database, the comparison range will be effectively narrowed, thereby further improving retrieval efficiency.

[0029] Therefore, before introducing the video retrieval method provided in this application, the database establishment method provided in this application will be described below in conjunction with the accompanying drawings. Figure 1 A schematic diagram illustrating an application scenario of the database creation method provided in this application embodiment is shown. For example... Figure 1As shown in (a) of the diagram, in some scenarios, to simplify the processing flow, after the Internet Protocol Camera (IPC) 10 records the monitored area to obtain a video stream, the IPC 10 directly executes the database establishment method provided in this application embodiment. The IPC 10 includes a Neural Processing Unit (NPU) and one or more processors. These processors may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement this application embodiment; no limitation is made here. The one or more processors included in the IPC can be of the same type, such as one or more CPUs; or they can be of different types, such as one or more CPUs and one or more ASICs; no limitation is made here.

[0030] like Figure 1 As shown in (b) of the diagram, in other scenarios, if the IPC 10 has limited computing power (or performance limitations) and cannot execute the database establishment method provided in this application embodiment, the IPC 10 will transmit the recorded video stream to the electronic device 20 via a wired / wireless network, so that the electronic device 20 can execute the database establishment method provided in this application embodiment. The electronic device 20 can be a network video recorder (NVR), a server, a computer, a smartphone, or other electronic device. The electronic device 20 includes an NPU and one or more processors. The processor may be a central processing unit (CPU), a specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application, without limitation herein. The one or more processors included in the electronic device 20 can be processors of the same type, such as one or more CPUs; or they can be processors of different types, such as one or more CPUs and one or more ASICs, without limitation herein.

[0031] Figure 2 A flowchart illustrating the database creation method provided in an embodiment of this application is shown. Figure 2 As shown, the database creation method includes the following steps 110 to 160.

[0032] Step 110: Obtain the video stream.

[0033] The video stream in this step can be a video stream recorded by IPC 10 of the monitored area, or it can be other video stream data.

[0034] Step 120: Perform object detection on the video frames in the video stream.

[0035] It is understandable that a video stream consists of a series of video frames captured in consecutive time. Specifically, in this step, the YOLO series of object detection models can be used to perform object detection on each video frame in the video stream to determine the objects included in each video frame.

[0036] Step 130: Determine whether the video frame includes the preset target object. If yes, proceed to step 140; otherwise, proceed to step 150.

[0037] As mentioned earlier, the target video frames that users actually need to retrieve are usually images containing a specific target object (such as a person, animal, etc.). Therefore, in this embodiment of the application, such specific target objects that users typically need to retrieve are identified as preset target objects. For example, preset target objects can be people, animals, vehicles, and other objects.

[0038] Based on the detection results of each video frame in step 120, this step specifically determines whether there is a video frame containing a preset target object among all the video frames constituting the video stream. If it is determined that at least one video frame containing the preset target object exists among all the video frames, proceed to step 140; if it is determined that none of the video frames contain the preset target object, proceed to step 150.

[0039] Step 140: Extract video frames from the video stream at the first frame extraction rate and use them as the first video frames for library construction.

[0040] The initial frame extraction rate is a relatively high rate, such as 80% or 60%. Since video frames containing the target object are highly relevant and relevant to the user, this step uses a high initial frame extraction rate for intensive extraction to maximize the preservation of the target object's motion details and activity trajectory, ensuring the completeness and high accuracy of subsequent searches. The extracted first batch of video frames will then be used in the subsequent database construction process.

[0041] Preferably, when performing frame extraction, video frames containing preset target objects are selected as the first database construction video frames. If the number of such video frames does not reach the target number corresponding to the first frame extraction rate, supplementary extraction is performed from video frames that do not contain preset target objects. If the number of video frames containing preset target objects is sufficient and exceeds the target number, video frames with discontinuous acquisition time are extracted from these video frames containing preset target objects as the first database construction video frames.

[0042] Step 150: Extract video frames from the video stream at the second frame extraction rate and use them as the first video frames for library construction.

[0043] The second frame extraction rate is lower than the first frame extraction rate, and its value can be 0, 10%, or 20%, etc. When the video frame does not contain the preset target object, its image is usually only the static background of the monitored area. Since the information of the static background changes little and the probability of user retrieval is low, by using a lower second frame extraction rate for sparse extraction, a large number of redundant background video frames can be effectively removed while retaining the basic scene information. Preferably, the second frame extraction rate is greater than zero. This is to consider that the user's actual retrieval needs may not be limited to the preset target category (such as people, animals, vehicles, etc.). For example, it may be necessary to retrieve scene images of non-preset target objects such as "fireworks," "snow," or "lightning." Therefore, setting a second frame extraction rate greater than zero can retain some background frames, thereby taking into account the retrieval needs of such special scenes. More preferably, when extracting video frames from the video stream at the second frame extraction rate, video frames with discontinuous acquisition time are extracted.

[0044] Step 160: Establish a database based on the first video frame used for database creation.

[0045] Specifically, the first video frame extracted in steps 140 and 150 is stored in the database to complete the database establishment.

[0046] In this embodiment, target detection is first performed on video frames in the video stream, and differentiated frame extraction strategies are adopted based on whether they contain preset target objects. On the one hand, a high frame extraction rate is used for video streams containing preset target objects such as people, vehicles, and animals, ensuring that key dynamic details are not lost and guaranteeing the accuracy of subsequent retrieval. On the other hand, a low frame extraction rate is used for video streams with static backgrounds or without video frames containing preset target objects, significantly eliminating invalid and redundant data and effectively reducing the amount of data in the database. Through the above method, the resource consumption caused by storing the entire amount of video data recorded throughout the day is avoided, and the data comparison range during subsequent retrieval is effectively narrowed by reducing the database base, thereby further improving the automated retrieval efficiency of target video frames. In addition, by preferentially setting a second frame extraction rate greater than zero, the retrieval needs of users for special scenes such as "fireworks" and "snowing" that are not preset target objects are also taken into account, improving the practicality and generalized retrieval capability of the database.

[0047] Understandably, step 120 involves significant computational overhead in performing object detection on video frames using the object detection model. Therefore, if the amount of video stream data acquired in step 110 is large, performing object detection on each video frame individually using the object detection model will be computationally intensive and time-consuming. Furthermore, this step-by-step detection method is too time-consuming and cannot meet the real-time requirement of simultaneous recording, detection, and database construction.

[0048] Therefore, in order to reduce computational overhead and meet real-time requirements, Figure 2 Based on the provided embodiments, in some embodiments, the video stream acquired in step 110 is a video stream captured at a first frame rate, and step 120 includes: performing target detection on the video frames in the video stream at a preset second frame rate, wherein the second frame rate is lower than the first frame rate. That is, in some embodiments, in step 120, target detection is not performed on all video frames in the video stream acquired in step 110 one by one, but rather frame-by-frame detection is performed at intervals according to a second frame rate lower than the first frame rate (for example, one frame is extracted for detection every N frames). By using this frequency reduction detection method, the amount of data processed by the target detection model can be effectively reduced without significantly missing video frames of the preset target object, thereby reducing computational overhead and time loss, and enabling synchronous recording, synchronous detection, and real-time database construction. The detected video frames form a first video frame set.

[0049] Based on the foregoing, step 140 includes: extracting video frames from the first video frame set at a first frame extraction rate, as the first library-building video frames. Step 150 includes: extracting video frames from the first video frame set at a second frame extraction rate, as the first library-building video frames. That is, in the embodiments of this application, the first frame extraction rate and the second frame extraction rate in steps 140 and 150 are both applied to the video frames in the first video frame set. Specifically, according to the judgment result of step 130, if there are video frames in the first video frame set that include a preset target object, then dense extraction is performed from the first video frame set at the first frame extraction rate. If there are no video frames in the first video frame set that include a preset target object, then sparse extraction is performed at the second frame extraction rate, and all extracted video frames are used as the first library-building video frames.

[0050] In the above embodiments, extracting a portion of video frames from all video frames for target detection effectively reduces the computational load of the target detection model and ensures the timeliness of synchronous recording and real-time database construction. Simultaneously, when determining the first video frame for database construction, extraction is performed from the first video frame set: when a video frame containing a preset target object exists in the first video frame set, a higher first extraction rate is used for intensive extraction to retain as many video frames containing the preset target object as possible; when no video frame containing the preset target object exists in the first video frame set, a lower second extraction rate is used for sparse extraction to reduce the number of redundant video frames added to the database, thereby effectively reducing the amount of data in the database.

[0051] Considering that video frames captured by IPC at adjacent time points may have high similarity in content, it is necessary to ensure that the video frames used to build the database have high differences. This would effectively reduce redundant data in the database and ensure that subsequent retrieval of target video frames from the database will yield the desired results, thereby improving the retrieval success rate. Figure 3 A flowchart illustrating a database creation method according to another embodiment of this application is shown. Figure 3 As shown, the database creation method includes the following steps 210 to 260.

[0052] Step 210: Acquire the video stream. This video stream is a video stream captured at the first frame rate.

[0053] Step 220: Perform target detection on video frames in the video stream at a preset second frame rate.

[0054] In this step, all video frames extracted and subjected to target detection form the first video frame set, which includes M video frames ordered by acquisition time, where M is a positive integer. For ease of explanation, the video frames in the first video frame set that include the preset target object are referred to as the first video frames.

[0055] Step 230: Determine whether the video frame includes the preset target object. If yes, proceed to any one of steps 242, 244, and 246; otherwise, proceed to step 250.

[0056] The principles and specific implementation methods of steps 210 to 230 can be found in the aforementioned related embodiments, and will not be repeated here.

[0057] In this embodiment of the application, the first frame rate is N1 / M, the second frame rate is N2 / M, N1, N2 and M are all positive integers, and N2 < N1 < M.

[0058] Step 242: If there are K first video frames in the first video frame set that include the preset target object, and K < N1, then obtain K first video frames, and extract N1-K video frames from the MK video frames in the first video frame set other than the first video frames, as the first database construction video frames.

[0059] The extraction of N1-K video frames satisfies the following condition: the K first video frames and the N1-K video frames are discontinuous video frames in the first video frame set. In other words, within the first video frame set, the K first video frames and the N1-K video frames are discontinuous with each other (i.e., the K first video frames form one part, and the N1-K video frames form another part; these two parts are discontinuous within the first video frame set). However, it should be noted that this only restricts the overall continuity between the two parts; the K first video frames or the N1-K video frames themselves can contain continuous video frames.

[0060] For example, if M is 7, N1 is 3, and K is 2, the first video frame set includes 7 video frames arranged in chronological order of acquisition time, namely P1 to P7. The first video frame P1 and the second video frame P2 both include the preset target object (i.e., the first video frame P1 and the second video frame P2 both belong to the first video frame). In this step, the first video frame P1 and the second video frame P2 are used as the first database construction video frames. To avoid selecting a video frame that is consecutively adjacent to the first video frame (i.e., the third video frame P3), one video frame is extracted from the fourth video frame P4 to the seventh video frame P7 as the first database construction video frame, thus obtaining 3 video frames.

[0061] Step 244: If there are K first video frames in the first video frame set, and K = N1, then obtain K first video frames as the first video frames for database construction.

[0062] When the number of video frames containing the preset target object in the first video frame set is exactly the target extraction number corresponding to the first extraction rate, all K first video frames can be directly used as the first library construction video frames, without the need to supplement extraction from other video frames that do not contain the preset target object.

[0063] Step 246: If there are K first video frames in the first video frame set, and K > N1, then extract N1 non-contiguous video frames from the K first video frames as the first video frames for database construction.

[0064] For example, if M is 7, N1 is 3, and K is 4, the first video frame set includes 7 video frames arranged in chronological order of acquisition time, namely P1 to P7. The first video frame P1, the second video frame P2, the third video frame P3, and the fourth video frame P4 all include a preset target object (i.e., the first video frame P1 to the fourth video frame P4 all belong to the first video frame). In this step, 3 non-contiguous video frames are extracted from the first video frame P1 to the fourth video frame P4. For example, the first video frame P1, the second video frame P2, and the fourth video frame P4 (or the first video frame P1, the third video frame P3, and the fourth video frame P4) can be extracted as the first video frames for database construction.

[0065] Step 250: Extract N2 non-contiguous video frames from the first video frame set as the first video frames for database construction.

[0066] If there is no first video frame containing the preset target object in the first video frame set, then N2 non-contiguous video frames are extracted as the first database construction video frames.

[0067] Step 260: Establish a database based on the first video frame used for database creation.

[0068] The principle and specific implementation method of this step can be found in step 160, and will not be repeated here.

[0069] for Figure 3 In the provided embodiments, when there are a large number of first video frames including a preset target object in the first video frame set (i.e., the number of first video frames is greater than the number of video frames to be extracted), discontinuous first video frames are extracted from the first video frames at a higher frame extraction rate as first database construction video frames; when the number of first video frames in the first video frame set is equal to the number of video frames to be extracted, all first video frames are directly used as first database construction video frames; when there are a small number of first video frames in the first video frame set (i.e., the number of first video frames is less than the number of video frames to be extracted), all first video frames are first used as first database construction video frames, and then the extraction process is initiated from the first... Video frames are extracted from non-first video frames in a video frame set, and the extracted non-first video frames must be discontinuous with the first video frame in the first video frame set. When the first video frame does not exist in the first video frame set, discontinuous video frames are extracted from the first video frame set at a lower extraction rate as the first database construction video frames. In this way, as many first database construction video frames with different content as possible can be extracted. Therefore, when building the database based on the first database construction video frames, redundant data can be reduced as much as possible, thereby improving the retrieval success rate when retrieving target video frames from the database.

[0070] Figure 4 A schematic flowchart of a database creation method according to another embodiment of this application is shown. Figure 4 As shown, the database creation method includes the following steps 310 to 350.

[0071] Step 310: Obtain the video stream.

[0072] This step is the same as step 110. The principle and specific implementation method of this step can be referred to step 110, and will not be repeated here.

[0073] Step 320: Determine whether the degree of image change between adjacent video frames in the video stream meets the preset conditions. If yes, proceed to step 330; otherwise, proceed to step 340.

[0074] If the degree of change in adjacent video frames does not meet the preset conditions, it indicates that the scene change between adjacent video frames is small, the content overlap is high, and the motion of objects in the frame is low (or the objects are in a relatively static state). Conversely, if the degree of change in adjacent video frames meets the preset conditions, it indicates that the scene change is large, the content overlap is low, and the motion of objects in the frame is high (or the objects have undergone significant displacement). For example, when the monitored image is static for a long time, the degree of change in adjacent video frames acquired at adjacent moments is almost zero, which does not meet the preset conditions. However, when a new object enters the monitored area, an existing object leaves the monitored area, or the position of an object in the frame changes (for example, a person who was in the middle of the frame at one moment appears in the lower right corner of the frame at the next moment), the degree of change in adjacent video frames acquired at adjacent moments is large, which meets the preset conditions. Based on this, this step determines the degree of scene change, content overlap, and the motion and position change of objects by judging whether the degree of change in adjacent video frames meets the preset conditions. Then, based on the judgment results, video frames are extracted from the video stream obtained in step 310 using a differentiated frame extraction rate to construct a database.

[0075] Specifically, in this step, an object detection model can be used to detect objects included in each video frame of the video stream acquired in step 310, and determine the position information of each object in the video frame. Then, based on the detected objects and their position information, the degree of image change between adjacent video frames is determined, so that different frame extraction rates can be adaptively used to extract video frames from the video stream to construct a database based on the degree of change. If adjacent video frames contain different objects, the degree of image change of adjacent video frames is determined to meet a preset condition. Alternatively, if adjacent video frames contain the same object, but the change in the position information of the object in adjacent video frames reaches a preset position information change threshold (or the positional offset of the object exceeds a preset range), the degree of image change of adjacent video frames is also determined to meet the preset condition. Conversely, if adjacent video frames contain completely identical objects, and the change in the position information of each object does not reach the preset position information change threshold, the degree of image change of adjacent video frames is determined not to meet the preset condition.

[0076] However, as mentioned above, when using the object detection model to detect objects included in video frames, the computational load is large and it takes a lot of time. Therefore, in order to improve the efficiency of this step, in this step, preferably, the degree of image change of adjacent video frames in the video stream is determined by the following steps a1 to a4 to determine whether the preset conditions are met.

[0077] Step a1: Calculate the average value of the pixel difference between each group of adjacent video frames in the video stream to obtain multiple average values ​​that correspond one-to-one with the multiple groups of adjacent video frames.

[0078] For example, the video stream obtained in step 310 includes 7 video frames, namely the first video frame P1 to the seventh video frame P7. These 7 video frames form 6 groups of adjacent video frames. The first group of adjacent video frames includes the first video frame P1 and the second video frame P2. The second group of adjacent video frames includes the second video frame P2 and the third video frame P3. The third group of adjacent video frames includes the third video frame P3 and the fourth video frame P4. The fourth group of adjacent video frames includes the fourth video frame P4 and the fifth video frame P5. The fifth group of adjacent video frames includes the fifth video frame P5 and the sixth video frame P6. The sixth group of adjacent video frames includes the sixth video frame P6 and the seventh video frame P7.

[0079] In this step, for each group of adjacent video frames, the difference between corresponding pixels in the two adjacent video frames within that group is calculated. After obtaining the differences for all pixels, the average of all the differences is calculated to obtain an average value for that group of adjacent video frames. If there are six groups of adjacent video frames, this step yields six average values ​​corresponding to each of the six groups of adjacent video frames.

[0080] Step a2: Determine if the average value is greater than a preset threshold. If yes, proceed to step a3; otherwise, proceed to step a4.

[0081] The preset threshold is used to measure the degree of difference between adjacent video frames. The preset threshold can be set to 10, 20, 30 or other values, or can be adaptively adjusted according to the resolution of the video stream, the frame rate, the complexity of the monitoring environment, and the sensitivity to scene changes in the actual application scenario.

[0082] In this step, specifically, it is determined whether each of the multiple average values ​​corresponding to multiple groups of adjacent video frames is greater than a preset threshold. If at least one of the multiple average values ​​is greater than the preset threshold, then proceed to step a3; if all the multiple average values ​​are less than or equal to the preset threshold, then proceed to step a4.

[0083] Step a3: Determine whether the degree of image change of a set of adjacent video frames corresponding to the average value meets the preset conditions.

[0084] If the average value is greater than the preset threshold, it indicates that the pixel difference between adjacent video frames is large, the scene has changed significantly, or the position of the object in the picture has changed significantly. In this step, the degree of picture change of adjacent video frames is determined to meet the preset conditions.

[0085] Step a4: Determine that the degree of image change in a set of adjacent video frames corresponding to the average value does not meet the preset conditions.

[0086] If the average value is less than or equal to the preset threshold, it indicates that the pixel difference between adjacent video frames is small, the scene is relatively stable, or the position of the object in the picture has not changed significantly. Therefore, the determination of the degree of picture change of adjacent video frames in this step does not meet the preset conditions.

[0087] Step 330: Extract video frames from the video stream at the third frame extraction rate as the second library construction video frames.

[0088] Specifically, if at least one set of adjacent video frames exhibits image changes that meet preset conditions, a video frame is extracted from the video stream at a higher third frame extraction rate to serve as the second database-building video frame. This extracted second database-building video frame will then proceed to the subsequent database-building process. It should be noted that the third frame extraction rate is related to... Figure 2 The first frame rate in each frame can be the same or different; there is no limitation here.

[0089] Step 340: Extract video frames from the video stream at a fourth frame extraction rate to serve as the second library-building video frames. The fourth frame extraction rate is lower than the third frame extraction rate.

[0090] If none of the adjacent video frames in a set of multiple sets of adjacent video frames meet the preset condition for image change, a video frame is extracted from the video stream at a fourth frame extraction rate lower than the third frame extraction rate, and used as the second database-building video frame. The extracted second database-building video frame will then proceed to the subsequent database-building process. It should be noted that the fourth frame extraction rate is related to... Figure 2 The second frame rate can be the same or different; there is no restriction here.

[0091] Step 350: Establish a database based on the second video frame used for database creation.

[0092] Steps 330 to 350 are similar to steps 140 to 160. Therefore, the principles and specific implementation methods of steps 330 to 350 can be found in steps 140 to 160, and will not be repeated here.

[0093] If the degree of image change between adjacent video frames in a video stream does not meet the preset conditions, it indicates that the scene change is small, the content overlap of the video frames is high, and the position of objects in the image has not changed significantly. Using a large number of such video frames as the second database construction video frames would lead to high data redundancy in the database built based on these second database construction video frames, and the database storage space would be occupied by a large amount of invalid or duplicate information, reducing the data's utilization value. Therefore, Figure 4In the provided embodiments, by determining whether the degree of change in adjacent video frames meets preset conditions, different frame extraction rates are used to extract video frames from the video stream based on the determination results to construct a database. When the degree of change in adjacent video frames does not meet the preset conditions, a lower fourth frame extraction rate is used to extract video frames, thereby reducing the input of duplicate data into the database. Conversely, when the degree of change in adjacent video frames meets the preset conditions, a higher third frame extraction rate is used to extract video frames, ensuring that key information can be densely captured during periods of drastic scene changes or significant object movement. This guarantees the representativeness and feature richness of the video frames input into the database, avoiding the omission of important scene details. This dynamic frame extraction method, based on the degree of scene change and the degree of change in the position of objects in the scene, effectively reduces data redundancy while ensuring the integrity of effective information in the database. It not only improves the utilization of storage space but also increases the retrieval success rate when retrieving target video frames from the database.

[0094] Furthermore, in Figure 4 Based on the provided embodiments and steps a1 to a4, in some other embodiments, the video stream acquired in step 310 includes L video frames ordered in chronological order of acquisition time, and the third frame extraction rate in step 330 is N3 / L, and the fourth frame extraction rate in step 340 is N4 / L, where L, N3, and N4 are all positive integers, and N4 < N3 < L. For ease of explanation, if the average value of a group of adjacent video frames is greater than a preset threshold, then the two video frames in that group of adjacent video frames are referred to as the second video frames. For example, if the average value of the third group of video frames is greater than the preset threshold, then the third video frame P3 and the fourth video frame P4 constituting the third group of video frames are both second video frames.

[0095] Following step a2, the database establishment method further includes: determining the total number P of second video frames in L video frames, where P is an integer and P≥0. Step 330 includes: if 0<P<N3, then obtaining P second video frames, and extracting N3-P video frames from LP video frames (excluding the second video frames) in the L video frames as the second database construction video frames, wherein the extraction of N3-P video frames satisfies the following condition: the P second video frames and the N3-P video frames are discontinuous video frames in the L video frames. That is, in the L video frames, the P second video frames and the N3-P non-second video frames are discontinuous (i.e., the P second video frames are one part, and the N3-P non-second video frames are another part, and these two parts are discontinuous in the L video frames); however, it should be noted that this only restricts the overall continuity between the two parts. The P second video frames themselves, or the N3-P non-second video frames, can contain continuous video frames. If P=N3, then acquire P second video frames as the second database construction video frames; if P>N3, then extract N3 non-contiguous video frames from the P second video frames sorted according to acquisition time order as the second database construction video frames. Step 340 includes: extracting N4 non-contiguous video frames from L video frames as the second database construction video frames.

[0096] In the above methods, when the number of second video frames is greater than the number of video frames to be extracted, extracting second video frames at intervals as second database construction video frames can further reduce data redundancy while retaining key scene information. When the number of second video frames is equal to the number of video frames to be extracted, using all second video frames as second database construction video frames can maximize the use of key video frames with significant scene changes, fully capture effective information, and achieve complete recording of key scenes. When the number of second video frames is less than the number of video frames to be extracted, prioritizing the use of second video frames as second database construction video frames and supplementing the insufficient number of video frames from the remaining non-second video frames as second database construction video frames, and ensuring that the final extracted non-second video frames and second video frames are not sequentially acquired in the video stream, effectively avoids high repetition of image content caused by continuous acquisition, thereby maximizing the richness and utilization value of the database while meeting the database quantity requirements. When a second video frame is not available, N3 non-contiguous video frames are extracted as the second database construction video frames. This ensures data diversity as much as possible and avoids the extracted second database construction video frames being too concentrated in the time dimension, thus ensuring that the established database can more comprehensively reflect the overall characteristics of the video stream. This method can improve the success rate of subsequently retrieving target video frames from the database.

[0097] It should be noted that the above Figure 2 and Figure 4 Two methods for creating a database are provided. If the storage space of the device executing the database creation method provided in this application is limited, only the storage space of the device can be used. Figure 2 and Figure 4 One of the provided methods for building the database is stored on the device. If the device has sufficient storage space, both methods can be stored simultaneously. In practical applications, the device can flexibly switch between or combine these two methods to build the database based on the current video stream characteristics or system resource status, thereby improving the comprehensiveness and retrieval efficiency of the database while considering storage costs.

[0098] Based on the database established in the above embodiments, the following will describe in detail, with reference to the accompanying drawings, how to automatically retrieve target video frames that meet the requirements from the database. Figure 5 A schematic flowchart of a video retrieval method provided in an embodiment of this application is shown. This video retrieval method is executed by a video retrieval device, which may be an IPC 10, an electronic device 20, or other devices. Figure 5 As shown, the method includes the following steps 410 to 430.

[0099] Step 410: Obtain a search request carrying search information.

[0100] In this process, after a user inputs a natural language description of the desired search scene into the video retrieval device, the device extracts search information from the user's input. This search information refers to the features or target image of the target object that the user wishes to retrieve. By inputting a target video frame with a natural language description into the video retrieval device executing this embodiment and performing a search selection operation (e.g., clicking the "Search" button), the video retrieval device can obtain a search request carrying the search information. For example, if a user inputs "a man wearing a red coat and a black hat" into the video retrieval device and clicks the "Search" button, the device can obtain a search request carrying the physical characteristics of that person (i.e., the search information).

[0101] Step 420: Retrieve the target video frame from the database based on the retrieval information.

[0102] In this step, target video frames matching the search information are retrieved from the database. Specifically, corresponding tag information can be pre-set for each video frame in the database, and this tag information includes the objects included in the corresponding video frame. If the search information includes the target search object, then the video frame corresponding to the tag information that includes the search object is determined as the target video frame. For example, if the first video frame in the database includes a person and a cat, then the corresponding tag information set for this video frame includes both a person and a cat. If the search information is: video frames including cats, then since the tag information corresponding to the first video frame includes cats, the first video frame is determined as the target video frame.

[0103] Alternatively, in other embodiments, the similarity between the target retrieval object in the search information and video frames in the database can be calculated, and then video frames that meet the similarity requirements can be identified as target video frames. Specifically, the target retrieval object is vectorized to obtain a text vector, and each video frame in the database is encoded to obtain an image vector corresponding to each video frame. Then, the similarity between the text vector and each image vector is calculated to obtain the similarity corresponding to each image vector. Finally, the video frames corresponding to the image vectors that meet the similarity requirements (e.g., similarity greater than a similarity threshold) are identified as target video frames.

[0104] Step 430: Output the target video frame.

[0105] Specifically, in this step, the target video frame is displayed on the display screen.

[0106] In this embodiment of the application, by obtaining retrieval information, video frames that match the retrieval information are automatically retrieved from the database. The entire process does not require manual intervention, thereby effectively improving retrieval efficiency and accuracy.

[0107] Users may need to retrieve different target video frames in the same database. In this application, the method of retrieving target video frames based on the similarity between the image vector of the video frame and the text vector of the target retrieval object will generate a lot of redundant calculations if the image vector of the video frame is recalculated every time, thus affecting the retrieval efficiency.

[0108] Based on this, in this application, preferably, after establishing the database, the visual encoding submodule in the multimodal model is used simultaneously to encode each video frame in the database to obtain the image vector corresponding to each video frame. All image vectors are then stored in the image vector database, along with the correspondence between video frames and image vectors. Subsequently, when it is necessary to calculate the similarity between a video frame and the target retrieval object, the image vector corresponding to that video frame is directly obtained from the image vector database. The text encoding submodule in the multimodal model is then used to encode the target retrieval object to obtain a text vector. By calculating the similarity between this text vector and the directly obtained image vector, rapid retrieval of the target video frame is achieved, effectively avoiding redundant determination of image vectors, reducing resource consumption, and improving retrieval efficiency.

[0109] To further improve the accuracy of retrieving target video frames, in some embodiments, a set of corresponding tag information is pre-set for each video frame in the database, and the correspondence between the set tag information, multiple video frames in the database, and multiple sets of tag information is stored in the database so that relevant video frames can be retrieved from the database based on the tag information later. Specifically, an object detection model can be used to detect the objects included in each video frame in the database to generate corresponding tag information. For example, if a video frame includes a person and a cat, then the person and cat can be identified as a set of tag information corresponding to that video frame (i.e., the set of tag information corresponding to that video frame includes a person and a cat). Alternatively, since cats belong to the animal category, the person and animal can be identified as a set of tag information corresponding to that video frame (i.e., the set of tag information corresponding to that video frame includes a person and an animal). Or, the person, animal, and cat can be identified as a set of tag information corresponding to that video frame (i.e., the set of tag information corresponding to that video frame includes a person, an animal, and a cat). It is worth noting that the tag information may include, but is not limited to, the category of the object included in the video frame (such as a person or a vehicle), appearance attributes (such as color or vehicle type), detection confidence, and the acquisition timestamp of the video frame.

[0110] Furthermore, the retrieval information carried in the retrieval request obtained in step 410 includes the target retrieval object. Moreover, in Figure 5 Based on this, step 420 includes the following steps b1 to b2.

[0111] Step b1: Retrieve candidate video frames from the database whose tag information matches the retrieval information.

[0112] Specifically, the target search object carried in the obtained search information is compared with the tag information corresponding to each video frame stored in the database. If the tag information of a video frame in the database contains a tag that is consistent with or belongs to the same category as the target search object, then the video frame is determined to match the search information and is identified as a candidate video frame. For example, if the target search object in the search information is "animal," then all video frames whose tag information contains "animal" will be filtered as candidate video frames; if the target search object is "red vehicle," then only video frames whose tag information must contain both "vehicle" and the corresponding color attribute "red" will be filtered out, thus initially narrowing down the subsequent search scope.

[0113] Step b2: Based on the target retrieval object, retrieve the target video frame from the candidate video frames.

[0114] Specifically, in this step, the target retrieval object can be vectorized to obtain a text vector, and each candidate video frame can be encoded to obtain an image vector corresponding to each video frame. Then, the similarity between the text vector and each image vector is calculated to obtain the similarity corresponding to each image vector. Finally, the candidate video frame corresponding to the image vector that meets the similarity requirement (e.g., similarity greater than the similarity threshold) is determined as the target video frame.

[0115] In this embodiment, tag information is first used for initial screening to quickly eliminate a large number of video frames irrelevant to the search information, retaining candidate video frames that match the search information. Then, the search continues from these candidate video frames. Compared to a single search mode, this hierarchical search method effectively improves the accuracy of target video frame retrieval. Furthermore, directly calculating the similarity between the text vector corresponding to the search information and the image vectors of all video frames in the database incurs significant computational overhead. This embodiment first uses tag information for initial screening to obtain candidate video frames, and then only calculates the vector similarity between the search information and the candidate video frames. This method reduces the amount of data involved in similarity calculation, effectively reducing computational load while ensuring high search accuracy, further improving search efficiency.

[0116] Category attributes include superordinate class objects and subordinate class objects. For example, cats, dogs, and rabbits all belong to the animal category, while cars, trucks, and motorcycles all belong to the vehicle category. In the classification hierarchy, animals and vehicles are superordinate class objects (or parent class objects), while cats, dogs, rabbits, cars, trucks, and motorcycles are their corresponding subordinate class objects (or child class objects). The mapping relationship between subordinate class objects and their superordinate class objects is defined as the category affiliation relationship. For example, the affiliation relationship between subordinate class objects such as cats, dogs, and rabbits and the superordinate class object "animal" is the category affiliation relationship.

[0117] When using an object detection model to detect objects contained in a video frame and setting label information for the video frame based on the detection results, if the object detection model can only identify the higher-order class of the object (such as only identifying animals, people, cars, etc.), it may not be able to accurately retrieve the video frame required by the user when the user actually needs to retrieve the lower-order class object (such as specifically a dog, child, car, etc.).

[0118] For example, if a video frame contains a cat, when an object detection model detects it, the model outputs "animal" and assigns it the hypernym label "animal." When a user enters hyponyms such as "cat" or "orange cat" for a search, the video frame cannot be accurately matched and retrieved because its label information does not contain specific hyponym features, resulting in a search failure.

[0119] Therefore, in order to improve the success rate of retrieval, another embodiment of the video retrieval method provided by this application will be described below with reference to the accompanying drawings. Figure 6 A flowchart illustrating a video retrieval method according to another embodiment of this application is shown. Figure 6 As shown, the method includes the following steps 510 to 560.

[0120] Step 510: Obtain a search request carrying search information.

[0121] This step is the same as step 410. Therefore, the principle and specific implementation method of this step can be referred to step 410, and will not be repeated here.

[0122] Step 520: Based on the preset category affiliation relationship, determine the category attribute of the target retrieval object in the category affiliation relationship.

[0123] In this step, based on the superordinate objects that the model can recognize, the subordinate objects included in each superordinate object are pre-determined, and the category affiliation relationship between the superordinate objects and subordinate objects is constructed. For example, if the superordinate objects that the model can recognize include people, vehicles, and animals, then a mapping table of superordinate objects and subordinate objects can be set up as shown in Table 1 below.

[0124] Table 1: Mapping Table of Superordinate Class Objects and Subordinate Class Objects

[0125] In this step, based on the pre-built category attribution relationships (such as Table 1 above), the category attribute of the target retrieval object in the category attribution relationship can be queried and determined. For example, if the target retrieval object is a cat, its category attribute in the category attribution relationship is a subordinate category object; if the target retrieval object is a person, its category attribute in the category attribution relationship is a superordinate category object.

[0126] Step 530: If the target retrieval object is a subordinate class object in the category attribution relationship, then map the target retrieval object to its subordinate target superior class object, and retrieve video frames whose category information in the tag information contains the category of the target superior class object from the database to obtain candidate video frames.

[0127] The target superordinate class object refers to a superordinate class object that has a direct affiliation with the target retrieval object according to a preset category affiliation relationship. Specifically, the target superordinate class object corresponding to the target retrieval object can be found in the preset category affiliation relationship (as shown in Table 1), and the target superordinate class object can be used as a search keyword to match the tag information of video frames in the database, thereby obtaining candidate video frames. Since the category in the tag information of the video frames in this embodiment is preferably set as the category of the superordinate class object, by mapping the subordinate class object (such as "cat") to the superordinate class object (such as "animal"), it is possible to successfully match the tag information in the database including the superordinate class object (such as "animal"), thereby retrieving all video frames belonging to that category (such as all video frames containing animals such as cats, dogs, and rabbits).

[0128] For example, suppose the database contains video frames P1 through P4. Frame P1 contains a person, a cat, and a dog, with tags indicating categories including "person" and "animal." Frame P2 contains a person and a rabbit, also with tags indicating categories including "person" and "animal." Frame P3 contains a person, with the tag indicating category "person." Frame P4 contains a person and a motorcycle, with tags indicating categories including "person" and "vehicle." If the target search object is "rabbit," its superordinate category is "animal" based on category attribution. Since both frame P1 and frame P2 contain the category "animal" in their tags, the candidate video frames determined after this step are frame P1 and frame P2.

[0129] Step 540: If the target retrieval object is a parent class object in the category attribution relationship, then retrieve video frames whose category information in the tag information contains the category of the target retrieval object from the database to obtain candidate video frames.

[0130] This step is similar to step 530. Therefore, the principle and specific implementation method of this step can be referred to step 530, and will not be repeated here.

[0131] Step 550: Based on the target retrieval object, retrieve the target video frame from the candidate video frames.

[0132] This step is the same as step b2 mentioned above. Therefore, the principle and specific implementation method of this step can be referred to step b2, and will not be repeated here.

[0133] Step 560: Output the target video frame.

[0134] This step is the same as step 430 above. Therefore, the principle and specific implementation method of this step can be referred to step 430, and will not be repeated here.

[0135] for Figure 6 The provided embodiment determines the subordinate objects included in each superior object based on the superior object that the model can identify, and constructs the category affiliation relationship between the superior and subordinate objects. This allows the user's search intent to be mapped to the corresponding superior object when the user inputs subordinate object search information. By utilizing the tag information of the superior objects identified by the model for preliminary retrieval and coarse screening, the model can effectively overcome the problem of retrieval failure caused by the model's inability to directly identify subordinate objects, thereby improving the hit rate and accuracy of video retrieval.

[0136] Furthermore, when the target retrieval object is a subordinate class object, in this embodiment, by continuing to retrieve the target video frame from the candidate video frames based on the target retrieval object in step 550, the candidate video frames can be subjected to secondary deep matching using semantic features or image features. In this way, interfering video frames that do not actually include the subordinate class target retrieval object can be removed from the candidate video frames, thereby accurately filtering out the target video frames that truly contain the target retrieval object, further improving the accuracy of the retrieval results.

[0137] Users may sometimes need to retrieve video frames collected within a specific time period; for example, a user might need to retrieve video frames containing cats from last Saturday. Therefore, to meet user needs and improve the practicality of the video retrieval method provided in this application, Figure 6 Based on the provided embodiments, in some embodiments, the tag information corresponding to the video frames stored in the database also includes the acquisition time of the video frames.

[0138] When the retrieval request carries retrieval information that also includes a target time period, step 530 includes: if the target retrieval object is a subordinate class object in the category attribution relationship, then the target retrieval object is mapped to the target superior class object to which it belongs, and video frames whose category information in the tag information contains the category of the target superior class object and whose acquisition time belongs to the target time period are retrieved from the database to obtain candidate video frames.

[0139] Specifically, the target superordinate class object corresponding to the target retrieval object can be determined first based on the preset category affiliation relationship; then, in the tag information of each video frame in the database, video frames that simultaneously meet the two conditions of "the category information contains the target superordinate class object" and "the collection time belongs to the target time period range" can be selected as candidate video frames.

[0140] Furthermore, step 540 includes: if the target retrieval object is a superordinate class object in the category attribution relationship, then retrieve video frames from the database whose category information in the tag information contains the category of the target retrieval object and whose acquisition time belongs to the target time period, to obtain candidate video frames.

[0141] Specifically, the target search object can be directly used as the search keyword, and combined with the time dimension of the target time period, the tag information of each video frame in the database can be matched and queried to filter out video frames that simultaneously meet the two conditions of "the category information includes the target search object" and "the video frame's acquisition time belongs to the target time period range", thus obtaining candidate video frames.

[0142] Furthermore, in scenarios with a large monitoring coverage area, users typically deploy multiple IPCs to monitor different areas. For example, users may deploy IPCs in three different areas: the garage, the porch, and the driveway. To easily distinguish between the IPCs, users usually assign identifiers to them. For instance, they might assign "Garage," "Porch," and "Driveway" identifiers to the three IPCs deployed in the garage, porch, and driveway, respectively, or assign "1," "2," and "3" identifiers to the three IPCs. In this case, users may sometimes need to retrieve video frames captured by a specific IPC. Based on this, to meet user needs and further improve the practicality of the video retrieval method provided in this application, Figure 6 Based on the provided embodiments, in other embodiments, the tag information corresponding to the video frame stored in the database also includes the identifier of the camera device that captured the video frame.

[0143] When the retrieval request includes a target identifier in the retrieval information, step 530 includes: if the target retrieval object is a subordinate class object in the category attribution relationship, then the target retrieval object is mapped to its subordinate target superior class object, and video frames whose category information in the tag information contains the category of the target superior class object and whose camera device identifier is the target identifier are retrieved from the database to obtain candidate video frames. Furthermore, step 540 includes: if the target retrieval object is a superior class object in the category attribution relationship, then video frames whose category information in the tag information contains the category of the target retrieval object and whose camera device identifier is the target identifier are retrieved from the database to obtain candidate video frames. This embodiment is similar to the embodiment where the retrieval information also includes a target time period; therefore, the principle and specific implementation of this embodiment can be referred to the aforementioned embodiment, and will not be repeated here.

[0144] In some scenarios, users may need to retrieve video frames captured by a specific IPC within a certain time period. For example, a user might need to find all video frames containing a man in red, captured by an IPC in a garage between 3 PM and 5 PM yesterday. In this case, the video retrieval device uses the user's input information, combined with the current system date, to determine the specific year, month, and day corresponding to "yesterday," and further parses the time period "3 PM to 5 PM," the identifier "garage," and the characteristic information of the target search object, "man in red." Subsequently, the video retrieval device combines these conditions into search information, filters the database for video frames that match that time period, were captured by an IPC identified as "garage," and contain the corresponding target search object characteristics, thus obtaining the target video frames.

[0145] When retrieving target video frames, the target retrieval object can be vectorized and encoded using the text encoding submodule of a multimodal model (e.g., a multimodal embedding model) to obtain text vectors. Then, each video frame is encoded using the visual encoding submodule of the multimodal model to obtain the corresponding image vector. Next, the similarity between the text vector and each image vector is calculated to obtain the similarity for each image vector. Finally, the video frames corresponding to image vectors with similarity requirements (e.g., similarity greater than a similarity threshold) are determined as the target video frames. Since the MobileCLIP model based on FastViT has advantages such as fewer model parameters, suitability for edge deployment, and strong image-text alignment capabilities, in this application, preferably, the visual encoding submodule uses the image encoder in the MobileCLIP model based on FastViT. The text encoding submodule and the visual encoding submodule use the same text encoding branch of the multimodal model, that is, the text encoder of the MobileCLIP model based on internal data fine-tuning, to ensure that the text vectors and image vectors are in a unified semantic space. Furthermore, the cosine similarity between the text vector and each image vector is preferred, but it is not limited to this. Dot product similarity or other vector similarity measures applicable to a unified semantic space can also be used.

[0146] However, the above methods involve multimodal models. For multimodal models with low accuracy, when determining target video frames using these methods, the model may fail to accurately extract deep semantic features of text or images. This can lead to inaccurate mapping between text vectors and image vectors, resulting in the calculated similarity not truly reflecting the matching degree between the target retrieval object and the video frame content. This can cause video frames containing the target object to be missed due to low similarity, or irrelevant video frames to be incorrectly retrieved due to inflated similarity, thus reducing the accuracy and reliability of the retrieval results.

[0147] Specifically, before utilizing a multimodal model, it is typically necessary to train and test it. For example, when testing a multimodal model, if the model outputs video frames containing the second search object but not the first search object when retrieving video frames based on the first search object, it indicates that the model has failed to accurately distinguish the features of the first and second search objects. In practical applications, if the target search object in the search information is the first search object, the model is highly likely to incorrectly identify video frames containing the second search object but not the target search object as the target video frames, thus returning incorrect search results. This situation will lead to a decrease in the accuracy of search results, causing users to obtain irrelevant and distracting information.

[0148] For example, when testing the trained multimodal model using test data, if the search object is "cat", the model outputs video frames containing dogs but not cats when searching for video frames containing cats. In subsequent practical applications, when the target search object is "cat", the model may retrieve video frames containing dogs but not cats, which will seriously affect the accuracy of the search results.

[0149] Therefore, in order to ensure the accuracy of the search results, in some embodiments, step 420 preferably includes the following steps c1 to c2.

[0150] Step c1: Determine the first verification object based on the target retrieval object.

[0151] The first verification object satisfies the following condition: when testing the multimodal model using test data, when the multimodal model retrieves video frames based on the target retrieval object, there is a case where the output contains the first verification object but does not contain the target retrieval object.

[0152] It is worth noting that a validation table as shown in Table 2 can be pre-set based on the test results of the multimodal model. This validation table includes the target retrieval object and the corresponding validation object. In this step, the first validation object can be determined based on the target retrieval object using this validation table.

[0153] Table 2: Verification Table

[0154] Step c2: Based on the target retrieval object and the first verification object, retrieve the target video frame from the database.

[0155] Specifically, this step involves using the text encoding submodule in the multimodal model to vectorize and encode the target retrieval object and the first verification object, respectively, to obtain the first text vector corresponding to the target retrieval object and the second text vector corresponding to the first verification object. Then, the visual encoding submodule in the multimodal model encodes each video frame in the database to obtain the image vector corresponding to each video frame. Next, the similarity between the first text vector and each image vector is calculated to obtain the first similarity for each video frame. Similarly, the similarity between the second text vector and each image vector is calculated to obtain the second similarity for each video frame. Finally, video frames that meet the requirements are identified as target video frames. Specifically, a video frame meeting these requirements is defined as one whose first similarity is greater than a first similarity threshold and whose first similarity is greater than a second similarity.

[0156] In this embodiment, by introducing a first verification object, and requiring that the target video frame not only has a high similarity to the target retrieval object (greater than the similarity threshold), but also that the first similarity with the target retrieval object is greater than the second similarity with the first verification object, when the model tends to confuse the target retrieval object with the first verification object due to insufficient accuracy, the comparison relationship between the first and second similarities can effectively eliminate video frames that contain the first verification object but not the target retrieval object. This solves the problem of not being able to effectively distinguish different objects due to inaccurate feature extraction by the multimodal model, reduces the false detection rate, and improves the accuracy and reliability of video retrieval results.

[0157] Figure 7 A flowchart illustrating a video retrieval method according to another embodiment of this application is shown. Figure 7 As shown, the method includes the following steps 610 to 670.

[0158] Step 610: Obtain a search request carrying search information.

[0159] Step 620: Determine the first verification object based on the target retrieval object.

[0160] Step 630: Determine the similarity between the video frame in the database and the target retrieval object, and obtain the first similarity corresponding to the video frame.

[0161] Step 640: Determine the similarity between the video frame in the database and the first verification object, and obtain the second similarity corresponding to the video frame.

[0162] The principles and specific implementation methods of steps 610 to 640 can be referred to the aforementioned related embodiments, and will not be repeated here.

[0163] Step 650: Determine the similarity between the video frame in the database and the second verification object to obtain the third similarity corresponding to the video frame. The second verification object includes the target retrieval object and the first verification object.

[0164] Specifically, the second verification object is vectorized and encoded by the text encoding submodule in the multimodal model to obtain the third text vector corresponding to the second verification object, and the similarity between the third text vector and each image vector is calculated to obtain the third similarity corresponding to each video frame.

[0165] Step 660: Determine the video frames that meet the conditions as the target video frames.

[0166] Among them, the video frames that meet this condition are: the maximum value of the first similarity, second similarity and third similarity of the video frame is the first similarity or the third similarity, and the first similarity is greater than the first similarity threshold.

[0167] Step 670: Output the target video frame.

[0168] When a video frame contains the first verification object, although its corresponding second similarity may be high, the third similarity between the video frame and the second verification object is more likely to be higher than the second similarity because the second verification object contains both the target retrieval object and the first verification object. Therefore, in this embodiment, by specifying in the filtering conditions that "the maximum value among the first similarity, second similarity, and third similarity is the first similarity or the third similarity" and "the first similarity is greater than the first similarity threshold," it is ensured that the target video frame is highly correlated with the target retrieval object, thereby enabling the retrieval of target video frames that truly contain the target retrieval object.

[0169] Figure 8 A flowchart illustrating a video retrieval method according to another embodiment of this application is shown. Figure 8 As shown, the method includes the following steps 710 to 750.

[0170] Step 710: Obtain a search request carrying search information.

[0171] Step 720: Retrieve candidate video frames from the database whose tag information matches the retrieval information.

[0172] Step 730: Determine the first verification object based on the target retrieval object.

[0173] Step 740: Based on the target retrieval object and the first verification object, retrieve the target video frame from the candidate video frames.

[0174] Step 750: Output the target video frame.

[0175] The specific implementation methods of steps 710 to 750 can be referred to the aforementioned related embodiments, and will not be repeated here.

[0176] for Figure 8 The illustrated embodiment first retrieves candidate video frames from the database whose tag information matches the search information. This process eliminates obviously irrelevant video frames, thus narrowing the search scope for subsequent processing. Based on this, further refined retrieval and verification are performed on the initially screened candidate video frames using the target search object and the first verification object. This effectively solves the problem of confusion between the target search object and the first verification object caused by insufficient accuracy of the multimodal model. This phased retrieval method, by filtering based on tag information in the first stage, avoids unnecessary subsequent similarity calculations and improves retrieval efficiency; and by using multimodal vectors for retrieval in the second stage, it ensures the accuracy of the retrieval results, thereby improving video retrieval efficiency while maintaining accuracy.

[0177] Figure 9 A flowchart illustrating a video retrieval method according to another embodiment of this application is shown. Figure 9 As shown, the method includes the following steps 810 to 880.

[0178] Step 810: Obtain a search request carrying search information.

[0179] Step 820: Retrieve candidate video frames from the database whose tag information matches the retrieval information.

[0180] Step 830: Determine the first verification object based on the target retrieval object.

[0181] Step 840: Determine the similarity between the candidate video frame and the target retrieval object, and obtain the first similarity corresponding to the candidate video frame.

[0182] Step 850: Determine the similarity between the candidate video frame and the first verification object to obtain the second similarity corresponding to the candidate video frame.

[0183] Step 860: Determine the similarity between the candidate video frame and the second verification object to obtain the third similarity corresponding to the candidate video frame. The second verification object includes the target retrieval object and the first verification object.

[0184] Step 870: Determine the candidate video frames that meet the conditions as the target video frames.

[0185] Step 880: Output the target video frame.

[0186] The specific implementation methods of steps 810 to 880 can be referred to the aforementioned related embodiments, and will not be repeated here.

[0187] exist Figure 8 Based on the illustrated embodiment, this application embodiment calculates the similarity between candidate video frames and the target retrieval object, the first verification object, and the second verification object containing both, and sets strict screening conditions: only when the first similarity between a video frame and the target retrieval object is greater than the first similarity threshold, and its similarity with the target retrieval object or the second verification object is not lower than its similarity with the first verification object, is it determined as a target video frame. This approach, while improving efficiency through preliminary retrieval based on tag information, further utilizes multiple similarity constraints to effectively solve the false detection problem caused by model feature confusion. It can more accurately eliminate interfering video frames that only contain confused objects but not the target retrieval object, thereby further improving the accuracy and reliability of video retrieval results while ensuring retrieval efficiency.

[0188] For the video retrieval method provided in this application, preferably, step 420 includes: determining the similarity between video frames in the database and the target retrieval object, obtaining the first similarity corresponding to the video frame, and determining video frames with a first similarity greater than a first similarity threshold as target video frames. Furthermore, step 430 includes the following steps d1 to d2. The specific method for determining the first similarity corresponding to the video frame can be referred to the foregoing embodiments, and will not be repeated here.

[0189] Step d1: Display the target video frame with a first similarity greater than the second similarity threshold in the first display format.

[0190] The first display format refers to directly presenting the complete target video frame on the display screen.

[0191] Step d2: Display the target video frames whose first similarity is less than or equal to the second similarity threshold in a second display format.

[0192] The second display format involves a collapsed display on the screen. When a user needs to query a specific target video frame, a preset operation (such as clicking, swiping, or expanding) performed on the screen will display the target video frame in the first display format, thus clearly showing the hidden target video frame. The second similarity threshold is higher than the first similarity threshold; the specific methods for determining the first and second similarity thresholds will be described in detail in other embodiments below.

[0193] In this embodiment, a differentiated display strategy is adopted for target video frames based on the comparison results of a first similarity, a first similarity threshold, and a second similarity threshold, thereby achieving intelligent allocation of screen display resources. For target video frames with a high first similarity (greater than the second similarity threshold), a first display format is used to present them directly and completely, allowing users to intuitively and conveniently obtain the target video frames that best match their search intent. For target video frames with a first similarity between the first and second similarity thresholds, a second display format is used for folded display, ensuring that relevant search results are not missed while effectively saving screen display space. When users need to view the folded content, they can unfold it through simple interactive operations. This hierarchical display method not only resolves the contradiction between the limited device screen size and the need for complete display of multiple video frames, but also optimizes the user search experience and improves the efficiency of information acquisition by highlighting target video frames with high similarity.

[0194] The following section will detail the specific methods for determining the first and second similarity thresholds. Specifically, the first and second similarity thresholds can be determined through the following steps e1 to e5.

[0195] Step e1: Obtain the sample set, which includes multiple sets of positive sample image-text pairs and multiple sets of negative sample image-text pairs.

[0196] Specifically, the sample set is constructed based on multiple sample images and multiple sample objects. A positive sample image-text pair includes a sample image and a sample object, with the sample image containing the sample object. A negative sample image-text pair also includes a sample image and a sample object, but the sample image does not contain the sample object. For example, suppose multiple sample images include a picture of a cat, and multiple sample objects include descriptions of both cats and dogs. When constructing a positive sample image-text pair, the sample image containing the cat is paired with the sample object described as a cat, ensuring the image content matches the text description. However, when constructing a negative sample image-text pair, the sample image containing the cat is paired with the sample object described as a dog, but the image content does not contain the dog, thus creating a mismatched negative sample image-text pair.

[0197] Step e2: Calculate the similarity between image and text pairs in each sample set to obtain the positive sample similarity set and the negative sample similarity set.

[0198] Specifically, for a set of positive sample image-text pairs: the text encoding submodule in the multimodal model is used to vectorize and encode the positive sample objects in the positive sample image-text pair to obtain the positive sample text vectors corresponding to the positive sample objects. Then, the visual encoding submodule in the multimodal model is used to encode the positive sample images in the positive sample image-text pair to obtain positive sample image vectors. Next, the similarity between the positive sample text vector and the positive sample image vector is calculated to obtain the positive sample similarity corresponding to the positive sample image-text pair. By performing the above processing on multiple sets of positive sample image-text pairs, a positive sample similarity set consisting of multiple positive sample similarities corresponding to the multiple sets of positive sample image-text pairs can be obtained.

[0199] The specific method for determining the negative sample similarity set is similar to that for determining the positive sample similarity set. Therefore, the negative sample similarity set can be determined by referring to the above method, and will not be repeated here.

[0200] Step e3: In the same coordinate system, fit the positive sample similarity of multiple positive samples in the positive sample similarity set to a positive sample similarity distribution curve, and fit the negative sample similarity of multiple negative samples in the negative sample similarity set to a negative sample similarity distribution curve.

[0201] Specifically, by statistically analyzing the frequency or density of each value in the positive sample similarity set, and using kernel density estimation (KDE) or histogram smoothing fitting algorithms, a positive sample similarity distribution curve reflecting the overall distribution characteristics of positive sample similarity can be obtained. The specific method for determining the negative sample similarity distribution curve is similar to that for determining the positive sample similarity distribution curve, and will not be elaborated here.

[0202] In this step, preferably, the normal distribution parameters of the positive sample similarity set and the negative sample similarity set are fitted respectively by the maximum likelihood estimation method to obtain the corresponding mean and standard deviation. Figure 10 This diagram illustrates the positive sample similarity distribution curves and negative sample similarity distribution curves provided in an embodiment of this application. Figure 10 As shown, the green curve represents the positive sample similarity distribution curve, the red curve represents the negative sample similarity distribution curve, and the blue dashed line represents the straight line where the positive sample similarity distribution curve and the negative sample similarity distribution curve intersect.

[0203] Step e4: Determine the similarity corresponding to the intersection of the positive sample similarity distribution curve and the negative sample similarity distribution curve as the first similarity threshold.

[0204] The reason for not directly selecting the interval with only negative sample similarity as the judgment boundary for the first similarity threshold is as follows: the interval with only negative sample similarity corresponds to a region of obviously irrelevant results, and its upper boundary is usually low. If this boundary is directly used as the first similarity threshold, a large number of candidate video frames in the mixed positive and negative sample intervals will be displayed in the first display format, resulting in too many potentially relevant results and increasing false recall. On the other hand, the boundary of the interval with only negative sample similarity is more sensitive to sample size, long-tail samples, and outliers, and its stability is relatively weak. Therefore, compared with this, using the intersection of the probability density distributions of positive sample similarity and negative sample similarity to determine the first similarity threshold can better balance the relationship between preserving potential positive samples and suppressing negative samples, and is therefore more suitable as the lower limit judgment criterion for whether it is worthwhile to display the target video frame in the first display format. Figure 10 In the data, the similarity at the intersection of the positive sample similarity distribution curve and the negative sample similarity distribution curve is 0.1473.

[0205] Step e5: Determine a similarity interval in the coordinate system that includes only positive sample similarity but not negative sample similarity, and determine one similarity value in the similarity interval as the second similarity threshold.

[0206] Specifically, after completing the statistical analysis of the similarity distribution of positive and negative samples, intervals within the high similarity region that contain only positive sample similarity and virtually no negative sample similarity are identified. The initial boundary of these intervals or the similarity value corresponding to a preset statistical position within these intervals is then determined as a high threshold. In this way, the second similarity threshold can correspond to search results with higher confidence, thus enabling the direct display of clearly relevant results.

[0207] In this application, the determination of the first and second similarity thresholds using the aforementioned method enables the adaptive construction of differentiation criteria based on the actual distribution characteristics of the sample data, avoiding the subjectivity and blindness of manually setting thresholds. Furthermore, the first similarity threshold, serving as a lower limit for filtering low-relevance content, effectively balances the risks of missed detections and false detections; the second similarity threshold, acting as an entry threshold for determining high-relevance content, ensures the absolute reliability of directly displayed results. This dual-threshold mechanism achieves hierarchical management of target video frames, guaranteeing both the immediate presentation of high-quality search results and preventing screen space from being occupied by low-confidence results, thereby maximizing user search efficiency and interactive experience within limited display resources.

[0208] It is worth noting that in some embodiments, the similarity value that maximizes the F1 score can be selected as the first similarity threshold based on receiver operating characteristic curve (ROC curve) analysis, and the similarity value corresponding to the accuracy reaching a preset target value (such as 95%) can be selected as the second similarity threshold. Furthermore, in addition to the MobileCLIP model based on FastViT, the visual and text encoding models in this application can also employ other lightweight native multimodal alignment models, including but not limited to TinyCLIP and CLIP-ViT-Base-32.

[0209] Figure 11 The diagram shows a structural schematic of a video retrieval device provided in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the video retrieval device.

[0210] like Figure 11 As shown, the video retrieval device 900 may include a processor 902 and a memory 904.

[0211] The memory 904 is used to store the computer program 906. The memory 904 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The computer program 906 may include computer-executable instructions.

[0212] The processor 902 is used to execute the computer program 906 to implement the above-described database establishment method embodiment and / or video retrieval method embodiment.

[0213] The processor 902 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The video retrieval device 900 includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0214] Finally, it is worth noting that the device executing the database establishment method provided in this application includes a video stream access submodule, a target detection submodule, a dynamic frame extraction submodule, a visual encoding submodule, an index storage submodule, and a tag construction submodule. Specifically, the video stream access submodule can acquire the video stream recorded and continuously output by IPC (e.g., step 110) and transmit the acquired video stream to the target detection submodule. The target detection submodule preferably uses a YOLO series target detection model to detect video frames according to a set detection frame rate to determine whether there are video frames containing a preset target object in the video content (e.g., step 120). The dynamic frame extraction submodule dynamically adjusts the frame extraction frequency based on the target detection results output by the target detection submodule to retain more key video frames corresponding to the appearance of the target and reduce the number of video frames without a target (e.g., steps 130-150). The tag construction submodule constructs corresponding tag information for the video frames in the database.

[0215] The visual encoding submodule encodes each video frame in the database to obtain the corresponding image vector, thus constructing an image vector database. The index storage submodule associates and stores the image vectors with their corresponding label information (such as target category labels, scene labels, and time information), constructing a vector index library (i.e., the image vector database) and a structured index library (i.e., a database storing video frames and label information). The vector index library stores the image vectors corresponding to each video frame to support subsequent similarity retrieval. The structured index library stores the target category labels, scene labels, and time information corresponding to each video frame, supporting subsequent structured initial screening. The two types of indexes are linked through video frame identifiers to enable joint use during the retrieval stage.

[0216] In this application, the device executing the video retrieval method provided includes a structured information extraction submodule, a structured initial screening submodule, a query generation submodule, a text encoding submodule, a similarity retrieval submodule, an exclusion processing submodule, and a result display submodule. Specifically, the structured information extraction submodule receives natural language search terms input by the user and parses the natural language search terms to identify the search content, scene information, and time information that may be contained therein (e.g., step 410). The structured information extraction submodule matches target search objects belonging to subordinate categories to determine the target category corresponding to the user's search terms (i.e., determine the target superordinate category object corresponding to the target search object, e.g., step 530). Furthermore, the structured information extraction submodule performs string literal matching between each element (i.e., the identifier of the camera device) in the user's preset list of monitoring device names or monitoring location names and the natural language search terms input by the user; when a match is successful, the corresponding scene tag information (i.e., the identifier of the camera device) is extracted as a structured search condition. For example, a user installs IPCs in three different scenarios: garage, porch, and driveway, and names them "Garage," "Porch," and "Driveway," respectively. When the user enters the search query "cat on the hood of a car in the garage," the system automatically extracts the corresponding scene tag (i.e., the identifier of the camera device) "Garage."

[0217] Time range information is used to convert natural language time expressions into computable structured time intervals, with a format consistent with the aforementioned structured index. To extract time range information, the structured information extraction submodule processes it based on a time parser and in conjunction with custom rules. Custom rules divide time expressions into basic time types and combined time types, where combined time types are formed by combining one or more basic time types. The basic time types include expressions such as today, yesterday, this week, last week, this month, last month, this year, last year, day of the week, last day of last week, several days ago, several weeks ago, several months ago, several years ago, past several days, past several weeks, past several months, past several years, a specific day of a specific month of a specific year, a specific month of a specific year, and a specific year. The time parser can directly extract basic time types. The combined time types include phrases such as "from...to...", "since...", "after...", and preset expression patterns where multiple basic time types appear side-by-side. The time parser cannot directly extract combined time types; therefore, a method of "basic time type identification + expression pattern boundary determination" is preferred. Specifically, firstly, a time parser is used to extract all basic time type objects from the search terms, then regular expression pattern matching is used to obtain the corresponding preset expression patterns, and the preset expression patterns are interpreted to determine the start and end times of each time segment, which are then merged to obtain a structured time interval.

[0218] Furthermore, the structured information extraction submodule is used to extract and remove time-related substrings from the original query statement. Since the training corpus of multimodal models typically mainly contains objective semantic descriptions of video content, but not specific time information that is not directly related to the content's semantics, retaining time-related substrings during text encoding can easily lead to a deviation between the input text features and the semantic space already learned by the model, thus affecting retrieval performance. Through the aforementioned text cleaning process, while extracting time-range information, a semantically cleaner content description text can be retained as input for subsequent text encoding, thereby improving the matching accuracy between text features and video content features.

[0219] Using the above methods, the system can automatically extract target category information, scene tag information, and time range information without requiring users to manually select category, scene, or time filter conditions, thereby improving the convenience of retrieval and providing input conditions for subsequent structured initial screening.

[0220] After the structured information extraction is completed, the structured preliminary screening submodule performs pre-screening on the index library built in the offline stage based on the extracted target category information, scene tag information, and time range information. Specifically, the structured preliminary screening submodule prioritizes video data whose target category tags match the target category information, whose scene tags match the scene tag information, and whose timestamps are within the time range, as candidate video frames (e.g., steps 520-540).

[0221] The extraction results of target category information can include three scenarios: extraction of a single target category, extraction of multiple target categories, and extraction of no target category. When a single or multiple target categories are extracted, the structured initial screening submodule retains video data whose target category labels contain at least one of the extracted target categories. When no target category is extracted, the target category dimension is not used as a limiting condition, and the search scope reverts to all video data corresponding to all target categories as well as video data that does not contain target category labels.

[0222] The filtering method for scene tag information is similar to that for target category information. When one or more scene tags are extracted, the structured initial screening module only retains video data whose scene tags belong to the extracted scene tag set; when no scene tags are extracted, the scene dimension is not used as a limiting condition, and the search scope falls back to the video data corresponding to all scenes.

[0223] The time range information is used to constrain video timestamps. When the time range information is extracted, the structured initial screening submodule only retains video data whose timestamps fall within the time range; when the time range information is not extracted, the time dimension is not used as a constraint.

[0224] Subsequent vector similarity retrieval is performed only on video frame vectors within the candidate dataset. By adding a structured initial screening step, the system can narrow down the candidate search range before semantic retrieval, reduce the number of invalid vector comparisons, and improve the consistency between search results and user search intent.

[0225] After the candidate video frames are determined, the query generation submodule generates positive query information (i.e., the text expression of the target retrieval object) and verification query information (i.e., the text expression of the first verification object and the text expression of the second verification object) based on the natural language search terms input by the user.

[0226] The query generation submodule matches the natural language search terms (i.e., the target retrieval objects) entered by the user with the entries in the rule dictionary (such as Table 2 above) to determine whether they match the preset easily confused terms. If they match, the first verification object and / or the second verification object are determined based on the preset correspondence in the rule dictionary. If they do not match, only the positive query information is retained for subsequent retrieval processing.

[0227] The text encoding submodule performs text vector encoding on the target retrieval object and the verification objects (first verification object and second verification object) respectively, resulting in the corresponding target retrieval object text vector and verification object text vector. The verification text vector may include the text vector of the first verification object and the text vector of the second verification object.

[0228] The similarity retrieval submodule calculates the similarity between the video frame vectors corresponding to candidate video frames and each text vector. Specifically, the similarity retrieval submodule calculates the positive similarity between the candidate video frame vector and the target retrieval object's text vector, and when a verification object exists, it further calculates the verification similarity between the candidate video frame vector and each verification object's text vector. The preferred similarity is cosine similarity, but it is not limited to this; dot product similarity or other vector similarity measures suitable for a unified semantic space can also be used.

[0229] The similarity retrieval submodule generates initial search results based on positive similarity and outputs the corresponding verification similarity to the subsequent rejection processing submodule. Thus, the system can not only recall candidate results based on positive semantics but also provide multiple sets of comparison criteria for subsequent verification processing, thereby improving its ability to distinguish easily confused semantic scenarios.

[0230] The exclusion processing submodule is used to verify the initial search results based on positive similarity and verification similarity. This verification process does not rely solely on a single exclusion semantic for elimination; instead, it combines target semantics, exclusion semantics, and inclusion semantics for joint discrimination. The result display submodule then performs tiered display of the excluded search results (e.g., step 430).

[0231] Preferably, the visual encoding submodule and the text encoding submodule use the MobileCLIP model, based on FastViT, as the base model, and are fine-tuned based on a self-built training set before deployment. The fine-tuning process uses a contrastive learning loss function. Since the original pre-trained model is mainly trained on a general dataset, its semantic understanding and retrieval performance in the security monitoring field is limited. Therefore, this application further constructs a business training set for security monitoring scenarios. The business training set is a subset of the aforementioned business dataset and is mutually exclusive with the aforementioned business test set. Its size ranges from tens of millions to hundreds of millions of image-text pairs, and the specific size can be adjusted according to actual resource conditions and scenario coverage requirements. Furthermore, the self-built training set is composed of a mixture of the general dataset and the business training set, thereby improving the retrieval capability in the vertical domain of security monitoring while maintaining the model's original semantic understanding capability for general scenarios as much as possible. The general dataset is an open-source image-text description dataset from the industry, with a size ranging from tens of millions to hundreds of billions of image-text pairs. Since the MobileCLIP model has the characteristics of small parameter size, strong native image-text alignment capability, and low inference overhead, it is suitable for deployment on resource-constrained edge devices.

[0232] In this application, the front-end device is preferably an IPC, and the back-end device is preferably an NVR. Based on the actual product form, computing power configuration, and business requirements, the system can adopt a variety of deployment methods in flexible combination, including deployment methods with high front-end load and low back-end load, deployment methods with high back-end load and low front-end load, front-end-only deployment methods, back-end-only deployment methods, front-end and back-end load balancing methods, and deployment methods that introduce client collaborative processing.

[0233] Specifically, the deployment targets include at least an object detection submodule, a visual encoding submodule, a text encoding submodule, and submodules related to index storage and retrieval. Depending on the hardware conditions and business requirements of the front-end devices, back-end devices, and clients, this application can deploy each module in any of the following ways: (1) Front-end deployment only: The target detection submodule, visual encoding submodule, text encoding submodule and index storage and retrieval related submodule are all deployed on the front-end device to complete the video content retrieval processing on the front-end side.

[0234] (2) Backend deployment only: The frontend device is mainly responsible for video acquisition and transmission. The target detection submodule, visual encoding submodule, text encoding submodule, and index storage and retrieval related submodules are all deployed on the backend device, and the backend device completes the detection, encoding, database construction and retrieval processing.

[0235] (3) Front-end and back-end load balancing deployment: The target detection submodule and the visual encoding submodule are deployed on the front-end device to reduce the pressure of uploading the original video and improve the real-time response capability; the text encoding submodule and the index storage and retrieval related submodules are deployed on the back-end device or client to utilize relatively sufficient computing and storage resources to complete query encoding, index management and retrieval processing.

[0236] Through the above-mentioned multiple deployment methods, this application can flexibly allocate the stages of target detection, video frame encoding, text encoding, index storage and retrieval processing according to different hardware resource conditions, thereby taking into account retrieval performance, system resource consumption and engineering deployment adaptability.

[0237] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described database establishment method embodiment and / or video retrieval method embodiment.

[0238] This application provides a computer program that can be executed by a processor to implement the above-described database establishment method embodiment and / or video retrieval method embodiment.

[0239] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described database establishment method embodiment and / or video retrieval method embodiment.

[0240] In the several embodiments provided in this application, any function, if implemented as a software functional module / unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of this application can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or other electronic device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0241] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0242] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In claims enumerating several means, several units or modules of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

[0243] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video retrieval method, characterized in that, The method includes: Obtain a search request carrying search information; Based on the search information, the target video frame is retrieved from the database; Output the target video frame.

2. The method according to claim 1, characterized in that, The retrieval information includes the target retrieval object, and the database includes multiple video frames and multiple sets of tag information, with each video frame corresponding to one of the multiple sets of tag information; The step of retrieving the target video frame from the database based on the retrieval information includes: Retrieve candidate video frames from the database whose tag information matches the retrieval information; Based on the target retrieval object, the target video frame is retrieved from the candidate video frames.

3. The method according to claim 2, characterized in that, The tag information includes the category information of the object in the video frame corresponding to the tag information; The step of retrieving candidate video frames from the database whose tag information matches the retrieval information includes: Based on a preset category affiliation relationship, the category attribute of the target retrieval object in the category affiliation relationship is determined, wherein the category attribute includes subordinate category objects and superior category objects, the category affiliation relationship is the affiliation relationship between subordinate category objects and superior category objects, and the category attribute of the category information in the tag information is the superior category object; If the target retrieval object is a subordinate class object in the category attribution relationship, then the target retrieval object is mapped to its subordinate target superior class object, and video frames containing the category of the target superior class object in the category information of the tag information are retrieved from the database to obtain the candidate video frames; If the target retrieval object is a parent class object in the category attribution relationship, then video frames whose category information in the tag information contains the category of the target retrieval object are retrieved from the database to obtain the candidate video frames.

4. The method according to claim 3, characterized in that, The retrieval information also includes the target time period, and the tag information also includes the acquisition time of the video frame corresponding to the tag information; If the target retrieval object is a subordinate class object in the category attribution relationship, then the target retrieval object is mapped to its subordinate target superordinate class object, and video frames whose category information in the tag information contains the category of the target superordinate class object are retrieved from the database to obtain the candidate video frames, including: If the target retrieval object is a subordinate class object in the category attribution relationship, then the target retrieval object is mapped to its subordinate target superior class object, and video frames whose category information in the tag information contains the category of the subordinate target class object and whose collection time belongs to the target time period are retrieved from the database to obtain the candidate video frames; If the target retrieval object is a parent class object in the category attribution relationship, then video frames whose category information in the tag information contains the category of the target retrieval object are retrieved from the database to obtain the candidate video frames, including: If the target retrieval object is a parent category object in the category attribution relationship, then the candidate video frames are obtained by retrieving the category information in the tag information from the database that contains the category of the target retrieval object and whose collection time belongs to the target time period.

5. The method according to claim 3, characterized in that, The retrieval information also includes target identifiers. The multiple video frames are acquired by multiple camera devices, and each of the multiple camera devices corresponds one-to-one with a multiple identifier. The tag information also includes the identifiers of the camera devices for the video frames corresponding to the tag information. If the target retrieval object is a subordinate class object in the category attribution relationship, then the target retrieval object is mapped to its subordinate target superordinate class object, and video frames whose category information in the tag information contains the category of the target superordinate class object are retrieved from the database to obtain the candidate video frames, including: If the target retrieval object is a subordinate class object in the category attribution relationship, then the target retrieval object is mapped to its subordinate target superordinate class object, and video frames in the tag information containing the category of the target superordinate class object and whose camera device identifier is the target identifier are retrieved from the database to obtain the candidate video frames; If the target retrieval object is a parent class object in the category attribution relationship, then video frames whose category information in the tag information contains the category of the target retrieval object are retrieved from the database to obtain the candidate video frames, including: If the target retrieval object is a parent category object in the category attribution relationship, then the candidate video frame is obtained by retrieving the category information in the tag information from the database that contains the category of the target retrieval object and the identification of the camera device is the target identification.

6. The method according to claim 1, characterized in that, The retrieval information includes the target retrieval object, and the database includes multiple video frames and multiple sets of tag information, with each video frame corresponding to one of the multiple sets of tag information; The step of retrieving the target video frame from the database based on the retrieval information includes: Based on the target retrieval object, the first verification object is determined; Based on the target retrieval object and the first verification object, the target video frame is retrieved from the database.

7. The method according to claim 6, characterized in that, The step of retrieving the target video frame from the database based on the target retrieval object and the first verification object includes: Determine the similarity between the video frame in the database and the target retrieval object to obtain the first similarity corresponding to the video frame; Determine the similarity between the video frame in the database and the first verification object to obtain the second similarity corresponding to the video frame; The target video frame is retrieved from the database based on the first similarity and the second similarity.

8. The method according to claim 7, characterized in that, The step of retrieving the target video frame from the database based on the first similarity and the second similarity includes: Determine the similarity between the video frame in the database and the second verification object to obtain the third similarity corresponding to the video frame, wherein the second verification object includes the target retrieval object and the first verification object; The video frame that meets the conditions is determined as the target video frame, wherein the video frame that meets the conditions is: the maximum value of the first similarity, the second similarity and the third similarity of the video frame is the first similarity or the third similarity, and the first similarity is greater than the first similarity threshold.

9. The method according to claim 1, characterized in that, The search information includes the target search object; The step of retrieving the target video frame from the database based on the retrieval information includes: Determine the similarity between the video frame in the database and the target retrieval object to obtain the first similarity corresponding to the video frame; The video frames with a first similarity greater than the first similarity threshold are determined as the target video frames; The output of the target video frame includes: The target video frames with a first similarity greater than the second similarity threshold are displayed in a first display format; The target video frames with a first similarity value less than or equal to a second similarity threshold are displayed in a second display format, wherein the second similarity threshold is higher than the first similarity threshold; wherein, The first similarity threshold and the second similarity threshold are predetermined through the following steps: Obtain a sample set, which includes multiple sets of positive sample image-text pairs and multiple sets of negative sample image-text pairs; Calculate the similarity between each image-text pair in the sample set to obtain a positive sample similarity set and a negative sample similarity set; In the same coordinate system, the similarity of multiple positive samples in the positive sample similarity set is fitted to a positive sample similarity distribution curve, and the similarity of multiple negative samples in the negative sample similarity set is fitted to a negative sample similarity distribution curve; The similarity corresponding to the intersection of the positive sample similarity distribution curve and the negative sample similarity distribution curve is determined as the first similarity threshold; In the coordinate system, a similarity interval is determined that includes only the positive sample similarity but not the negative sample similarity, and one similarity value in the similarity interval is determined as the second similarity threshold.

10. The method according to claim 1, characterized in that, The database is pre-established through the following steps: Acquire video stream; Target detection is performed on the video frames in the video stream; If a video frame in the video stream is detected to include a preset target object, then a video frame is extracted from the video stream at a first frame extraction rate and used as the first library construction video frame. If no video frame in the video stream is detected to include the preset target object, then a video frame is extracted from the video stream at a second frame extraction rate as the first library construction video frame, wherein the second frame extraction rate is lower than the first frame extraction rate; The database is established based on the first video frame used for database creation.

11. The method according to claim 10, characterized in that, The video stream is a video stream captured at a first frame rate, and the target detection on the video frames in the video stream includes: Target detection is performed on video frames in the video stream at a preset second frame rate, wherein the detected video frames form a first video frame set, and the second frame rate is lower than the first frame rate; If a video frame in the video stream is detected to include a preset target object, then a video frame is extracted from the video stream at a first extraction rate as the first video frame for library construction, including: If a video frame in the video stream is detected to include the preset target object, then a video frame is extracted from the first video frame set at the first frame extraction rate and used as the first library construction video frame. If no video frame in the video stream is detected to include the preset target object, then a video frame is extracted from the video stream at a second frame extraction rate as the first library construction video frame, including: If no video frame in the video stream is detected to include the preset target object, then a video frame is extracted from the first video frame set at the second frame extraction rate and used as the first library construction video frame.

12. The method according to claim 11, characterized in that, The first video frame set includes M video frames ordered in chronological order of acquisition time. The first frame extraction rate is N1 / M, and the second frame extraction rate is N2 / M, where N1, N2, and M are all positive integers, and N2 < N1 < M. If a video frame in the video stream is detected to include the preset target object, then a video frame is extracted from the first video frame set at the first extraction rate as the first library-building video frame, including: If there are K first video frames in the first video frame set that include the preset target object, and K < N1, then K first video frames are obtained, and N1-K video frames are extracted from MK video frames other than the first video frames in the first video frame set as the first library construction video frames, where K is a positive integer, and the extraction of the N1-K video frames satisfies the following: the K first video frames and the N1-K video frames are non-contiguous video frames in the first video frame set; If there are K first video frames in the first video frame set, and K = N1, then obtain K first video frames as the first database construction video frames. If there are K video frames in the first video frame set, and K > N1, then N1 non-contiguous video frames are extracted from the K video frames and used as the first video frames for database construction. If no video frame in the video stream is detected to include the preset target object, then a video frame is extracted from the first video frame set at the second frame extraction rate as the first library construction video frame, including: If the first video frame is not present in the first video frame set, then N2 non-contiguous video frames are extracted from the first video frame set as the first database construction video frames.

13. The method according to claim 1, characterized in that, The database is pre-established through the following steps: Acquire video stream; Determine whether the degree of image change between adjacent video frames in the video stream meets preset conditions; If the degree of image change of adjacent video frames in the video stream meets the preset condition, then video frames are extracted from the video stream at the third frame extraction rate and used as the second library video frames. If the degree of image change of adjacent video frames in the video stream does not meet the preset condition, then a video frame is extracted from the video stream at a fourth frame extraction rate as the second library video frame, wherein the fourth frame extraction rate is lower than the third frame extraction rate. The database is established based on the second video frame used for database creation.

14. The method according to claim 13, characterized in that, The step of determining whether the degree of image change between adjacent video frames in the video stream meets preset conditions includes: Calculate the average value of the pixel difference between each group of adjacent video frames in the video stream to obtain multiple average values ​​that correspond one-to-one with the multiple groups of adjacent video frames. Determine whether the average value is greater than a preset threshold; If the average value is greater than the preset threshold, then it is determined that the degree of image change of a group of adjacent video frames corresponding to the average value meets the preset condition.

15. The method according to claim 14, characterized in that, The video stream includes L video frames ordered in chronological order of capture time. The third frame extraction rate is N3 / L, the fourth frame extraction rate is N4 / L, and L, N3, and N4 are all positive integers, with N4 < N3 < L. After determining whether the average value is greater than a preset threshold, the method further includes: Determine the total number P of the second video frames in the L video frames, wherein the second video frame is a video frame in each group of adjacent video frames whose average value is greater than the preset threshold, P is an integer and P≥0; If the degree of image change between adjacent video frames in the video stream meets the preset condition, then video frames are extracted from the video stream at a third frame extraction rate as the second library video frames, including: If 0 < P < N3, then obtain P second video frames, and extract N3-P video frames from LP video frames other than the second video frames in the L video frames as the second library construction video frames. The extraction of the N3-P video frames satisfies that the P second video frames and the N3-P video frames are non-contiguous video frames in the L video frames. If P=N3, then obtain P of the second video frames as the second video frames for database construction; If P > N3, then N3 non-contiguous video frames are extracted from the P second video frames sorted in order of acquisition time, and used as the second database construction video frames. If the degree of image change in adjacent video frames in the video stream does not meet the preset condition, then video frames are extracted from the video stream at a fourth frame extraction rate as the second library-building video frames, including: If P=0, then N4 non-contiguous video frames are extracted from the L video frames and used as the second database construction video frames.

16. A video retrieval device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the video retrieval method according to any one of claims 1 to 15.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the video retrieval method according to any one of claims 1 to 15.

Images