Video retrieval method and device and electronic equipment

Abstract

The invention discloses a video retrieval method and device and electronic equipment. The video retrieval method comprises the following steps: converting image features of at least one to-be-retrieved video into uniform image grading features; respectively calculating a feature matching degree between a target retrieval object and the image grading feature corresponding to the to-be-retrieved video; and determining a video clip matched with the target retrieval object from the to-be-retrieved video according to the feature matching degree to serve as a target retrieval result. According to the method, the image features are converted into the unified image grading features, so that the specific source and format of the original image features do not need to be concerned in the subsequent retrieval process, the feature matching degree can be directly calculated based on the unified image grading features, and the retrieval speed is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and more particularly to video retrieval methods and apparatus, and electronic devices. Background Technology

[0002] With the development of technology, video retrieval technology has been widely used in fields such as autonomous driving, intelligent security, and smart cities. Video retrieval technology helps users quickly find the video clips they need from massive amounts of video, improving work efficiency and quality of life. However, existing video retrieval technologies have some problems that limit their further development and application. For example, existing video retrieval technologies deal with video image features from different sources, which vary in format and are difficult for the system to reuse directly. Often, features need to be extracted again, significantly reducing retrieval efficiency and increasing time consumption. Summary of the Invention

[0003] This invention provides a video retrieval method, apparatus, and electronic device to solve the problem of low retrieval efficiency in existing video retrieval technologies.

[0004] In a first aspect, the present invention provides a video retrieval method. The video retrieval method includes: converting image features of at least one video to be retrieved into uniform image hierarchical features; calculating the feature matching degree between the target retrieval object and the image hierarchical features corresponding to the video to be retrieved; and determining, based on the feature matching degree, a video segment from the video to be retrieved that matches the target retrieval object as the target retrieval result.

[0005] In one implementation of the first aspect, converting image features of at least one video to be retrieved into unified image grading features includes: obtaining format information of the image features of the video to be retrieved; determining a business scenario matching degree calculation function associated with the target retrieval object; calculating a business scenario matching degree using the business scenario matching degree calculation function; classifying the image features into grading features based on the format information of the image features and the business scenario matching degree; and extracting feature descriptors of the grading features of the image features, and processing the feature descriptors of the grading features according to the grading level to output the image grading features.

[0006] In one implementation of the first aspect, calculating the feature matching degree between the target retrieval object and the image classification feature corresponding to the video to be retrieved includes: calculating a first feature matching degree between the target retrieval object and the image classification feature using a high-level semantic feature comparison function; calculating a second feature matching degree between the target retrieval object and the image classification feature using a mid-level image feature comparison function; and calculating an image feature matching degree based on the first feature matching degree and the second feature matching degree, combined with the business scenario, and outputting it as the feature matching degree.

[0007] In one implementation of the first aspect, determining a video segment matching the target retrieval object from the video to be retrieved based on the feature matching degree as the target retrieval result includes: determining the retrieved video frame based on the feature matching degree, and querying the video information of the video frame, the video information including at least one of video timestamp, video file name, and video image thumbnail; obtaining the timeline of the retrieved video frame based on the video information, and constructing at least one video segment based on the continuous video frames according to the timeline; and locating the corresponding video position based on the timeline according to the specification for the video segment, so as to decode and render the video frame of the video segment.

[0008] In one implementation of the first aspect, the video retrieval method further includes: in response to a target retrieval request, determining the target retrieval object and an image feature set corresponding to the target retrieval request, wherein the image feature set includes image features of the video to be retrieved.

[0009] In one implementation of the first aspect, converting the image features of at least one video to be retrieved into unified image classification features includes: determining a business scenario corresponding to the target retrieval object, and determining a business scenario matching degree calculation function corresponding to the business scenario; calculating the business scenario matching degree between each image feature of the video to be retrieved and the business scenario according to the business scenario matching degree calculation function; classifying each image feature into a corresponding classification level according to the business scenario matching degree; and performing filtering and dimensionality reduction operations on the image features based on the classified classification levels to determine the image classification features.

[0010] In one implementation of the first aspect, the image features are filtered and dimensionality reduced based on the divided hierarchical levels to determine the image hierarchical features, including: obtaining a preset level threshold and the hierarchical levels of all image features in the image feature set; filtering target image features from the image feature set, wherein the target image features are image features whose hierarchical levels are higher than the level threshold; and performing a dimensionality reduction operation on the target image features to use the dimensionality-reduced target image features as the image hierarchical features.

[0011] In one implementation of the first aspect, the target retrieval object is a target image or a target text string.

[0012] In one implementation of the first aspect, calculating the feature matching degree of the image hierarchical features corresponding to the target retrieval object and the video to be retrieved includes: obtaining a first event tag of the target retrieval object and determining a second event tag for each video to be retrieved; comparing the first event tag with the second event tag to determine the comparison result; adjusting the weight value of the video to be retrieved according to the comparison result, and adjusting the calculation order of each video to be retrieved according to the weight value; and performing feature matching degree calculation operations on the image hierarchical features of each video to be retrieved in sequence according to the calculation order.

[0013] In one implementation of the first aspect, the operation of calculating the feature matching degree for the image classification features of each video to be retrieved in sequence includes: if the weight value of the video to be retrieved is greater than a preset weight threshold, then a frame-by-frame strategy is used to calculate the feature matching degree for the image classification features of the video to be retrieved; and if the weight value of the video to be retrieved is less than or equal to the preset weight threshold, then a frame-skipping strategy is used to calculate the feature matching degree for the image classification features of the video to be retrieved.

[0014] In one implementation of the first aspect, the feature matching degree calculation operation includes: determining a high-level semantic feature comparison function and a mid-level image feature comparison function; calculating a first feature matching degree between the target retrieval object and the image hierarchical features based on the high-level semantic feature comparison function; calculating a second feature matching degree between the target retrieval object and the image hierarchical features based on the mid-level image feature comparison function; and determining the feature matching degree between the target retrieval object and the image hierarchical features based on the first feature matching degree and the second feature matching degree.

[0015] In one implementation of the first aspect, determining a video segment matching the target retrieval object from the video to be retrieved based on the feature matching degree includes: obtaining all feature matching degrees corresponding to all image grading features of the target retrieval object; sorting all feature matching degrees and filtering out target grading features based on the sorting results, wherein the target grading features are image grading features with feature matching degrees greater than a preset matching degree threshold; and determining the video segment based on the target grading features.

[0016] In one implementation of the first aspect, determining a video segment matching the target retrieval object from the video to be retrieved based on the feature matching degree as the target retrieval result further includes: displaying the video segment in the form of a timeline; and, in response to a user's click operation on the video segment, decoding the corresponding video segment and rendering and playing the video segment.

[0017] Secondly, the present invention provides a video retrieval device. The video retrieval device includes: a hierarchical feature conversion module configured to convert image features of at least one video to be retrieved into unified image hierarchical features; a hierarchical feature retrieval module configured to calculate the feature matching degree between a target retrieval object and the image hierarchical features corresponding to the video to be retrieved; and a retrieval result presentation module configured to determine, based on the feature matching degree, a video segment from the video to be retrieved that matches the target retrieval object as the target retrieval result.

[0018] Thirdly, the present invention provides an electronic device. The electronic device includes: a memory configured to store an executable program; and a processor configured to execute the program to perform the video retrieval method described in any one of the preceding claims.

[0019] According to embodiments of this disclosure, by converting image features of videos from different sources and in different formats into unified image hierarchical features, subsequent retrieval processes do not require concern about the specific source and format of the original image features. Retrieval can be directly performed based on the unified and standardized image hierarchical features, avoiding the repetitive decoding and feature extraction operations required in existing technologies to ensure compatibility with different image features. This significantly reduces computational redundancy and time overhead, improving retrieval efficiency. Furthermore, by calculating the feature matching degree between the target object and the unified image hierarchical features, video segments can be quickly and effectively located, thereby retrieving videos related to the target object from the videos to be retrieved. Attached Figure Description

[0020] Figure 1 This is a flowchart of a video retrieval method according to an embodiment of the present invention.

[0021] Figure 2 This is a block diagram of a video retrieval device according to an embodiment of the present invention.

[0022] Figure 3 This is a system architecture diagram of a video retrieval system according to an embodiment of the present invention. Detailed Implementation

[0023] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0024] With the development of technology, video retrieval technology has been widely used in fields such as autonomous driving, intelligent security, smart cities, smart communities, smart offices, and home IPCs. In practical applications, users can input images as search criteria to quickly locate highly relevant video segments from massive amounts of video data.

[0025] However, existing video retrieval technologies have several drawbacks: (1) High retrieval cost: Due to the need to decode, extract features, and compare similarity of massive amounts of video data, the entire process is computationally complex and usually takes hours, making it difficult to meet the demand for high-frequency retrieval. (2) Low retrieval efficiency: The video image features extracted by different systems are diverse, and existing solutions cannot directly utilize existing feature data, often requiring re-extraction, resulting in redundant calculations and further increasing retrieval time. (3) Limited functional support: Existing video retrieval technologies typically only support retrieval based on low-level image features (such as color and texture) and lack support for multimodal retrieval (such as text-image bimodal retrieval).

[0026] To address at least the aforementioned technical problems, this disclosure provides a video retrieval method. According to this disclosure, by converting image features of videos from different sources and in different formats into unified image hierarchical features, subsequent retrieval processes do not require concern about the specific source and format of the original image features. Retrieval can be directly performed based on the unified and standardized image hierarchical features, avoiding the repetitive decoding and feature extraction operations required in existing technologies to ensure compatibility with different image features. This significantly reduces computational redundancy and time overhead, improving retrieval efficiency. Furthermore, by calculating the feature matching degree between the target object and the unified image hierarchical features, video segments can be quickly and effectively located, thereby retrieving videos related to the target object from the videos to be retrieved.

[0027] In the following, the technical solutions according to this disclosure will be described with reference to specific embodiments and in conjunction with the accompanying drawings.

[0028] Figure 1 This is a flowchart illustrating a video retrieval method 100 according to an embodiment of the present disclosure. (Refer to...) Figure 1 The method 100 includes the following steps 102 to 106.

[0029] Step 102: Convert the image features of at least one video to be retrieved into uniform image hierarchical features.

[0030] The image features refer to image features extracted from different sources, algorithms, or devices, which may differ in format, dimension, encoding method, etc. In practical applications, the image features and the corresponding videos to be retrieved can be uniformly stored in an image feature database for easy and unified retrieval during subsequent feature conversion.

[0031] It should be noted that when converting image features into unified image grading features, either feature quality grading or feature value grading can be used. Feature quality grading refers to grading based on feature quality attributes (such as feature size, feature probability, feature correlation, etc.). Feature value grading refers to grading based on feature value attributes (i.e., the feature's contribution to the matching of the business scenario). This embodiment mainly introduces how to perform grading based on feature value attributes.

[0032] Step 104: Calculate the feature matching degree of the image hierarchical features corresponding to the target retrieval object and the video to be retrieved.

[0033] The target retrieval object is the specific content the user wants to find, which can be a target image (such as a picture or screenshot) or a target text string (such as a text description). For example, the target retrieval object could be: a red car driving in the rain. In practical applications, because the target retrieval object can be a target image or a target text string, this invention has multimodal retrieval capabilities (i.e., image-text dual-modal retrieval), which can adapt to different types of retrieval needs and improve practicality.

[0034] The feature matching degree refers to the similarity measure between the image hierarchical features of the target retrieval object (such as the target image) and the video to be retrieved. The higher the feature matching degree, the stronger the similarity.

[0035] Step 106: Based on the feature matching degree, determine the video segments from the videos to be retrieved that match the target retrieval object, and use them as the target retrieval results. The target retrieval results refer to the final output set of matching video segments, which may include information such as video timestamps for user viewing and verification.

[0036] In some embodiments, converting image features of at least one video to be retrieved into unified image grading features includes: obtaining format information of the image features of the video to be retrieved; determining a business scenario matching degree calculation function associated with the target retrieval object; calculating a business scenario matching degree using the business scenario matching degree calculation function; classifying the image features into grading features based on the format information of the image features and the business scenario matching degree; and extracting feature descriptors of the grading features of the image features, and processing the feature descriptors of the grading features according to the grading level to output the image grading features. The format information refers to metadata information describing the image features, such as data structure, encoding method, dimensions, and type.

[0037] The business scenario matching degree calculation function is a preset algorithm function used to calculate the relevance between the image features and the business scenario. The business scenario refers to a specific usage scenario or application environment, such as intelligent security, autonomous driving, or intelligent office. The business scenario matching degree refers to the degree of relevance between a certain image feature and the specific application scenario pointed to by the current target retrieval object, evaluated and quantified by the business scenario matching degree calculation function. In practical applications, the business scenario matching degree and the format information of the pre-acquired image features can be combined to perform hierarchical decision-making, classifying the original, heterogeneous image features into different level categories. That is, the image features of the video to be retrieved are converted into unified hierarchical features with clear labels indicating the strength of business relevance. For example, image features with a business scenario matching degree higher than 70% can be set as Level 1, image features with a business scenario matching degree between 30% and 70% as Level 2, and image features with a business scenario matching degree lower than 30% as Level 3.

[0038] The feature descriptor refers to the specific numerical vector or statistical information of the image feature. The hierarchical level refers to the level label (e.g., level 1, level 2, level 3) after the image feature is divided, which reflects the value weight of the feature in a specific business scenario. In practical applications, the feature descriptors of the hierarchical features can be processed according to the hierarchical level to output the image hierarchical features. Specifically, the processing can be based on the hierarchical level to which the hierarchical feature belongs. For example, for high-level (e.g., level 1) hierarchical features, their complete descriptors may be retained or only lightweight compression may be performed to ensure the integrity of the information; for medium-level hierarchical features, moderate dimensionality reduction or information reduction may be performed; for low-level hierarchical features, significant information filtering or direct discarding may be performed.

[0039] As described above, by calculating the matching degree of business scenarios, classifying hierarchical features, and processing feature descriptors, the standardized transformation of image features from different sources and with different structures is achieved, providing structurally unified and quality-controllable feature inputs for subsequent efficient retrieval.

[0040] In some embodiments, calculating the feature matching degree between the target retrieval object and the image classification feature corresponding to the video to be retrieved includes: calculating a first feature matching degree between the target retrieval object and the image classification feature using a high-level semantic feature comparison function; calculating a second feature matching degree between the target retrieval object and the image classification feature using a mid-level image feature comparison function; and calculating an image feature matching degree based on the first feature matching degree and the second feature matching degree, combined with the business scenario, and outputting it as the feature matching degree.

[0041] The advanced semantic feature comparison function (such as the cosine similarity function) is a function used to calculate the similarity between advanced semantic features in the target retrieval object and advanced semantic features in the image hierarchical features. These advanced semantic features are features extracted from images or text that express abstract concepts and semantic information (such as "car," "dog," and "running"). Correspondingly, the first feature matching degree is the matching degree calculated based on the advanced semantic feature comparison function, which reflects the semantic similarity between the target retrieval object and the image hierarchical features.

[0042] The intermediate-level image feature comparison function is a function used to calculate the similarity between intermediate-level image features in the target retrieval object and intermediate-level image features in the image hierarchy features. Intermediate-level image features lie between low-level pixels and high-level semantics, and typically include information such as keypoints, edges, and texture statistics. In practical applications, this intermediate-level image feature comparison function can be Euclidean distance, structured cross-correlation, or a structural similarity function. Correspondingly, the second feature matching degree is the matching degree calculated based on the intermediate-level image feature comparison function, which reflects the similarity between the target retrieval object and the image hierarchy features at the visual feature level.

[0043] The feature matching degree is a comprehensive matching degree obtained by fusing the first and second feature matching degrees. In practical applications, different weights can be assigned to the first and second feature matching degrees according to the business scenario corresponding to the target retrieval object, and then a comprehensive score can be calculated, which is the feature matching degree. Of course, for different business scenarios, the weight ratio of semantic and visual features can be dynamically adjusted for fusion. For example, in the security scenario, semantic matching can be strengthened to achieve scenario-adaptive optimization of the retrieval strategy, significantly reducing the false detection rate and false negative rate.

[0044] As described above, the matching degree between the target retrieval object and the image hierarchical features is calculated using a high-level semantic feature comparison function and a mid-level image feature comparison function, respectively. The two matching degrees are then fused together to integrate the semantic information and visual details of the image, thereby improving the comprehensiveness and accuracy of feature comparison and enhancing the robustness of retrieval for complex scenes and ambiguous targets.

[0045] In some embodiments, determining a video segment matching the target retrieval object from the video to be retrieved based on the feature matching degree as the target retrieval result includes: determining the retrieved video frames based on the feature matching degree, and querying the video information of the video frames, the video information including at least one of video timestamp, video file name, and video image thumbnail; obtaining the timeline of the retrieved video frames based on the video information, and constructing at least one video segment based on the continuous video frames according to the timeline; and locating the corresponding video position based on the timeline according to the specification for the video segment, so as to decode and render the video frames of the video segment.

[0046] In practical applications, after determining the feature matching degree between the target retrieval object and various image hierarchical features, feature filtering can be performed based on this matching degree to select target hierarchical features with a matching degree greater than a certain threshold. Next, the retrieved video frames are determined based on these target hierarchical features, and the corresponding video information is queried. This video information may include video timestamps, file names, and image thumbnails. Then, cluster analysis is performed based on the video timestamps in the video information to obtain a corresponding timeline. Several consecutive video frames (i.e., adjacent video frames on the timeline) constitute a video segment for visualization to the user. When the user clicks on a video segment, the system calls the video decoder, locates the video position of the nearest keyframe according to the timeline, and begins decoding from that point. During decoding, the system discards video frames that do not meet the time requirements until the precise video position is located. Finally, the system renders the video image from that video position, completing the accurate playback and presentation of the target retrieval results.

[0047] As described above, by querying video information and aggregating timelines, scattered frame-level matching results are aggregated into continuous video segments, avoiding the return of a large number of isolated frames. Based on this, for a specific video segment, location is performed based on the timeline, allowing the video decoder to directly jump to keyframes for decoding and rendering, avoiding redundant calculations in full-segment decoding and improving response efficiency.

[0048] In some embodiments, prior to step 102, the video retrieval method may further include: in response to a target retrieval request, determining a target retrieval object and an image feature set corresponding to the target retrieval request, wherein the image feature set includes image features of the video to be retrieved. The target retrieval request is a retrieval instruction to find a target retrieval object, which is the specific content the user wants to find.

[0049] As described above, when the target retrieval request is received, the target retrieval object and the image feature set of the video to be retrieved are determined so that the target retrieval object can be quickly retrieved based on the image features in the image feature set.

[0050] In some embodiments, converting image features of at least one video to be retrieved into unified image grading features includes: determining a business scenario corresponding to the target retrieval object, and determining a business scenario matching degree calculation function corresponding to the business scenario; calculating the business scenario matching degree between each image feature of the video to be retrieved and the business scenario according to the business scenario matching degree calculation function; classifying each image feature into a corresponding grading level according to the business scenario matching degree; and performing filtering and dimensionality reduction operations on the image features based on the divided grading levels to determine the image grading features.

[0051] As described above, calculating the matching degree between each image feature of the video to be retrieved and the business scenario, and classifying them according to the matching degree, can effectively filter out image features that are highly relevant to the business scenario. This allows for differentiated filtering and purification of image features based on the value orientation of different retrieval tasks. In other words, based on the current retrieval intent, the most relevant and valuable image features are retained, while irrelevant or low-value information is discarded. This makes subsequent feature processing more targeted, reduces interference from irrelevant features, and improves retrieval efficiency and the relevance of retrieval results.

[0052] In some embodiments, determining the image classification features by performing filtering and dimensionality reduction operations on the image features based on the defined classification levels includes: obtaining a preset level threshold and the classification levels of all image features in the image feature set; filtering target image features from the image feature set, wherein the target image features are image features whose classification level is higher than the level threshold; and performing dimensionality reduction operations on the target image features to use the dimensionality-reduced target image features as the image classification features. It should be noted that the filtering operation is an operation that retains high-level features (i.e., high-value features) and discards low-level features (i.e., low-value or redundant features), which can effectively reduce the number of features that need to be processed subsequently.

[0053] The grading level is a grade value determined based on the matching degree of the business scenario, which is used to guide the filtering operation of image features (i.e., retention or rejection). In practical applications, image features can be graded based on a preset grading system, which can be various grading systems, such as a three-level grading system or a five-level grading system. Taking a three-level grading system as an example, its specific division criteria can be as follows: Level 1: Image features with a business scenario matching degree greater than 70%; Level 2: Image features with a business scenario matching degree greater than 30% and less than or equal to 70%; Level 3: Image features with a business scenario matching degree less than or equal to 30%.

[0054] The grading threshold is a preset grading threshold used to filter out high-value features. For example, if the grading threshold is set to three levels in a three-level grading system, then during the filtering operation, only image features at the first and second grading levels will be retained, while image features corresponding to the third level will be filtered out.

[0055] Dimensionality reduction refers to the process of reducing the dimensionality of features. In practical applications, it can reduce the dimensionality of feature descriptors for target image features, such as reducing a 64-level histogram description to a 48-level or 36-level histogram description, thus resulting in shorter feature vectors for the image grading features. It should be noted that during dimensionality reduction, the feature descriptors of the target image features can be simplified or optimized to different degrees according to the grading level; that is, different intensities of dimensionality reduction are applied based on the grading level. For example, for high-level (e.g., Level 1) target image features, a slight dimensionality reduction can be performed on their feature descriptors to retain more detailed information, such as reducing a 64-level histogram description to 48 levels. For medium / low-level target image features (e.g., Level 2), a more significant dimensionality reduction can be performed on their feature descriptors, such as reducing a 64-level histogram description to a 16-level histogram description, retaining only basic features to maximize subsequent retrieval speed.

[0056] As described above, by setting a level threshold to filter image features, target image features are obtained, thus retaining high-value features and removing low-value features, effectively reducing the number of features and computational complexity. Furthermore, dimensionality reduction processing is applied to the target image features to further reduce feature complexity, facilitating subsequent rapid retrieval.

[0057] In some embodiments, calculating the feature matching degree of the image grading features corresponding to the target retrieval object and the video to be retrieved includes: obtaining a first event tag of the target retrieval object and determining a second event tag for each video to be retrieved; comparing the first event tag with the second event tag to determine the comparison result; adjusting the weight value of the video to be retrieved according to the comparison result, and adjusting the calculation order of each video to be retrieved according to the weight value; and performing feature matching degree calculation operations on the image grading features of each video to be retrieved in sequence according to the calculation order.

[0058] The first event tag is a semantic tag identified and extracted from the input target retrieval object, used to describe the event or content category in the target retrieval object. It can be extracted through corresponding machine learning or deep learning models. The second event tag is a semantic tag describing the content that is pre-added to video frames or video segments by the front-end algorithm at the video capture end. This second event tag can usually be directly embedded in the video bitstream of the video to be retrieved.

[0059] The weight value is a numerical value assigned to the video to be retrieved based on the comparison results, used to adjust the retrieval order (i.e., retrieval priority) of the videos. In practical applications, if the comparison result is: event tag mismatch, the weight value of the video to be retrieved can be reduced; if the comparison result is: event tag match, the weight value of the video to be retrieved can be increased. After determining the weight values ​​of each video to be retrieved, they are sorted according to the size of the weight values, so that videos with high weight values ​​are calculated first. This ensures that users can see the most likely relevant results faster by providing incremental retrieval results (i.e., gradually returning retrieval results), alleviating users' waiting anxiety. It should be noted that since the calculation and comparison of event tags are relatively fast and have a short processing time, sorting the videos to be retrieved in advance using event tags can indirectly improve retrieval speed and efficiency, and reduce users' waiting anxiety.

[0060] As described above, by introducing a pre-comparison mechanism for event tags, weights are assigned to the videos to be retrieved and their calculation priorities are dynamically adjusted. This allows for the priority processing of image grading features corresponding to high-weight videos to be retrieved, thereby prioritizing the presentation of highly relevant search results. This reduces the waiting time for highly relevant search results, alleviates user anxiety, and improves the user experience.

[0061] In some embodiments, the operation of calculating the feature matching degree for the image classification features of each video to be retrieved sequentially includes: if the weight value of the video to be retrieved is greater than a preset weight threshold, then a frame-by-frame strategy is used to calculate the feature matching degree for the image classification features of the video to be retrieved; and if the weight value of the video to be retrieved is less than or equal to the preset weight threshold, then a frame-skipping strategy is used to calculate the feature matching degree for the image classification features of the video to be retrieved. It should be noted that the preset weight threshold is a preset weight threshold, which can be set according to actual conditions and is not limited here.

[0062] The frame-by-frame strategy refers to performing feature matching calculations on every frame of the video to be retrieved. The frame-skipping strategy refers to skipping some video image frames during feature matching calculations, such as skipping one frame every N frames. This can reduce computational load and improve retrieval efficiency, and is suitable for videos with low weight values ​​to be retrieved.

[0063] As described above, by employing a frame-by-frame or frame-skipping strategy to calculate feature matching degree based on video weight values, computational resources are allocated rationally, maximizing retrieval efficiency while ensuring overall retrieval performance. Specifically: for videos with high weight values, meticulous frame-by-frame calculations are used to ensure retrieval accuracy and completeness of results; for videos with low weight values, a frame-skipping strategy is employed, sacrificing a small amount of accuracy for faster retrieval and avoiding excessive processing time.

[0064] In some embodiments, the feature matching degree calculation operation includes: determining a high-level semantic feature comparison function and a mid-level image feature comparison function; calculating a first feature matching degree between the target retrieval object and the image grading feature based on the high-level semantic feature comparison function; calculating a second feature matching degree between the target retrieval object and the image grading feature based on the mid-level image feature comparison function; and determining a feature matching degree between the target retrieval object and the image grading feature based on the first feature matching degree and the second feature matching degree.

[0065] In practical applications, before calculating the first feature matching degree based on the high-level semantic feature comparison function, high-level feature descriptors corresponding to the target retrieval object and image hierarchical features can be obtained separately. Then, the first feature matching degree is calculated based on the corresponding high-level feature descriptors and the high-level semantic feature comparison function. It should be noted that this high-level feature descriptor includes semantic feature tokens representing high-level semantic features and their associated information. Similarly, before calculating the second feature matching degree based on the intermediate-level image feature comparison function, intermediate-level feature descriptors corresponding to the target retrieval object and image hierarchical features are also obtained separately. Then, the second feature matching degree is calculated based on the corresponding intermediate-level feature descriptors and the intermediate-level image feature comparison function. It should be noted that this intermediate-level feature descriptor includes image keypoints representing intermediate-level features and image statistical information near the keypoints.

[0066] As described above, calculating the matching degree using an intermediate-level image feature comparison function ensures the accuracy of image matching in terms of detail and structure, while calculating the matching degree using a high-level semantic feature comparison function guarantees the ability to understand the deep semantics of image content. Furthermore, fusing these two methods allows for a more comprehensive and accurate assessment of the similarity between the target retrieval object and the corresponding image frames of the image hierarchy features, ensuring the accuracy and reliability of the retrieval results.

[0067] In some embodiments, determining a video segment matching the target retrieval object from the video to be retrieved based on the feature matching degree includes: obtaining all feature matching degrees corresponding to all image classification features of the target retrieval object; sorting all feature matching degrees and filtering out target classification features based on the sorting results, wherein the target classification features are image classification features with feature matching degrees greater than a preset matching degree threshold; and determining the video segment based on the target classification features. It should be noted that the matching degree threshold can be set according to actual conditions and is not limited here.

[0068] In practical applications, after obtaining the feature matching degree between the target retrieval object and each image hierarchical feature, feature filtering can be performed based on this matching degree to select target hierarchical features with a matching degree greater than a threshold. Next, the retrieved video frames (i.e., the matched video frames) are determined based on these target hierarchical features, and the corresponding video information (such as video timestamps) is queried. Then, cluster analysis is performed based on the video timestamps in the video information, and several consecutive video frames constitute a video segment for visualization presentation to the user.

[0069] As described above, sorting and filtering image hierarchical features based on feature matching degree can quickly locate video segments that are highly relevant to the target retrieval object, avoid returning a large number of irrelevant low-quality results, and ensure the accuracy and simplicity of the target retrieval results.

[0070] In some embodiments, determining a video segment that matches the target retrieval object from the video to be retrieved based on the feature matching degree as the target retrieval result further includes: displaying the video segment in the form of a timeline; and, in response to a user's click operation on the video segment, decoding the corresponding video segment and rendering and playing the video segment.

[0071] In practical applications, when a user clicks on a video segment on the timeline, the video decoder locates the nearest upstream keyframe in the video stream based on the video timestamp corresponding to that segment, using it as the decoding starting point. The video decoder then begins decoding from this starting point, discarding irrelevant frames, until it precisely decodes to the actual video position of the segment, and then renders and plays the video from that actual position.

[0072] As described above, displaying a timeline makes it easier for users to quickly browse and locate video clips of interest, thus improving usability.

[0073] Figure 2 This is a schematic diagram illustrating the structure of a video retrieval device according to an embodiment of the present disclosure. Figure 2 As shown, the video retrieval device 200 includes a hierarchical feature conversion module 210, a hierarchical feature retrieval module 220, and a retrieval result presentation module 230.

[0074] The hierarchical feature conversion module 210 is configured to convert the image features of at least one video to be retrieved into unified image hierarchical features.

[0075] The hierarchical feature retrieval module 220 is configured to calculate the feature matching degree of the image hierarchical features corresponding to the target retrieval object and the video to be retrieved.

[0076] Furthermore, the search result presentation module 230 is configured to determine, based on the feature matching degree, a video segment from the video to be searched that matches the target search object as the target search result.

[0077] In some embodiments, the video retrieval device 200 may further include a determining module. The obtaining module is configured to, in response to a target retrieval request, determine the target retrieval object and an image feature set corresponding to the target retrieval request, wherein the image feature set includes image features of the video to be retrieved.

[0078] It should be understood that the hierarchical feature conversion module 210, hierarchical feature retrieval module 220, retrieval result presentation module 230, and determination module can be further configured to perform the corresponding steps or actions in the video retrieval method described in the above embodiments, which will not be repeated here.

[0079] Figure 3 This is an architectural diagram illustrating a video retrieval system according to an embodiment of the present disclosure. (Refer to...) Figure 3 The system includes an image feature integration service module, a video retrieval device as described above, and an optional image feature database.

[0080] In such Figure 3 In the video retrieval system shown, the video retrieval device includes a hierarchical feature conversion module, a hierarchical feature retrieval module, and a retrieval result presentation module. In some embodiments, the video retrieval device may further include a determination module.

[0081] In addition, the image feature integration service module is used to receive and store image features from the hierarchical feature encoder. The hierarchical feature encoder mainly includes two types: locally deployed image hierarchical feature encoders and image hierarchical feature encoders accessed remotely via a network.

[0082] Furthermore, the image feature database is used to store image features from the image feature integration service module. Storing image features in this database reduces the coupling between system components and isolates the feature storage logic and feature retrieval logic of the video retrieval system. It should be noted that the image feature database can be optionally configured. Of course, in some implementations, an image feature queue or adjacency list can be used instead of the image feature database to achieve a similar purpose.

[0083] Based on the above-mentioned image feature integration service module, video retrieval device, and optional image feature database, the video retrieval system of the present invention can support three typical system architecture strategies: (1) Real-time processing mode: After the video to be retrieved is decoded, the locally deployed image hierarchical feature encoder is immediately invoked to extract features, and the generated image features are directly transmitted to the video retrieval device for real-time analysis through the image feature integration service module. (2) Remote feature access mode: The image features generated by the remotely deployed image hierarchical feature encoder are obtained through the interface, and the generated image features are directly input into the video retrieval device for processing through the image feature integration service module. (3) Database middleware mode: Regardless of whether the feature encoder is deployed locally or remotely, the extracted image features are stored in a unified image feature database through the image feature integration service module. When the video retrieval device searches, it directly queries the image feature database to obtain the image features and complete the video retrieval task.

[0084] According to another aspect of the invention, this disclosure also provides an electronic device. The electronic device includes a memory and a processor. The memory is configured to store an executable program. The processor is communicatively connected to the memory and configured to execute the program to perform the methods described above.

[0085] In summary, the video retrieval method, apparatus, and electronic device provided by the present invention: (1) solves the problem of deep learning image feature conversion through the hierarchical feature conversion module, avoiding the waste of retrieval time by re-extracting video features, which indirectly solves the problem of image feature reuse, enabling the retrieval system to support multiple different types of image features. At the same time, through the hierarchical feature retrieval module, video retrieval is performed based on the intermediate and high-level semantic features of the image, improving the user retrieval experience. In addition, it can also support image and text dual-modal retrieval. (2) Integrating IPC event tags accelerates the video retrieval process, using IPC event tags to quickly filter video segments, further improving the video retrieval speed, enhancing the user retrieval experience, and alleviating waiting anxiety.

[0086] It should be understood that the terms "first," "second," etc., are used in this invention to describe various types of information, but these terms are not limited to them and are only used to distinguish information of the same type from each other. For example, without departing from the scope of this invention, "first" information can also be referred to as "second" information, and similarly, "second" information can also be referred to as "first" information.

[0087] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A video retrieval method, characterized in that, include: Convert the image features of at least one video to be retrieved into uniform image hierarchical features; Calculate the feature matching degree of the image hierarchical features corresponding to the target retrieval object and the video to be retrieved; as well as Based on the feature matching degree, video segments that match the target retrieval object are determined from the video to be retrieved and used as the target retrieval result.

2. The video retrieval method according to claim 1, characterized in that, Converting at least one image feature of a video to be retrieved into uniform image hierarchical features includes: Obtain the format information of the image features of the video to be retrieved; Determine the business scenario matching degree calculation function associated with the target retrieval object; The business scenario matching degree is calculated using the business scenario matching degree calculation function, and the image features are divided into hierarchical features based on the format information of the image features and the business scenario matching degree; and Extract the feature descriptors of the hierarchical features of the image features, and process the feature descriptors of the hierarchical features according to the hierarchical level to output the image hierarchical features.

3. The video retrieval method according to claim 1, characterized in that, Calculating the feature matching degree of the image hierarchical features corresponding to the target retrieval object and the video to be retrieved includes: The first feature matching degree between the target retrieval object and the image hierarchical features is calculated using an advanced semantic feature comparison function; The second feature matching degree between the target retrieval object and the image hierarchical features is calculated using an intermediate-level image feature comparison function; and Based on the first feature matching degree and the second feature matching degree, the image feature matching degree is calculated by combining the business scenario, and the output is used as the feature matching degree.

4. The video retrieval method according to claim 1, characterized in that, Determining video segments from the video to be retrieved that match the target retrieval object based on the feature matching degree as target retrieval results includes: The retrieved video frames are determined based on the feature matching degree, and the video information of the video frames is queried. The video information includes at least one of the following: video timestamp, video file name, and video image thumbnail. Obtain the timeline of the retrieved video frames based on the video information, and construct at least one video segment based on the timeline and consecutive video frames; and Based on the specified video segment, the corresponding video position is located on the timeline to decode and render the video frames of the video segment.

5. The video retrieval method according to claim 1, characterized in that, Also includes: In response to a target retrieval request, the target retrieval object and image feature set corresponding to the target retrieval request are determined, wherein the image feature set includes image features of the video to be retrieved.

6. The video retrieval method according to claim 5, characterized in that, Converting at least one image feature of a video to be retrieved into uniform image hierarchical features includes: Determine the business scenario corresponding to the target retrieval object, and determine the business scenario matching degree calculation function corresponding to the business scenario; Based on the business scenario matching degree calculation function, calculate the matching degree between each image feature of the video to be retrieved and the business scenario. Based on the matching degree of the business scenario, each image feature is classified into a corresponding hierarchical level; and The image features are filtered and reduced in dimensionality based on the hierarchical classification to determine the image hierarchical features.

7. The video retrieval method according to claim 6, characterized in that, The image features are filtered and reduced in dimensionality based on the hierarchical classification to determine the image hierarchical features, including: Obtain the preset level threshold and the level of all image features in the image feature set; Target image features are selected from the image feature set, wherein the target image features are image features whose classification level is higher than the classification threshold; and The target image features are subjected to dimensionality reduction to use the dimensionality-reduced target image features as the image classification features.

8. The video retrieval method according to claim 1, characterized in that, The target retrieval object is a target image or a target text string.

9. The video retrieval method according to claim 1, characterized in that, Calculating the feature matching degree of the image hierarchical features corresponding to the target retrieval object and the video to be retrieved includes: Obtain the first event tag of the target retrieval object, and determine the second event tag of each video to be retrieved; The first event tag is compared with the second event tag to determine the comparison result; The weight values ​​of the videos to be retrieved are adjusted based on the comparison results, and the calculation order of each video to be retrieved is adjusted based on the weight values; and According to the calculation order, feature matching degree calculation operations are performed on the image hierarchical features of each video to be retrieved in sequence.

10. The video retrieval method according to claim 9, characterized in that, The feature matching degree calculation operation for the image hierarchical features of each video to be retrieved includes: If the weight value of the video to be retrieved is greater than a preset weight threshold, then a frame-by-frame strategy is used to calculate the feature matching degree of the image hierarchical features of the video to be retrieved; and If the weight value of the video to be retrieved is less than or equal to the preset weight threshold, a frame skipping strategy is used to calculate the feature matching degree of the image hierarchical features of the video to be retrieved.

11. The video retrieval method according to claim 10, characterized in that, The feature matching degree calculation operation includes: Determine the high-level semantic feature matching function and the mid-level image feature matching function; Based on the advanced semantic feature comparison function, the first feature matching degree between the target retrieval object and the image hierarchical features is calculated; Based on the intermediate image feature comparison function, calculate the second feature matching degree between the target retrieval object and the image hierarchical features; and Based on the first feature matching degree and the second feature matching degree, the feature matching degree between the target retrieval object and the image hierarchical features is determined.

12. The video retrieval method according to claim 1, characterized in that, Determining video segments that match the target retrieval object from the video to be retrieved based on the feature matching degree includes: Obtain the feature matching degree between the target retrieval object and all image hierarchical features; All feature matching degrees are sorted, and target hierarchical features are selected based on the sorting results. The target hierarchical features are image hierarchical features whose feature matching degree is greater than a preset matching degree threshold. The video segment is determined based on the target classification features.

13. The video retrieval method according to claim 12, characterized in that, The method of determining video segments that match the target retrieval object from the video to be retrieved based on the feature matching degree as target retrieval results also includes: The video clips are displayed in a timeline format; and In response to a user's click on the video segment, the corresponding video segment is decoded and rendered for playback.

14. A video retrieval device, characterized in that, include: The hierarchical feature conversion module is configured to convert image features of at least one video to be retrieved into uniform image hierarchical features; The hierarchical feature retrieval module is configured to calculate the feature matching degree of the image hierarchical features corresponding to the target retrieval object and the video to be retrieved, respectively. as well as The search results presentation module is configured to determine, based on the feature matching degree, a video segment from the video to be searched that matches the target search object as the target search result.

15. An electronic device, characterized in that, include: The memory is configured to store executable programs; as well as A processor is configured to execute the program to cause the electronic device to perform the video retrieval method according to any one of claims 1 to 13.

Images