Video retrieval method, device, equipment and product

Abstract

The invention discloses a video retrieval method, device, equipment and product, and relates to the technical field of artificial intelligence, and the video retrieval method comprises the following steps: performing video frame extraction on a video to be retrieved according to a fixed sampling frequency to obtain a first video frame sequence; determining a target sampling frequency according to the content complexity and / or the system resource utilization rate of the first video frame sequence, and performing video frame extraction on the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence; and performing feature extraction on the video frames in the second video frame sequence, and retrieving a similar video corresponding to the to-be-retrieved video from a video library based on the extracted features, thereby realizing the balance between the analysis precision of the video and the use of computing resources.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to video retrieval methods, devices, equipment and products. Background Art

[0002] Video retrieval refers to the use of artificial intelligence technology to determine whether a given video is a duplicate or identical to other videos. It is widely used in scenarios such as video infringement judgment and similar video recommendation.

[0003] In the related art, the video retrieval device can first extract the frame sequence of the video to be retrieved to obtain the video frame sequence, then extract the image features of the video frames in the video frame sequence to obtain the image feature sequence, and then extract the fixed-length target video features from the image feature sequence to obtain the representation of the video to be retrieved on the video element, and finally retrieve similar videos from the video library based on the target video features, that is, based on the representation of the video to be retrieved on the video element. In this way, it is possible to determine whether the videos are similar based on the video elements without performing timing logic judgment, so that whether it is a general video with normal timing, a mixed video with chaotic timing, or a variable length video, it is possible to effectively perform video retrieval, which ultimately improves the effectiveness of video retrieval.

[0004] However, during the video analysis process, computing resources are limited. If all videos are segmented using the same preset frame interval, the analysis accuracy of some videos may be too high and consume a lot of computing resources, while other videos may have too low analysis accuracy due to insufficient resources. This unbalanced resource allocation will reduce the overall analysis efficiency. In other words, the above method cannot achieve a balance between video analysis accuracy and computing resource usage. Summary of the invention

[0005] The main purpose of this application is to provide a video retrieval method, device, equipment and product, aiming to achieve a balance between video analysis accuracy and computing resource usage.

[0006] To achieve the above objectives, the present application proposes a video retrieval method, which comprises:

[0007] Extracting video frames from the video to be retrieved at a fixed sampling frequency to obtain a first video frame sequence; determining a target sampling frequency according to content complexity and / or system resource usage of the first video frame sequence, and extracting video frames from the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence;

[0008] Feature extraction is performed on the video frames in the second video frame sequence, and similar videos corresponding to the video to be retrieved are retrieved from a video library based on the extracted features.

[0009] In one embodiment, determining the target sampling frequency according to the content complexity of the first video frame sequence includes:

[0010] Determining the complexity of the video frames per second in the first video frame sequence;

[0011] Determining a target sampling frequency per second according to the complexity of the video frames per second;

[0012] Alternatively, based on the complexity of the video frames per second, the number of seconds in the first video frame sequence in which the video frame complexity is greater than a preset complexity is determined; if the number of seconds is greater than or equal to the preset number of seconds, the sampling frequency is set as the target sampling frequency per second in the first video frame sequence.

[0013] In one embodiment, the step of determining the complexity of the video frames per second in the first video frame sequence comprises:

[0014] Performing edge detection processing on video frames within each second in the first video frame sequence to obtain edge pixel density of the video frame, and determining the complexity of the video frame according to the edge pixel density;

[0015] Alternatively, color detection processing is performed on video frames within each second in the first video frame sequence to obtain color information of the video frames, and the complexity of the video frames is determined according to the color information.

[0016] In one embodiment, determining the target sampling frequency according to the content complexity of the first video frame sequence and the system resource usage rate includes:

[0017] Determining a content complexity level according to the content complexity of the first video frame sequence, and determining a system resource state according to the system resource usage rate;

[0018] The target sampling frequency is determined according to the content complexity level and the system resource status, and the corresponding relationship among the content complexity level, the system resource status and the sampling frequency.

[0019] In one embodiment, the step of extracting features from the video frames in the second video frame sequence and retrieving similar videos corresponding to the video to be retrieved from the video library based on the extracted features includes:

[0020] Extracting video features of each video frame in the second video frame sequence based on a video feature extraction model;

[0021] Determine, according to the video features of each of the video frames, a video key frame in the second video frame sequence, wherein the similarity between the video key frame and two adjacent video frames is greater than a preset similarity;

[0022] Retrieving each suspected similar video of the to-be-retrieved video from the video library based on the similarity between the video feature of the video key frame and the video feature of the video key frame of each video in the video library;

[0023] The repeatability of each of the suspected similar videos is respectively verified with the video to be retrieved, and a similar video corresponding to the video to be retrieved is determined according to the repeatability verification result.

[0024] In one embodiment, the step of retrieving each suspected similar video of the video to be retrieved from the video library based on the similarity between the video feature of the video key frame and the video feature of the video key frame of each video in the video library includes:

[0025] The video features of the video key frames are reduced in dimension by a principal component analysis method to obtain the video features of the video key frames after the dimension reduction;

[0026] Based on the similarity between the video features of the video key frames after dimensionality reduction and the video features of the video key frames of each video in the video library, each suspected similar video of the video to be retrieved is retrieved from the video library.

[0027] In one embodiment, before the step of extracting the video features of each video frame in the second video frame sequence based on the video feature extraction model, the step further includes:

[0028] Acquire video samples, wherein the video samples include similar video pairs and dissimilar video pairs;

[0029] Extracting video features of two similar videos in the similar video pair, and extracting video features of two dissimilar videos in the dissimilar video pair;

[0030] Determining a first similarity between the video features of the two similar videos, and determining a second similarity between the video features of the two dissimilar videos;

[0031] A noise contrast estimation loss function is calculated based on the first similarity and the second similarity, and the noise contrast estimation loss function is used to optimize the EfficientNetV2M network to obtain the video feature extraction model.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a video retrieval device, the video retrieval device comprising:

[0033] A first sampling module, used for extracting video frames of the video to be retrieved according to a fixed sampling frequency to obtain a first video frame sequence;

[0034] a second sampling module, configured to determine a target sampling frequency according to content complexity and / or system resource usage of the first video frame sequence, and extract video frames from the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence;

[0035] The retrieval module is used to extract features from the video frames in the second video frame sequence, and retrieve similar videos corresponding to the video to be retrieved from a video library based on the extracted features.

[0036] In addition, to achieve the above objectives, the present application also proposes a video retrieval device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video retrieval method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the video retrieval method described above are implemented.

[0038] Compared with the related art that uses the same preset frame interval to segment all videos, which makes it impossible to achieve a balance between video analysis accuracy and computing resource usage, the present application first extracts video frames for the video to be retrieved at a fixed sampling frequency to obtain a first video frame sequence, dynamically determines the target sampling frequency according to the content complexity of the first video frame sequence and / or the system resource usage rate, and performs secondary video frame extraction on the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence. Finally, feature extraction is performed on the video frames in the second video frame sequence, and similar videos corresponding to the video to be retrieved are retrieved from the video library based on the extracted features. Since the sampling frequency can be dynamically determined according to the complexity of the video content and / or the system resource usage rate, the sampling frequency can be dynamically adjusted and changed according to changes in the complexity of the video content and / or the system resource usage rate, thereby achieving an optimized balance between analysis accuracy and computing resource usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1A schematic diagram of a flow chart provided for the first embodiment of the video retrieval method of the present application;

[0042] Figure 2 A schematic diagram of a flow chart provided for the second embodiment of the video retrieval method of the present application;

[0043] Figure 3 Another flowchart diagram provided for the second embodiment of the video retrieval method of the present application;

[0044] Figure 4 A schematic diagram of a process flow provided for Embodiment 3 of the video retrieval method of the present application;

[0045] Figure 5 A flowchart of the fourth embodiment of the video retrieval method of the present application is provided;

[0046] Figure 6 This is a schematic diagram of the module structure of the video retrieval device according to an embodiment of the present application;

[0047] Figure 7 It is a structural diagram of the video retrieval device in the embodiment of the present application.

[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] In the related art, the video retrieval device can first extract the frame sequence of the video to be retrieved to obtain the video frame sequence, then extract the image features of the video frames in the video frame sequence to obtain the image feature sequence, and then extract the fixed-length target video features from the image feature sequence to obtain the representation of the video to be retrieved on the video element, and finally retrieve similar videos from the video library based on the target video features, that is, based on the representation of the video to be retrieved on the video element. In this way, it is possible to judge whether the video is similar based on the video element without performing a timing logic judgment, so that whether it is a general video with normal timing, a mixed video with chaotic timing, or a lengthened video, it can effectively perform video retrieval, and finally improve the effectiveness of video retrieval. However, in the video analysis process, computing resources are limited. If all videos are segmented using the same preset frame interval, the analysis accuracy of some videos may be too high and consume a lot of computing resources, while other videos may have too low analysis accuracy due to insufficient resources. This unbalanced resource allocation will reduce the overall analysis efficiency, that is, the above method cannot achieve a balance between the analysis accuracy of the video and the use of computing resources.

[0052] In response to the above problems, the present application proposes a video retrieval method, and the main technical solutions include: extracting video frames of the video to be retrieved at a fixed sampling frequency to obtain a first video frame sequence; determining the target sampling frequency according to the content complexity and / or system resource utilization of the first video frame sequence, and extracting video frames of the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence; extracting features of the video frames in the second video frame sequence, and retrieving similar videos corresponding to the video to be retrieved from the video library based on the extracted features.

[0053] Compared with the related art that uses the same preset frame interval to segment all videos, which makes it impossible to achieve a balance between video analysis accuracy and computing resource usage, the present application can dynamically determine the sampling frequency according to the complexity of the video content and / or the system resource usage rate, so that the sampling frequency can be dynamically adjusted and changed according to the changes in the complexity of the video content and / or the system resource usage rate, thereby achieving an optimized balance between analysis accuracy and computing resource usage.

[0054] In addition, in order to speed up the extraction of video features and ensure the quality of video features, the present application calculates a noise contrast estimation loss function based on a first similarity between video features of two similar videos and a second similarity between video features of two dissimilar videos, and uses the noise contrast estimation loss function to optimize the EfficientNetV2M network to obtain a video feature extraction model. The video feature extraction model of the EfficientNetV2M network is used to extract features, which can maintain a high accuracy while reducing the demand for computing resources, which is particularly important for large-scale video processing.

[0055] In addition, in order to reduce the amount of data processing, the present application reduces the dimension of the video features of the video key frames through the principal component analysis method to obtain the reduced-dimensional video features, thereby streamlining the video feature dimensions, reducing the amount of data, reducing the computational complexity, and accelerating the video retrieval process. At the same time, it removes features that are not highly distinguishable from the video content, making the matching more accurate.

[0056] In addition, the overall solution of this application combines multiple efficient technologies and strategies, from video processing to feature extraction, key frame recognition, feature vector dimension reduction, retrieval, and final duplication determination, forming a highly optimized processing flow. This end-to-end optimization not only improves the processing speed, but also ensures the accuracy of the deduplication effect, which has obvious advantages over traditional single technology applications or simple comparison methods.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a video retrieval device, etc. The following takes the video retrieval device as an example to illustrate this embodiment and the following embodiments.

[0058] It should be noted that the video retrieval method of this application can be applied to scenarios such as video infringement judgment, content monitoring, and similar video recommendation. For example, in the video infringement judgment method, by comparing the video content, pirated or infringing content can be quickly identified to protect the rights and interests of the original creator. In terms of content monitoring, such as news media and social platforms, video retrieval technology can be used to monitor and filter out illegal and bad content to maintain the health of platform content.

[0059] It should be noted that after retrieving similar videos, the ultimate goal of this application is to automatically identify and eliminate duplicate or highly similar video clips in the database to save video storage space and improve video retrieval efficiency.

[0060] Based on this, the embodiment of the present application provides a video retrieval method, referring to Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the video retrieval method of the present application.

[0061] In this embodiment, the video retrieval method includes steps S10 to S30:

[0062] Step S10, extracting video frames from the video to be retrieved according to a fixed sampling frequency to obtain a first video frame sequence;

[0063] The embodiment of the present application is implemented in the scenario of performing similar video retrieval on a video, for example, performing video retrieval on a remixed video uploaded by a user to determine whether the remixed video uploaded by the user infringes upon copyright, or performing video retrieval on a video that the user has watched to recommend similar videos to the user, etc. In the embodiment of the present application, after obtaining the video to be retrieved, the video retrieval device will first extract video frames from the video to be retrieved to obtain a video frame sequence or multiple different video frame sequences, and the extracted video frame sequences are collectively referred to as first video frame sequences.

[0064] Sampling frequency, also known as frame rate, is an important concept in video processing. It indicates the number of times the video screen is refreshed per second, and the unit is usually frames per second or hertz. The higher the sampling frequency, the more pictures are displayed per second, the smoother the video looks, and the better the dynamic performance is. The lower the sampling frequency, the larger the interval between pictures, and the video may appear stuck or jumpy, affecting the viewing experience. The fixed sampling frequency can be determined according to the actual situation. It can be set to collect video frames for each second of video at a fixed sampling frequency. For example, 5 video frames are collected per second, and these 5 video frames form a sub-video frame sequence in seconds. Among them, the first video frame sequence here includes the sub-video frame sequence corresponding to each second. Since a video exists for multiple seconds, each second can include multiple video frames. Therefore, a video has multiple sub-video frame sequences, and these sub-video frame sequences are combined to obtain the first video frame sequence.

[0065] In addition to extracting video frames from the video to be retrieved at a fixed sampling frequency to obtain a first video frame sequence, other video frame extraction rules can also be used to extract video frames. For example, video frames can be extracted from the video at fixed time intervals (such as every second, every two seconds, etc.) to obtain a first video frame sequence.

[0066] Step S20, determining a target sampling frequency according to the content complexity and / or system resource usage of the first video frame sequence, and performing video frame extraction on the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence;

[0067] In this embodiment, the target sampling frequency may be determined according to the content complexity of the first video frame sequence, or according to the system resource usage, or according to the content complexity of the first video frame sequence and the system resource usage.

[0068] The target sampling frequency can be dynamically changed according to the content complexity of the first video frame sequence and / or the system resource utilization rate, that is, when the content complexity of the first video frame sequence changes and / or the system resource utilization rate changes, the corresponding target sampling frequency also changes accordingly. The target sampling frequency can be set to be less than or equal to the fixed sampling frequency.

[0069] The content complexity of the first video frame sequence can be represented by the content complexity of the video frames per second in the first video frame sequence, or the content complexity of the entire video to be retrieved can be used to represent the content complexity of the first video frame sequence. The specific method is referred to the second embodiment and will not be described here.

[0070] For each video frame, the content complexity of the video frame may also be determined by using edge pixel density of the video frame, color information of the video frame, etc.

[0071] Step S30: extracting features from the video frames in the second video frame sequence, and retrieving similar videos corresponding to the video to be retrieved from a video library based on the extracted features.

[0072] The present embodiment provides a video retrieval method. Compared with the related art that uses the same preset frame interval to segment all videos, which makes it impossible to achieve a balance between video analysis accuracy and computing resource usage, the present application first extracts video frames for the video to be retrieved at a fixed sampling frequency to obtain a first video frame sequence, dynamically determines the target sampling frequency according to the content complexity of the first video frame sequence and / or the system resource usage rate, and performs secondary video frame extraction on the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence. Finally, feature extraction is performed on the video frames in the second video frame sequence, and similar videos corresponding to the video to be retrieved are retrieved from the video library based on the extracted features. Since the sampling frequency can be dynamically determined according to the video content complexity and / or the system resource usage rate, the sampling frequency can be dynamically adjusted and changed according to changes in the video content complexity and / or the system resource usage rate, thereby achieving an optimized balance between analysis accuracy and computing resource usage.

[0073] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 In step S20, determining the target sampling frequency according to the content complexity of the first video frame sequence includes steps S21 to S23:

[0074] Step S21, determining the complexity of the video frames per second in the first video frame sequence;

[0075] Step S22, determining a target sampling frequency per second according to the complexity of the video frames per second;

[0076] The strategy can be determined based on the sampling frequency at the second level, that is, the content complexity analysis is performed based on the video frames per second, and then the sampling frequency is adjusted at the second level. The sampling frequency corresponding to each second changes dynamically according to the content complexity. The target sampling frequency corresponding to each second can be the same or different. Subsequently, during the second sampling, different or the same sampling frequency is used to sample each second in the first video frame sequence to obtain a second video frame sequence.

[0077] It should be noted that the above unit of one second is only an example, and other durations, such as 2 seconds, may also be used.

[0078] The complexity of the video frames per second mentioned above can be the complexity of all the video frames per second, or the complexity of the video frames extracted at intervals per second. For example, assuming that 1 second includes 5 video frames, the content complexity corresponding to the 5 video frames in this second can be determined, and the target sampling frequency for this second can be determined based on the content complexity corresponding to the 5 video frames in this second. Alternatively, the 1st, 3rd and 5th video frames in this second are extracted, and the content complexity corresponding to the 1st, 3rd and 5th video frames is determined, and then the target sampling frequency for this second is determined based on the content complexity corresponding to the 1st, 3rd and 5th video frames. This process will produce content complexity results of video frames at the second level.

[0079] Alternatively, in step S23, based on the complexity of the video frames per second, the number of seconds in the first video frame sequence in which the complexity of the video frames is greater than a preset complexity is determined; if the number of seconds is greater than or equal to the preset number of seconds, the sampling frequency is set as the target sampling frequency per second in the first video frame sequence.

[0080] In addition to adopting the above-mentioned sampling frequency determination strategy based on the second level, a sampling frequency determination strategy based on the complete video can also be adopted. When performing complexity analysis on the entire video, the content complexity corresponding to each second can be determined based on the complexity of the video frames per second, and then the number of seconds in the first video frame sequence where the complexity of the video frames is greater than the preset complexity can be determined based on the content complexity corresponding to each second. Among them, the calculation method of the content complexity corresponding to each second can be: determining the content complexity corresponding to each second based on the ratio of the number of video frames with a complexity greater than the preset complexity per second to the total number of video frames per second, the larger the ratio, the higher the content complexity corresponding to each second; or determining the content complexity corresponding to each second based on the number of video frames with a complexity greater than the preset complexity per second, the larger the number, the higher the content complexity corresponding to each second.

[0081] The above-mentioned preset number of seconds may be half of the total number of seconds of the video to be retrieved.

[0082] If the number of seconds is greater than or equal to the preset number of seconds, it is determined that the target sampling frequency per second in the first video frame sequence is consistent, and subsequently during the second sampling, the first video frame sequence is sampled at the same sampling frequency per second to obtain a second video frame sequence.

[0083] In this embodiment, the sampling frequency of the second sampling may be determined by adopting a sampling frequency determination strategy based on the second level or a sampling frequency determination strategy based on the complete video, so that the second sampling is more flexible.

[0084] Further, in a feasible implementation manner, step S21 includes step S211 or step S212:

[0085] Step S211, performing edge detection processing on the video frames in each second in the first video frame sequence to obtain edge pixel density of the video frames, and determining the complexity of the video frames according to the edge pixel density;

[0086] The above-mentioned edge pixel density is an indicator used to describe the edge density in a video frame. In the field of image processing, edge pixel density usually refers to the number of edge points per unit area, which is obtained by processing the video frame with an edge detection algorithm, counting the number of edge points and normalizing them. First, edge detection is performed on the video frame. Commonly used edge detection algorithms include Sobel, Canny, etc. Then, the number of edge points in the edge detection result is counted. Finally, the number of edge points is divided by the total area of ​​the image (or the total number of pixels), and normalized to obtain the edge density value, which is the edge pixel density. The larger the edge density value of the video frame, the more complex the video frame is. If the edge density value of the video frame is smaller, the simpler the video frame is.

[0087] In a feasible implementation, a threshold value for edge pixel density may be set. When the edge pixel density of a video frame is greater than or equal to the threshold value, the complexity of the video frame is determined to be high. When the edge pixel density of a video frame is less than the threshold value, the complexity of the video frame is determined to be low.

[0088] Alternatively, in step S212, color detection processing is performed on video frames within each second in the first video frame sequence to obtain color information of the video frames, and the complexity of the video frames is determined according to the color information.

[0089] The color information of the video frame mentioned above includes but is not limited to the color richness, color contrast, interaction between color and texture, shape, etc. of the video frame. These color information can be used to evaluate the content complexity of the video frame. The following will introduce the impact of various color information on the content complexity of the video frame:

[0090] Regarding the evaluation of the content complexity of the video frame due to color richness: the colors in the video frame can be simplified to a limited number of primary colors through color quantization techniques such as K-means clustering and color histogram segmentation. The more primary colors there are, the higher the color richness of the video frame is, which may increase the content complexity of the video frame. Analyze the spatial distribution of each primary color in the video frame. If a certain color distribution is very concentrated, it may mean that there are fewer objects in the video frame and the content complexity of the video frame is relatively low; if the color distribution is scattered and messy, it may mean that the video frame contains more objects and the content complexity of the video frame is high.

[0091] Regarding the evaluation of the content complexity of video frames based on color contrast: Color contrast is also an important factor affecting image complexity. High-contrast video frames usually have more distinct color differences and may contain more details and layers, thereby increasing the complexity of the video frames. The contrast level of the video frames can be evaluated by calculating the contrast between different colors in the video frames, such as the difference based on the RGB color space.

[0092] Evaluation of the interaction between color, texture and shape on the content complexity of video frames: The color in a video frame does not exist in isolation, but is closely related to the texture, shape and other features of the video frame. For example, texture features can be regarded as a combination of multiple color primitives. The stronger the regularity of the primitive combination, the simpler the texture features; conversely, the more complex. Shape features are also affected by color. A large number of colors may mean that the video frame contains a large number of objects, thereby increasing the complexity of shape feature extraction and matching.

[0093] In another embodiment, the complexity of the video frame may be determined according to the density of edge pixels and the complexity of the video frame may be determined according to color information, weights may be assigned to the complexity of the video frame determined in the two ways, and finally the complexity of the video frame may be determined comprehensively.

[0094] In this embodiment, by determining the complexity of the video frame according to the density of the edge pixels, or determining the complexity of the video frame according to the color information of the video frame, the complexity of the video frame is detected, thereby improving the detection accuracy of the complexity of the video frame.

[0095] In another embodiment, referring to Figure 3 In step S20, determining the target sampling frequency according to the content complexity of the first video frame sequence and the system resource usage rate includes steps S24 to S25:

[0096] Step S24, determining a content complexity level according to the content complexity of the first video frame sequence, and determining a system resource state according to the system resource usage rate;

[0097] The content complexity level may include high content complexity and low content complexity, and may be further divided into more complexity levels according to actual conditions.

[0098] The system resource status may include sufficient system resources and insufficient system resources.

[0099] Step S25, determining the target sampling frequency according to the content complexity level and the system resource status, and the corresponding relationship among the content complexity level, the system resource status and the sampling frequency.

[0100] For example, if the content complexity is high and the system resources are sufficient, a higher target sampling frequency is used, for example, 5 frames per second are sampled, where the sampling frequency is less than or equal to the fixed sampling frequency.

[0101] If the content complexity is low and the system resources are sufficient, a lower target sampling frequency is used, for example, sampling 3 frames per second to save resources.

[0102] If the content complexity is high and the system resources are insufficient, a target sampling frequency that is as high as possible can be adopted as long as the system can support it. For example, the target sampling frequency can be determined based on the ratio of available resources of the system.

[0103] If the content complexity is low and the system resources are insufficient, a lower target sampling frequency is used, for example, 2 frames per second are sampled.

[0104] In this embodiment, the sampling frequency can be dynamically determined according to the complexity of the video content and / or the system resource utilization rate, so that the sampling frequency can be dynamically adjusted and changed according to the changes in the complexity of the video content and the system resource utilization rate, thereby achieving an optimized balance between analysis accuracy and computing resource utilization.

[0105] In another embodiment, determining the target sampling frequency according to the system resource usage rate in step S20 includes: determining whether the system resources are sufficient according to the system resource usage rate, and when the system resources are sufficient, determining the first preset sampling frequency as the target sampling frequency, and when the system resources are insufficient, determining the second preset sampling frequency as the target sampling frequency. The first preset sampling frequency is greater than the second preset sampling frequency, and the first preset sampling frequency and the second preset sampling frequency are both less than or equal to the fixed sampling frequency. Thus, the sampling frequency can be dynamically determined and adjusted according to the system resource usage rate.

[0106] Based on the above embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 4 , step S30 includes steps S31 to S34:

[0107] Step S31, extracting video features of each video frame in the second video frame sequence based on a video feature extraction model;

[0108] The video feature extraction model here can be: a convolutional neural network model, which automatically learns and extracts high-level features from raw video frames through operations such as convolutional layers, pooling layers, and fully connected layers. It can also be a recurrent neural network and its variants such as LSTM and GRU: mainly used to process sequence data, such as time series analysis, natural language processing, etc., to extract features by capturing long-term dependencies in the sequence.

[0109] The video feature extraction model of the present application is preferably an EfficientNetV2M network model. The EfficientNetV2M model is used as a video feature extraction tool to perform feature extraction on each video frame in the second video frame sequence. This process converts each video frame in the second video frame sequence into a high-dimensional feature vector with a size of [N, D], thereby effectively capturing the core visual information of the video frame.

[0110] Among them, the EfficientNetV2M network model uses a compound scaling strategy to simultaneously adjust the depth, width, and resolution of the model to find the best model architecture, thereby improving performance while maintaining model efficiency. Its feature extraction process includes:

[0111] (1) Input processing: The video frames in the second video frame sequence are taken as input and necessary preprocessing is performed, such as resizing and normalization, to meet the input requirements of the EfficientNetV2M network model.

[0112] (2) Convolutional layer processing: The video frames in the second video frame sequence first pass through a series of convolutional layers, which are the core of feature extraction in EfficientNetV2M. EfficientNetV2M uses improved MBConv modules that combine depthwise separable convolution and pointwise convolution to extract features in an efficient manner. In EfficientNetV2M, the Fused-MBConv module is also introduced, which fuses the 1x1 convolution and Depthwise convolution in the MBConv module into a 3x3 convolution operation to further improve computational efficiency.

[0113] (3) Compound scaling strategy: EfficientNetV2M uses a compound scaling strategy to simultaneously adjust the depth, width, and resolution of the model. This means that during the feature extraction process, as the number of network layers increases, the number of channels (width) and resolution of the feature map will also be adjusted accordingly to better capture the information in the image.

[0114] (4) Feature map transfer: During the feature extraction process, the output (feature map) of each layer will be used as the input of the next layer. These feature maps contain information of different levels and scales of the video frame.

[0115] (5) Activation function and regularization: EfficientNetV2M also uses activation functions such as ReLU or SILU to increase the nonlinearity of the network, as well as regularization techniques to prevent overfitting and improve the generalization ability of the model.

[0116] Through its deep convolutional neural network structure and compound scaling strategy, EfficientNetV2M can efficiently extract features of video frames, which are crucial for subsequent tasks such as video retrieval.

[0117] Step S32, determining a video key frame in the second video frame sequence according to the video features of each of the video frames, wherein the similarity between the video key frame and its two adjacent video frames is greater than a preset similarity;

[0118] The purpose of selecting video key frames in a video is to extract the most representative frames from the second video frame sequence for use in video retrieval applications.

[0119] The principles for selecting video key frames include: video key frames should be able to represent the main content of the video shot or scene, avoid selecting video key frames with similar or repeated content, and select an appropriate number of video key frames based on the complexity and needs of the video content.

[0120] The method of selecting the video key frames in the second video frame sequence includes: analyzing the image features of the video frames such as color, texture, shape, etc., and selecting frames with significant features or changes as video key frames. Alternatively, by analyzing the motion information between video frames such as optical flow, motion vector, etc., the video key frames are selected when the amount of motion reaches a local minimum or maximum. Alternatively, the features of the video frames are clustered and the frames corresponding to each cluster center are used as video key frames. This method can automatically determine the number and position of video key frames. Alternatively, the inter-frame difference method is used to determine the video key frames in the second video frame sequence. This method is simple and effective.

[0121] The present application preferably uses the frame difference method to determine the video key frame in the second video frame sequence. Specifically, it includes:

[0122] (1) Calculate the difference between two adjacent frames in the second video frame sequence. The difference can be calculated by the sum of the absolute values ​​of the pixel differences, or by other more complex measurement methods, such as structural similarity or pixel difference method. In this case, the structural information of the video frames is taken into account, the similarity between the two video frames is calculated, and the difference value is obtained by the inverse of the similarity or after a certain transformation. Alternatively, the pixel difference method is used to subtract each pixel point of the two adjacent frames, and then the absolute value is taken, and finally the absolute values ​​of all the pixels are summed to obtain the difference value between the frames.

[0123] (2) Determine a threshold based on experience or experiment to determine whether the scene has changed. The setting of the threshold directly affects the extraction effect of the video key frame.

[0124] (3) If the inter-frame difference value is greater than the threshold, it is considered that the scene has changed and the current frame is marked as a candidate video key frame.

[0125] (4) The final video key frame is further selected from the candidate video key frames. This step can be performed using a variety of methods, such as fixed interval sampling, adaptive threshold, clustering algorithm, etc. Among them, one frame is selected as a video key frame every fixed number of frames. Alternatively, the threshold is dynamically adjusted according to the size of the difference value between frames, so as to extract more video key frames in places where the scene changes greatly. Alternatively, the candidate key frames are clustered and the center of each cluster is selected as the video key frame.

[0126] (5) Post-processing the extracted video key frames, such as removing redundant frames and adjusting the interval between frames, to improve the quality and representativeness of the video key frames.

[0127] In addition to the above method, the feature difference between two adjacent frames in the second video frame sequence can also be calculated, and the feature vectors can be compared using a measurement method such as Euclidean distance or cosine similarity to determine the similarity between frames. When the feature difference exceeds a preset threshold, the frame is selected as a video key frame. Specifically, for each video frame in the second video frame sequence, the similarity between the video frame and the two adjacent video frames on the left and right of the video frame is calculated respectively. If the similarity between the video frame and the two adjacent video frames on the left and right of the video frame is greater than the preset similarity, the frame is determined to be a video key frame.

[0128] Step S33, based on the similarity between the video features of the video key frames and the video features of the video key frames of each video in the video library, retrieving each suspected similar video of the video to be retrieved from the video library;

[0129] The system will perform a comprehensive search in the video feature library and use an efficient similarity search algorithm to identify the video features that are closest to the video features of the video key frames. This step ensures that potential duplicate videos can be quickly located even in massive data, and the video corresponding to the closest video features can be identified as a suspected similar video.

[0130] It should be noted that the video library pre-stores preset video features of the video key frames of each video, that is, it pre-stores multiple videos, preset video key frames corresponding to each video, and preset video features corresponding to each video key frame. The process of retrieving suspected similar videos is actually to compare the similarity between the video features of the video key frames. The preset video features whose similarity with the video features of the video key frames of the video to be retrieved in the video library is less than the set similarity can be obtained, and the specific preset video key frames are located according to the preset video features to further determine the video to which the preset video key frames belong, and finally determine the suspected similar videos of the video to be retrieved.

[0131] Step S34, performing repeatability verification on each of the suspected similar videos and the video to be retrieved, and determining similar videos corresponding to the video to be retrieved according to the repeatability verification results.

[0132] For the retrieved suspected similar videos, the similarity between the video features of each video key frame of each suspected similar video and the video features of each video key frame of the video to be retrieved is calculated by means of matrix operations, and the similarity matrix of the video features of the video key frames of each video pair (the video pair consisting of the video to be retrieved and the suspected similar video) is constructed. On this basis, a strict threshold is set, and the proportion of similarities exceeding the threshold in the statistical matrix. If this ratio reaches the preset standard, it is determined that there is a significant repetition between the video to be retrieved and the corresponding suspected similar video. On the contrary, if the proportion of high similarity does not meet the standard, the video to be retrieved is judged to be non-repetitive content.

[0133] In this embodiment, the video feature extraction model of the EfficientNetV2M network is used to extract features, which can maintain high accuracy while reducing computing resource requirements, which is particularly important for large-scale video processing.

[0134] Further, step S33 includes step S331 and step S332:

[0135] Step S331, reducing the dimension of the video features of the video key frames by principal component analysis to obtain the video features of the video key frames after the dimension reduction;

[0136] In order to further improve processing efficiency and reduce computational complexity, principal component analysis technology is used to perform dimensionality reduction operations on the video features of the extracted video key frames to remove redundant information and retain the feature dimensions that best represent the video content.

[0137] Principal component analysis is a commonly used data dimensionality reduction technology that can map multidimensional data to a lower dimensional space through linear transformation while retaining the main feature information of the original data as much as possible. When processing the video features of video key frames, principal component analysis can reduce the number of features and simplify the subsequent processing process. Specifically, the specific steps of principal component analysis include:

[0138] First, the feature vectors of all key frames are organized into a matrix, where each row represents a feature vector of a key frame and each column represents a feature dimension. Since the dimensions and ranges of different features may be different, they need to be standardized, that is, each feature is scaled to the same scale. This is usually achieved by subtracting the mean and then dividing by the standard deviation. Ensuring that the mean of the data is 0 is the premise for the principal component analysis method to calculate the covariance matrix or correlation matrix. Next, the covariance matrix of the standardized data is calculated, where the covariance matrix measures the correlation between different features, the eigenvalues ​​of the covariance matrix represent the importance of each principal component, and the eigenvector represents the direction of the new coordinate axis, that is, the direction of the principal component. Next, the first k principal components are selected according to the size of the eigenvalue. Finally, the video features of the video key frame are projected onto these k principal components, that is, the projection of the video features of the video key frame in the direction of the k eigenvectors is calculated to obtain the video features after dimensionality reduction.

[0139] Step S332: based on the similarity between the video features of the video key frames after dimensionality reduction and the video features of the video key frames of each video in the video library, retrieve each suspected similar video of the video to be retrieved from the video library.

[0140] The system will perform a comprehensive search in the video feature library and use an efficient similarity search algorithm to identify the video features that are closest to the video features after dimensionality reduction. This step ensures that potential duplicate videos can be quickly located even in massive data, and the video corresponding to the closest video features can be identified as a suspected similar video.

[0141] In this embodiment, in order to reduce the amount of data processing, the video features of the video key frames are reduced in dimension through principal component analysis to obtain the reduced-dimensional video features, thereby streamlining the video feature dimensions, reducing the amount of data, reducing the computational complexity, and accelerating the video retrieval process. At the same time, features that are not highly distinguishable from the video content are removed, making the matching more accurate.

[0142] Based on the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the above embodiment can refer to the above introduction, and will not be repeated in the following. Figure 5 , before step S31, it also includes steps S110 to S140:

[0143] Step S110, obtaining video samples, wherein the video samples include similar video pairs and dissimilar video pairs;

[0144] Similar video pairs are the positive sample pairs mentioned below, and dissimilar video pairs are the negative sample pairs mentioned below.

[0145] Step S120, extracting video features of two similar videos in the similar video pair, and extracting video features of two dissimilar videos in the dissimilar video pair;

[0146] Step S130, determining a first similarity between the video features of the two similar videos, and determining a second similarity between the video features of the two dissimilar videos;

[0147] Step S140, calculating a noise contrast estimation loss function based on the first similarity and the second similarity, and optimizing the EfficientNetV2M network using the noise contrast estimation loss function to obtain the video feature extraction model.

[0148] In order to make the EfficientNet model more effectively serve the video retrieval task, this application adopts the noise contrast estimation loss function to optimize its performance. The EfficientNetV2M network adopts the strategy of the noise contrast estimation loss function to optimize its feature representation ability during the training process. Specifically, if the EfficientNetV2M network is used in a self-supervised learning task and needs to extract effective feature representation through contrast learning, it may use the noise contrast estimation loss function as the loss function during the training process. In this way, the EfficientNetV2M network can make full use of the advantages of the noise contrast estimation loss function in distinguishing positive and negative samples, and further improve its feature representation ability and model performance.

[0149] The noise contrast estimation loss function mentioned above is a loss function commonly used in contrastive learning. Its core idea is to learn meaningful feature representations by maximizing the similarity of similar sample pairs and minimizing the similarity of dissimilar sample pairs. Among them, the noise contrast estimation loss function usually contains a temperature coefficient, which is used to adjust the distribution of similarity scores, thereby affecting the model's discrimination of samples. The larger the temperature coefficient, the lower the model's discrimination of negative samples, and more negative samples can be included; conversely, more attention is paid to particularly difficult negative samples. By comparing the similarity of positive and negative samples, the noise contrast estimation loss function can prompt the model to learn more robust and meaningful feature representations. Through training, the EfficientNet network can extract highly discriminative feature representations from video key frames for subsequent similarity comparisons. The following will introduce the training process of the video feature extraction model in detail:

[0150] First, collect a large number of videos of different categories and styles and extract video key frames from them as training samples. Make sure that the dataset contains a certain ratio of similar video pairs to dissimilar video pairs to cover a wide range of video content differences. Label the image data to determine which frame pairs are positive sample pairs (from the same video or very similar videos) and negative sample pairs (from different videos).

[0151] Next, the noise contrastive estimation loss function is used as the loss function, which encourages the model to bring positive sample pairs closer and push negative sample pairs away in contrastive learning.

[0152]

[0153] Among them, z i and is the feature vector of the positive sample pair, sim() measures the similarity of two feature vectors (such as cosine similarity), and T is the temperature coefficient.

[0154] Next, train the model using the Adam or RMSprop optimizer, set the initial learning rate to a low level (e.g., 1e-4 to 1e-3), and adopt a learning rate decay strategy such as CosineAnnealingLR based on the training progress. Use a batch construction method with multiple frame pairs, each batch contains multiple positive sample pairs (different frames of the same video) and negative sample pairs (frames of different videos), and increase diversity through data augmentation. Plan to train for multiple epochs, evaluate the model performance on the validation set after each epoch, and monitor the changes in InfoNCE loss and possible other indicators (such as accuracy). Set an early stopping condition to terminate training when the performance on the validation set no longer improves, and finally select the model weights that perform best on the validation set.

[0155] Through the above scheme, a video feature extraction model optimized for video content recognition can be effectively trained, and its performance on the video feature extraction task can be improved by using the noise contrast estimation loss function.

[0156] Based on the same inventive concept, the present application also provides a video retrieval device, please refer to Figure 6 , the video retrieval device comprises:

[0157] A first sampling module 10 is used to extract video frames of the video to be retrieved according to a fixed sampling frequency to obtain a first video frame sequence;

[0158] A second sampling module 20 is used to determine a target sampling frequency according to the content complexity of the first video frame sequence and / or the system resource usage rate, and perform video frame extraction on the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence;

[0159] The retrieval module 30 is used to extract features from the video frames in the second video frame sequence, and retrieve similar videos corresponding to the video to be retrieved from a video library based on the extracted features.

[0160] Optionally, the second sampling module 20 is further used for:

[0161] Determining the complexity of the video frames per second in the first video frame sequence;

[0162] Determining a target sampling frequency per second according to the complexity of the video frames per second;

[0163] Alternatively, based on the complexity of the video frames per second, the number of seconds in the first video frame sequence in which the video frame complexity is greater than a preset complexity is determined; if the number of seconds is greater than or equal to the preset number of seconds, the sampling frequency is set as the target sampling frequency per second in the first video frame sequence.

[0164] Optionally, the second sampling module 20 is further used for:

[0165] Performing edge detection processing on video frames within each second in the first video frame sequence to obtain edge pixel density of the video frame, and determining the complexity of the video frame according to the edge pixel density;

[0166] Alternatively, color detection processing is performed on video frames within each second in the first video frame sequence to obtain color information of the video frames, and the complexity of the video frames is determined according to the color information.

[0167] Optionally, the second sampling module 20 is further used for:

[0168] Determining a content complexity level according to the content complexity of the first video frame sequence, and determining a system resource state according to the system resource usage rate;

[0169] The target sampling frequency is determined according to the content complexity level and the system resource status, and the corresponding relationship among the content complexity level, the system resource status and the sampling frequency.

[0170] Optionally, the retrieval module 30 is further used for:

[0171] Extracting video features of each video frame in the second video frame sequence based on a video feature extraction model;

[0172] Determine, according to the video features of each of the video frames, a video key frame in the second video frame sequence, wherein the similarity between the video key frame and two adjacent video frames is greater than a preset similarity;

[0173] Retrieving each suspected similar video of the to-be-retrieved video from the video library based on the similarity between the video feature of the video key frame and the video feature of the video key frame of each video in the video library;

[0174] The repeatability of each of the suspected similar videos is respectively verified with the video to be retrieved, and a similar video corresponding to the video to be retrieved is determined according to the repeatability verification result.

[0175] Optionally, the retrieval module 30 is further used for:

[0176] The video features of the video key frames are reduced in dimension by a principal component analysis method to obtain the video features of the video key frames after the dimension reduction;

[0177] Based on the similarity between the video features of the video key frames after dimensionality reduction and the video features of the video key frames of each video in the video library, each suspected similar video of the video to be retrieved is retrieved from the video library.

[0178] Optionally, the video retrieval device further comprises a model optimization module, which is connected 30 to the retrieval module and is used to:

[0179] Acquire video samples, wherein the video samples include similar video pairs and dissimilar video pairs;

[0180] Extracting video features of two similar videos in the similar video pair, and extracting video features of two dissimilar videos in the dissimilar video pair;

[0181] Determining a first similarity between the video features of the two similar videos, and determining a second similarity between the video features of the two dissimilar videos;

[0182] A noise contrast estimation loss function is calculated based on the first similarity and the second similarity, and the noise contrast estimation loss function is used to optimize the EfficientNetV2M network to obtain the video feature extraction model.

[0183] The video retrieval device provided by the present application adopts the video retrieval method in the above embodiment, which can solve the technical problem that the analysis accuracy of the video and the use of computing resources cannot be balanced. Compared with the prior art, the beneficial effects of the video retrieval device provided by the present application are the same as the beneficial effects of the video retrieval method provided by the above embodiment, and other technical features in the video retrieval device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0184] The present application provides a video retrieval device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video retrieval method in the above-mentioned embodiment one.

[0185] Reference below Figure 7 , which shows a schematic diagram of the structure of a video retrieval device suitable for implementing the embodiment of the present application. The video retrieval device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The video retrieval device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0186] like Figure 7 As shown, the video retrieval device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the video retrieval device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the video retrieval device to communicate wirelessly or wired with other devices to exchange data. Although the video retrieval device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or less systems can be implemented or have alternatively.

[0187] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0188] The video retrieval device provided by the present application adopts the video retrieval method in the above embodiment, which can solve the technical problem that the analysis accuracy of the video and the use of computing resources cannot be balanced. Compared with the prior art, the beneficial effects of the video retrieval device provided by the present application are the same as the beneficial effects of the video retrieval method provided by the above embodiment, and other technical features in the video retrieval device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0189] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0190] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0191] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the video retrieval method in the above-mentioned embodiment.

[0192] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0193] The computer-readable storage medium may be included in the video retrieval device; or may exist independently without being assembled into the video retrieval device.

[0194] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the video retrieval device, the video retrieval device is enabled to: extract video frames of the video to be retrieved at a fixed sampling frequency to obtain a first video frame sequence; determine the target sampling frequency according to the content complexity and / or system resource utilization rate of the first video frame sequence, and extract video frames of the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence; extract features of the video frames in the second video frame sequence, and retrieve similar videos corresponding to the video to be retrieved from the video library based on the extracted features.

[0195] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0196] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0197] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0198] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video retrieval method, and can solve the technical problem that the analysis accuracy of the video and the use of computing resources cannot be balanced. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the video retrieval method provided in the above-mentioned embodiment, and will not be repeated here.

[0199] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video retrieval method when executed by a processor.

[0200] The computer program product provided by the present application can solve the technical problem of the inability to balance the analysis accuracy of the video and the use of computing resources. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the video retrieval method provided by the above embodiment, which will not be repeated here.

[0201] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A video retrieval method, characterized in that: The method comprises: Extracting video frames from the video to be retrieved according to a fixed sampling frequency to obtain a first video frame sequence; Determine a target sampling frequency according to content complexity and / or system resource usage of the first video frame sequence, and perform video frame extraction on the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence; Feature extraction is performed on the video frames in the second video frame sequence, and similar videos corresponding to the video to be retrieved are retrieved from a video library based on the extracted features.

2. The method according to claim 1, characterized in that Determining the target sampling frequency according to the content complexity of the first video frame sequence includes: Determining the complexity of video frames per second in the first video frame sequence; Determining a target sampling frequency per second according to the complexity of the video frames per second; Alternatively, based on the complexity of the video frames per second, the number of seconds in the first video frame sequence in which the video frame complexity is greater than a preset complexity is determined; if the number of seconds is greater than or equal to the preset number of seconds, the sampling frequency is set as the target sampling frequency per second in the first video frame sequence.

3. The method according to claim 2, characterized in that The step of determining the complexity of the video frames per second in the first video frame sequence comprises: Performing edge detection processing on video frames within each second in the first video frame sequence to obtain edge pixel density of the video frame, and determining the complexity of the video frame according to the edge pixel density; Alternatively, color detection processing is performed on video frames within each second in the first video frame sequence to obtain color information of the video frames, and the complexity of the video frames is determined according to the color information.

4. The method according to claim 1, characterized in that Determining the target sampling frequency according to the content complexity of the first video frame sequence and the system resource usage rate includes: Determining a content complexity level according to the content complexity of the first video frame sequence, and determining a system resource state according to the system resource usage rate; The target sampling frequency is determined according to the content complexity level and the system resource status, and the corresponding relationship among the content complexity level, the system resource status and the sampling frequency.

5. The method according to any one of claims 1 to 4, characterized in that: The step of extracting features from the video frames in the second video frame sequence and retrieving similar videos corresponding to the video to be retrieved from the video library based on the extracted features includes: Extracting video features of each video frame in the second video frame sequence based on a video feature extraction model; Determine, according to the video features of each of the video frames, a video key frame in the second video frame sequence, wherein the similarity between the video key frame and two adjacent video frames is greater than a preset similarity; Retrieving each suspected similar video of the to-be-retrieved video from the video library based on the similarity between the video feature of the video key frame and the video feature of the video key frame of each video in the video library; The repeatability of each of the suspected similar videos is respectively verified with the video to be retrieved, and a similar video corresponding to the video to be retrieved is determined according to the repeatability verification result.

6. The method according to claim 5, characterized in that The step of retrieving each suspected similar video of the to-be-retrieved video from the video library based on the similarity between the video feature of the video key frame and the video feature of the video key frame of each video in the video library comprises: The video features of the video key frames are reduced in dimension by a principal component analysis method to obtain the video features of the video key frames after the dimension reduction; Based on the similarity between the video features of the video key frames after dimensionality reduction and the video features of the video key frames of each video in the video library, each suspected similar video of the video to be retrieved is retrieved from the video library.

7. The method according to claim 5, characterized in that Before the step of extracting the video features of each video frame in the second video frame sequence based on the video feature extraction model, the method further includes: Acquire video samples, wherein the video samples include similar video pairs and dissimilar video pairs; Extracting video features of two similar videos in the similar video pair, and extracting video features of two dissimilar videos in the dissimilar video pair; Determining a first similarity between the video features of the two similar videos, and determining a second similarity between the video features of the two dissimilar videos; A noise contrast estimation loss function is calculated based on the first similarity and the second similarity, and the noise contrast estimation loss function is used to optimize the EfficientNetV2M network to obtain the video feature extraction model.

8. A video retrieval device, characterized in that: The device comprises: A first sampling module, used for extracting video frames of the video to be retrieved according to a fixed sampling frequency to obtain a first video frame sequence; a second sampling module, configured to determine a target sampling frequency according to content complexity and / or system resource usage of the first video frame sequence, and extract video frames from the first video frame sequence based on the target sampling frequency to obtain a second video frame sequence; The retrieval module is used to extract features from the video frames in the second video frame sequence, and retrieve similar videos corresponding to the video to be retrieved from a video library based on the extracted features.

9. A video retrieval device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video retrieval method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the video retrieval method according to any one of claims 1 to 7 are implemented.

Images