Video frame extraction method and device, equipment and medium
Through the composite hashing processing and deep learning model of the video frame extraction method, the problem of insufficient identification accuracy in complex scenarios is solved in the prior art, and more efficient video frame extraction and filtering effects are achieved.
Patent Information
- Application Number
- CN202411189758.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-06
AI Technical Summary
Existing video processing technology is difficult to deal with color changes and multimodal dynamic scenes in complex scenarios, resulting in insufficient recognition accuracy and robustness and poor filtering effect.
The video frame extraction method is adopted, and the color histogram hash value and perceived hash value are obtained for series processing, and the composite hash value is generated, and the frame extraction frequency is dynamically adjusted according to the sound change amplitude, and the image features are used using a deep learning model. Extraction and similarity calculation, dynamically adjusting the threshold to suit different scenarios.
It improves the ability to recognize color changes, accurately recognize and filter similar frames, improves the efficiency and effect of video frame extraction, enhances the filtering effect in different scenarios, and meets the personalized needs of users.
Smart Images

Figure CN119094682A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of video processing technology, and in particular to a video frame extraction method, device, equipment and medium. Background Art
[0002] In video processing, it is often necessary to filter redundant video frames to reduce the amount of data and improve processing efficiency.
[0003] Existing methods mainly rely on perceptual hashing algorithms of images to identify and filter similar frames by comparing the differences between hash values. However, traditional perceptual hashing methods have certain limitations when facing complex scenes and have difficulty handling color changes and multi-modal dynamic scenes. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a video frame extraction method, device, equipment and medium.
[0005] The present disclosure provides a video frame extraction method, the method comprising:
[0006] Perform frame extraction on the video to be processed according to the target frequency to obtain multiple candidate video frames;
[0007] Obtaining a color histogram hash value and a perceptual hash value corresponding to each of the candidate video frames;
[0008] Performing serial processing based on the color histogram hash value and the perceptual hash value to obtain a composite hash value corresponding to each of the candidate video frames;
[0009] Determine a target distance between candidate video frames based on the composite hash value corresponding to each of the candidate video frames;
[0010] A target video frame is determined from the multiple candidate video frames based on the target distance and a preset target threshold.
[0011] Optionally, the video frame extraction method further includes:
[0012] Performing audio analysis on the video to be processed to obtain a sound change amplitude;
[0013] When the sound change amplitude is greater than or equal to the target sound amplitude, setting the target frequency to a first frequency;
[0014] When the sound change amplitude is smaller than the target sound amplitude, the target frequency is set to a second frequency; wherein the first frequency is greater than the second frequency.
[0015] Optionally, obtaining a color histogram hash value corresponding to each of the candidate video frames includes:
[0016] Convert each of the candidate video frames from the RGB color space to the HSV color space, and calculate histograms of the color channels corresponding to the HSV color space to obtain multiple histograms corresponding to each of the candidate video frames;
[0017] Normalizing the multiple histograms corresponding to each of the candidate video frames to obtain multiple normalized histograms corresponding to each of the candidate video frames;
[0018] A plurality of normalized histograms corresponding to each of the candidate video frames are quantized to obtain a plurality of binary character strings corresponding to each of the candidate video frames, and the plurality of binary character strings are combined to obtain a color histogram hash value corresponding to each of the candidate video frames.
[0019] Optionally, obtaining the perceptual hash value corresponding to each of the candidate video frames includes:
[0020] Scaling each of the candidate video frames to a target size grayscale image;
[0021] Perform discrete cosine transform calculation based on the pixels of the target size grayscale image to obtain a discrete cosine transform matrix corresponding to each of the candidate video frames;
[0022] Determine a discrete cosine transform value of a target frequency based on a discrete cosine transform matrix corresponding to each of the candidate video frames, and determine a binary string corresponding to each of the candidate video frames based on the discrete cosine transform value of the target frequency and a target average value;
[0023] The perceptual hash value is determined based on the binary string corresponding to each of the candidate video frames.
[0024] Optionally, the method further includes:
[0025] Performing image feature extraction on each of the target video frames based on a preset deep learning model to obtain image features corresponding to each of the target video frames, and performing cosine similarity calculation based on the image features corresponding to each of the target video frames to obtain cosine similarity between the target video frames;
[0026] The target video frame is filtered based on the cosine similarity and the target similarity.
[0027] Optionally, the video frame extraction method further includes:
[0028] Determining a target number of video frames according to a set target sliding window; wherein the video frames are the candidate video frames and / or the target video frames;
[0029] determining a distance average based on the target distance between the video frames and the number of targets;
[0030] When the distance average value is greater than or equal to the target value, adjusting the target threshold to a first threshold and / or the target similarity to a first similarity; wherein the first threshold is less than the target threshold, and the first similarity is less than the target similarity;
[0031] When the distance average is less than the target value, the target threshold is adjusted to a second threshold and / or the target similarity is adjusted to a second similarity; wherein the second threshold is greater than the target threshold, and the second similarity is greater than the target similarity.
[0032] Optionally, the video frame extraction method further includes:
[0033] In response to the video frame extraction setting request, displaying a video frame extraction setting interface;
[0034] Receive deduplication setting information, threshold information and video information input by a user based on the video frame extraction setting interface;
[0035] Based on the deduplication setting information and the threshold information, video frame extraction and deduplication processing is performed on the video corresponding to the video information.
[0036] The embodiment of the present disclosure also provides a video frame extraction device, including:
[0037] A frame extraction module is used to extract frames of the video to be processed according to the target frequency to obtain multiple candidate video frames;
[0038] A first acquisition module, used to obtain a color histogram hash value corresponding to each of the candidate video frames;
[0039] A second acquisition module is used to obtain a perceptual hash value corresponding to each of the candidate video frames;
[0040] A processing module, configured to perform serial processing based on the color histogram hash value and the perceptual hash value to obtain a composite hash value corresponding to each of the candidate video frames;
[0041] A first determination module, configured to determine a target distance between candidate video frames based on the composite hash value corresponding to each of the candidate video frames;
[0042] The second determination module is configured to determine a target video frame from the plurality of candidate video frames based on the target distance and the target threshold.
[0043] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the video frame extraction method provided in the embodiment of the present disclosure.
[0044] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the video frame extraction method provided by the embodiment of the present disclosure.
[0045] The embodiment of the present disclosure further provides a computer program product, including a computer program, wherein the computer program implements the video frame extraction method described in the above aspect when executed by a processor.
[0046] Compared with the prior art, the technical solution provided by the embodiment of the present disclosure has the following advantages: the video frame extraction scheme provided by the embodiment of the present disclosure performs frame extraction processing on the processed video according to the target frequency to obtain multiple candidate video frames, obtains the color histogram hash value and the perceptual hash value corresponding to each candidate video frame, performs serial processing based on the color histogram hash value and the perceptual hash value, obtains the composite hash value corresponding to each candidate video frame, determines the target distance between the candidate video frames based on the composite hash value corresponding to each candidate video frame, and determines the target video frame from multiple candidate video frames based on the target distance and the preset target threshold. Adopting the above technical solution, the ability to recognize color changes is improved through the composite hash value, so that similar frames can be accurately identified and filtered, and the efficiency and effect of video frame extraction are improved. In addition, the frame extraction frequency is dynamically adjusted in combination with the amplitude of sound changes, so that the frame extraction frequency can adapt to the changes in video content, and the perceptual hash and deep learning models are combined to achieve more refined image comparison, improve the accuracy and efficiency of filtering, and dynamically adjust the threshold using a sliding window according to the video content to enhance the filtering effect in different scenes, and the video frame extraction processing can be performed according to the personalized needs of users through the video frame extraction setting interface to set relevant information, further meet the user's use needs, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0048] Figure 1 A schematic diagram of a flow chart of a video frame extraction method provided in an embodiment of the present disclosure;
[0049] Figure 2Another example diagram of video frame extraction provided by the embodiment of the present disclosure;
[0050] Figure 3 An example diagram of a video frame extraction provided by an embodiment of the present disclosure;
[0051] Figure 4 A schematic diagram of the structure of a video frame extraction device provided in an embodiment of the present disclosure;
[0052] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0053] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0054] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0055] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0056] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0057] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0058] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0059] Based on the background technology description, existing technologies such as perceptual hashing are mainly based on the photometric information of the image, generating hash values by calculating the low-frequency components of the DCT (Discrete Cosine Transform) of the image, and performing similarity comparisons through the Hamming distance. This method works well when processing simple scenes, but the recognition accuracy and robustness are insufficient for color changes and complex multimodal dynamic scenes.
[0060] In other words, the existing technology has the problems of low recognition accuracy. Traditional perceptual hashing methods are difficult to accurately distinguish color changes and are prone to missed detections or false detections. They lack adaptability and fixed thresholds are difficult to adapt to changes in different video scenes, affecting the filtering effect. The processing efficiency is low and a single hashing algorithm cannot fully utilize multi-dimensional image features, resulting in poor filtering effects.
[0061] The video frame extraction method of the disclosed embodiment performs frame extraction processing on the processed video according to the target frequency to obtain multiple candidate video frames, obtains the color histogram hash value and the perceptual hash value corresponding to each candidate video frame, performs serial processing based on the color histogram hash value and the perceptual hash value, obtains the composite hash value corresponding to each candidate video frame, determines the target distance between the candidate video frames based on the composite hash value corresponding to each candidate video frame, and determines the target video frame from multiple candidate video frames based on the target distance and the preset target threshold. Adopting the above technical scheme, the ability to recognize color changes is improved through the composite hash value, so that similar frames can be accurately identified and filtered, and the efficiency and effect of video frame extraction are improved. In addition, combined with the amplitude of sound changes, the frame extraction frequency is dynamically adjusted so that the frame extraction frequency can adapt to the changes in video content, combined with perceptual hash and deep learning models, more sophisticated image comparison is achieved, the accuracy and efficiency of filtering are improved, and the threshold is dynamically adjusted using a sliding window according to the video content, and the filtering effect in different scenes is enhanced, and the video frame extraction processing can be performed according to the personalized needs of users through the video frame extraction setting interface. Related information is set to further meet user needs and improve user experience.
[0062] Figure 1 The present invention provides a flow chart of a video frame extraction method provided in an embodiment of the present invention. The method can be executed by a video frame extraction device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:
[0063] Step 101: Perform frame extraction processing on the video to be processed according to the target frequency to obtain multiple candidate videos.
[0064] In the disclosed embodiments, the target frequency can be set according to the application scenario. As an example, the processed video is subjected to audio analysis to obtain the sound change amplitude. When the sound change amplitude is greater than or equal to the target sound amplitude, the target frequency is set to the first frequency, and when the sound change amplitude is less than the target sound amplitude, the target frequency is set to the second frequency; wherein the first frequency is greater than the second frequency, and the frame extraction frequency is dynamically adjusted by combining the sound change amplitude, so that the frame extraction frequency can adapt to the changes in the video content. As another example, the processed video is subjected to dialogue analysis to obtain the analysis result, and the target frequency is set based on the dialogue analysis result.
[0065] In the embodiment of the present disclosure, after the target frequency is determined, the video to be processed is subjected to frame extraction processing according to the target frequency, for example, five frames per second, etc., to obtain a plurality of candidate video frames.
[0066] Step 102: Obtain the color histogram hash value and the perceptual hash value corresponding to each candidate video frame.
[0067] In some embodiments, there are many ways to obtain the color histogram hash value corresponding to each candidate video frame. As an example, each candidate video frame is converted from the RGB color space to the HSV color space, and the histograms are calculated respectively in the color channels corresponding to the HSV color space to obtain multiple histograms corresponding to each candidate video frame. The multiple histograms corresponding to each candidate video frame are normalized to obtain multiple normalized histograms corresponding to each candidate video frame. The multiple normalized histograms corresponding to each candidate video frame are quantized to obtain multiple binary strings corresponding to each candidate video frame, and the multiple binary strings are combined to obtain the color histogram hash value corresponding to each candidate video frame. As another example, each candidate video frame is input into a pre-trained color histogram hash value calculation model to obtain the color histogram hash value corresponding to each candidate video frame.
[0068] In some embodiments, there are many ways to obtain the perceptual hash value corresponding to each candidate video frame. As an example, each candidate video frame is scaled to a grayscale image of a target size, and a discrete cosine transform is calculated based on the pixels of the grayscale image of the target size to obtain a discrete cosine transform matrix corresponding to each candidate video frame. The discrete cosine transform value of the target frequency is determined based on the discrete cosine transform matrix corresponding to each candidate video frame, and the binary string corresponding to each candidate video frame is determined based on the discrete cosine transform value of the target frequency and the target average value. Based on the binary string corresponding to each candidate video frame, the perceptual hash value is determined. As another example, each candidate video frame is input into a pre-trained perceptual hash value calculation model to obtain the perceptual hash value corresponding to each candidate video frame.
[0069] Step 103: Perform serial processing based on the color histogram hash value and the perceptual hash value to obtain a composite hash value corresponding to each candidate video frame.
[0070] Step 104: Determine a target distance between candidate video frames based on the composite hash value corresponding to each candidate video frame.
[0071] In the embodiment of the present disclosure, the color histogram hash value and the perceptual hash value corresponding to each candidate video frame are processed in series to obtain a composite hash value corresponding to each candidate video frame.
[0072] In the embodiments of the present disclosure, there are many ways to determine the target distance between candidate video frames based on the composite hash value corresponding to each candidate video frame. For example, the target distance is obtained by calculating the number of 1s in the result after the XOR operation of two composite hash values. The target distance represents the different number of bits between the two composite hash values.
[0073] Step 105 : determining a target video frame from a plurality of candidate video frames based on the target distance and a preset target threshold.
[0074] In an embodiment of the present disclosure, a target threshold can be set according to an application scenario. As an example, a target number of multiple candidate video frames is determined according to a set target sliding window, and a distance average is determined based on the target distance and the target number between the multiple candidate video frames. When the distance average is greater than or equal to the target value, the target threshold is adjusted to a first threshold, wherein the first threshold is less than the target threshold; when the distance average is less than the target value, the target threshold is adjusted to a second threshold; wherein the second threshold is greater than the target threshold, and a sliding window is used to dynamically adjust the threshold according to the video content to enhance the filtering effect in different scenarios; as another example, historical video frame extraction thresholds for different scenarios are obtained, and the optimal video frame extraction thresholds are marked according to different scenarios, so as to determine the optimal video frame extraction threshold corresponding to the current video frame extraction scenario, i.e., the target threshold.
[0075] In the disclosed embodiment, if the target distance between the candidate video frames is less than the target distance, it indicates that the two candidate video frames are similar, and the candidate video frames obtained later can be filtered, that is, the candidate video frames are deleted from multiple candidate video frames, and the remaining ones are used as target video frames, that is, the retained key video frames.
[0076] The video frame extraction scheme provided by the embodiment of the present disclosure performs frame extraction processing on the processed video according to the target frequency to obtain multiple candidate video frames, obtains the color histogram hash value and the perceptual hash value corresponding to each candidate video frame, performs serial processing based on the color histogram hash value and the perceptual hash value to obtain the composite hash value corresponding to each candidate video frame, determines the target distance between the candidate video frames based on the composite hash value corresponding to each candidate video frame, and determines the target video frame from the multiple candidate video frames based on the target distance and the preset target threshold. The above technical scheme is adopted to improve the recognition ability of color changes through the composite hash value, so that similar frames can be accurately identified and filtered, and the efficiency and effect of video frame extraction are improved.
[0077] Figure 2 This is a flow chart of another video frame extraction method provided in an embodiment of the present disclosure. Based on the above embodiment, this embodiment further optimizes the above video frame extraction method.
[0078] like Figure 2 As shown, the method includes:
[0079] Step 201, perform audio analysis on the video to be processed to obtain the sound change amplitude, when the sound change amplitude is greater than or equal to the target sound amplitude, set the target frequency to the first frequency, and when the sound change amplitude is less than the target sound amplitude, set the target frequency to the second frequency; wherein the first frequency is greater than the second frequency.
[0080] In the embodiment of the present disclosure, the sound change amplitude refers to the amplitude of the overall audio ripple, which represents the sound intensity; the target sound amplitude can be selected and set according to the actual application scenario.
[0081] Specifically, the input video to be processed is pre-processed with audio analysis. The audio analysis is used to detect the amplitude of sound changes in order to dynamically adjust the frame frequency: when the sound changes greatly, the frame frequency is increased to capture more details; when the sound changes slightly, the frame frequency is reduced to the default frequency. For example, when extracting frames from a car accident video on a highway section, more pictures can be extracted within the critical time period of the accident through the amplitude of the audio changes.
[0082] Therefore, the frame extraction frequency is dynamically adjusted in combination with the amplitude of sound changes, so that the frame extraction frequency can adapt to changes in video content.
[0083] Step 202: extract frames of the video to be processed according to the target frequency to obtain multiple candidate video frames, convert each candidate video frame from the RGB color space to the HSV color space, and calculate histograms in the color channels corresponding to the HSV color space to obtain multiple histograms corresponding to each candidate video frame, and normalize the multiple histograms corresponding to each candidate video frame to obtain multiple normalized histograms corresponding to each candidate video frame.
[0084] Step 203: quantize the multiple normalized histograms corresponding to each candidate video frame to obtain multiple binary character strings corresponding to each candidate video frame, and combine the multiple binary character strings to obtain a color histogram hash value corresponding to each candidate video frame.
[0085] In the disclosed embodiment, the video frame obtained by frame extraction is in RGB (color of three channels of red, green and blue) format, and then the candidate video frame is converted from the RGB color space to the HSV (hue, saturation, value) color space, and then the histogram is calculated on the three channels of the HSV space respectively to obtain multiple histograms corresponding to each candidate video frame, and the multiple histograms corresponding to each candidate video frame are normalized to obtain multiple normalized histograms corresponding to each candidate video frame, and finally the multiple normalized histograms corresponding to each candidate video frame are quantized to obtain multiple binary strings corresponding to each candidate video frame, and the color histogram hash value corresponding to each candidate video frame is obtained by combining. Among them, the color histogram is an image processing technology that represents the color characteristics of the image by statistically analyzing the number distribution of pixels of different colors in the image.
[0086] Specifically, the candidate video frame is read and converted from the RGB color space to the HSV color space. The HSV color space is more consistent with human visual perception and is more robust to changes in brightness. The color histogram of the candidate video frame is calculated. Usually, the histogram is calculated separately on the three channels of the HSV space. For example, 8 bins can be set for each channel, so that the size of the entire histogram is 8x8x8=512. That is to say, when calculating the color histogram of the candidate video frame, the bins are used to divide the color value into several intervals so as to count the number of pixels in each interval. The purpose is to divide the continuous The color value data is discretized for easier statistics and analysis; the histogram is normalized so that the histogram is insensitive to changes in image size, that is, no matter how the image size changes, the histogram value can remain consistent, for example, the number of pixels in each bin is divided by the total number of all pixels in the image; the histogram is quantized, and the normalized histogram is quantized. The quantization process maps each normalized histogram value to a binary bit (for example, the part greater than 0.5 is 1, and vice versa), thereby forming a binary string, and finally these binary strings are combined together to form a color histogram hash value.
[0087] Step 204: Scale each candidate video frame to a grayscale image of a target size, perform discrete cosine transform calculation based on the pixels of the grayscale image of the target size, and obtain a discrete cosine transform matrix corresponding to each candidate video frame.
[0088] Step 205: determine the discrete cosine transform value of the target frequency based on the discrete cosine transform matrix corresponding to each candidate video frame, determine the binary string corresponding to each candidate video frame based on the discrete cosine transform value of the target frequency and the target average value, and determine the perceptual hash value based on the binary string corresponding to each candidate video frame.
[0089] Among them, the target size is set according to the needs of the application scenario, the target frequency is usually the low-frequency part frequency, and the perceptual hash value is a hash algorithm based on the visual characteristics of the image, which is used to generate fingerprint information of the image for similarity comparison.
[0090] Specifically, the candidate video frame is read and scaled, for example, the candidate video frame is scaled to a 32x32 grayscale image, thereby simplifying the calculation while maintaining the main visual features of the image; the discrete cosine transform (DCT) moment of the candidate video frame pixel is calculated, and the DCT moment is a method of converting image information in the spatial domain to the frequency domain, which can well extract the overall features of the image; since the main information of the candidate video frame is concentrated in the low-frequency part of the DCT moment, only the first 8x8 part of the DCT matrix can be retained, and the average value of the low-frequency part of the DCT moment is calculated, and then each value is compared with the average value, and the value greater than the average value is 1, and the value less than or equal to the average value is 0, and finally a 64-bit binary string, i.e., the perceptual hash value, is generated.
[0091] Step 206: Perform serial processing based on the color histogram hash value and the perceptual hash value to obtain a composite hash value corresponding to each candidate video frame, and determine a target distance between the candidate video frames based on the composite hash value corresponding to each candidate video frame.
[0092] In the embodiment of the present disclosure, the color histogram hash and the perceptual hash are connected in series to form a composite hash value. That is, on the basis of the traditional perceptual hash, the color histogram is fused, by converting the video frame into a grayscale image and calculating the perceptual hash value, then calculating the color histogram of the video frame and generating a color histogram hash value. Finally, the perceptual hash value and the color histogram hash value are combined to generate a composite hash value, thereby improving the accuracy and robustness of recognition.
[0093] Specifically, by comparing the composite hash values through a target distance, such as the Hamming distance, the similarity of two candidate video frames can be measured. The Hamming distance represents the number of different bits in two binary strings of the same length. The smaller the distance, the more similar the candidate video frames are.
[0094] Therefore, the color histogram is introduced and combined with the traditional perceptual hashing to generate a composite hash value to improve the ability to recognize color changes.
[0095] Step 207, determining a target number of video frames according to the set target sliding window, wherein the video frames are candidate video frames and / or target video frames, determining a distance average value based on the target distance and the target number between the video frames, and when the distance average value is greater than or equal to the target value, adjusting the target threshold to a first threshold value and / or the target similarity to a first similarity; wherein the first threshold value is less than the target threshold value, and the first similarity is less than the target similarity; when the distance average value is less than the target value, adjusting the target threshold to a second threshold value and / or the target similarity to a second similarity; wherein the second threshold value is greater than the target threshold value, and the second similarity is greater than the target similarity.
[0096] Step 208: Determine a target video frame from multiple candidate video frames based on the target distance and a preset target threshold.
[0097] Step 209: extract image features of each target video frame based on a preset deep learning model to obtain image features corresponding to each target video frame, calculate cosine similarity based on the image features corresponding to each target video frame to obtain cosine similarity between target video frames, and filter the target video frames based on the cosine similarity and target similarity.
[0098] In the disclosed embodiments, the deep learning model may be a convolutional neural network (CNN), a deep learning model widely used in image recognition and processing tasks, or other deep learning models such as MobileNet, ResNet, EfficientNet, etc., to further improve the effects of image feature extraction and similarity comparison.
[0099] In the disclosed embodiment, multi-dimensional hashing is first used for preliminary filtering, and then the remaining frames are compared in detail using a more complex image comparison algorithm, such as using a CNN model (such as MobileNet) to extract image features and calculate cosine similarity. Multi-level filtering is used to improve the accuracy and efficiency of filtering. Among them, cosine similarity is a method for measuring the directional similarity of two vectors, and is often used to compare the similarity of image feature vectors.
[0100] Among them, the target similarity can be selected and set according to the application scenario. If the cosine similarity between the target video frames is greater than the target similarity, it means that the two target video frames are similar. The target video frames obtained later can be filtered, that is, the target video frames are deleted from multiple target video frames, and the remaining target video frames are used as the final frame extraction results.
[0101] Therefore, by combining perceptual hashing and deep learning models, more detailed image comparisons can be achieved, improving the accuracy and efficiency of filtering.
[0102] In the embodiments of the present disclosure, the target sliding window and the target value can be selected and set according to the application scenario.
[0103] Specifically, in the deduplication stage, an adaptive threshold algorithm can be designed to dynamically adjust the Hamming distance and similarity thresholds according to the video content and context. Specifically, the image change rate is calculated based on the sliding window. In fast-paced action scenes, the threshold is lowered to retain more details; in slower-paced scenes, the threshold is increased to reduce redundancy.
[0104] Specifically, the target sliding window can use a sliding window in the process of calculating the Hamming distance of the hash value and the similarity of the image features, and the window contains a certain number of continuous frames. For example, the window size can be set to 10, which means that 10 consecutive candidate video frames will be processed each time. Users can adjust the window size according to the actual scene and the different requirements of the frame extraction effect to obtain different effects.
[0105] Specifically, calculate the rate of change: in each target sliding window, the Hamming distance or similarity between each video frame and its previous video frame can be calculated, and then the average of these distances can be calculated. This average value can be regarded as the rate of change of the image in the window; adjust the target threshold: you can set a target threshold, and then adjust the target threshold according to the rate of change. If the rate of change is greater than a certain value (for example, half of the target threshold), then lower the target threshold so that more frames can be retained; if the rate of change is less than a certain value (for example, half of the target threshold), then increase the target threshold so that fewer frames can be retained; filter similar frames: use the adjusted target threshold to filter similar frames. If the Hamming distance between a video frame and its previous video frame is less than the target threshold, then the two video frames are considered to be similar, and the latter video frame can be filtered out.
[0106] Therefore, the threshold is dynamically adjusted using a sliding window according to the video content to enhance the filtering effect in different scenes.
[0107] Thus, by combining perceptual hashing, color histogram and deep learning models such as convolutional neural network (CNN) model and sound change amplitude, multi-dimensional similar image filtering and dynamic frame extraction are achieved.
[0108] Based on the description of the above embodiments, the traditional fixed-frequency frame extraction is prone to generate a large amount of redundant data, while the embodiment of the present disclosure adopts a dynamic frame extraction method combined with audio information to reduce the amount of data and calculation. The traditional perceptual hash algorithm usually generates hash values based only on the photometric information of the image, that is, brightness or grayscale value. This method is more effective at the geometric shape and structural level of the image, but it does not consider the color information of the image. Therefore, if the image only changes in color, such as the traffic light changes from red to green, and the photometric level is similar, the traditional perceptual hash may not be able to effectively distinguish. The multidimensional hash analysis of the embodiment of the present disclosure solves this problem by introducing a color histogram. The color histogram can record the frequency and distribution of various colors in the image, so that the hash value reflects the color change, so that the color switch of the traffic light can be distinguished. Because the multidimensional hash analysis method adds more image feature dimensions (such as color histogram, edge information and texture features), the method is more effective and accurate than the traditional perceptual hash in identifying color changes. Each dimension can capture the different characteristics of the image. Combining these characteristics can generate a more expressive hash value, thereby greatly improving the accuracy of image recognition and comparison.
[0109] Specifically, the video frame extraction of the embodiment of the present disclosure can be used to extract key frames (such as emergency braking, emergency steering, traffic light start and stop, etc.) from the video recorded by the vehicle-side sensor, which can be used for the training of autonomous driving models, as well as image deduplication, video summary generation, surveillance video analysis and other fields.
[0110] As an example, Figure 3 As shown, the web page interface displays the video recorded by the vehicle sensor, and the user selects the video to perform frame extraction and deduplication. In some embodiments, in response to a video frame extraction setting request, a video frame extraction setting interface is displayed, and deduplication setting information, threshold information and video information input by the user are received based on the video frame extraction setting interface, and video frame extraction and deduplication processing is performed on the video corresponding to the video information based on the deduplication setting information and threshold information.
[0111] Specifically, the user can open the video frame extraction setting interface according to actual application needs, and select the deduplication setting information in the video frame extraction setting interface, such as Figure 3 The multidimensional hash and CNN shown in the figure are used, and threshold information such as a similarity threshold of 95% is selected, and video information such as a name or link of the video to be processed is selected, so that the video corresponding to the video information is subjected to video frame extraction and deduplication processing based on the deduplication setting information and threshold information input by the user, that is, the video frame extraction processing described in the aforementioned embodiment; the user can also select the output data set name, that is, the name of the video frame after deduplication processing, such as Figure 3 The output data set name shown can be selected and input by the user to further meet the user's usage needs.
[0112] As a result, the recognition accuracy is improved, that is, the system can accurately identify and filter similar frames through multi-dimensional hash analysis and multi-level filtering mechanism; the adaptability is enhanced, and the adaptive threshold mechanism can dynamically adjust the filtering threshold to adapt to different video scenes and enhance the filtering effect; the processing efficiency is improved, the dynamic frame extraction frequency and multi-level filtering mechanism reduce the amount of calculation and improve the processing efficiency while ensuring the filtering effect.
[0113] Figure 4 This is a schematic diagram of the structure of a video frame extraction device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 4 As shown, the device comprises:
[0114] A frame extraction module 401 is used to extract frames of the video to be processed according to a target frequency to obtain multiple candidate video frames;
[0115] A first acquisition module 402 is used to obtain a color histogram hash value corresponding to each candidate video frame;
[0116] A second acquisition module 403 is used to obtain a perceptual hash value corresponding to each candidate video frame;
[0117] A processing module 404 is used to perform serial processing based on the color histogram hash value and the perceptual hash value to obtain a composite hash value corresponding to each of the candidate video frames;
[0118] A first determination module 405, configured to determine a target distance between candidate video frames based on the composite hash value corresponding to each of the candidate video frames;
[0119] The second determination module 406 is configured to determine a target video frame from the plurality of candidate video frames based on the target distance and the target threshold.
[0120] Optionally, the video frame extraction device further includes:
[0121] An analysis module, used for performing audio analysis on the video to be processed to obtain a sound change amplitude;
[0122] A setting module, configured to set the target frequency to a first frequency when the sound change amplitude is greater than or equal to the target sound amplitude;
[0123] The setting module is used to set the target frequency to a second frequency when the sound change amplitude is smaller than the target sound amplitude; wherein the first frequency is greater than the second frequency.
[0124] Optionally, the first acquisition module 402 is specifically configured to:
[0125] Convert each of the candidate video frames from the RGB color space to the HSV color space, and calculate histograms of the color channels corresponding to the HSV color space to obtain multiple histograms corresponding to each of the candidate video frames;
[0126] Normalizing the multiple histograms corresponding to each of the candidate video frames to obtain multiple normalized histograms corresponding to each of the candidate video frames;
[0127] A plurality of normalized histograms corresponding to each of the candidate video frames are quantized to obtain a plurality of binary character strings corresponding to each of the candidate video frames, and the plurality of binary character strings are combined to obtain a color histogram hash value corresponding to each of the candidate video frames.
[0128] Optionally, the second acquisition module 403 is specifically configured to:
[0129] Scaling each of the candidate video frames to a target size grayscale image;
[0130] Perform discrete cosine transform calculation based on the pixels of the target size grayscale image to obtain a discrete cosine transform matrix corresponding to each of the candidate video frames;
[0131] Determine a discrete cosine transform value of a target frequency based on a discrete cosine transform matrix corresponding to each of the candidate video frames, and determine a binary string corresponding to each of the candidate video frames based on the discrete cosine transform value of the target frequency and a target average value;
[0132] The perceptual hash value is determined based on the binary string corresponding to each of the candidate video frames.
[0133] Optionally, after obtaining the color histogram hash value and the perceptual hash value corresponding to each of the candidate video frames, the device further includes:
[0134] An extraction and calculation module, used to extract image features from each of the target video frames based on a preset deep learning model to obtain image features corresponding to each of the target video frames, and to perform cosine similarity calculation based on the image features corresponding to each of the target video frames to obtain cosine similarity between the target video frames;
[0135] A filtering module is used to filter the target video frame based on the cosine similarity and the target similarity.
[0136] Optionally, the video frame extraction device further includes:
[0137] A third determination module is used to determine a target number of video frames according to a set target sliding window; wherein the video frames are the candidate video frames and / or the target video frames;
[0138] A fourth determination module determines a distance average based on the target distance between the video frames and the number of targets;
[0139] An adjustment module, configured to adjust the target threshold to a first threshold and / or the target similarity to a first similarity when the distance average value is greater than or equal to a target value; wherein the first threshold is less than the target threshold, and the first similarity is less than the target similarity;
[0140] The adjustment module is further used to adjust the target threshold to a second threshold and / or the target similarity to a second similarity when the distance average is less than the target value; wherein the second threshold is greater than the target threshold, and the second similarity is greater than the target similarity.
[0141] Optionally, the video frame extraction device further includes:
[0142] A response display module, used for displaying a video frame extraction setting interface in response to a video frame extraction setting request;
[0143] A receiving module, used for receiving deduplication setting information, threshold information and video information input by a user based on the video frame extraction setting interface;
[0144] A deduplication module is used to perform video frame extraction and deduplication processing on the video corresponding to the video information based on the deduplication setting information and the threshold information.
[0145] The video frame extraction device provided in the embodiments of the present disclosure can execute the video frame extraction method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.
[0146] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the video frame extraction method provided by any embodiment of the present disclosure.
[0147] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 5, which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure. The electronic device 500 in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0148] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0149] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0150] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the video frame extraction method of the embodiment of the present disclosure are executed.
[0151] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0152] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0153] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0154] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains an audio file identifier, and obtains a source audio file in an object storage service according to the audio file identifier, processes the source audio file to obtain a decrypted audio stream, obtains a target audio format, and determines a video frame extraction adapter to process the decrypted audio stream according to the source audio format and the target audio format of the source audio file to obtain an audio stream in a target format.
[0155] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0157] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.
[0158] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0159] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0160] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:
[0161] processor;
[0162] a memory for storing instructions executable by the processor;
[0163] The processor is used to read the executable instructions from the memory and execute the instructions to implement any video frame extraction method provided in the present disclosure.
[0164] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute any of the video frame extraction methods provided by the present disclosure.
[0165] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0166] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0167] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A video frame extraction method, characterized in that: include: Perform frame extraction on the video to be processed according to the target frequency to obtain multiple candidate video frames; Obtaining a color histogram hash value and a perceptual hash value corresponding to each of the candidate video frames; Performing serial processing based on the color histogram hash value and the perceptual hash value to obtain a composite hash value corresponding to each of the candidate video frames; Determine a target distance between candidate video frames based on the composite hash value corresponding to each of the candidate video frames; A target video frame is determined from the multiple candidate video frames based on the target distance and a preset target threshold.
2. The video frame extraction method according to claim 1, characterized in that: Also includes: Performing audio analysis on the video to be processed to obtain a sound change amplitude; When the sound change amplitude is greater than or equal to the target sound amplitude, setting the target frequency to a first frequency; When the sound change amplitude is smaller than the target sound amplitude, the target frequency is set to a second frequency; wherein the first frequency is greater than the second frequency.
3. The video frame extraction method according to claim 1, characterized in that: The obtaining of the color histogram hash value corresponding to each of the candidate video frames includes: Convert each of the candidate video frames from the RGB color space to the HSV color space, and calculate histograms of the color channels corresponding to the HSV color space to obtain multiple histograms corresponding to each of the candidate video frames; Normalizing the multiple histograms corresponding to each of the candidate video frames to obtain multiple normalized histograms corresponding to each of the candidate video frames; A plurality of normalized histograms corresponding to each of the candidate video frames are quantized to obtain a plurality of binary character strings corresponding to each of the candidate video frames, and the plurality of binary character strings are combined to obtain a color histogram hash value corresponding to each of the candidate video frames.
4. The video frame extraction method according to claim 1, characterized in that: The obtaining of the perceptual hash value corresponding to each of the candidate video frames includes: Scaling each of the candidate video frames to a target size grayscale image; Perform discrete cosine transform calculation based on the pixels of the target size grayscale image to obtain a discrete cosine transform matrix corresponding to each of the candidate video frames; Determine a discrete cosine transform value of a target frequency based on a discrete cosine transform matrix corresponding to each of the candidate video frames, and determine a binary string corresponding to each of the candidate video frames based on the discrete cosine transform value of the target frequency and a target average value; The perceptual hash value is determined based on the binary string corresponding to each of the candidate video frames.
5. The video frame extraction method according to claim 1, characterized in that: Also includes: Performing image feature extraction on each of the target video frames based on a preset deep learning model to obtain image features corresponding to each of the target video frames, and performing cosine similarity calculation based on the image features corresponding to each of the target video frames to obtain cosine similarity between the target video frames; The target video frame is filtered based on the cosine similarity and the target similarity.
6. The video frame extraction method according to claim 5, characterized in that: Also includes: Determining a target number of video frames according to a set target sliding window; wherein the video frames are the candidate video frames and / or the target video frames; determining a distance average based on the target distance between the video frames and the number of targets; When the distance average value is greater than or equal to the target value, adjusting the target threshold to a first threshold and / or the target similarity to a first similarity; wherein the first threshold is less than the target threshold, and the first similarity is less than the target similarity; When the distance average is less than the target value, the target threshold is adjusted to a second threshold and / or the target similarity is adjusted to a second similarity; wherein the second threshold is greater than the target threshold, and the second similarity is greater than the target similarity.
7. The video frame extraction method according to claim 1, characterized in that: Also includes: In response to the video frame extraction setting request, displaying a video frame extraction setting interface; Receive deduplication setting information, threshold information and video information input by a user based on the video frame extraction setting interface; Based on the deduplication setting information and the threshold information, video frame extraction and deduplication processing is performed on the video corresponding to the video information.
8. A video frame extraction device, characterized in that: include: A frame extraction module is used to extract frames of the video to be processed according to the target frequency to obtain multiple candidate video frames; A first acquisition module, used to obtain a color histogram hash value corresponding to each of the candidate video frames; A second acquisition module is used to obtain a perceptual hash value corresponding to each of the candidate video frames; A processing module, configured to perform serial processing based on the color histogram hash value and the perceptual hash value to obtain a composite hash value corresponding to each of the candidate video frames; A first determination module, configured to determine a target distance between candidate video frames based on the composite hash value corresponding to each of the candidate video frames; The second determination module is configured to determine a target video frame from the plurality of candidate video frames based on the target distance and the target threshold.
9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the video frame extraction method described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the video frame extraction method described in any one of claims 1-7.
Citation Information
Cited By
Self-adaptive video frame extraction method, equipment and medium
CN121691809A
