Video key frame extraction method and device, equipment and storage medium

Abstract

The invention provides a video key frame extraction method and device, equipment and a storage medium, and the method comprises the steps: obtaining an original frame sequence of an input video; performing content analysis on the original frame sequence, and identifying whether a user-defined attention target exists or not; if the concerned target is not recognized, selecting a key frame from the original frame sequence according to a preset extraction strategy; and if the attention target is identified, constructing a candidate frame set based on the frames containing the attention target and calculating visual difference measurement among the frames in the candidate frame set, and selecting a key frame from the candidate frame set based on the visual difference measurement and outputting the key frame. By adopting the method, the key frame extraction efficiency, pertinence and information density can be improved, and the accuracy of subsequent video analysis is further improved.

Description

Technical Field

[0001] This application relates to the field of video processing, and more specifically, to a method, apparatus, device, and storage medium for extracting keyframes from a video. Background Technology

[0002] With the rapid popularization of scenarios such as intelligent monitoring and video content retrieval, the demand for video data processing continues to grow. As a core step in video preprocessing, keyframe extraction directly affects the efficiency of subsequent video analysis and the accuracy of content understanding. It has become one of the key technologies for reducing video data redundancy and improving the processing performance of multimodal models.

[0003] In existing technologies, the most widely used method is the fixed time interval frame extraction method. This method selects a frame sequence evenly from the input video by pre-setting a fixed time interval (such as extracting one frame every 2 seconds) or a fixed number of frames (such as extracting one frame every 10 frames), and uses this as the set of key frames for the video.

[0004] The existing technology has obvious flaws: its frame extraction logic only relies on fixed intervals of time or frame number, completely ignoring the dynamic changes and differences in video content. If there are long static segments in the video, a large number of redundant frames with repeated content will be extracted. When the target that the user is interested in (such as an abnormal object that appears briefly in the monitoring scene) only appears in the gap of the interval period, the key frame containing the target may be missed, causing the subsequent video analysis to fail to capture the core content. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a video keyframe extraction method, apparatus, device and storage medium that can improve the efficiency, targeting and information density of keyframe extraction, thereby improving the accuracy of subsequent video analysis.

[0006] In a first aspect, embodiments of this application provide a method for extracting video keyframes, the method comprising: Obtain the raw frame sequence of the input video; Content analysis is performed on the original frame sequence to identify whether a user-defined target of interest exists; If no target of interest is identified, key frames are selected from the original frame sequence using a preset extraction strategy. If a target of interest is identified, a candidate frame set is constructed based on the frames containing the target of interest, and a visual difference metric between each frame in the candidate frame set is calculated. Based on the visual difference metric, a key frame is selected from the candidate frame set and output.

[0007] Optionally, the step of performing content analysis on the original frame sequence to identify whether a user-defined target of interest exists includes: Each frame in the original frame sequence is detected using a target detection model configured to detect targets of at least one preset category, which includes at least one of pedestrians, vehicles, animals, or specific objects. When at least one target of the preset category is detected in any frame, it is determined that the target of interest has been identified.

[0008] Optionally, the step of selecting keyframes from the original frame sequence using a preset extraction strategy includes: Determine the total number K of keyframes to be selected; Calculate the uniform time interval or frame interval based on the total duration or total number of frames of the original frame sequence and the quantity K. Based on the interval, K positions are selected at equal intervals on the time axis of the original frame sequence, and a frame is randomly selected from the frame interval corresponding to each position to form the key frame.

[0009] Optionally, constructing a candidate frame set based on frames containing the target of interest includes: The frame containing the target of interest is taken as the target frame, and the target frame, the first frame of the original frame sequence, and the last frame of the original frame sequence are merged to form an initial frame set; Determine whether the number of frames in the initial frame set meets the preset key frame number requirement; If the target frame is not reached, supplementary frames are selected from the original frame sequence excluding the target frame in chronological order. The supplementary frame is added to the initial frame set to obtain the candidate frame set.

[0010] Optionally, calculating the visual difference metric between frames in the candidate frame set includes: For each frame in the candidate frame set, the image of the frame is scaled to a predetermined size and converted into a grayscale image. The difference hash value of the frame is generated by comparing the grayscale value of each pixel in the grayscale image with its neighboring pixels pixel by pixel. For any two frames in the candidate frame set, calculate the Hamming distance between the difference hash values ​​corresponding to the two frames, and use it as a measure of visual difference between the two frames.

[0011] Optionally, selecting keyframes from the candidate frame set based on the visual difference metric includes: Select one frame from the candidate frame set as the initial keyframe and add it to the selected keyframe set; In the candidate frame set, among the remaining frames excluding the selected keyframe set, calculate the average of the visual difference measure between each remaining frame and all frames in the selected keyframe set; Add the remaining frames with the largest average value to the selected keyframe set; Repeat the process of calculating the average value and selecting frames until the number of frames in the selected keyframe set reaches the preset number of keyframes. At this point, the selected keyframe set is the final selected keyframe set.

[0012] Optionally, the method further includes: Provide a user configuration interface, through which the user can set at least one of the following parameters: The rules of the preset extraction strategy, the categories of the user-defined target of interest, the number of keyframes to be output, and the difference threshold used to judge frame similarity.

[0013] Secondly, embodiments of this application provide a video keyframe extraction device, the device comprising: The raw frame sequence acquisition module is used to acquire the raw frame sequence of the input video; The target of interest identification module is used to perform content analysis on the original frame sequence to identify whether there is a user-defined target of interest. The first keyframe selection module is used to select keyframes from the original frame sequence using a preset extraction strategy if no target of interest is identified. The second keyframe selection module is used to construct a candidate frame set based on the frames containing the target of interest if a target of interest is identified, calculate the visual difference metric between each frame in the candidate frame set, select a keyframe from the candidate frame set based on the visual difference metric, and output it.

[0014] Optionally, the step of performing content analysis on the original frame sequence to identify whether a user-defined target of interest exists includes: Each frame in the original frame sequence is detected using a target detection model configured to detect targets of at least one preset category, which includes at least one of pedestrians, vehicles, animals, or specific objects. When at least one target of the preset category is detected in any frame, it is determined that the target of interest has been identified.

[0015] Optionally, the step of selecting keyframes from the original frame sequence using a preset extraction strategy includes: Determine the total number K of keyframes to be selected; Calculate the uniform time interval or frame interval based on the total duration or total number of frames of the original frame sequence and the quantity K. Based on the interval, K positions are selected at equal intervals on the time axis of the original frame sequence, and a frame is randomly selected from the frame interval corresponding to each position to form the key frame.

[0016] Optionally, constructing a candidate frame set based on frames containing the target of interest includes: The frame containing the target of interest is taken as the target frame, and the target frame, the first frame of the original frame sequence, and the last frame of the original frame sequence are merged to form an initial frame set; Determine whether the number of frames in the initial frame set meets the preset key frame number requirement; If the target frame is not reached, supplementary frames are selected from the original frame sequence excluding the target frame in chronological order. The supplementary frame is added to the initial frame set to obtain the candidate frame set.

[0017] Optionally, calculating the visual difference metric between frames in the candidate frame set includes: For each frame in the candidate frame set, the image of the frame is scaled to a predetermined size and converted into a grayscale image. The difference hash value of the frame is generated by comparing the grayscale value of each pixel in the grayscale image with its neighboring pixels pixel by pixel. For any two frames in the candidate frame set, calculate the Hamming distance between the difference hash values ​​corresponding to the two frames, and use it as a measure of visual difference between the two frames.

[0018] Optionally, selecting keyframes from the candidate frame set based on the visual difference metric includes: Select one frame from the candidate frame set as the initial keyframe and add it to the selected keyframe set; In the candidate frame set, among the remaining frames excluding the selected keyframe set, calculate the average of the visual difference measure between each remaining frame and all frames in the selected keyframe set; Add the remaining frames with the largest average value to the selected keyframe set; Repeat the process of calculating the average value and selecting frames until the number of frames in the selected keyframe set reaches the preset number of keyframes. At this point, the selected keyframe set is the final selected keyframe set.

[0019] Optionally, the device further includes: A parameter configuration module is used to provide a user configuration interface, through which the user can receive settings for at least one of the following parameters: The rules of the preset extraction strategy, the categories of the user-defined target of interest, the number of keyframes to be output, and the difference threshold used to judge frame similarity.

[0020] Thirdly, embodiments of this application provide a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the video keyframe extraction method described in any of the optional embodiments of the first aspect are performed.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the video keyframe extraction method described in any of the optional embodiments of the first aspect.

[0022] The technical solution provided in this application includes, but is not limited to, the following beneficial effects: Obtaining the original frame sequence of the input video has the advantage of providing a complete and comprehensive source of basic frames for the subsequent keyframe extraction process. This avoids the early omission of potential key content in the original video due to missing initial frames, and ensures that subsequent content analysis and keyframe selection operations can be carried out based on the complete frame information of the video, thus laying the foundation for the effectiveness of the entire process at the input level.

[0023] Content analysis of the original frame sequence identifies whether there are user-defined targets of interest. The beneficial effect is that it anchors the keyframe extraction logic to the actual needs of users. It is no longer a generic frame extraction without distinction, but rather uses the user's target of interest as the core judgment criterion, so that the subsequent keyframe selection strategy revolves around the user's actual content of interest, thereby improving the matching degree between keyframes and user needs.

[0024] If no target of interest is identified, key frames are selected from the original frame sequence using a preset extraction strategy. The advantage of this strategy is that it adapts to the needs of scenarios without a target of interest: key frame selection is completed quickly through a preset strategy without the need for complex content depth processing, which effectively saves computing resources; at the same time, it ensures the uniform distribution of key frames in the time dimension, which can maintain the basic content coverage of the video without a target of interest, balancing the need for processing efficiency and the preservation of basic information.

[0025] If a target of interest is identified, a candidate frame set is constructed based on frames containing the target of interest, and a visual difference metric is calculated between each frame in the candidate frame set. Based on the visual difference metric, key frames are selected from the candidate frame set and output. The beneficial effects are that it ensures that the key frames cover the user's target of interest, that is, by constructing a candidate frame set containing the target of interest, it ensures that the key frames focus on the core content; and it improves the information quality of the key frames, that is, by filtering out frames with significant content differences through the visual difference metric, it covers the richest target-related information with the fewest number of key frames, reduces the content redundancy of the key frames, and increases the information density of the key frames.

[0026] In summary, this application, through its scenario- and step-by-step keyframe extraction logic, not only adapts to different video content scenarios, improving processing efficiency when there is no target of interest, but also focuses on core content when there is a target of interest; it also takes into account the demand matching degree and information quality of keyframes, ultimately achieving a synergistic improvement in keyframe extraction in terms of efficiency, targeting, and information density.

[0027] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 The flowchart of a video keyframe extraction method provided in Embodiment 1 of this application is shown; Figure 2 A flowchart of a target identification method provided in Embodiment 1 of this application is shown; Figure 3 A flowchart of a keyframe selection method provided in Embodiment 1 of this application is shown; Figure 4 A flowchart of a candidate frame set construction method provided in Embodiment 1 of this application is shown; Figure 5 A flowchart of a visual difference measurement calculation method provided in Embodiment 1 of this application is shown; Figure 6 A schematic diagram of a Hamming distance calculation process provided in Embodiment 1 of this application is shown; Figure 7A flowchart of the second keyframe selection method provided in Embodiment 1 of this application is shown; Figure 8 This paper shows a complete flowchart of a video keyframe extraction method provided in Embodiment 1 of this application. Figure 9 This illustration shows a system architecture and data flow diagram provided in Embodiment 1 of this application; Figure 10 This shows a schematic diagram of the structure of a video keyframe extraction device provided in Embodiment 2 of this application; Figure 11 A schematic diagram of the structure of a computer device provided in Embodiment 3 of this application is shown. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0031] Example 1 To facilitate understanding of this application, the following is combined with... Figure 1 The flowchart illustrating a video keyframe extraction method provided in Embodiment 1 of this application will be used to describe Embodiment 1 of this application in detail.

[0032] See Figure 1 As shown, Figure 1 A flowchart of a video keyframe extraction method provided in Embodiment 1 of this application is shown, wherein the method includes steps S101 to S103: S101: Obtain the original frame sequence of the input video.

[0033] Specifically, this step involves collecting video data through a camera terminal, then uniformly extracting frames at a fixed rate (F frames per second) configured by the user through the client or preset by the system to generate an original frame list containing all extracted frames, providing a complete basic input for subsequent content analysis and key frame selection.

[0034] S102: Perform content analysis on the original frame sequence to identify whether there are user-defined targets of interest.

[0035] Specifically, content analysis relies on the "N-in-one algorithm," which is the decision trigger for the entire process. The form can be flexibly adapted; it can be a single model integrating multiple detection capabilities or multiple models in parallel, depending on the user's set of algorithm capabilities.

[0036] The objects to be detected are determined entirely based on the user's business scenario. Users can define the categories of targets they are interested in, including but not limited to pedestrians, vehicles, animals, fires, smoke, or specific defective objects in industrial inspections, which are closely aligned with actual usage needs.

[0037] S103: If no target of interest is identified, key frames are selected from the original frame sequence using a preset extraction strategy.

[0038] Specifically, the preset extraction strategy is a random, equally spaced extraction strategy. This strategy can avoid in-depth processing of worthless video content, save computing resources from the source, and solve the problem of computational redundancy in existing technologies.

[0039] Its execution logic is as follows: first, read the total number of frames of the input video, calculate the frame interval based on the user-preset final number of keyframes K, and randomly select 1 frame in each of the K intervals evenly divided on the time axis to ensure that the keyframes are evenly distributed in the time dimension.

[0040] S104: If a target of interest is identified, a candidate frame set is constructed based on the frames containing the target of interest, and a visual difference metric between each frame in the candidate frame set is calculated. Based on the visual difference metric, a key frame is selected from the candidate frame set and output.

[0041] Specifically, the construction of the candidate frame set follows the merging rule of "target frame + first and last frame". First, the frames containing the target of interest are selected as target frames, and then merged with the first frame (first frame) and the last frame (last frame) of the original frame sequence to form the initial frame set. The forced inclusion of the first and last frames is a heuristic rule that incorporates domain knowledge and can effectively capture key information such as the start of the action and state changes.

[0042] If the number of frames in the initial frame set does not reach the preset number of key frames K, supplementary frames will be selected from the candidate frames other than the target frame in the original frame sequence in chronological order until the K frame requirement is met, ensuring information integrity.

[0043] Visual difference measurement is achieved by combining dhash (Difference Hash algorithm) with Hamming distance. The combination of the two provides a fast and efficient means of quantifying inter-frame visual differences and is the mathematical basis for achieving "maximum diversity" selection.

[0044] Keyframe selection employs an incremental selection strategy that maximizes the difference: first, initial frames with high "overall difference" are selected from the candidate frames, then the frames with the largest average Hamming distance from the selected frame set are selected iteratively until K frames are selected, and finally the frames are output in the original frame time order to ensure that the richest information is covered with the fewest frames.

[0045] In an optional implementation, see Figure 2 As shown, Figure 2 The flowchart of a target of interest identification method provided in Embodiment 1 of this application is shown, wherein the step of performing content analysis on the original frame sequence to identify whether a user-defined target of interest exists includes steps S201~S202: S201: Detect each frame in the original frame sequence using a target detection model, the target detection model being configured to detect targets of at least one preset category, the preset category including at least one of pedestrians, vehicles, animals, or specific objects.

[0046] Specifically, the target detection model is the "N-in-one algorithm", whose detection capabilities can be flexibly switched according to the user's business scenario. For example, it can be adapted to pedestrian and vehicle detection in security scenarios, specific object detection in industrial defect detection scenarios, and pet detection in pet monitoring scenarios.

[0047] S202: When at least one target of the preset category is detected in any frame, it is determined that the target of interest has been identified.

[0048] Specifically, it is not necessary for all frames in the original frame sequence to contain the target of interest. As long as any frame contains at least one preset category target, the subsequent optimization frame extraction process is triggered to avoid missing sudden key content and solve the problem that dynamically changing content is not deeply integrated in the existing technology.

[0049] In an optional implementation, see Figure 3 As shown, Figure 3 The flowchart of a keyframe selection method provided in Embodiment 1 of this application is shown, wherein the keyframes are selected from the original frame sequence using a preset extraction strategy, including steps S301-S302: S301: Determine the total number K of keyframes to be selected.

[0050] Specifically, the total number of keyframes K is one of the core parameters. Users can set it themselves through PC (Personal Computer) front-end, APP (Application) mobile client, etc., or use the system default value to adapt to the input frame requirements of different multimodal large language models (MLLMs).

[0051] S302: Calculate a uniform time interval or frame interval based on the total duration or total number of frames of the original frame sequence and the quantity K.

[0052] Specifically, if calculated based on the total number of frames, the frame interval = total number of frames in the original frame sequence ÷ K (if not divisible, take an approximate value); if calculated based on the total duration, the time interval = total video duration ÷ K, ensuring that the interval allocation fits the actual distribution of the video and avoiding the rigidity of fixed intervals.

[0053] S303: Based on the interval, select K positions at equal intervals on the time axis of the original frame sequence, and randomly select a frame from the frame interval corresponding to each position to form the key frame.

[0054] Specifically, the frame interval corresponding to each position is the set of frames within the adjacent interval range before and after that position. For example, when the frame interval is 10 frames, the first interval is frames 1-10, the second is frames 11-20, and so on. Random frame selection can avoid the risk of missing key information caused by fixed frame selection.

[0055] In an optional implementation, see Figure 4 As shown, Figure 4 The flowchart illustrates a method for constructing a candidate frame set according to Embodiment 1 of this application, wherein the step of constructing a candidate frame set based on frames containing the target of interest includes steps S401 to S404: S401: Take the frame containing the target of interest as the target frame, and merge the target frame, the first frame of the original frame sequence and the last frame of the original frame sequence to form an initial frame set.

[0056] Specifically, even if no target of interest is detected in the first or last frame, it must still be forcibly included in the initial frame set. This is because in scenarios such as behavior analysis and event detection, the first and last frames often contain key information such as the start of an event and the end of a state, which can improve the reliability of capturing key information.

[0057] S402: Determine whether the number of frames in the initial frame set meets the preset key frame number requirement.

[0058] Specifically, the preset keyframe quantity requirement is the user-configured or system-default K value. The judgment logic is that if the number of frames in the initial frame set is ≥ K, it is considered to meet the requirement. Otherwise, it needs to be improved by supplementing frames to ensure that the number of keyframes is sufficient to support the understanding needs of the multimodal model.

[0059] S403: If not reached, then supplementary frames are selected from the original frame sequence excluding the target frame in chronological order.

[0060] Specifically, the selection of supplementary frames strictly follows the chronological order of the original frames, prioritizing frames that are temporally adjacent to the target frame and the first and last frames, to ensure that the supplemented frame set maintains continuity in the time dimension and avoids information fragmentation.

[0061] S404: Add the supplementary frame to the initial frame set to obtain the candidate frame set.

[0062] Specifically, after the supplementary frames are added, the original temporal order of all frames in the candidate frame set must be maintained, and the arrangement relationship of the frames must not be changed, so as to provide an ordered input for subsequent visual difference measurement calculation and incremental selection strategy.

[0063] In an optional implementation, see Figure 5 As shown, Figure 5 The flowchart of a visual difference measurement calculation method provided in Embodiment 1 of this application is shown, wherein the calculation of the visual difference measurement between each frame in the candidate frame set includes steps S501~S502: S501: For each frame in the candidate frame set, the image of the frame is scaled to a predetermined size and converted into a grayscale image. The difference hash value (dhash, Difference Hash algorithm) of the frame is generated by comparing the grayscale value of each pixel in the grayscale image with its neighboring pixels pixel by pixel.

[0064] Specifically, the predetermined size is set to 9×8 pixels, which is the optimal value in experience that balances comparison accuracy and computational load. If the size is too large (such as 16×17), it will increase the computational burden, and if it is too small (such as 4×5), it will reduce the accuracy of difference recognition.

[0065] Each pixel in a grayscale image represents a grayscale value using an integer from 0 to 255. When comparing pixels one by one, only the current pixel is compared with its right-hand neighboring pixel. No comparisons are made between different rows. Finally, a 64-bit sequence of 0s and 1s is generated, which is the "visual fingerprint" of the image.

[0066] S502: For any two frames in the candidate frame set, calculate the Hamming distance between the difference hash values ​​corresponding to the two frames, and use it as a measure of visual difference between the two frames.

[0067] Specifically, Hamming distance is a fundamental concept in information theory and computer science for quantifying the degree of difference between two sequences of equal length. Its general calculation formula is: ; in, and Representing sequences respectively and In the The character or bit value at each position (in this scheme, the corresponding bits of the 64-bit differential hash value); It is an indicator function, when Not equal to The function value is 1 when they are equal and 0 when they are equal. For sequence and The length of the nth element (n=64 in this scheme).

[0068] See Figure 6 As shown, Figure 6 This paper illustrates a flowchart of a Hamming distance calculation method provided in Embodiment 1 of this application, wherein... Figure 6 The calculation process for Hamming distance in S502 of this scheme is as follows: The process starts with "Start Calculation". "Input two sequences s1 and s2" corresponds to the 64-bit difference hash sequence of the two frames in this scheme (i.e., the "visual fingerprint" of the frame). The subsequent "Are the sequence lengths equal?" check is a necessary preliminary step in this scheme - because Hamming distance is the core indicator for quantifying the visual differences between frames and needs to be calculated based on sequences of equal length. This step can avoid calculation errors caused by mismatched sequence lengths. If the check fails, "an exception is thrown / an error is reported that the calculation cannot be performed", ensuring the validity of the calculation. If the check result is "yes", then "the result value is calculated and returned" to obtain the Hamming distance.

[0069] The calculation logic of Hamming distance is to count the number of different bits at the same position in two 64-bit difference hash values. The more bits there are, the greater the visual difference between the two frames, and vice versa.

[0070] When the selected frame set contains multiple frames, the Hamming distance between the candidate frame and each frame in the set is calculated and then averaged (i.e., the distance between the candidate frame and each frame in the set is calculated using the formula above). Then, the arithmetic mean is calculated. The formula contains a 1 / N factor (N is the number of frames in the selected frame set), which objectively reflects the overall difference between the candidate frames and the selected set.

[0071] In an optional implementation, see Figure 7 As shown, Figure 7The flowchart of the second keyframe selection method provided in Embodiment 1 of this application is shown, wherein the step of selecting keyframes from the candidate frame set based on the visual difference metric includes steps S701 to S704: S701: Select a frame from the candidate frame set as the initial keyframe and add it to the selected keyframe set.

[0072] Specifically, the initial keyframe is selected from the candidate frame set with high "overall difference". For example, by calculating the average Hamming distance between all candidate frames, the frame with the largest average value is selected as the initial point to ensure that the initial frame has strong information representativeness.

[0073] S702: In the remaining frames of the candidate frame set excluding the selected keyframe set, calculate the average value of the visual difference measure between each remaining frame and all frames in the selected keyframe set.

[0074] Specifically, this average calculation avoids the one-sidedness of judging differences in a single frame. By comprehensively considering the differences with all frames in the selected set, it ensures that the selected frame can supplement the set with new and effective information.

[0075] S703: Add the remaining frames with the largest average value to the selected keyframe set.

[0076] Specifically, selecting the frame with the highest average value essentially means prioritizing the frame with the most significant visual difference from the selected set. This maximizes the information coverage of the selected set, avoids redundancy issues with multiple similar frames, and solves the limitation of insufficient utilization of dynamic information in existing technologies.

[0077] S704: Repeat the process of calculating the average value and selecting frames until the number of frames in the selected keyframe set reaches the preset number of keyframes. At this time, the selected keyframe set is the final selected keyframe set.

[0078] Specifically, the number of frames in the selected keyframe set increases by 1 after each iteration. The iteration terminates when the number of frames is exactly equal to the value of K. Finally, the frames are reordered according to the time order of the original frame sequence before being output to adapt to the requirements of the multimodal large model for the time logic of the input frames.

[0079] In an optional implementation, the method further includes: A user configuration interface is provided, through which the user sets at least one of the following parameters: the rules of the preset extraction strategy, the category of the user-defined target of interest, the number of keyframes to be output, and the difference threshold used to judge frame similarity.

[0080] Specifically, the user configuration interface corresponds to the operation interface of the PC front-end and the APP mobile terminal. In addition to the parameters mentioned above, it also supports users to set parameters such as the original frame extraction frequency F and the selection of key target algorithms, so as to realize customized frame extraction for professional fields and solve the problem of the lack of customized strategies in existing technologies.

[0081] There is no fixed standard for the difference threshold used to judge frame similarity. The system provides default empirical values ​​such as 0.75 and 0.8, which users can flexibly adjust according to specific business scenarios such as security and industrial inspection to ensure that they meet the accuracy requirements of different scenarios.

[0082] To better illustrate the video keyframe extraction method provided in this application, please refer to [link / reference]. Figure 8 As shown, Figure 8 This document illustrates a complete flowchart of a video keyframe extraction method provided in Embodiment 1 of this application. The process begins with "reading video," and proceeds through the step of "extracting video frames at F frames per second to generate an original frame list," corresponding to S101, "obtaining the original frame sequence of the input video." This is followed by the "N-in-one algorithm for key target detection," corresponding to S102, a target of interest identification step. If "no target of interest" is determined, a preset extraction strategy is executed, "directly and randomly extracting K frames from the original frames at equal intervals as keyframes and returning them," corresponding to S103. If "a target of interest" is determined, target frames are first filtered by "selecting target frames containing the target of interest based on the original frame list and returning them in the order of the original frames," and then the initial frame set is constructed by "merging the selected target frames and the first and last frames to generate candidate frames, with the remaining frames as alternative frames," corresponding to S401. Afterwards, the process continues... The process involves checking whether the number of candidate frames meets the minimum frame count requirement K. If the result is "no", then "randomly select insufficient frames from the candidate frames and merge them with the candidate frames, returning a total of K frames as keyframes", corresponding to the supplementary frame logic in S403-S404. If the result is "yes", then "traverse the candidate frame sequence and calculate the hash of each frame", corresponding to the difference hash value generation in S501. Next, "use Hamming distance as the inter-frame difference distance metric and select an image with high 'overall difference' as the initial point". Then, "each time, select the image with the largest nearest distance from the selected set among the unselected images, and repeat the above steps until the number of selected frames reaches K", corresponding to the incremental keyframe selection strategy in S601-S604. Finally, "return K keyframes in the original frame order" to complete the entire process.

[0083] The video keyframe extraction method provided in this application is implemented based on a hardware / software support platform. See [link to relevant documentation]. Figure 9 As shown, Figure 9 This paper illustrates a system architecture and data flow diagram provided in Embodiment 1 of this application, wherein... Figure 9The “user camera” corresponds to the raw video data acquisition stage of S101; “Kafka” is responsible for the transmission and flow of acquired data; the “AI intelligent platform” is the core processing module, whose internal modules such as “human detection” and “vehicle detection” correspond to the “N-in-one algorithm” of S102 (to realize the recognition of multiple categories of targets of interest); the “gateway” connects to the “PC front-end” and “APP mobile terminal” (corresponding to the user configuration interface, supporting the setting of parameters such as the number of keyframes K and the category of targets of interest in S301); “Redis (Remote Dictionary Server)” and “MySQL (MySQL, relational database management system)” serve as caching and storage components, supporting the queue management of input data, avoiding overload of the AI ​​intelligent platform, and ensuring the efficient operation of the S103-S104 keyframe extraction process.

[0084] To better illustrate the video keyframe extraction method provided in this application, a specific example of video keyframe extraction is provided. Assume a user configures keyframe extraction parameters via an app in a security monitoring scenario: original frame extraction frequency F = 5 frames / second, number of keyframes K = 8, target category is "pedestrian," difference threshold is 0.8 by default, and a 10-second monitoring video is input (no pedestrians in the first 3 seconds, pedestrians appearing consecutively from seconds 3 to 7, and no pedestrians in the last 3 seconds). The specific frame extraction process is as follows: first, extract 50 frames (Frame1~Frame50) from the 10-second video at F = 5 frames / second; then, use the "N-in-one algorithm" (target detection model) to detect each frame, determining that Frame1~Frame15 (first 3 seconds) have no pedestrians, Frame16~Frame35 (seconds 3-7) are target frames containing pedestrians, and Frame36~Frame50 (last 3 seconds) have no pedestrians. Subsequently, a candidate frame set is constructed—the target frames (Frame1~Frame50) are selected from the target frames. The frames 16 through 35 are merged with the first frame 1 and the last frame 50 to form an initial set of 22 frames (satisfying the requirement of K=8, no additional frames are needed). Then, each candidate frame is scaled to 9×8 pixels and converted to grayscale. A 64-bit difference hash value is generated by comparing each pixel. The Hamming distance between the difference hash values ​​of any two frames is then calculated as a measure of visual difference. An incremental selection strategy is then adopted: first, Frame25 (where the pedestrian is in the center of the image and has the highest overall difference) is selected as the initial keyframe. Then, the average Hamming distance between the remaining frames and the selected set is calculated iteratively. The frames with the largest average distances are selected in sequence: Frame18, Frame32, Frame16, Frame35, Frame1, Frame40, and Frame50. Finally, a set of 8 keyframes {Frame1, Frame16, Frame18, Frame25, Frame32, Frame35, Frame40, Frame50} is formed and output in the original frame time order. This not only covers the core segment where the pedestrian appears but also retains the scene information at the beginning and end of the video, without redundant similar frames.

[0085] Example 2 See Figure 10 As shown, Figure 10 This illustration shows a structural schematic diagram of a video keyframe extraction device according to Embodiment 2 of this application, wherein the device includes: The raw frame sequence acquisition module 1001 is used to acquire the raw frame sequence of the input video; The target of interest identification module 1002 is used to perform content analysis on the original frame sequence to identify whether there is a user-defined target of interest. The first keyframe selection module 1003 is used to select keyframes from the original frame sequence using a preset extraction strategy if no target of interest is identified. The second keyframe selection module 1004 is used to construct a candidate frame set based on the frames containing the target of interest if a target of interest is identified, calculate the visual difference measure between each frame in the candidate frame set, select a keyframe from the candidate frame set based on the visual difference measure, and output it.

[0086] In an optional implementation, the content analysis of the original frame sequence to identify whether a user-defined target of interest exists includes: Each frame in the original frame sequence is detected using a target detection model configured to detect targets of at least one preset category, which includes at least one of pedestrians, vehicles, animals, or specific objects. When at least one target of the preset category is detected in any frame, it is determined that the target of interest has been identified.

[0087] In an optional implementation, the step of selecting keyframes from the original frame sequence using a preset extraction strategy includes: Determine the total number K of keyframes to be selected; Calculate the uniform time interval or frame interval based on the total duration or total number of frames of the original frame sequence and the quantity K. Based on the interval, K positions are selected at equal intervals on the time axis of the original frame sequence, and a frame is randomly selected from the frame interval corresponding to each position to form the key frame.

[0088] In an optional implementation, constructing a candidate frame set based on frames containing the target of interest includes: The frame containing the target of interest is taken as the target frame, and the target frame, the first frame of the original frame sequence, and the last frame of the original frame sequence are merged to form an initial frame set; Determine whether the number of frames in the initial frame set meets the preset key frame number requirement; If the target frame is not reached, supplementary frames are selected from the original frame sequence excluding the target frame in chronological order. The supplementary frame is added to the initial frame set to obtain the candidate frame set.

[0089] In an optional implementation, calculating the visual difference metric between frames in the candidate frame set includes: For each frame in the candidate frame set, the image of the frame is scaled to a predetermined size and converted into a grayscale image. The difference hash value of the frame is generated by comparing the grayscale value of each pixel in the grayscale image with its neighboring pixels pixel by pixel. For any two frames in the candidate frame set, calculate the Hamming distance between the difference hash values ​​corresponding to the two frames, and use it as a measure of visual difference between the two frames.

[0090] In an optional implementation, selecting keyframes from the candidate frame set based on the visual difference metric includes: Select one frame from the candidate frame set as the initial keyframe and add it to the selected keyframe set; In the candidate frame set, among the remaining frames excluding the selected keyframe set, calculate the average of the visual difference measure between each remaining frame and all frames in the selected keyframe set; Add the remaining frames with the largest average value to the selected keyframe set; Repeat the process of calculating the average value and selecting frames until the number of frames in the selected keyframe set reaches the preset number of keyframes. At this point, the selected keyframe set is the final selected keyframe set.

[0091] In an optional implementation, the device further includes: A parameter configuration module is used to provide a user configuration interface, through which the user can receive settings for at least one of the following parameters: The rules of the preset extraction strategy, the categories of the user-defined target of interest, the number of keyframes to be output, and the difference threshold used to judge frame similarity.

[0092] Example 3 Based on the same application concept, see [link / reference] Figure 11 As shown, Figure 11 This illustration shows a structural schematic diagram of a computer device provided in Embodiment 3 of this application, wherein, as shown... Figure 11 As shown, the computer device 1100 provided in Embodiment 3 of this application includes: The system includes a processor 1101, a memory 1102, and a bus 1103. The memory 1102 stores machine-readable instructions executable by the processor 1101. When the computer device 1100 is running, the processor 1101 and the memory 1102 communicate via the bus 1103. When the machine-readable instructions are executed by the processor 1101, the steps of the video keyframe extraction method shown in Embodiment 1 are performed.

[0093] Example 4 Based on the same concept, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the video keyframe extraction method described in any of the above embodiments.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0095] The computer program product for extracting keyframes from video provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0096] The video keyframe extraction device provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0097] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0098] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0100] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0102] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for extracting keyframes from a video, characterized in that, The method includes: Obtain the raw frame sequence of the input video; Content analysis is performed on the original frame sequence to identify whether a user-defined target of interest exists; If no target of interest is identified, key frames are selected from the original frame sequence using a preset extraction strategy. If a target of interest is identified, a candidate frame set is constructed based on the frames containing the target of interest, and a visual difference metric between each frame in the candidate frame set is calculated. Based on the visual difference metric, a key frame is selected from the candidate frame set and output.

2. The method according to claim 1, characterized in that, The step of performing content analysis on the original frame sequence to identify whether there are user-defined targets of interest includes: Each frame in the original frame sequence is detected using a target detection model configured to detect targets of at least one preset category, which includes at least one of pedestrians, vehicles, animals, or specific objects. When at least one target of the preset category is detected in any frame, it is determined that the target of interest has been identified.

3. The method according to claim 1, characterized in that, The step involves selecting key frames from the original frame sequence using a preset extraction strategy, including: Determine the total number K of keyframes to be selected; Calculate the uniform time interval or frame interval based on the total duration or total number of frames of the original frame sequence and the quantity K. Based on the interval, K positions are selected at equal intervals on the time axis of the original frame sequence, and a frame is randomly selected from the frame interval corresponding to each position to form the key frame.

4. The method according to claim 1, characterized in that, The construction of a candidate frame set based on frames containing the target of interest includes: The frame containing the target of interest is taken as the target frame, and the target frame, the first frame of the original frame sequence, and the last frame of the original frame sequence are merged to form an initial frame set; Determine whether the number of frames in the initial frame set meets the preset key frame number requirement; If the target frame is not reached, supplementary frames are selected from the original frame sequence excluding the target frame in chronological order. The supplementary frame is added to the initial frame set to obtain the candidate frame set.

5. The method according to claim 1, characterized in that, The calculation of the visual difference metric between frames in the candidate frame set includes: For each frame in the candidate frame set, the image of the frame is scaled to a predetermined size and converted into a grayscale image. The difference hash value of the frame is generated by comparing the grayscale value of each pixel in the grayscale image with its neighboring pixels pixel by pixel. For any two frames in the candidate frame set, calculate the Hamming distance between the difference hash values ​​corresponding to the two frames, and use it as a measure of visual difference between the two frames.

6. The method according to claim 1, characterized in that, The step of selecting keyframes from the candidate frame set based on the visual difference metric includes: Select one frame from the candidate frame set as the initial keyframe and add it to the selected keyframe set; In the candidate frame set, among the remaining frames excluding the selected keyframe set, calculate the average of the visual difference measure between each remaining frame and all frames in the selected keyframe set; Add the remaining frames with the largest average value to the selected keyframe set; Repeat the process of calculating the average value and selecting frames until the number of frames in the selected keyframe set reaches the preset number of keyframes. At this point, the selected keyframe set is the final selected keyframe set.

7. The method according to claim 1, characterized in that, The method further includes: Provide a user configuration interface, through which the user can set at least one of the following parameters: The rules of the preset extraction strategy, the categories of the user-defined target of interest, the number of keyframes to be output, and the difference threshold used to judge frame similarity.

8. A video keyframe extraction device, characterized in that, The device includes: The raw frame sequence acquisition module is used to acquire the raw frame sequence of the input video; The target of interest identification module is used to perform content analysis on the original frame sequence to identify whether there is a user-defined target of interest. The first keyframe selection module is used to select keyframes from the original frame sequence using a preset extraction strategy if no target of interest is identified. The second keyframe selection module is used to construct a candidate frame set based on the frames containing the target of interest if a target of interest is identified, calculate the visual difference metric between each frame in the candidate frame set, select a keyframe from the candidate frame set based on the visual difference metric, and output it.

9. A computer device, characterized in that, include: The system includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the video keyframe extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the video keyframe extraction method as described in any one of claims 1 to 7.

Images