Video frame extraction method and device and electronic equipment
By identifying and accurately extracting key indicator values for shot transitions from video frames, the problem of poor image analysis results caused by extraction at fixed time intervals is solved, achieving more efficient image analysis and preservation of motion details.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-11
- Publication Date
- 2026-04-07
Smart Images
Figure CN121814977A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computer network streaming media, in particular, to a video frame extraction method and device and electronic equipment. BACKGROUND
[0002] For various video live streams, especially video streams of security monitoring live, stored video files, more and more visual AI algorithms are introduced for analysis, such as personnel recognition, behavior recognition, and cap recognition according to safety regulations. Before introducing visual AI algorithms for analysis on these video streams or video files, frame extraction (or screenshot) may be needed on the video streams or video files. The usual method is to extract video frames at fixed time intervals, convert them into pictures, and then use them as inputs for AI algorithms. This method of extracting video frames at fixed time intervals may result in poor image analysis effect.
[0003] Currently, no effective solution has been proposed for the above problems. SUMMARY
[0004] Embodiments of the present application provide a video frame extraction method, device and electronic equipment to at least solve the technical problem that video frame extraction at fixed time intervals in the related art results in poor image analysis effect.
[0005] According to an aspect of an embodiment of the present application, a video frame extraction method is provided, including: obtaining a to-be-processed video frame in a video sequence; in a case where the to-be-processed video frame is not a first frame in the video sequence, determining a shot cut key indicator value of the to-be-processed video frame; determining whether the to-be-processed video frame is a shot transition frame according to the shot cut key indicator value to obtain a first determination result; and in a case where the first determination result indicates that the to-be-processed video frame is a shot transition frame, extracting and outputting the to-be-processed video frame.
[0006] Optionally, after obtaining the to-be-processed video frame in the video sequence, the method further includes: in a case where the to-be-processed video frame is a first frame in the video sequence, extracting and outputting the to-be-processed video frame.
[0007] Optionally, the determining the shot-cut key indicator value of the to-be-processed video frame comprises: obtaining a previous video frame of the to-be-processed video frame, and obtaining a number of tracked feature points of the previous video frame and a number of successfully tracked feature points of the to-be-processed video frame; determining a tracking point loss rate of the to-be-processed video frame according to the number of tracked feature points of the previous video frame and the number of successfully tracked feature points of the to-be-processed video frame; obtaining an initial frame in the video sequence, and obtaining a first position of an i-th feature point in a successfully tracked feature point subset of the initial frame and a second position of the i-th feature point in a successfully tracked feature point subset of the to-be-processed video frame, wherein i is a positive integer; determining a position mutation value of the to-be-processed video frame according to the first position, the second position and the number of successfully tracked feature points of the to-be-processed video frame, wherein the position mutation value is used to determine whether a shot cut occurs; obtaining a motion direction angle of the i-th feature point in the successfully tracked feature point subset of the to-be-processed video frame and a motion direction mean value of all feature points in the successfully tracked feature point subset of the to-be-processed video frame; determining a motion direction variance value of the to-be-processed video frame according to the motion direction angle, the motion direction mean value and the number of successfully tracked feature points of the to-be-processed video frame; and determining the shot-cut key indicator value of the to-be-processed video frame according to the tracking point loss rate, the position mutation value and the motion direction variance value.
[0008] Optionally, the determining whether the to-be-processed video frame is a shot-cut frame according to the shot-cut key indicator value comprises: determining that the to-be-processed video frame is a shot-cut frame when the tracking point loss rate is greater than a first threshold value, or the position mutation value is greater than a second threshold value, or the motion direction variance value is greater than a third threshold value.
[0009] Optionally, the method further comprises: in a case where the first determination result indicates that the to-be-processed video frame is not a shot-cut frame, obtaining a first coordinate of a first feature point in the successfully tracked feature point subset of the to-be-processed video frame, and obtaining a second coordinate of a next feature point of the first feature point in a feature point set of the to-be-processed video frame, wherein the first feature point is any one of the successfully tracked feature point subset of the to-be-processed video frame; determining a vector amplitude of an optical flow vector of the first feature point according to a difference between the second coordinate and the first coordinate; obtaining vector amplitudes of all feature points in the successfully tracked feature point subset of the to-be-processed video frame to obtain a vector amplitude set; and determining an average amplitude corresponding to the vector amplitude set, wherein the average amplitude is used to represent an average displacement of the feature points between the to-be-processed video frame and a next video frame of the to-be-processed video frame.
[0010] Optionally, after determining the average amplitude corresponding to the set of vector amplitudes, the method further comprises: in a case where the average amplitude is greater than a first amplitude threshold, determining that the motion intensity level of the video frame to be processed is a first level; in a case where the average amplitude is less than or equal to the first amplitude threshold and greater than a second amplitude threshold, determining that the motion intensity level of the video frame to be processed is a second level, wherein the second level is lower than the first level; in a case where the average amplitude is less than or equal to the second amplitude threshold and greater than a third amplitude threshold, determining that the motion intensity level of the video frame to be processed is a third level, wherein the third level is lower than the second level; and in a case where the average amplitude is less than or equal to the third amplitude threshold, determining that the motion intensity level of the video frame to be processed is a fourth level, wherein the fourth level is lower than the third level.
[0011] Optionally, the method further comprises: obtaining a video type of the video sequence; and determining a target motion intensity threshold according to the video type, wherein the target motion intensity threshold is any one of the first amplitude threshold, the second amplitude threshold and the third amplitude threshold; and in a case where the average amplitude is greater than or equal to the target motion intensity threshold, extracting and outputting the video frame to be processed.
[0012] According to another aspect of the embodiments of the present application, a video frame extraction device is further provided, which comprises: an obtaining module configured to obtain a video frame to be processed in a video sequence; a determining module configured to determine a shot transition key indicator value of the video frame to be processed in a case where the video frame to be processed is not a first frame in the video sequence; a judging module configured to judge whether the video frame to be processed is a shot transition frame according to the shot transition key indicator value, to obtain a first judgment result; and an extracting module configured to extract and output the video frame to be processed in a case where the first judgment result indicates that the video frame to be processed is the shot transition frame.
[0013] According to still another aspect of the embodiments of the present application, an electronic device is further provided, which comprises: a memory configured to store program instructions; and a processor connected with the memory, configured to execute the program instructions to realize the following functions: obtaining a video frame to be processed in a video sequence; determining a shot transition key indicator value of the video frame to be processed in a case where the video frame to be processed is not a first frame in the video sequence; judging whether the video frame to be processed is a shot transition frame according to the shot transition key indicator value, to obtain a first judgment result; and extracting and outputting the video frame to be processed in a case where the first judgment result indicates that the video frame to be processed is the shot transition frame.
[0014] According to still another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, which comprises a stored computer program, wherein a device in which the non-volatile storage medium is located executes the video frame extraction method by running the computer program.
[0015] According to still another aspect of the embodiments of the present application, a computer program product is provided, which comprises computer instructions, and the computer instructions are executed by a processor to implement the video frame extraction method.
[0016] In the embodiments of the present application, a to-be-processed video frame in a video sequence is acquired; in a case where the to-be-processed video frame is not a first frame in the video sequence, a shot switching key indicator value of the to-be-processed video frame is determined; whether the to-be-processed video frame is a shot transition frame is determined according to the shot switching key indicator value, to obtain a first determination result; in a case where the first determination result indicates that the to-be-processed video frame is a shot transition frame, the to-be-processed video frame is extracted and output, so that the purpose of accurately capturing a shot switching instant in a video is achieved, thereby achieving the technical effects of improving the accuracy and efficiency of subsequent image analysis, and further solving the technical problem in the prior art that the extraction of video frames according to a fixed time interval leads to poor effects of subsequent image analysis. BRIEF DESCRIPTION OF DRAWINGS
[0017] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application, and illustrate the embodiments of the present application and the description thereof, which serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:
[0018] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a video frame extraction method according to an embodiment of the present application;
[0019] Figure 2 is a flowchart of a video frame extraction method according to an embodiment of the present application;
[0020] Figure 3 is a logic flowchart of a video frame extraction method according to an embodiment of the present application;
[0021] Figure 4 is a structure diagram of a video frame extraction device according to an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.
[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0024] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:
[0025] Optical flow is the instantaneous velocity of pixels moving on the imaging plane of a moving object in space.
[0026] Optical flow is a method that uses the changes of pixels in an image sequence over time and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.
[0027] Video frame extraction, also known as video frame reduction or video frame trimming, refers to the technique of extracting a portion of frames (images) from a video at certain intervals or under certain conditions. Simply put, it involves extracting parts of a video to create a new video with fewer frames, or extracting each frame from a video as a still image.
[0028] Live streaming: an important online video live streaming technology that enables real-time transmission and viewing of audio and video content through the processes of pushing and pulling streams.
[0029] Video frame: The basic unit of video data transmission and processing. It represents a single image in a video sequence. During video playback, these static images are displayed continuously at a certain rate, thus producing dynamic visual effects.
[0030] A shot (spot) is a continuous, uninterrupted sequence of images captured by a camera without jumps or breaks. It has visual continuity, meaning it is a set of consecutive frames in a video with minimal pixel differences.
[0031] Scene: A scene consists of a series of shots that present a semantically coherent part of the story. It is a collection of consecutive frames in a video with little semantic difference.
[0032] Shot transition frames, also known as shot change frames, typically refer to the frames in a video where a significant change in scene or perspective occurs. During video shooting or editing, shot transition frames are created when the camera moves from one shot to another, or when the camera switches from one shooting angle to another during video editing.
[0033] The method of extracting video frames at fixed time intervals in related technologies has two drawbacks. Firstly, for videos with rapid motion, it ignores the fast changes in the scene within a short period, thus losing motion details and resulting in a loss of detail in the output image. Secondly, for videos with minimal changes, extracting images at fixed time intervals leads to excessive repetition in the output, which in turn increases the burden on backend AI analysis and inefficiently utilizes computing power. These two aspects contribute to the poor performance of subsequent image analysis.
[0034] To address the problems existing in related technologies, this application provides a method for video frame extraction, which can be implemented in... Figure 1 The computer terminal shown is explained below.
[0035] The video frame extraction method embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal for implementing a video frame extraction method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0036] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video frame extraction method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned video frame extraction method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0038] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.
[0039] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.
[0040] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.
[0041] In the above operating environment, this application provides a video frame extraction method embodiment. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0042] Figure 2 This is a flowchart of a video frame extraction method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:
[0043] Step S202: Obtain the video frames to be processed from the video sequence.
[0044] In step S202 above, the video frame to be processed is a frame selected or read from a series of consecutive video frames (i.e., video sequence) for specific processing or analysis during the video processing or analysis process.
[0045] Step S204: If the video frame to be processed is not the first frame in the video sequence, determine the key indicator value of the shot switching of the video frame to be processed.
[0046] In step S204 above, determining the key indicators for shot switching involves analyzing the changes between consecutive video frames to determine whether a shot switch has occurred. Key indicators for shot switching include, but are not limited to, tracking point loss rate, degree of positional abruptness (or positional abruptness), and variance of motion direction.
[0047] Step S206: Determine whether the video frame to be processed is a transition frame based on the key indicator value of the shot switching, and obtain the first judgment result.
[0048] In step S206 above, when the key indicators for shot switching include tracking point loss rate, position change degree (or position change amount) and motion direction variance, a first judgment result is obtained based on the relationship between the above three indicators and the corresponding threshold. For example, if any one of the above three indicators exceeds the corresponding threshold, then the frame (the video frame to be processed) is a shot transition frame.
[0049] Step S208: If the first judgment result indicates that the video frame to be processed is a shot transition frame, extract and output the video frame to be processed.
[0050] In step S208 above, when the video frame to be processed is a shot transition frame, it is necessary to extract the video frame to be processed. The purpose of video frame extraction is to ensure that subsequent processing stages such as video analysis or video encoding can obtain representative frames containing different scene or perspective changes.
[0051] Through steps S202 to S208 described above, the goal of accurately capturing the moment of shot transition in the video is achieved, thereby improving the accuracy and efficiency of subsequent image analysis. This solves the technical problem in related technologies where video frames are extracted at fixed time intervals, resulting in poor image analysis performance. The following is a further explanation.
[0052] In some embodiments of this application, the symbols used below are defined as follows:
[0053] set up and +1 For two consecutive video frames, t = { ,1 , ,2 , ..., ,n}for The set of feature points detected in the middle, ,i = ( ,i , ,i ) indicates the first Each feature point in coordinates in t t In order to be in The subset of feature points successfully tracked in the middle t = | t | indicates the number of feature points successfully tracked.
[0054] In step S202 of the above video frame extraction method, after obtaining the video frame to be processed in the video sequence, the method further includes: extracting and outputting the video frame to be processed when the video frame to be processed is the first frame in the video sequence.
[0055] In some embodiments of this application, the first frame of the video sequence serves as a starting point, providing a benchmark for subsequent inter-frame comparisons. In scenarios such as optical flow analysis and shot transition detection, the data from the first frame is used to initialize tracking points and motion models, providing a reference for the analysis of subsequent frames. Each frame in the video sequence carries visual information at a specific moment. Processing video frames sequentially starting from the first frame ensures the continuity of the analysis and the accuracy of the results. Therefore, regardless of whether a video frame meets the criteria for shot transition or the intensity of motion, if the video frame to be processed is the first frame in the video sequence, it is directly extracted and output because the first frame carries all the information from the beginning of the video, and its importance can be determined without complex optical flow analysis. If the first frame is not extracted, it will affect the understanding of the video content and the contextual relevance, resulting in the loss of crucial starting information in the analysis process.
[0056] In step S204 of the above video frame extraction method, determining the key indicator value of the shot switching of the video frame to be processed includes: obtaining the previous video frame of the video frame to be processed, and obtaining the number of tracking feature points in the previous video frame, and the number of feature points successfully tracked in the video frame to be processed; determining the tracking point loss rate of the video frame to be processed based on the number of tracking feature points in the previous video frame and the number of feature points successfully tracked in the video frame to be processed; obtaining the initial frame in the video sequence, and obtaining the first position of the i-th feature point in the subset of feature points successfully tracked in the initial frame and the second position of the i-th feature point in the subset of feature points successfully tracked in the video frame to be processed, where i is a positive integer. The following steps are taken: First, determine the positional abrupt change of the video frame to be processed based on the first position, the second position, and the number of feature points successfully tracked in the video frame. This positional abrupt change is used to determine whether a shot change has occurred. Second, obtain the motion direction angle of the i-th feature point in the successfully tracked feature point subset of the video frame to be processed, and the mean motion direction of all feature points in the successfully tracked feature point subset. Third, determine the motion direction variance of the video frame to be processed based on the motion direction angle, the mean motion direction, and the number of feature points successfully tracked in the video frame. Finally, determine the key indicator value for shot change in the video frame to be processed based on the tracking point loss rate, the positional abrupt change, and the motion direction variance.
[0057] In some embodiments of this application, feature points are salient points in an image, which may be corner points, edge points, or other unique points. Feature points serve as reference points for motion in optical flow analysis.
[0058] The mathematical model for calculating whether a video frame is a shot transition frame (or shot switching frame) using optical flow needs to determine the following three key shot switching metrics:
[0059] 1. Tracking point loss rate (Lost Ratio):
[0060] The mathematical model or formula is expressed as follows:
[0061] (1)
[0062] in, This indicates the number of feature points successfully tracked in the previous video frame of the video frame to be processed. This indicates the number of feature points successfully tracked in the current frame (i.e., the video frame to be processed). This indicates the tracking point loss rate.
[0063] 2. Position Mutation:
[0064] The mathematical model or formula is expressed as follows:
[0065] (2)
[0066] in, This represents the first position of the i-th point in the subset of feature points successfully tracked in the initial frame of the video sequence. This indicates the second position of the i-th feature point in the subset of feature points successfully tracked in the current frame (i.e., the video frame to be processed). For Euclidean distance, This indicates the amount of positional mutation.
[0067] 3. Variance of direction of motion:
[0068] The mathematical model or formula is expressed as follows:
[0069] (3)
[0070] Formula (3) above represents the average of the squares of the minimum angular difference, where, This indicates the number of feature points successfully tracked in the video frame to be processed. This represents the motion direction angle (in radians) of the i-th feature point in the subset of feature points successfully tracked in the video frame to be processed. This represents the mean (in radians) of the motion directions of all feature points in the successfully tracked feature point subset of the video frame to be processed. `min(.)` indicates taking the minimum angular difference on the circumference (e.g., the actual angular difference between 10° and 350° is 20°, not 340°). `|.|` represents the absolute value. This represents the variance of the direction of motion.
[0071] Based on the above formulas (1) to (3), the tracking point loss rate, position change amount and motion direction variance are obtained respectively, thereby determining the key index values of the lens switching of the video frame to be processed.
[0072] In step S206 of the above video frame extraction method, determining whether the video frame to be processed is a transitional frame based on the key indicator value of the shot switching includes: determining that the video frame to be processed is a transitional frame when the tracking point loss rate is greater than a first threshold, or the position change amount is greater than a second threshold, or the motion direction variance value is greater than a third threshold.
[0073] In some embodiments of this application, the criteria for determining the tracking point loss rate are as follows:
[0074] (4)
[0075] in, The first threshold mentioned above is used to detect large-scale tracking point loss. The physical meaning of formula (4) is: when tracking of points exceeding the set first threshold fails, a camera switch is triggered. First threshold For surveillance video scenarios, it can be set to 0.7.
[0076] The criteria for determining the amount of positional mutation are as follows:
[0077] (5)
[0078] in, The second threshold mentioned above is used to detect sudden changes in position. The physical meaning of formula (5) is: when the average Euclidean distance between the tracking point and the feature point in the initial frame exceeds the second threshold. When this occurs, the camera angle is switched. For typical surveillance videos, the second threshold can be set to 20 (unit: pixels).
[0079] The criteria for determining the variance of the direction of motion are as follows:
[0080] (6)
[0081] in, The third threshold mentioned above is used to detect sudden changes in motion patterns. The physical meaning of formula (6) is: when the variance of the motion direction exceeds the third threshold... (Unit: radians) 2 When the consistency of the motion direction is disrupted (i.e., the variance exceeds the third threshold), a camera switch is triggered.
[0082] Low variance ): All points move in the same direction and height (such as translational movement), for example: →→→→→→→→.
[0083] High variance ( ): Movement direction diverges (such as camera transitions or complex movements). For example: →→←→↑↓←→.
[0084] Third threshold recommendation:
[0085] Typical value range: Calibration formula: ,in For example, the average angle deviation of surveillance videos can be set to... , .
[0086] The final shot transition condition (the condition for determining the transition frame) is a logical OR combination of the three factors, expressed by the following formula:
[0087] (7)
[0088] Formula (7) indicates that if any of the three key indicators for lens switching exceeds the corresponding threshold, then this frame (the video frame to be processed) is determined to be a lens transition frame.
[0089] In the aforementioned video frame extraction method, the method further includes: when the first judgment result indicates that the video frame to be processed is not a transition frame, obtaining the first coordinates of the first feature point in the feature point subset successfully tracked by the video frame to be processed, and obtaining the second coordinates of the next feature point in the set of all feature points of the video frame to be processed, wherein the first feature point is any feature point in the feature point subset successfully tracked by the video frame to be processed; determining the vector amplitude of the optical flow vector of the first feature point based on the difference between the second coordinates and the first coordinates; obtaining the vector amplitudes of all feature points in the feature point subset successfully tracked by the video frame to be processed, to obtain a set of vector amplitudes; determining the average amplitude corresponding to the set of vector amplitudes, wherein the average amplitude is used to represent the average displacement of the feature point between the video frame to be processed and the next video frame of the video frame to be processed.
[0090] In some embodiments of this application, the formal representation of the mathematical model for calculating the motion intensity of video frames using the optical flow method is as follows:
[0091] 1. Calculation of optical flow vector magnitude:
[0092] For each successfully tracked feature point ,i ∈ t Its optical flow vector is:
[0093]
[0094] The vector magnitude (length) is:
[0095] (8)
[0096] in, This indicates the first feature point in the feature point subset successfully tracked in the video frame to be processed. This represents the first coordinate of the first feature point. This indicates the next feature point in the set of all feature points of the first feature point in the video frame to be processed. This represents the second coordinate.
[0097] 2. Average amplitude (overall exercise intensity):
[0098] (9)
[0099] in, This represents the number of all feature points in the subset of feature points successfully tracked in the video frame to be processed. The above average amplitude is represented. The physical meaning of formula (9) is that the feature point represents the average displacement between the video frame to be processed and the next video frame, reflecting the overall motion speed. The larger the average amplitude value, the more intense the overall motion.
[0100] In the above steps, after determining the average amplitude corresponding to the vector amplitude set, the method further includes: if the average amplitude is greater than a first amplitude threshold, determining the motion intensity of the video frame to be processed as a first level; if the average amplitude is less than or equal to the first amplitude threshold and greater than a second amplitude threshold, determining the motion intensity of the video frame to be processed as a second level, wherein the second level is lower than the first level; if the average amplitude is less than or equal to the second amplitude threshold and greater than a third amplitude threshold, determining the motion intensity of the video frame to be processed as a third level, wherein the third level is lower than the second level; if the average amplitude is less than or equal to the third amplitude threshold, determining the motion intensity of the video frame to be processed as a fourth level, wherein the fourth level is lower than the third level.
[0101] In some embodiments of this application, based on average amplitude The threshold is used to classify the motion intensity of each video frame, as shown in the following formula:
[0102] (10)
[0103] in, Indicates the first amplitude threshold. This indicates the second amplitude threshold. This indicates the third amplitude threshold. (Very fast) indicates the first level. (Fast) indicates the second level. (Medium) indicates the third level. (Slow) indicates the fourth level. This is an empirical threshold based on the actual video content. For example, for surveillance videos, the threshold can be set as follows: .
[0104] In the above video frame extraction method, the method further includes: obtaining the video type of the video sequence; determining the target motion intensity threshold based on the video type, wherein the target motion intensity threshold is any one of the first amplitude threshold, the second amplitude threshold, and the third amplitude threshold; and extracting and outputting the video frames to be processed when the average amplitude is greater than or equal to the target motion intensity threshold.
[0105] In some embodiments of this application, it is necessary to identify the video type of the currently processed video sequence. Video types include, but are not limited to, security surveillance videos, live sports broadcasts, movies, animations, news reports, etc. Each video type has different motion characteristics; for example, surveillance videos have less motion while sports videos have more intense motion. Based on the video type, a target motion intensity threshold is determined, which serves as the standard for judging the motion intensity of video frames. The selection of the threshold needs to consider the average motion state of the video content. For example, for surveillance videos with less motion, the threshold is set lower; for sports videos with intense motion, the threshold is set higher. By calculating the relationship between the average amplitude of the video frames to be processed and the target motion intensity threshold, it is determined whether to extract and output the video frames to be processed. For example, if the average amplitude is greater than or equal to the target motion intensity threshold, the video frames to be processed are extracted and output. This method can intelligently adjust the frame extraction standard according to the specific type of the video, ensuring that the frames extracted from the video sequence accurately reflect the motion details of the video, without losing important information or generating redundant data due to overly general or inapplicable frame extraction strategies.
[0106] To better understand the video frame extraction method in this application, the following is combined with... Figure 3 The video frame extraction process will be explained. Figure 3 This is a logic flowchart of video frame extraction according to an embodiment of this application. Specifically, it includes the following processes:
[0107] 1. Initialize and start the video frame extraction process.
[0108] 2. Are there any frames in the video sequence (or queue) that need to be processed?
[0109] 3. If there are no more frames in the video sequence that need to be processed, then the process ends.
[0110] 4. If there are video frames in the video sequence that need to be processed, then read a frame from the video sequence (or queue).
[0111] 5. If this frame is the first frame in the sequence, extract and output this frame.
[0112] 6. If it is not the first frame, use formulas (1)(2)(3) to calculate the key indicator values for the three shot switching in this frame.
[0113] 7. Based on the three key indicator values for shot switching, use formula (7) to determine whether the current frame is a shot transition frame. If it is, extract and output the current frame.
[0114] 8. If it is not a transition frame, use formulas (8) and (9) to calculate the average amplitude value (or optical flow value) of the current frame.
[0115] 9. Use formula (10) to determine whether the average amplitude value of this frame reaches or exceeds the preset threshold of motion intensity (such as the target motion intensity threshold). If it reaches or exceeds the preset threshold of motion intensity, extract and output this frame.
[0116] 10. If the preset threshold for exercise intensity is not reached or exceeded, return to step 2 to continue processing.
[0117] The video frame extraction method provided in this application analyzes the input video frames using optical flow. It extracts frames of rapid motion and outputs them, while not extracting frames of less motion. It directly extracts and outputs frames of shot transitions (or shot transition frames). This solves the problem of traditional fixed-time-interval frame extraction methods losing motion details in video images, ensuring that the output extracted frames reflect the motion details of the video. The video frame extraction method in this application can be used for pre-processing in machine vision AI analysis to optimize AI analysis results, and can also be used to preserve motion details during video compression.
[0118] Figure 4 This is a structural diagram of a video frame extraction device according to an embodiment of this application, such as... Figure 4 As shown, the device includes:
[0119] Acquisition module 40 is used to acquire video frames to be processed in the video sequence;
[0120] The determination module 42 is used to determine the key indicator value of the shot switching of the video frame to be processed when the video frame to be processed is not the first frame in the video sequence.
[0121] The judgment module 44 is used to determine whether the video frame to be processed is a transition frame based on the key indicator value of the shot switching, and to obtain the first judgment result.
[0122] Extraction module 46 is used to extract and output the video frame to be processed when the first judgment result indicates that the video frame to be processed is a shot transition frame.
[0123] By using the acquisition module, determination module, judgment module, and extraction module in the aforementioned video frame extraction device, the goal of accurately capturing the moment of shot switching in the video is achieved, thereby improving the accuracy and efficiency of subsequent image analysis. This solves the technical problem in related technologies where video frames are extracted at fixed time intervals, resulting in poor image analysis performance.
[0124] In the acquisition module of the aforementioned video frame extraction device, the acquisition module is further used to extract and output the video frame to be processed when the video frame to be processed is the first frame in the video sequence.
[0125] In the determining module of the aforementioned video frame extraction device, the determining module is further configured to acquire the previous video frame of the video frame to be processed, and acquire the number of tracking feature points in the previous video frame, as well as the number of feature points successfully tracked in the video frame to be processed; determine the tracking point loss rate of the video frame to be processed based on the number of tracking feature points in the previous video frame and the number of feature points successfully tracked in the video frame to be processed; acquire the initial frame in the video sequence, and acquire the first position of the i-th feature point in the subset of feature points successfully tracked in the initial frame and the second position of the i-th feature point in the subset of feature points successfully tracked in the video frame to be processed, where i is a positive integer; and determine the tracking point loss rate of the video frame to be processed based on the first position. The system calculates the positional abrupt change of the video frame to be processed based on the second position and the number of feature points successfully tracked in the video frame. The positional abrupt change is used to determine whether a shot change has occurred. It also obtains the motion direction angle of the i-th feature point in the subset of successfully tracked feature points in the video frame, and the mean motion direction of all feature points in the subset. Based on the motion direction angle, the mean motion direction, and the number of feature points successfully tracked in the video frame, it determines the motion direction variance of the video frame. Finally, based on the tracking point loss rate, the positional abrupt change, and the motion direction variance, it determines the key indicator value for shot change in the video frame.
[0126] In the judgment module of the aforementioned video frame extraction device, the judgment module is also used to determine that the video frame to be processed is a lens transition frame when the tracking point loss rate is greater than a first threshold, or the position change amount is greater than a second threshold, or the motion direction variance value is greater than a third threshold.
[0127] The aforementioned video frame extraction device further includes a processing module 48, which is further configured to: obtain the first coordinates of a first feature point in the feature point subset successfully tracked by the video frame to be processed, and obtain the second coordinates of the next feature point in the set of all feature points of the video frame to be processed, when the first judgment result indicates that the video frame to be processed is not a transition frame; determine the vector amplitude of the optical flow vector of the first feature point based on the difference between the second coordinates and the first coordinates; obtain the vector amplitudes of all feature points in the feature point subset successfully tracked by the video frame to be processed, thereby obtaining a set of vector amplitudes; and determine the average amplitude corresponding to the set of vector amplitudes, wherein the average amplitude is used to represent the average displacement of the feature point between the video frame to be processed and the next video frame of the video frame to be processed.
[0128] In the processing module of the aforementioned video frame extraction device, the processing module is further configured to: determine the motion intensity of the video frame to be processed as a first level when the average amplitude is greater than a first amplitude threshold; determine the motion intensity of the video frame to be processed as a second level when the average amplitude is less than or equal to the first amplitude threshold and greater than a second amplitude threshold, wherein the second level is lower than the first level; determine the motion intensity of the video frame to be processed as a third level when the average amplitude is less than or equal to the second amplitude threshold and greater than a third amplitude threshold, wherein the third level is lower than the second level; and determine the motion intensity of the video frame to be processed as a fourth level when the average amplitude is less than or equal to the third amplitude threshold, wherein the fourth level is lower than the third level.
[0129] In the processing module of the aforementioned video frame extraction device, the processing module is further used to obtain the video type of the video sequence; determine the target motion intensity threshold based on the video type, wherein the target motion intensity threshold is any one of the first amplitude threshold, the second amplitude threshold, and the third amplitude threshold; and extract and output the video frames to be processed when the average amplitude is greater than or equal to the target motion intensity threshold.
[0130] It should be noted that, Figure 4 The video frame extraction device shown is used to perform Figure 2 The video frame extraction method shown above is also applicable to this video frame extraction device, and will not be repeated here.
[0131] This application also provides an electronic device, which includes a memory and a processor. The memory is used to store program instructions. The processor is connected to the memory and is used to execute program instructions that perform the following functions: acquiring a video frame to be processed in a video sequence; determining a key indicator value for shot transition of the video frame to be processed when the video frame to be processed is not the first frame in the video sequence; determining whether the video frame to be processed is a shot transition frame based on the key indicator value for shot transition, and obtaining a first determination result; and extracting and outputting the video frame to be processed when the first determination result indicates that the video frame to be processed is a shot transition frame.
[0132] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The video frame extraction method shown above is also applicable to this electronic device, and will not be repeated here.
[0133] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following video frame extraction method by running the computer program: acquiring a video frame to be processed in a video sequence; determining a key shot transition index value for the video frame to be processed if the video frame to be processed is not the first frame in the video sequence; determining whether the video frame to be processed is a shot transition frame based on the key shot transition index value, and obtaining a first determination result; and extracting and outputting the video frame to be processed if the first determination result indicates that the video frame to be processed is a shot transition frame.
[0134] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The video frame extraction method shown above is also applicable to this non-volatile storage medium, and will not be repeated here.
[0135] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the video frame extraction method in various embodiments of this application.
[0136] This application also provides a computer program that, when executed by a processor, implements the steps of the video frame extraction method in various embodiments of this application.
[0137] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0138] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0139] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0141] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0142] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0143] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A video frame extraction method, characterized in that, include: Retrieve the video frames to be processed from the video sequence; If the video frame to be processed is not the first frame in the video sequence, determine the key indicator value of the shot switching of the video frame to be processed. Based on the key indicator value of the shot switching, determine whether the video frame to be processed is a shot transition frame, and obtain the first judgment result; If the first determination result indicates that the video frame to be processed is the transition frame of the shot, the video frame to be processed is extracted and output.
2. The method according to claim 1, characterized in that, After obtaining the video frames to be processed from the video sequence, the method further includes: If the video frame to be processed is the first frame of the video sequence, the video frame to be processed is extracted and output.
3. The method according to claim 1, characterized in that, Determining the key image switching metrics for the video frame to be processed includes: Obtain the previous video frame of the video frame to be processed, and obtain the number of tracked feature points of the previous video frame, as well as the number of feature points successfully tracked in the video frame to be processed. Based on the number of tracked feature points in the previous video frame and the number of successfully tracked feature points in the video frame to be processed, the tracking point loss rate of the video frame to be processed is determined. Obtain the initial frame in the video sequence, and obtain the first position of the i-th feature point in the feature point subset successfully tracked in the initial frame and the second position of the i-th feature point in the feature point subset successfully tracked in the video frame to be processed, where i is a positive integer; Based on the first position, the second position, and the number of feature points successfully tracked in the video frame to be processed, the position change amount of the video frame to be processed is determined, wherein the position change amount is used to determine whether a shot change has occurred; Obtain the motion direction angle of the i-th feature point in the feature point subset successfully tracked in the video frame to be processed, and the average motion direction of all feature points in the feature point subset successfully tracked in the video frame to be processed; The motion direction variance of the video frame to be processed is determined based on the motion direction angle, the mean of the motion direction, and the number of feature points successfully tracked in the video frame to be processed. Based on the tracking point loss rate, the position change amount, and the motion direction variance, the key indicator values for shot switching of the video frame to be processed are determined.
4. The method according to claim 3, characterized in that, Determining whether the video frame to be processed is a transition frame based on the key indicator value of the shot switching includes: If the tracking point loss rate is greater than a first threshold, or the position change is greater than a second threshold, or the motion direction variance is greater than a third threshold, the video frame to be processed is determined to be a shot transition frame.
5. The method according to claim 1, characterized in that, The method further includes: If the first judgment result indicates that the video frame to be processed is not the lens transition frame, the first coordinate of the first feature point in the feature point subset successfully tracked by the video frame to be processed is obtained, and the second coordinate of the next feature point of the first feature point in the set of all feature points of the video frame to be processed is obtained, wherein the first feature point is any feature point in the feature point subset successfully tracked by the video frame to be processed. The magnitude of the optical flow vector of the first feature point is determined based on the difference between the second coordinate and the first coordinate. Obtain the vector magnitudes of all feature points in the successfully tracked feature point subset of the video frame to be processed, and obtain the vector magnitude set. Determine the average amplitude corresponding to the vector amplitude set, wherein the average amplitude is used to represent the average displacement of the feature points between the video frame to be processed and the next video frame of the video frame to be processed.
6. The method according to claim 5, characterized in that, After determining the average magnitude corresponding to the set of vector magnitudes, the method further includes: If the average amplitude is greater than the first amplitude threshold, the motion intensity of the video frame to be processed is determined to be at the first level. If the average amplitude is less than or equal to the first amplitude threshold and greater than the second amplitude threshold, the motion intensity of the video frame to be processed is determined to be a second level, wherein the second level is lower than the first level; If the average amplitude is less than or equal to the second amplitude threshold and greater than the third amplitude threshold, the motion intensity of the video frame to be processed is determined to be a third level, wherein the third level is lower than the second level; If the average amplitude is less than or equal to the third amplitude threshold, the motion intensity of the video frame to be processed is determined to be a fourth level, wherein the fourth level is lower than the third level.
7. The method according to claim 6, characterized in that, The method further includes: Obtain the video type of the video sequence; A threshold for the intensity of target motion is determined based on the video type, wherein the threshold for the intensity of target motion is any one of the first amplitude threshold, the second amplitude threshold, and the third amplitude threshold; If the average amplitude is greater than or equal to the target motion intensity threshold, the video frame to be processed is extracted and output.
8. A video frame extraction device, characterized in that, include: The acquisition module is used to acquire the video frames to be processed in the video sequence; The determination module is used to determine the key indicator value of the shot switching of the video frame to be processed when the video frame to be processed is not the first frame in the video sequence. The judgment module is used to determine whether the video frame to be processed is a transition frame based on the key indicator value of the shot switching, and to obtain a first judgment result; An extraction module is used to extract and output the video frame to be processed when the first judgment result indicates that the video frame to be processed is the transition frame of the shot.
9. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor, connected to the memory, is configured to execute program instructions to perform the following functions: acquire a video frame to be processed from a video sequence; if the video frame to be processed is not the first frame in the video sequence, determine a key indicator value for shot transition of the video frame to be processed; determine whether the video frame to be processed is a shot transition frame based on the key indicator value for shot transition, and obtain a first determination result; if the first determination result indicates that the video frame to be processed is the shot transition frame, extract and output the video frame to be processed.
10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the video frame extraction method according to any one of claims 1 to 7 by running the computer program.
11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the video frame extraction method according to any one of claims 1 to 7.