Deep learning-based target key frame intelligent clipping method and system

Abstract

The application belongs to the technical field of video analysis and artificial intelligence, and particularly relates to a target key frame intelligent clipping method and system based on deep learning, which comprises the following steps: calculating a corresponding fluctuation index for each frame image in a video sequence to be clipped; obtaining at least one candidate frame image according to the fluctuation index; for any candidate frame image, performing spatial semantic analysis on a context pixel sequence using a preset lightweight convolutional network model according to pixel information of the candidate frame image and pixel information of front and rear frame images of the candidate frame image, obtaining a spatial semantic vector, calculating a feature saliency score of the candidate frame image according to a modulation vector, and traversing to obtain the feature saliency score of each candidate frame image; screening the at least one candidate frame image according to the feature saliency score, obtaining a target key frame, and completing video clipping according to the target key frame. The application is helpful to improve the clipping efficiency.

Description

Technical Field

[0001] This invention relates to the fields of video analysis and artificial intelligence. More specifically, this invention relates to a method and system for intelligent editing of target keyframes based on deep learning. Background Technology

[0002] With the development of the Internet, video analytics technology has been widely applied in many fields such as short video review and urban security monitoring. In practical applications, edge devices (i.e., hardware devices deployed at the network edge, close to the video data source, with certain computing, storage, and network communication capabilities) are typically used to perform deep neural network processing on the video stream to identify video frames containing target actions as target keyframes.

[0003] However, edge devices generally suffer from limited computing power and GPU memory. Meanwhile, video streams contain numerous static or slowly changing frames, which contribute almost nothing to the identification of keyframes. Furthermore, conventional deep neural networks are complex and computationally expensive. If all frames in the video stream were fed into a conventional deep learning network for processing, the GPU memory and computing power of the edge device would be quickly exhausted, severely impacting the identification of keyframes and consequently affecting editing efficiency.

[0004] Therefore, how to improve editing efficiency under the constraints of limited resources on edge devices has become an urgent technical problem to be solved. Summary of the Invention

[0005] In the existing technology, how to improve editing efficiency under the constraint of limited resources of edge devices has become an urgent technical problem to be solved. To solve this problem, the present invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention provides a target keyframe intelligent editing method based on deep learning, comprising:

[0007] Obtain the video sequence to be edited;

[0008] For each frame in the video sequence to be edited, the corresponding fluctuation index is calculated based on the grayscale gradient change between the image and its neighboring frames.

[0009] Based on the fluctuation index, all frame images in the video sequence to be edited are filtered to obtain at least one candidate frame image;

[0010] For any candidate frame image, a context pixel sequence is constructed based on the pixel information of the candidate frame image and the pixel information of the frames before and after the candidate frame image. A preset lightweight convolutional network model is used to perform spatial semantic analysis on the context pixel sequence to obtain a spatial semantic vector. The spatial semantic vector and the fluctuation index corresponding to the candidate frame image are fused to obtain a modulation vector. The feature saliency score of the candidate frame image is calculated based on the modulation vector. The feature saliency score of each candidate frame image is obtained by traversing.

[0011] The at least one candidate frame image is filtered based on the feature saliency score to obtain the target keyframe, and the video editing is completed based on the target keyframe.

[0012] Secondly, the present invention provides a target keyframe intelligent editing system based on deep learning, comprising:

[0013] The acquisition module is used to acquire the video sequence to be edited;

[0014] The fluctuation calculation module is used to calculate the corresponding fluctuation index for each frame of the video sequence to be edited, based on the grayscale gradient change between the image and its adjacent frames.

[0015] The filtering module is used to filter all frame images in the video sequence to be edited according to the fluctuation index to obtain at least one candidate frame image;

[0016] The saliency calculation module is used to construct a context pixel sequence for any candidate frame image based on the pixel information of the candidate frame image and the pixel information of the frames before and after the candidate frame image, perform spatial semantic analysis on the context pixel sequence using a preset lightweight convolutional network model to obtain a spatial semantic vector, perform feature fusion on the spatial semantic vector and the fluctuation index corresponding to the candidate frame image to obtain a modulation vector, calculate the feature saliency score of the candidate frame image based on the modulation vector, and iterate to obtain the feature saliency score of each candidate frame image.

[0017] The target keyframe determination module is used to filter the at least one candidate frame image according to the feature saliency score to obtain the target keyframe, and to complete the video editing based on the target keyframe.

[0018] The advantages of this invention over existing technologies are as follows:

[0019] This invention first calculates a fluctuation index based on the grayscale gradient change of each frame relative to its adjacent frames. This fluctuation index is used to initially screen all frames in the video sequence to be edited, obtaining at least one candidate frame image with greater depth processing value. Next, a context pixel sequence of the candidate frame image is constructed based on its context pixel information. A pre-defined lightweight convolutional network model is then used to perform spatial semantic analysis on the context pixel sequence, yielding a spatial semantic vector. Then, the fluctuation index, representing the grayscale gradient change characteristics, and the spatial semantic vector, representing the pixel spatial semantic information, are fused to obtain a modulation vector. This modulation vector is then used to calculate the feature saliency score of the candidate frame image. Finally, the target keyframe is selected based on the feature saliency score. Compared to existing technologies that feed all frames in the video stream into a conventional deep learning network for processing, this invention's two-stage screening, construction of the context pixel sequence, and application of a lightweight convolutional network model can reduce the consumption of GPU memory and computing power on edge devices, improving editing efficiency. Attached Figure Description

[0020] Figure 1 This is a flowchart of the steps of a deep learning-based intelligent editing method for target keyframes provided by the present invention;

[0021] Figure 2 This is a schematic diagram of the structure of a target keyframe intelligent editing system based on deep learning provided by the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0024] S1: Obtain the video sequence to be edited.

[0025] It should be further explained that the sequence to be edited is the raw video data stream or video file from which target keyframes need to be extracted. It can be a live stream captured in real time by a camera, or a locally stored historical video file. This sequence consists of multiple consecutive frames arranged in chronological order.

[0026] S2: For each frame in the video sequence to be edited, calculate the corresponding fluctuation index based on the grayscale gradient change between the image and its neighboring frames.

[0027] Optionally, for each frame in the video sequence to be edited, a fluctuation index is calculated based on the grayscale gradient change between the image and its neighboring frames, including:

[0028] Take any frame from the video sequence to be edited as the target frame image;

[0029] The gray-level gradient matrix of the target frame image and the gray-level gradient matrix of the adjacent previous frame image are calculated using the edge detection operator.

[0030] For any pixel in the gray-level gradient matrix of the target frame image, the gray-level absolute difference is calculated by combining the gray-level gradient value of the corresponding pixel in the gray-level gradient matrix of the adjacent previous frame image. The gray-level absolute difference is obtained by traversing the gray-level gradient matrix of the target frame image for each pixel. The average value of all gray-level absolute differences is used as the fluctuation index of the target frame image.

[0031] Iterate through the video sequence to obtain the fluctuation index of each frame.

[0032] It should be noted that the gray-level gradient matrix of the target frame image is obtained using the edge detection operator. grayscale gradient matrix of the adjacent previous frame image ,in, The acquisition time of the target frame image. The acquisition time interval between the target frame image and the adjacent previous frame image.

[0033] For example, the formula for calculating the fluctuation index of the target frame image is as follows:

[0034]

[0035] In the above formula (1), The fluctuation index of the target frame image; The total number of pixels in the target frame image. The pixel width of the target frame image. The pixel height of the target frame image. , The coordinates in the target frame image are The grayscale gradient value of the pixel. Represents the coordinates in the adjacent previous frame image. The grayscale gradient value of the pixel.

[0036] In the above formula (1), The coordinates in the target frame image are The absolute difference in grayscale values ​​of the pixels.

[0037] In the above formula (1), It is the average of all absolute differences in gray levels in the target frame image.

[0038] It should be noted that during the video stream input process, if the scene is basically static between adjacent image frames, the edge distribution and gray-scale gradient structure of objects within the frame are highly similar in the time direction; however, when there is target movement, shot switching, or sudden change in lighting within the frame, the gray-scale gradient amplitude of local pixels will change significantly.

[0039] It is worth noting that the volatility index is used to quantify the degree of change in image content between the current frame and its predecessor. The larger the value, the more obvious the change in grayscale gradient structure between adjacent frames, and the more likely there are dynamic events in the picture, such as target movement, camera switching, or sudden changes in lighting; the smaller the value, the more static the picture or the extremely slow the change.

[0040] It should be further explained that the fluctuation index of the first frame in the video sequence to be edited is preset to a fixed threshold (for example, set to 1.0, or set to a value greater than the preset fluctuation threshold) so that it will definitely be selected as a candidate frame image.

[0041] S3: Filter all frame images in the video sequence to be edited according to the fluctuation index to obtain at least one candidate frame image.

[0042] Optionally, all frame images in the video sequence to be edited are filtered according to the fluctuation index to obtain at least one candidate frame image, including;

[0043] When the volatility index is less than the preset volatility threshold, the corresponding frame image is used as a static redundant frame.

[0044] When the volatility index is not less than the preset volatility threshold, the corresponding frame image is used as a candidate frame image.

[0045] It should be noted that when the fluctuation index is lower than the preset fluctuation threshold, it indicates that the content of the current frame has hardly changed in terms of gradient structure compared to the previous frame, and the scene is likely to be in a static or near-static state. At this time, the corresponding frame is judged as a static redundant frame, and its pixel data is directly discarded, only the timestamp is retained. At the same time, the original pixel matrix data block occupied by the frame is released or discarded from memory and is no longer transmitted to the lightweight convolutional network and subsequent processing modules.

[0046] Only frames whose fluctuation index reaches or exceeds the preset fluctuation threshold are retained as candidate frame images (the content of the candidate frame image shows obvious changes in grayscale gradient structure compared to the previous frame, such changes are usually caused by dynamic events such as target movement, camera switching or sudden changes in lighting), and enter the subsequent depth feature extraction process.

[0047] It should be noted that the preset fluctuation threshold is determined by the implementer through offline calibration based on the typical gradient change differences between the static background and the dynamic target in the target scene, and can also be adjusted according to the time redundancy of the actual video material.

[0048] S4: For any candidate frame image, a context pixel sequence is constructed based on the pixel information of any candidate frame image and the pixel information of the frames before and after any candidate frame image. A spatial semantic analysis is performed on the context pixel sequence using a preset lightweight convolutional network model to obtain a spatial semantic vector. The spatial semantic vector and the fluctuation index corresponding to the candidate frame image are fused to obtain a modulation vector. The feature saliency score of any candidate frame image is calculated based on the modulation vector. The feature saliency score of each candidate frame image is obtained by traversing through the frames.

[0049] Optionally, a context pixel sequence is constructed based on the pixel information of any candidate frame image and the pixel information of the frames preceding and following any candidate frame image, including:

[0050] Obtain the three-channel pixel matrix of any candidate frame image, the three-channel pixel matrix of the preceding frame image adjacent to any candidate frame image, and the three-channel pixel matrix of the following frame image adjacent to any candidate frame image, respectively.

[0051] Concatenate all three-channel pixel matrices to obtain the context pixel sequence.

[0052] It should be noted that the three channels refer to the red, green, and blue color channels, and the pixel values ​​of each channel have been pre-normalized to [value missing]. The interval. For any candidate frame image, its corresponding three-channel pixel matrix is ​​used... ,in The acquisition time corresponding to this candidate frame image is used to represent the three-channel pixel matrix of the preceding frame image. This indicates that the three-channel pixel matrix of the next adjacent frame image of the candidate frame image is represented by... The context pixel sequence corresponding to the candidate frame image is represented by... express, The overall shape is Its nine channels are, in order, the R, G, and B channels of the preceding frame, the R, G, and B channels of the candidate frame image, and the R, G, and B channels of the following frame.

[0053] Optionally, the lightweight convolutional network model includes at least one depthwise separable convolutional layer, a global average pooling layer, and an output fully connected layer. A pre-defined lightweight convolutional network model is used to perform spatial semantic analysis on the context pixel sequence to obtain a spatial semantic vector, including:

[0054] By combining the preset convolution stride parameters, at least one depth-separable convolutional layer is used to sequentially extract spatial features and expand the channel dimension of the context pixel sequence to obtain a multi-channel compressed feature map.

[0055] The average compression vector is obtained by calculating the pixel average value of the compressed feature map corresponding to each channel in the multi-channel compressed feature map through a global average pooling layer.

[0056] The average compressed vector is mapped to a preset dimension using the output fully connected layer to obtain a spatial semantic vector.

[0057] It should be noted that the lightweight convolutional network model uses a depthwise separable convolutional structure as the feature extraction backbone. The depthwise separable convolutional structure is a well-known technique and will not be elaborated further. This model has a small number of parameters and fast computation speed, making it suitable for deployment on edge devices.

[0058] More specifically, this application sets up three depthwise separable convolutional layers. Each depthwise separable convolutional layer includes channel-wise convolution and pointwise convolution. All depthwise separable convolutional layers have the same convolution stride parameter, for example, 2. By setting the convolution stride parameter, the context pixel sequence is spatially downsampled (i.e., spatial size compression). The context pixel sequence passes through the three depthwise separable convolutional layers in sequence to complete three downsampling processes, resulting in a multi-channel compressed feature map.

[0059] It should be noted that a global average pooling layer is used to transform the compressed feature map corresponding to each channel into a single numerical value to eliminate spatial location sensitivity. Finally, an output fully connected layer maps these channel values ​​to a value of length [missing information]. The vector, where For preset dimensions, dimensions The empirical value of 256 is used, but it can also be adjusted by the implementer based on the target complexity and feature expression requirements of the actual application scenario.

[0060] It should be noted that the collection time was... The spatial semantic vector corresponding to the candidate frame image is used This indicates that the forward propagation process of a lightweight convolutional network model is equivalent to using... As input, visual semantic information such as edges, textures, object parts, and spatial layout is parsed through layer-by-layer convolution, and then mapped through global pooling and fully connected layers to finally generate a compact deep feature representation. .

[0061] It should be noted that spatial semantic vectors It is a fixed-dimensional floating-point vector output by a lightweight convolutional network, used to compactly describe the spatial visual semantic features of the current candidate frame in its local temporal context, such as the object category, target pose, scene layout, and other information contained in the image.

[0062] Optionally, feature fusion is performed on the spatial semantic vector and the fluctuation index corresponding to the candidate frame image to obtain the modulation vector, including:

[0063] The modulation vector is obtained by multiplying the fluctuation index corresponding to the candidate frame image with the spatial semantic vector.

[0064] It should be further explained that feature fusion combines the fluctuation index (scalar) with the spatial semantic vector through scalar multiplication. Specifically, the fluctuation index is multiplied by each element of the semantic vector, so that the overall amplitude of the semantic vector reflects the strength of the temporal fluctuation. The modulation vector is the composite feature vector obtained after feature fusion, carrying both temporal fluctuation energy and spatial visual semantic information. The amplitude of the modulation vector is... Scaling: When the fluctuation index is large, the frame content changes drastically, and the amplitude of the modulation vector increases accordingly; conversely, it decreases, so that the overall amplitude of the output vector reflects the strength of the temporal fluctuation.

[0065] Optionally, a feature saliency score for any candidate frame image is calculated based on the modulation vector, including:

[0066] The modulation vector is mapped to a high-dimensional non-orthogonal space through a pre-defined high-dimensional fully connected layer to obtain a high-dimensional feature vector;

[0067] The high-dimensional feature vector is compressed to obtain the corresponding high-dimensional real value, and the normalized high-dimensional real value is used as the feature significance score.

[0068] It should be added that the feature saliency score is a scalar value in the interval (0,1) after the modulation vector is input into a high-dimensional fully connected layer and an output fully connected layer, and then processed by nonlinear mapping and Sigmoid activation. It represents the confidence (probability) that the current frame is the target keyframe.

[0069] The aforementioned high-dimensional fully connected layer and output fully connected layer can be understood as a pre-built fully connected network. This fully connected network has been trained offline using a labeled video action dataset. The network sequentially performs the mapping from the modulation vector to the high-dimensional feature vector, and then to the feature saliency score. Specifically:

[0070] First, the modulation vector passes through a high-dimensional fully connected layer, which contains the weight matrix. and bias vector ,in The dimension is , for dimensional vector, For the aforementioned preset dimensions, The empirical value of 512 is taken; it is dimensionless and can be adjusted by the implementer according to the requirements of feature decoupling.

[0071] The calculation for this layer is as follows:

[0072]

[0073] In the above formula (2), The time of data collection is The high-dimensional feature vector corresponding to the candidate frame image, It is a rectified linear unit activation function, that is, taking the value of each input element... Introducing nonlinear transformation This is the weight matrix of the high-dimensional fully connected layer. The time of data collection is The fluctuation index corresponding to the candidate frame image. The time of data collection is The spatial semantic vector corresponding to the candidate frame image. The time of data collection is The modulation vector corresponding to the candidate frame image, This is the bias vector for the high-dimensional fully connected layer.

[0074] Regarding the above formula (2), it should be noted that... The row vectors are usually in a non-orthogonal state after training, therefore The components of each dimension do not have orthogonal constraints. The linear coupling between temporal fluctuations and spatial semantics is transformed into a set of coordinate coefficients on a non-orthogonal basis, no longer subject to structural constraints under orthogonal space.

[0075] Next, the high-dimensional feature vector The input is fed into the output fully connected layer of a pre-built fully connected network, which contains a weight vector. and bias scalar ,in The dimension is ,Will The linear projection of the 3D feature is a real value. This real value is then passed through a sigmoid activation function. Compress to The interval is used to obtain the feature significance score. The calculation formula is:

[0076]

[0077] In the above formula (3), The collection time is The feature saliency score corresponding to the candidate frame image, It is a sigmoid activation function. This is the weight vector of the output fully connected layer. The collection time is The high-dimensional feature vector corresponding to the candidate frame image, This outputs the bias scalar of the fully connected layer.

[0078] Regarding the above formula (3), it should be noted that... Its output range is strictly 1 , dimensionless.

[0079] Regarding the above formula (3), it should be noted that... It is a scalar whose magnitude comprehensively reflects the joint saliency of the current frame in terms of temporal fluctuation and spatial semantics: the closer the value is to 1, the richer the dynamic information contained in the frame and the more consistent it is with the key frame criterion; the closer the value is to 0, the weaker the spatiotemporal features of the frame.

[0080] The following two specific examples illustrate the application of feature saliency scoring in this invention:

[0081] In a candidate frame, a car is turning. A lightweight convolutional network model extracts spatial semantic vectors containing semantics such as "vehicle," "wheel," and "road" from its context pixel sequence. Simultaneously, the frame has a high volatility index (0.32). After fusing the two and performing saliency calculations, the resulting feature saliency score is 0.87, indicating that this frame is highly likely to be the target keyframe.

[0082] Although the other candidate frame has a high fluctuation index (0.28), the image only shows leaves being blown by the wind. The semantic vectors extracted by the lightweight convolutional network are mainly meaningless targets such as "leaves" and "swaying". After fusion, the saliency score calculated by the network is only 0.31, which may not be identified as the target keyframe.

[0083] It should be noted that the offline training process of the pre-built fully connected network is as follows: a dataset containing target keyframes and static redundant frames is pre-collected and labeled. The samples labeled as target keyframes are assigned label 1, and the static redundant frames are assigned label 0.

[0084] For each frame in the dataset, its fluctuation index and spatial semantic vector are calculated, and the resulting modulation vector is used as the input to a pre-built fully connected network.

[0085] The forward computation of a pre-built fully connected network outputs a feature saliency score. The error between the label and the true label is calculated using the binary cross-entropy loss function, which is defined as follows:

[0086]

[0087] In the aforementioned binary cross-entropy loss function, The number of samples in the dataset is dimensionless. For the first The label of each sample, which can be 0 or 1, is dimensionless; For pre-built fully connected networks to the first The feature significance score of the predicted output for each sample, dimensionless; It is the natural logarithm; This represents the loss function value.

[0088] Calculated using the backpropagation algorithm about , , , The gradient is calculated, and the Adam optimizer is used to update these parameters. This process is repeated until the loss function value converges.

[0089] After training converges, all network parameters are fixed and used for inference computation in this step.

[0090] S5: Filter at least one candidate frame image based on the feature saliency score to obtain the target keyframe, and complete the video editing based on the target keyframe.

[0091] Optionally, at least one candidate frame image is filtered based on feature saliency scores to obtain target keyframes, and video editing is performed based on the target keyframes, including:

[0092] Collect real-time video memory usage and determine the dynamic scoring threshold based on the real-time video memory usage.

[0093] Candidate frame images with feature saliency scores greater than the score threshold are used as target keyframes.

[0094] Optionally, a dynamic scoring threshold can be determined by combining real-time video memory usage, including:

[0095] Get historical video memory usage;

[0096] The real-time memory usage rate and the historical memory usage rate are weighted and fused to obtain a smooth memory usage rate.

[0097] Calculate the product of smooth memory usage and preset adjustment coefficient, and use the product of the exponential function value corresponding to the product and the preset benchmark value as the backup threshold.

[0098] The upper limit is truncated on the backup threshold and the upper limit of the preset threshold to obtain the dynamic scoring threshold.

[0099] It should be noted that real-time video memory utilization refers to the ratio of the video memory capacity used by the edge device at the current moment to the total video memory capacity, expressed as a percentage or decimal (ranging from 0 to 1). This value reflects the current hardware resource scarcity and is the core basis for subsequent adaptive adjustment of the filtering intensity. In this invention, "current moment" refers to the moment when the video sequence to be edited is being processed. The system collects the raw value of the video memory utilization ratio at the current moment in real time through the hardware monitoring interface at the same time step as the video frame rate; that is, the system re-collects the video memory utilization ratio when processing each frame; and the system recalculates the dynamic scoring threshold based on the latest collected video memory utilization ratio when processing each frame. Therefore, each frame has a dynamic scoring threshold "specific to the processing moment of that frame".

[0100] Historical memory usage rate refers to the memory usage rate value collected at one or more sampling times prior to the current moment. In this invention, it specifically refers to the memory usage rate value at the previous moment.

[0101] For example, the collection time is The smoothed video memory usage corresponding to the candidate frame image is used It indicates that the collection time is The real-time video memory usage corresponding to the candidate frame image is used This indicates that its corresponding historical video memory usage rate is used express.

[0102] ,in, The smoothing coefficient is taken as an empirical value of 0.2. It is dimensionless and is a preset parameter, but it can also be adjusted by the implementer according to the typical spectral characteristics of memory fluctuations.

[0103] It should be noted that the weighted fusion uses an exponential moving average algorithm. Its core idea is to assign higher weights (such as 0.8) to historical data and lower weights (such as 0.2) to the current original value, thereby suppressing high-frequency spike fluctuations in video memory usage caused by instantaneous memory allocation / release operations.

[0104] For each candidate frame image, the corresponding smooth memory usage is obtained. Then, a nonlinear mapping function with exponential operation and upper limit truncation is used to calculate the dynamic scoring threshold. The formula is as follows:

[0105]

[0106] In the above formula (4), The time of data collection is The dynamic scoring threshold corresponding to the candidate frame image; Preset threshold upper limit; The preset baseline value is 0.55, which is an empirical value and is dimensionless. It is a preset parameter that corresponds to the basic screening threshold when the video memory usage rate is close to 0. It can also be adjusted by the implementer according to the overall requirements of the keyframe density of the target scene. The preset adjustment coefficient is taken as an empirical value of 1.2. It is dimensionless and is a preset parameter that determines the steepness of the threshold change with the video memory occupancy rate. The larger the value, the more sensitive the threshold will be to rise. It can also be adjusted by the implementer based on the total video memory capacity of the edge device and the processing performance margin. It is an exponential function with the natural constant e as its base. The time of data collection is The smoothed video memory usage corresponding to the candidate frame image; This is a backup threshold. This indicates that the smaller value between the backup threshold and the preset upper limit is used to perform hard upper limit truncation.

[0107] Regarding the above formula (4), it should be noted that when the smoothing memory usage increases, the dynamic scoring threshold rises rapidly, indirectly increasing the difficulty of keyframe admission, thereby automatically reducing the amount of data entering deep processing. The dynamic scoring threshold is determined in real time by the current hardware load status, reflecting the minimum requirement for the spatiotemporal salience of candidate frames under this resource constraint.

[0108] In video analytics scenarios, if the memory usage of edge devices continues to rise, it usually means that the inference engine is processing a large number of high-fluctuation frames, and the queue of data to be processed is backed up. If key frames are still filtered with a fixed threshold at this time, the output rate of key frames will not be automatically reduced, the demand for computing resources will continue to increase, which will exacerbate memory shortage and even trigger memory overflow or inference engine deadlock.

[0109] It should be further explained that this invention achieves precise closed-loop control of keyframe screening intensity and hardware load status by constructing an adaptive dynamic threshold feedback mechanism based on the real-time video memory occupancy rate of edge devices. First, an exponential moving average algorithm is used to perform temporal smoothing filtering on the original video memory occupancy rate, effectively suppressing instantaneous spikes and disturbances generated during video memory allocation and release, avoiding instability in screening results caused by frequent threshold jitter. Then, by non-linearly mapping the smoothed video memory occupancy rate to a dynamic scoring threshold, the system automatically relaxes screening conditions to retain more candidate frames when video memory resources are abundant, and automatically tightens screening conditions to control the inference task injection rate when video memory resources are scarce, thereby proactively avoiding video memory overflow and inference engine deadlock caused by a large influx of highly volatile frames during peak activity periods. This invention significantly improves the long-term operational stability and reliability of edge devices in resource-fluctuating scenarios while ensuring keyframe recall rate.

[0110] After obtaining at least one target keyframe, if the target keyframe is the first keyframe identified in the video sequence to be edited, the first clip boundary marker is generated using the timestamp of the start frame of the video sequence as the start time and the timestamp of the target keyframe as the end time. If the target keyframe is not the first keyframe, a new clip boundary marker is generated using the most recently recorded end time (i.e., the timestamp of the previous target keyframe) as the start time of the current clip and the timestamp of the target keyframe as the end time. The system maintains an ordered set of clip boundary markers in memory to record the start and end boundary information of all determined clips. This completes the editing of the video sequence to be edited.

[0111] In one embodiment, a deep learning-based intelligent editing system for target keyframes is provided, which corresponds one-to-one with the deep learning-based intelligent editing method for target keyframes described in the above embodiments. For example... Figure 2 As shown, this deep learning-based intelligent editing system for target keyframes includes a source data acquisition module 201, a fluctuation calculation module 202, a filtering module 203, a saliency calculation module 204, and a target keyframe determination module 205. Detailed descriptions of each functional module are as follows:

[0112] Module 201 is used to acquire the video sequence to be edited;

[0113] The fluctuation calculation module 202 is used to calculate the corresponding fluctuation index for each frame of the video sequence to be edited based on the grayscale gradient change between the image and its neighboring frames.

[0114] The filtering module 203 is used to filter all frame images in the video sequence to be edited according to the fluctuation index to obtain at least one candidate frame image;

[0115] The saliency calculation module 204 is used to construct a context pixel sequence for any candidate frame image based on the candidate frame image and the previous and next frame images of the candidate frame image, perform spatial semantic analysis on the context pixel sequence using a preset lightweight convolutional network model to obtain a spatial semantic vector, perform feature fusion on the spatial semantic vector and the fluctuation index corresponding to the candidate frame image to obtain a modulation vector, calculate the feature saliency score of any candidate frame image based on the modulation vector, and iterate to obtain the feature saliency score of each candidate frame image.

[0116] The target keyframe determination module 205 is used to filter at least one candidate frame image based on feature saliency score to obtain the target keyframe, and to complete video editing based on the target keyframe.

[0117] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and substitutions without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based intelligent editing method for target keyframes, characterized in that, include: Obtain the video sequence to be edited; For each frame in the video sequence to be edited, the corresponding fluctuation index is calculated based on the grayscale gradient change between the image and its neighboring frames. Based on the fluctuation index, all frame images in the video sequence to be edited are filtered to obtain at least one candidate frame image; For any candidate frame image, a context pixel sequence is constructed based on the pixel information of the candidate frame image and the pixel information of the frames before and after the candidate frame image. A preset lightweight convolutional network model is used to perform spatial semantic analysis on the context pixel sequence to obtain a spatial semantic vector. The spatial semantic vector and the fluctuation index corresponding to the candidate frame image are fused to obtain a modulation vector. The feature saliency score of the candidate frame image is calculated based on the modulation vector. The feature saliency score of each candidate frame image is obtained by traversing. The at least one candidate frame image is filtered based on the feature saliency score to obtain the target keyframe, and the video editing is completed based on the target keyframe.

2. The intelligent editing method for target keyframes based on deep learning according to claim 1, characterized in that, For each frame in the video sequence to be edited, the fluctuation index is calculated based on the grayscale gradient change between the image and its neighboring frames, including: Take any frame image from the video sequence to be edited as the target frame image; The gray-level gradient matrix of the target frame image and the gray-level gradient matrix of the adjacent previous frame image of the target frame image are calculated using the edge detection operator. For any pixel in the gray-level gradient matrix of the target frame image, the gray-level absolute difference is calculated by combining the gray-level gradient value of the corresponding pixel in the gray-level gradient matrix of the adjacent previous frame image. The gray-level absolute difference is obtained by traversing the gray-level gradient matrix of the target frame image and the average value of all the gray-level absolute differences is used as the fluctuation index of the target frame image. The fluctuation index of each frame in the video sequence to be edited is obtained by iterating through the sequence.

3. The intelligent editing method for target keyframes based on deep learning according to claim 1, characterized in that, The step of filtering all frame images in the video sequence to be edited according to the fluctuation index to obtain at least one candidate frame image includes: When the fluctuation index is less than the preset fluctuation threshold, the corresponding frame image is used as a static redundant frame. When the fluctuation index is not less than the preset fluctuation threshold, the corresponding frame image is used as a candidate frame image.

4. The intelligent editing method for target keyframes based on deep learning according to claim 1, characterized in that, The step of constructing a context pixel sequence based on the pixel information of any candidate frame image and the pixel information of the preceding and following frames of any candidate frame image includes: The three-channel pixel matrix of any candidate frame image, the three-channel pixel matrix of the preceding frame image adjacent to any candidate frame image, and the three-channel pixel matrix of the following frame image adjacent to any candidate frame image are obtained respectively. By concatenating all the aforementioned three-channel pixel matrices, the context pixel sequence is obtained.

5. The intelligent editing method for target keyframes based on deep learning according to claim 1, characterized in that, The lightweight convolutional network model includes at least one depthwise separable convolutional layer, a global average pooling layer, and an output fully connected layer. The spatial semantic analysis of the context pixel sequence using the preset lightweight convolutional network model to obtain a spatial semantic vector includes: By combining the preset convolution stride parameters, the spatial features of the context pixel sequence are extracted and the channel dimension is expanded sequentially using the at least one depth-separable convolutional layer to obtain a multi-channel compressed feature map. The average pixel value of the compressed feature map corresponding to each channel in the multi-channel compressed feature map is calculated by the global average pooling layer to obtain the average compression vector; The average compressed vector is mapped to a preset dimension using the output fully connected layer to obtain a spatial semantic vector.

6. The intelligent editing method for target keyframes based on deep learning according to claim 1, characterized in that, The step of fusing features between the spatial semantic vector and the fluctuation index corresponding to the candidate frame image to obtain a modulation vector includes: The modulation vector is obtained by multiplying the fluctuation index corresponding to the candidate frame image with the spatial semantic vector.

7. The intelligent editing method for target keyframes based on deep learning according to claim 1, characterized in that, The step of calculating the feature saliency score of any candidate frame image based on the modulation vector includes: The modulation vector is mapped to a high-dimensional non-orthogonal space through a preset high-dimensional fully connected layer to obtain a high-dimensional feature vector; The high-dimensional feature vector is compressed to obtain the corresponding high-dimensional real value, and the normalized high-dimensional real value is used as the feature significance score.

8. The intelligent editing method for target keyframes based on deep learning according to claim 1, characterized in that, The step of filtering the at least one candidate frame image based on the feature saliency score to obtain the target keyframe, and completing video editing based on the target keyframe, includes: Collect real-time video memory usage and determine the dynamic scoring threshold based on the real-time video memory usage. Candidate frame images with feature saliency scores greater than the score threshold are used as target keyframes.

9. The intelligent editing method for target keyframes based on deep learning according to any one of claims 1 to 8, characterized in that, The step of determining the dynamic scoring threshold by combining the real-time video memory usage rate includes: Get historical video memory usage; The real-time video memory utilization rate and the historical video memory utilization rate are weighted and fused to obtain a smooth video memory utilization rate; Calculate the product of the smoothed video memory usage rate and the preset adjustment coefficient, and use the product of the exponential function value corresponding to the product and the preset benchmark value as the standby threshold. The upper limit is truncated by the backup threshold and the preset threshold upper limit to obtain the dynamic scoring threshold.

10. A target keyframe intelligent editing system based on deep learning, characterized in that, include: The acquisition module is used to acquire the video sequence to be edited; The fluctuation calculation module is used to calculate the corresponding fluctuation index for each frame of the video sequence to be edited, based on the grayscale gradient change between the image and its adjacent frames. The filtering module is used to filter all frame images in the video sequence to be edited according to the fluctuation index to obtain at least one candidate frame image; The saliency calculation module is used to construct a context pixel sequence for any candidate frame image based on the pixel information of the candidate frame image and the pixel information of the frames before and after the candidate frame image, perform spatial semantic analysis on the context pixel sequence using a preset lightweight convolutional network model to obtain a spatial semantic vector, perform feature fusion on the spatial semantic vector and the fluctuation index corresponding to the candidate frame image to obtain a modulation vector, calculate the feature saliency score of the candidate frame image based on the modulation vector, and iterate to obtain the feature saliency score of each candidate frame image. The target keyframe determination module is used to filter the at least one candidate frame image according to the feature saliency score to obtain the target keyframe, and to complete the video editing based on the target keyframe.

Images