Video frame extraction method and device based on adaptive regularization, equipment and medium

Abstract

The invention relates to the technical field of image detection, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a video frame extraction method, device and equipment based on self-adaptive regularization and a medium. Performing frame normalization processing on the plurality of initial video frames to generate a plurality of standard video frames; extracting spatial features and time features of the standard video frames, and constructing a multi-resolution video feature map of the standard video frames according to the spatial features and the time features; performing multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain a plurality of map regularization coefficients; performing adaptive regularization constraint processing on the standard video frame according to the image regularization coefficient to generate a plurality of regularization video frames; and performing target frame selection on the plurality of regularized video frames to obtain a target video frame corresponding to the initial video stream data. According to the invention, the accuracy of video frame extraction can be improved.

Description

Adaptive regularization-based video frame extraction method, apparatus, equipment, and medium Technical Field

[0001] This invention relates to the field of image detection, and in particular to a method, apparatus, device, and medium for video frame extraction based on adaptive regularization. Background Technology

[0002] In today's era of rapid digital information development, video data, with its rich and intuitive content, is widely used in many fields such as video surveillance, film and television production, online education, and live video streaming. With the popularization of video acquisition equipment and the maturity of generation technology, the amount of video data has exploded. This massive amount of video not only places extremely high demands on storage resources, but also faces enormous computational pressure and efficiency challenges in subsequent analysis, processing, and retrieval.

[0003] Video frame extraction, a crucial step in video processing, aims to extract representative and key information frames from the raw video stream to achieve a concise representation of the video data. For example, in video surveillance, extracting keyframes can quickly pinpoint the time of abnormal events and key scenes, reducing manual review workload; in film and television production, keyframe extraction facilitates editors' rapid browsing of content for editing and special effects processing.

[0004] Traditional video frame extraction methods are mainly based on fixed-interval sampling or simple regularized difference analysis. Fixed-interval sampling extracts frames at preset time or frame intervals, which is simple and easy to implement, but it cannot flexibly adjust the strategy according to the actual changes in video content. This can easily lead to key frames not accurately reflecting important information, missing key events, or including too many redundant frames. Regularization, as an effective machine learning technique, can help constrain the video frame extraction process when introduced into the field of video frame extraction. However, research on applying regularization techniques to video frame extraction is still in its early stages. Most existing regularization methods use fixed regularization parameters, which cannot adaptively adjust the regularization strength according to the dynamic changes in video data, making it difficult to achieve optimal video frame extraction results in different scenarios.

[0005] For example, in the medical and health field, there is an urgent need for massive video storage and rapid analysis, including surgical videos and dynamic recordings of medical images. Traditional frame extraction methods or fixed-interval sampling are prone to missing important information such as key surgical steps and changes in lesions, while simple inter-frame difference analysis is easily affected by noise.

[0006] For example, in the field of fintech, key frames need to be accurately extracted from transaction monitoring videos and customer identity verification videos to achieve efficient risk identification and identity verification. However, existing methods lack adaptability and are difficult to cope with complex and ever-changing scenarios. While regularization technology can constrain the model, existing regularization methods with fixed parameters cannot dynamically and adaptively adjust according to the video.

[0007] Therefore, how to implement a video frame extraction method based on adaptive regularization to improve the accuracy of video frame extraction has become an urgent problem to be solved. Summary of the Invention

[0008] This invention provides a video frame extraction method, apparatus, device, and medium based on adaptive regularization, the main purpose of which is to solve the problem of low accuracy in video frame extraction.

[0009] In a first aspect, to achieve the above objective, the present invention provides a video frame extraction method based on adaptive regularization, comprising: decomposing initial video stream data into multiple consecutive initial video frames, and performing frame normalization processing on the multiple initial video frames to generate multiple standard video frames; extracting spatial and temporal features of the standard video frames, and constructing a multi-resolution video feature map of the standard video frames based on the spatial and temporal features; performing multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients; performing adaptive regularization constraint processing on the standard video frames based on the graph regularization coefficients to generate multiple regularized video frames; and selecting target frames from the multiple regularized video frames to obtain target video frames corresponding to the initial video stream data.

[0010] Secondly, the present invention also provides a video frame extraction device based on adaptive regularization, comprising: a video frame normalization module, used to decompose initial video stream data into multiple consecutive initial video frames, and perform frame normalization processing on the multiple initial video frames to generate multiple standard video frames; a spatiotemporal feature extraction module, used to extract spatial and temporal features of the standard video frames, and construct a multi-resolution video feature map of the standard video frames based on the spatial and temporal features; an attention calculation module, used to perform multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients; a regularization constraint module, used to perform adaptive regularization constraint processing on the standard video frames based on the graph regularization coefficients to generate multiple regularized video frames; and a target frame selection module, used to select target frames from the multiple regularized video frames to obtain the target video frames corresponding to the initial video stream data.

[0011] Thirdly, the present invention also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the aforementioned adaptive regularization-based video frame extraction method.

[0012] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the above-described adaptive regularization-based video frame extraction method.

[0013] In this embodiment of the invention, by decomposing and normalizing the initial video stream, inter-frame inconsistencies caused by changes in shooting equipment, environment, or encoding differences are effectively eliminated. This makes content-based feature comparisons (such as structural similarity or pixel differences) more accurate and reliable, thereby significantly improving the accuracy and robustness of subsequent keyframe detection. The multi-resolution feature map simultaneously captures local details and global temporal context, enabling the algorithm not only to identify significant visual content changes within a single frame but also to make comprehensive judgments based on the motion trajectories and scene evolution trends of preceding and following frames. This effectively avoids false detections caused by brief screen flicker or localized motion. This significantly improves the completeness and semantic accuracy of subsequent keyframe extraction. By performing multi-head self-attention regularization on multi-resolution video feature maps, it can adaptively learn and enhance the association of key information in different spatial and temporal dimensions in the feature maps, effectively suppressing noise and redundant information in the features. The resulting graph regularization coefficients assign differentiated weights to different feature regions, enabling the subsequent keyframe extraction algorithm to focus on the most representative feature regions, thereby more accurately identifying the frame sequences that truly reflect the structural changes in video content, improving the robustness of keyframe extraction, and reducing the misjudgment rate of non-keyframes.

[0014] Specifically, by using adaptive regularization constraints, the graph regularization coefficients are transformed into customized regularization constraints for each frame, which effectively enhances the structural quality and semantic consistency of video frames. Furthermore, inter-frame constraints ensure visual coherence in the temporal dimension. The resulting regularized video frames have clearer edge textures and more stable temporal features, providing high-quality input with lower noise, more prominent structure, and more coherent content for subsequent keyframe extraction. This significantly improves the accuracy and robustness of keyframe recognition. By intelligently selecting the regularized video frame sequence, the accuracy of keyframe extraction can be significantly improved, ensuring that the selected target video frames can completely and without redundancy summarize the core content of the initial video stream data. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 is a schematic diagram of an application environment for a video frame extraction method based on adaptive regularization according to an embodiment of the present invention; Figure 2 is a flowchart of a video frame extraction method based on adaptive regularization according to an embodiment of the present invention; Figure 3 is a flowchart of constructing a multi-resolution video feature map of a standard video frame based on spatial and temporal features according to an embodiment of the present invention; Figure 4 is a functional block diagram of a video frame extraction device based on adaptive regularization according to an embodiment of the present invention; Figure 5 is a structural schematic diagram of an electronic device for implementing a video frame extraction method based on adaptive regularization according to an embodiment of the present invention; Figure 6 is another structural schematic diagram of an electronic device for implementing a video frame extraction method based on adaptive regularization according to an embodiment of the present invention.

[0017] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0019] It should be noted that the terms "first," "second," etc., used in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0020] This application provides a video frame extraction method based on adaptive regularization. The execution entity of this method includes, but is not limited to, at least one electronic device that can be configured to execute the device provided in this application, such as a server or a terminal. In other words, the video frame extraction method based on adaptive regularization can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0021] This invention presents a video frame extraction method based on adaptive regularization, which can be applied in the application environment shown in Figure 1. In this environment, the client communicates with the server via a network. The server can obtain initial video stream data from the client. By decomposing and normalizing the initial video stream, it effectively eliminates inter-frame inconsistencies caused by changes in shooting equipment, environment, or encoding differences. This makes content-based feature comparisons (such as structural similarity or pixel differences) more accurate and reliable, thus significantly improving the accuracy and robustness of subsequent keyframe detection. Multi-resolution feature maps simultaneously capture local details and global temporal context, effectively avoiding false detections caused by brief screen flicker or local motion, thereby greatly improving the completeness and semantic accuracy of subsequent keyframe extraction. By performing multi-head self-attention regularization on multi-resolution video feature maps, it can adaptively learn and enhance the association of key information in different spatial and temporal dimensions in the feature maps, effectively suppressing noise and redundant information in the features. The resulting graph regularization coefficients assign differentiated weights to different feature regions, enabling subsequent keyframe extraction algorithms to focus on the most representative feature regions, thereby more accurately identifying frame sequences that truly reflect changes in the video content structure, improving the robustness of keyframe extraction, and reducing the false positive rate for non-keyframes.

[0022] Specifically, by using adaptive regularization constraint processing, the graph regularization coefficients are transformed into customized regularization constraints for each frame, which effectively enhances the structural quality and semantic consistency of video frames. Furthermore, inter-frame constraints ensure visual coherence in the temporal dimension. The resulting regularized video frames have clearer edge textures and more stable temporal features, providing high-quality input with lower noise, more prominent structure, and more coherent content for subsequent keyframe extraction. This significantly improves the accuracy and robustness of keyframe recognition. By intelligently selecting the regularized video frame sequence, the accuracy of keyframe extraction can be significantly improved, ensuring that the selected target video frames can completely and without redundancy summarize the core content of the initial video stream data. Finally, the target video frames are output and fed back to the client.

[0023] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0024] Referring to Figure 2, a flowchart illustrating a video frame extraction method based on adaptive regularization according to an embodiment of the present invention is shown. In this embodiment, the video frame extraction method based on adaptive regularization includes: S1, decomposing the initial video stream data into multiple consecutive initial video frames, and performing frame normalization processing on the multiple initial video frames to generate multiple standard video frames.

[0025] In this embodiment of the invention, the initial video frame refers to a single static image obtained by splitting the initial video stream data according to the time dimension. A complete initial video stream consists of several consecutive initial video frames. The number of frames and the frame rate determine the playback duration and smoothness of the video. The frame normalization processing refers to the operation of standardizing and adjusting the split initial video frames. The purpose is to eliminate the differences between different initial video frames (such as pixel value range, resolution size, color space format, etc.) so that the processed video frames have uniform specifications, which facilitates subsequent calculation and analysis.

[0026] In this embodiment of the invention, the step of decomposing the initial video stream data into multiple consecutive initial video frames includes: parsing the format of the initial video stream data to obtain the video encoding format and video encapsulation parameters corresponding to the initial video stream data; performing frame separation processing on the initial video stream data according to the video encapsulation parameters to obtain a video basic stream; decoding the video basic stream using a video decoder corresponding to the video encoding format to obtain multiple decoded video frames; and arranging the multiple decoded video frames in chronological order to form multiple consecutive initial video frames.

[0027] In this embodiment of the invention, the initial video stream data is parsed to determine its encoding format and encapsulation parameters. During parsing, the header information of the video stream data is read. This information typically contains key clues about the video encoding format (e.g., H.264, H.265) and encapsulation parameters (e.g., frame rate, resolution, keyframe interval). Based on the parsed video encapsulation parameters, the initial video stream data undergoes frame separation processing, that is, continuous video stream data is divided into independent video frames. Based on keyframe positions, timestamps, and other information in the encapsulation parameters, the video stream data is precisely divided into multiple video elementary streams.

[0028] In detail, the video elementary stream is decoded using a video decoder corresponding to the video encoding format. The purpose is to restore the compressed video data to the original video frames. Based on the previously parsed video encoding format, a suitable video decoder is selected to decode the video elementary stream. The decoder parses and processes the video elementary stream according to the encoding format specifications, ultimately obtaining multiple decoded video frames. These multiple decoded video frames are arranged in chronological order to form multiple consecutive initial video frames. During the arrangement process, they are sorted according to the timestamp information in the decoded video frames, thus obtaining multiple consecutive initial video frames. These video frames can be used for subsequent video processing and analysis tasks.

[0029] In this embodiment of the invention, the step of performing frame normalization processing on multiple initial video frames to generate multiple standard video frames includes: performing size normalization processing on the frame size of each initial video frame to obtain a size-normalized video frame; performing pixel linear mapping processing on the pixel values ​​of each size-normalized video frame to obtain a pixel-normalized video frame; converting the color space of each pixel-normalized video frame to a standard color space to obtain a color-normalized video frame; and performing image enhancement processing on the color-normalized video frames to obtain multiple standard video frames.

[0030] In this embodiment of the invention, a standard size reference suitable for subsequent processing is determined. For initial video frames larger than this reference, they are processed by proportionally reducing their size. During the reduction process, pixels within the frame are sampled and fused appropriately to avoid image content distortion or edge blurring. For initial video frames smaller than this reference, they are processed by proportionally enlarging their size. New pixels are generated by interpolating information from the surrounding areas of existing pixels, ensuring that the enlarged image does not have obvious pixel blocks or content breaks. After this processing, the initial video frames of varying sizes are converted into size-normalized video frames of uniform size.

[0031] In detail, during the initial video frame acquisition process, the distribution range of pixel values ​​may fluctuate significantly due to differences in camera sensitivity parameters and ambient brightness. The maximum and minimum pixel values ​​within each size-normalized video frame are identified, and then, through linear mapping, the values ​​of all pixels within that video frame are uniformly mapped to a preset standard value range.

[0032] Specifically, for pixels with values ​​higher than the upper limit of the standard range, they are adjusted to the maximum value of the range; for pixels with values ​​lower than the lower limit of the standard range, they are adjusted to the minimum value of the range; and for pixels between the original value range and the standard range, the corresponding standard value is calculated according to a linear proportional relationship. This process ensures that video frames with different brightness and contrast have a uniform pixel value distribution, avoiding calculation deviations caused by excessive differences in pixel values ​​during subsequent processing, ultimately resulting in pixel-normalized video frames.

[0033] Specifically, a predefined standard color space is defined, and for each pixel in the normalized video frame, the color components of each pixel within the frame are converted one by one according to the color mapping rules between this color space and the standard color space. During the conversion process, the component parameters that express the same color perception in both color spaces are precisely matched to ensure that the converted video frame does not have color shifts or color casts. For example, the original red will not become orange, and the blue will not have an excessively high grayscale value. After the conversion, the color information of all video frames is unified under the standard color space, forming a color-normalized video frame.

[0034] Furthermore, the image enhancement includes edge enhancement and noise suppression. By identifying edge, texture, and other detailed feature regions within the video frame, the pixel contrast of these regions is appropriately adjusted to make originally blurry edges clearer and subtle texture features easier for the computer to recognize. Noise suppression analyzes the distribution characteristics of surrounding pixels to distinguish noise points from normal image pixels within the video frame, smoothing noise points while preserving the original information of normal pixels to avoid loss of image details due to noise reduction. After such image enhancement processing, a standard video frame that meets the requirements of subsequent computer vision tasks is formed.

[0035] In this embodiment of the invention, by decomposing and normalizing the initial video stream, the inconsistencies between frames caused by changes in shooting equipment, environment or encoding are effectively eliminated, making content-based feature comparison (such as structural similarity or pixel differences) more accurate and reliable, thereby significantly improving the accuracy and robustness of subsequent keyframe detection.

[0036] S2. Extract the spatial and temporal features of the standard video frame, and construct a multi-resolution video feature map of the standard video frame based on the spatial and temporal features.

[0037] In this embodiment of the invention, the spatial features include the shape, edges, texture, color distribution, and spatial positional relationships of standard video frames, reflecting the static structural attributes of the video image at a certain moment; the temporal features refer to the correlation features between multiple consecutive standard video frames in the time dimension, which are dynamic change information extracted based on the temporal continuity of video frames, including the motion trajectory of targets, action changes, and inter-frame differences and correlations of image content between adjacent frames, reflecting the dynamic evolution attributes of the video image over time.

[0038] In this embodiment of the invention, the extraction of spatial and temporal features of the standard video frame includes: performing inter-frame alignment processing on the standard video frame to obtain aligned video frames; using a preset spatial feature extraction network to extract spatial feature maps of each aligned video frame; performing multi-scale pooling processing on the spatial feature maps to generate spatial features of the standard video frame; performing cyclic convolution processing on the spatial feature maps according to a preset temporal dimension to generate inter-frame motion trajectories and pixel change trends of the standard video frame; performing temporal pooling processing on the inter-frame motion trajectories and pixel change trends to obtain target inter-frame motion trajectories and target pixel change trends; and using the target inter-frame motion trajectories and target pixel change trends as temporal features of the standard video frame.

[0039] In this embodiment of the invention, a frame in a continuous standard video frame sequence is selected as a reference frame. The spatial coordinates of the reference frame are set as a reference standard. For each subsequent frame to be aligned, key feature points within the frame (such as edge corners of objects in the image, or areas with significant texture) are identified. The corresponding positions of these feature points in the reference frame are compared, and the offset direction and magnitude of the frame to be aligned relative to the reference frame are calculated. Based on the calculated offset parameters, the entire frame to be aligned is subjected to spatial transformation processing, including translation, rotation, scaling, etc., so that the key feature points within the frame to be aligned completely coincide with the corresponding feature points of the reference frame, thereby converting it into an aligned video frame that perfectly matches the spatial position between frames.

[0040] In detail, the pre-defined spatial feature extraction network is trained on a large number of image samples and has the ability to automatically identify static features at different levels in an image. During processing, the aligned video frame is input into this network, which processes the pixel information within the frame layer by layer from the bottom to the top. The bottom layer network prioritizes identifying basic visual features within the frame, such as the edge contours of the target, local textures, and color patches. The middle layer network combines the basic features extracted from the bottom layer to form more discernible features, such as the local shape of the target and the structure of its components. The top layer network further integrates the middle layer features to generate abstract features that can characterize the global attributes of the target throughout the entire frame. These features at different levels are integrated into a multi-dimensional feature matrix, which is a spatial feature map that comprehensively reflects the static spatial attributes of the aligned video frame.

[0041] Specifically, spatial feature maps contain a wealth of feature information from the bottom layer to the top layer. Multi-scale pooling processing employs multiple different pooling scales simultaneously to process the spatial feature maps in parallel. For each pooling scale, the feature map is divided into multiple corresponding feature regions. Then, the feature information within each region is filtered and integrated, such as retaining the most representative feature values ​​or calculating the average level of feature values ​​within the region, thereby achieving dimensionality reduction and refinement of feature information. Small-scale pooling retains more fine-grained features, such as edge texture variations of the target; large-scale pooling focuses on global macro-features, such as the overall shape and spatial layout of the target. Subsequently, the features obtained from pooling at different scales are fused, removing redundant information, and finally generating standard video frame spatial features that can take into account both fine-grained details and global attributes.

[0042] Furthermore, the preset time dimension clearly defines the range of consecutive frames to be analyzed. For example, several adjacent frames are selected as a temporal analysis window. The recurrent convolutional network has the ability to remember temporal information. During processing, the spatial feature maps of consecutive aligned video frames within the temporal window are input into the network frame by frame in chronological order. The network first performs convolution processing on the current input feature map to extract the spatial feature details of that frame. At the same time, it combines the temporal memory information stored when previous frames were input to compare the differences between the feature maps of the current frame and those of historical frames. Through this frame-by-frame comparison and memorization method, the network can identify the changing patterns of features between consecutive frames: on the one hand, it tracks the positional changes of the same feature in consecutive frames, thus forming the motion trajectory of the target between frames; on the other hand, it analyzes the strength and distribution changes of the feature values ​​corresponding to pixels in the feature map, thus obtaining the changing trends of pixels between frames. Finally, the network outputs two types of information that reflect the dynamic changes of consecutive frames: the inter-frame motion trajectory and the pixel change trend.

[0043] The temporal pooling process divides continuous temporal information into multiple temporal segments according to a preset time interval. Then, it filters and integrates the motion trajectories and pixel change trends within each segment. For inter-frame motion trajectories, it removes abnormal trajectory points that deviate from the overall motion direction within a segment, retaining trajectory information that reflects the true motion state of the target. For pixel change trends, it filters out irregular random fluctuations within segments, extracting trend information that reflects the overall direction of pixel feature changes. Finally, the processed information from all temporal segments is summarized to form a more stable and representative inter-frame motion trajectory and target pixel change trend.

[0044] Specifically, the motion trajectory between target frames reflects the dynamic changes in the spatial position of the target within consecutive frames, and the trend of target pixel changes reflects the dynamic changes in the strength distribution of pixel features within consecutive frames. By structurally integrating these two types of information, a unified set of temporal features is formed, which is defined as the temporal features of a standard video frame.

[0045] For example, in the healthcare field, for patient motion monitoring video processing, the first step is to perform inter-frame alignment on standard video frames. For instance, optical flow can be used to align key parts of the patient's body in different frames, resulting in aligned video frames. A pre-trained convolutional neural network is then used as a spatial feature extraction network to extract spatial feature maps from each aligned video frame. This captures spatial information such as the patient's posture and movement amplitude. Multi-scale pooling is then applied to the spatial feature maps to generate spatial features, better preserving spatial details at different levels. Based on a preset time dimension, recurrent convolution is performed on the spatial feature maps to analyze the temporal continuity of the patient's movements, obtaining inter-frame motion trajectories and pixel change trends, such as the patient's limb movement path and changes in movement amplitude. Finally, temporal pooling is performed to obtain the target inter-frame motion trajectory and target pixel change trends, serving as temporal features. These spatial and temporal features can be used to assess the standardization of patient rehabilitation training movements and assist doctors in identifying lesions or abnormal areas.

[0046] As shown in Figure 3, in this embodiment of the invention, the step of constructing a multi-resolution video feature map of the standard video frame based on the spatial features and the temporal features includes: S21, performing multi-scale spatial downsampling on the spatial features to generate multiple spatial resolution feature maps; S22, performing multi-granularity temporal pooling on the temporal features to generate multiple temporal resolution feature maps; S23, identifying the spatial resolution levels of the multiple spatial resolution feature maps and the temporal resolution level of the temporal resolution feature maps; S24, performing feature fusion between the corresponding spatial resolution feature maps and the temporal resolution feature maps based on the spatial resolution level and the temporal resolution level to obtain a primary multi-resolution fusion feature map; S25, performing cross-resolution feature enhancement on the primary multi-resolution fusion feature map to obtain multiple enhanced resolution video feature maps; S26, concatenating the multiple enhanced resolution video feature maps along a preset channel dimension to obtain the multi-resolution video feature map of the standard video frame.

[0047] In this embodiment of the invention, pooling operations, such as max pooling or average pooling, are employed. By setting different pooling window sizes and strides, feature information at different scales is extracted from the original spatial feature map, generating multiple spatial resolution feature maps. For example, using a larger pooling window and stride can yield a low-resolution feature map, which can capture the general structure and global information of the image; while a smaller pooling window and stride generates a high-resolution feature map, preserving more local detail information.

[0048] Multi-granular temporal pooling is applied to temporal features. Temporal pooling is similar to spatial pooling, but it targets features in the time dimension. Different pooling parameters can be set according to different time intervals to segment the temporal features, thereby generating multiple temporal resolution feature maps. Longer pooling intervals can extract trend features over a long period of time, reflecting the overall dynamic changes of the video; shorter pooling intervals focus on detailed changes within a short period of time, capturing more refined temporal features.

[0049] In detail, by analyzing the size and feature information of the feature maps, the spatial and temporal scale levels they represent are determined. For example, the spatial resolution level can be determined based on the width and height of the feature map, while the temporal resolution level can be determined based on the sampling interval and aggregation degree of the temporal features in the temporal dimension. Based on the identified spatial and temporal resolution levels, the corresponding spatial resolution feature maps and temporal resolution feature maps are fused. Common fusion methods include feature concatenation and element-wise addition. By fusing spatial and temporal features at the same resolution level, the spatial and temporal information of video frames can be considered simultaneously, resulting in a primary multi-resolution fusion feature map.

[0050] Specifically, an attention mechanism is employed. By calculating the correlation between feature maps of different resolutions, different weights are assigned to each feature map, thereby enhancing important feature information and suppressing irrelevant noise. For example, high-resolution feature maps may contain more local details but may lack global information, while low-resolution feature maps are the opposite. Through the attention mechanism, high-resolution feature maps can focus more on global information that complements low-resolution feature maps, while low-resolution feature maps can learn from local details in high-resolution feature maps, achieving cross-resolution feature enhancement.

[0051] This involves stitching together multiple enhanced resolution video feature maps along a preset channel dimension. Different channels typically represent different feature representations. Stitching feature maps enhanced at different resolutions along the channel dimension can integrate feature information at various resolution levels, forming a richer and more comprehensive multi-resolution video feature map.

[0052] In this embodiment of the invention, the multi-resolution feature map captures both local details and global temporal context, enabling the algorithm to not only identify significant visual content changes within a single frame, but also to make a comprehensive judgment by combining the motion trajectory and scene evolution trend of the preceding and following frames. This effectively avoids false detections caused by brief screen flickering or local motion, thereby significantly improving the completeness and semantic accuracy of subsequent key frame extraction.

[0053] S3. Perform multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients.

[0054] In this embodiment of the invention, the graph regularization coefficient is a set of feature optimization parameters generated by multi-head self-attention regularization. Its function is to weight different feature regions of the multi-resolution video feature map, which can quantify the importance of each region in the feature map, highlight the contribution of key feature regions, reduce the interference of invalid or noisy feature regions, and provide a precise adjustment basis for subsequent feature processing.

[0055] In this embodiment of the invention, the step of performing multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients includes: performing multi-head self-attention calculation on the multi-resolution video feature map to generate a query matrix, a key matrix, and a value matrix corresponding to each attention head; calculating the multi-head attention distribution matrix of the multi-resolution video feature map based on the query matrix and the key matrix; performing matrix normalization on the multi-head attention distribution matrix to obtain the attention weight matrix of each attention head; multiplying the attention weight matrix of each attention head with the corresponding value matrix to obtain the multi-head attention feature map of each attention head; performing global average pooling on each of the multi-head attention feature maps to obtain pooled feature maps; and performing linear fully connected computation on the pooled feature maps to obtain multiple graph regularization coefficients.

[0056] In this embodiment of the invention, the input feature map is decomposed into multiple different "heads", and attention is calculated independently for each head. Specifically, for the input multi-resolution video feature map, a query matrix, a key matrix, and a value matrix corresponding to each attention head are generated by linear transformation. The query matrix is ​​used to represent other features that the current feature needs to pay attention to, the key matrix is ​​used to calculate the similarity between features, and the value matrix contains the actual information of the feature.

[0057] In detail, a dot product attention mechanism is used to perform a dot product operation on the query matrix and the key matrix to obtain a matrix representing the similarity between features. This similarity matrix is ​​scaled to avoid numerical instability in the subsequent softmax function calculation. The scaled matrix is ​​then normalized by the softmax function so that each element in the matrix represents a probability value, that is, the attention weight of one feature to another. In this way, a multi-head attention distribution matrix is ​​obtained, which describes the attention distribution between different features under different attention heads.

[0058] Specifically, matrix normalization is performed on the multi-head attention distribution matrix to ensure that the attention weight matrix of each attention head satisfies certain distribution characteristics, such as a sum of 1. A common normalization method is softmax normalization, which maps each element in the matrix to the interval [0,1] and ensures that the sum of elements in the same row or column is 1. Through normalization, the attention weights become more reasonable and stable, facilitating subsequent feature fusion operations.

[0059] Furthermore, the value matrix is ​​weighted and summed according to the attention weight matrix, so that each feature can aggregate relevant information based on its similarity (attention weight) with other features. Thus, each attention head can extract different feature representations, resulting in a multi-head attention feature map. By averaging all pixel values ​​in the multi-head attention feature map, the feature information extracted by different attention heads can be aggregated to obtain a more compact feature representation.

[0060] Specifically, a linear fully connected layer is performed on the pooled feature map. A linear fully connected layer is a simple neural network layer that performs a linear transformation on the input features through matrix multiplication and bias term addition, mapping the pooled feature map to a new feature space, thus obtaining graph regularization coefficients. These graph regularization coefficients can be used for subsequent regularization operations, such as constraining the model's parameters to prevent overfitting.

[0061] For example, in medical image-assisted diagnosis, for the processing of multimodal medical images (such as multi-resolution video feature maps fused from X-rays and CT scans), multi-head self-attention calculation is performed on the multi-resolution video feature maps to generate query, key, and value matrices corresponding to each attention head. Through query and key matrix operations, a multi-head attention distribution matrix is ​​obtained, which reflects the correlation strength between different image regions. The distribution matrix is ​​normalized to obtain an attention weight matrix, clarifying the degree of attention received by each region. The weight matrix is ​​multiplied by the value matrix to obtain a multi-head attention feature map, highlighting key image features. The feature map is then transformed into a pooled feature map to reduce dimensionality while retaining overall features. Finally, linear fully connected computation is performed to obtain multiple graph regularization coefficients, which can be used to adjust the image feature weights, assisting doctors in more accurately identifying the location and nature of lesions, and improving diagnostic accuracy and efficiency.

[0062] For example, in financial transaction risk assessment, the analysis of multi-dimensional transaction data (such as multi-resolution video feature maps fused from transaction amount, time, and transaction party information) first involves multi-head self-attention calculation to generate a correlation matrix, then calculating a multi-head attention distribution matrix to understand the correlation between different transaction data dimensions, normalizing to obtain an attention weight matrix to determine the importance of each dimension, multiplying with the value matrix to obtain a multi-head attention feature map, highlighting key transaction features, and then performing linear fully connected computation after global average pooling to obtain graph regularization coefficients. These coefficients can adjust the weights of transaction features, helping the risk assessment model to more accurately identify abnormal transactions, such as fraudulent transactions and illegal operations, and provide timely warnings to ensure the safety and stability of financial transactions.

[0063] In this embodiment of the invention, by performing multi-head self-attention regularization calculation on multi-resolution video feature maps, the key information associations in different spatial and temporal dimensions of the feature maps can be adaptively learned and enhanced, effectively suppressing noise and redundant information in the features. The resulting graph regularization coefficients assign differentiated weights to different feature regions, enabling subsequent keyframe extraction algorithms to focus on the most representative feature regions, thereby more accurately identifying frame sequences that truly reflect changes in the video content structure, improving the robustness of keyframe extraction, and reducing the misjudgment rate of non-keyframes.

[0064] S4. Perform adaptive regularization constraint processing on the standard video frame according to the graph regularization coefficient to generate multiple regularized video frames.

[0065] In this embodiment of the invention, the step of performing adaptive regularization constraint processing on the standard video frames according to the graph regularization coefficients to generate multiple regularized video frames includes: determining a regularization intensity factor corresponding to each standard video frame according to the multiple graph regularization coefficients; generating an adaptive regularization constraint function for each standard video frame based on the regularization intensity factor; performing local structure regularization processing on each standard video frame using the adaptive regularization constraint function to obtain a locally regularized video frame; applying inter-frame consistency constraints to the locally regularized video frames to obtain constrained video frames; calculating the pixel gradient change of the constrained video frames, and determining the regularized video frames in the constrained video frames based on the pixel gradient change.

[0066] In this embodiment of the invention, the graph regularization coefficients reflect the characteristics and importance of different standard video frames within the overall features. By analyzing these coefficients, such as observing their size distribution and relative differences, the strength of regularization required for each standard video frame can be assessed. For example, if a graph regularization coefficient is large, it indicates that the standard video frame has more prominent or unique properties within the overall features, and may require stronger regularization to constrain its feature changes and avoid excessive deviation from the overall feature pattern; conversely, standard video frames with smaller coefficients may only require weaker regularization. Based on this analytical logic, a suitable regularization strength factor can be determined for each standard video frame, which will serve as an important basis for subsequent regularization processing.

[0067] In detail, since the regularization intensity factors differ across standard video frames, an adaptive constraint function needs to be generated for each frame. This typically combines prior knowledge with mathematical models. Prior knowledge might include understanding the distribution and variation patterns of video frame features; for example, knowing that adjacent pixels in video frames are generally similar spatially and continuous temporally. Based on this prior knowledge, and combined with the regularization intensity factor, a function is generated through mathematical construction methods, such as building functions based on distance metrics or probability distributions, that can adaptively adjust the constraint strength according to the regularization intensity factor. This function can flexibly apply different levels of constraints tailored to the characteristics of each standard video frame, ensuring that the regularization process is effective without excessively damaging the original features of the frame.

[0068] Specifically, local structure regularization primarily focuses on the local region features within a standard video frame. Each standard video frame is divided into multiple local regions, such as small pixel blocks or specific structural regions. For each local region, an adaptive regularization constraint function is applied. Based on the region's position and features within the overall frame, as well as the corresponding regularization intensity factor, the pixel values ​​of the local region are adjusted. In this way, the local structure of each standard video frame is optimized, resulting in locally regularized video frames, making the local features within the frame more reasonable and stable.

[0069] Furthermore, inter-frame consistency constraints are applied to the locally regularized video frames. Since a video is composed of a series of consecutive frames, adjacent frames usually have similar scene and object motion information. After obtaining the locally regularized video frames, the feature differences between adjacent frames are analyzed, such as changes in pixel values, changes in object position and shape, etc. Based on these differences, inter-frame consistency constraint methods, such as motion compensation and feature matching, are used to adjust the locally regularized video frames.

[0070] Motion compensation estimates the motion trajectory of an object between adjacent frames and performs corresponding translation, rotation, or scaling operations on the frames to make the position and shape of the object more consistent in different frames. Feature matching compares feature points in adjacent frames to ensure that the same feature is represented consistently in different frames. Through these inter-frame consistency constraints, discontinuities and abrupt changes between frames are eliminated, resulting in constrained video frames, making the entire video sequence smoother and more natural.

[0071] Finally, the pixel gradient change of the constrained video frame is calculated. The pixel gradient change reflects the degree of spatial variation of pixel values ​​in the constrained video frame. By calculating the gradient of all pixels in each constrained video frame, a gradient matrix is ​​obtained. Then, various characteristics of the gradient matrix, such as the average value, maximum value, and variance of the gradient, are statistically analyzed as metrics for the pixel gradient change. Based on these metrics, the smoothness and feature variation of the constrained video frame are evaluated. Generally speaking, frames with smaller pixel gradient changes indicate that their pixel values ​​change more gently and their features are relatively stable, while frames with larger changes may have more details or noise. Based on a preset threshold or judgment criterion, frames with pixel gradient changes that meet the requirements are selected from the constrained video frames as regularized video frames.

[0072] For example, in medical image analysis, the processing of standard video frames for monitoring patient lesions involves first determining the regularization intensity factor for each standard video frame based on multiple graph regularization coefficients. For instance, a larger coefficient indicates that the lesion features in that frame are key, and the intensity factor is also larger. An adaptive regularization constraint function is generated based on the intensity factor. When the intensity factor is large, the function retains key structures such as image edges more effectively. This function is then used to perform local structural regularization processing on each standard video frame, highlighting the local features of the lesion and obtaining locally regularized video frames. Next, inter-frame consistency constraints are applied to ensure that the changes in the position and shape of the lesion between different frames are reasonable, resulting in constrained video frames. The pixel gradient change of the constrained video frames is calculated. If the pixel gradient change of a certain frame is small, it indicates that the image is smooth and has less noise, and it is determined as a regularized video frame. These regularized video frames can present lesion information more clearly and accurately, assisting doctors in diagnosing the condition.

[0073] For example, in financial transaction monitoring, standard video frames of a transaction scenario are analyzed. A regularization strength factor is determined based on graph regularization coefficients. A larger coefficient reflects more key transaction information, resulting in a larger strength factor. An adaptive regularization constraint function is generated to preserve important transaction information (such as transaction amount display and transaction party identification). Local structural regularization is then performed using the function to obtain locally regularized video frames. Inter-frame consistency constraints are applied to ensure the transaction process is coherent and reasonable across video frames. The pixel gradient change of the constrained video frames is calculated. Frames with small and stable pixel gradient changes indicate clear and interference-free transaction footage and are identified as regularized video frames. These regularized video frames accurately record key transaction information, facilitating subsequent auditing and risk assessment, and ensuring the security and compliance of financial transactions.

[0074] In this embodiment of the invention, by using adaptive regularization constraint processing, the graph regularization coefficients are transformed into customized optimization strategies for each frame, which can effectively enhance the structural quality and semantic consistency of video frames. Furthermore, the inter-frame constraints ensure visual coherence in the temporal dimension. The resulting regularized video frames have clearer edge textures and more stable temporal features, providing high-quality input with lower noise, more prominent structure, and more coherent content for subsequent keyframe extraction. This significantly improves the recognition accuracy and robustness of keyframes.

[0075] S5. Select target frames from multiple regularized video frames to obtain the target video frames corresponding to the initial video stream data.

[0076] In this embodiment of the invention, the step of selecting a target video frame from multiple regularized video frames to obtain the target video frame corresponding to the initial video stream data includes: calculating the image clarity feature value and content clarity feature value of each regularized video frame; calculating the inter-frame difference feature value between the regularized video frames based on the temporal position of each regularized video frame; performing weighted feature value fusion on the image clarity feature value, the content clarity feature value, and the inter-frame difference feature value to obtain a target feature value; selecting frames from the regularized video frames based on a preset adaptive feature threshold and the target feature value to obtain multiple candidate keyframes; and performing content redundancy removal processing on the multiple candidate keyframes to obtain the target video frame.

[0077] In this embodiment of the invention, the image sharpness feature value is used to measure the visual sharpness of a regularized video frame. This is typically achieved using edge detection methods. Edges are regions in an image where grayscale changes drastically; sharp images often have more pronounced and regular edges. For example, edge detection operators such as the Sobel operator and the Canny operator are used to process the regularized video frame, calculating the intensity and number of edges in the image. Higher edge intensity and a greater number of edges indicate, to some extent, a sharper image. Through statistical and computational methods, this edge information is transformed into a specific image sharpness feature value.

[0078] Content clarity features focus on evaluating the recognizability and completeness of the content contained in a regularized video frame, achieved by analyzing elements such as objects and scenes in the image. For example, an image segmentation algorithm is used to divide the image into different regions, and then the features within each region are analyzed, such as color distribution and texture features. Regions containing rich and clearly identifiable objects are given a higher score, and the content clarity feature value of the entire frame is obtained by combining the scores of all regions.

[0079] In detail, inter-frame difference feature values ​​are calculated by comparing pixel value differences and feature point matching between adjacent frames. For example, for two adjacent frames, the grayscale value difference between their corresponding pixels is calculated, and the number of pixels with large differences is counted; or feature point detection algorithms (such as SIFT, SURF, etc.) are used to extract feature points in two frames, and then feature point matching is performed. The inter-frame difference is measured based on the number of successfully matched feature points and the change in their positions.

[0080] The process involves weighted feature fusion of image clarity feature value, content clarity feature value, and inter-frame difference feature value. For example, in scenarios where image clarity is highly critical, the weight of the image clarity feature value can be set to be larger, while in scenarios that emphasize content changes, the weight of the inter-frame difference feature value may be higher. The three feature values ​​are multiplied by their corresponding weights, and the results are summed to obtain a comprehensive target feature value. This value can fully reflect the characteristics of each regularized video frame in multiple dimensions.

[0081] Furthermore, regularized video frames are selected based on preset adaptive feature thresholds and target feature values. The adaptive feature thresholds are dynamically adjusted according to the overall characteristics of the video and actual application requirements. For each regularized video frame, its target feature value is compared with the adaptive feature threshold. If the target feature value is greater than the threshold, it indicates that the frame has high comprehensive characteristics in terms of image clarity, content clarity, and inter-frame differences, and it is selected as a candidate keyframe. In this way, representative and important frames can be screened from a large number of regularized video frames, resulting in multiple candidate keyframes.

[0082] Specifically, content redundancy removal from multiple candidate keyframes can be achieved by comparing the similarity between frames. For example, metrics such as structural similarity (SSIM) or mean squared error (MSE) can be calculated between candidate keyframes. If the similarity between two frames exceeds a certain threshold, they are considered redundant. Redundant frames are then removed according to a certain strategy (such as retaining frames that are earlier in time or have higher target feature values), ultimately resulting in a set of content-rich and non-redundant target video frames that better represent the key information of the initial video stream data.

[0083] In this embodiment of the invention, by intelligently selecting the regularized video frame sequence, the accuracy of key frame extraction can be significantly improved, ensuring that the selected target video frames can completely and without redundancy summarize the core content of the initial video stream data.

[0084] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0085] Figure 4 shows a functional block diagram of a video frame extraction device based on adaptive regularization provided in an embodiment of the present invention.

[0086] In this embodiment, an adaptive regularization-based video frame extraction device is provided, which corresponds one-to-one with the adaptive regularization-based video frame extraction method described in the above embodiments. As shown in Figure 4, the adaptive regularization-based video frame extraction device 100 can be installed in an electronic device. According to the functions implemented, the adaptive regularization-based video frame extraction device 100 includes a video frame normalization module 101, a spatiotemporal feature extraction module 102, an attention calculation module 103, a regularization constraint module 104, and a target frame selection module 105. The functional modules are described in detail below: Video frame standardization module 101 is used to decompose the initial video stream data into multiple consecutive initial video frames, and perform frame normalization processing on the multiple initial video frames to generate multiple standard video frames; Spatiotemporal feature extraction module 102 is used to extract the spatial and temporal features of the standard video frames, and construct a multi-resolution video feature map of the standard video frames based on the spatial and temporal features; Attention calculation module 103 is used to perform multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients; Regularization constraint module 104 is used to perform adaptive regularization constraint processing on the standard video frames based on the graph regularization coefficients to generate multiple regularized video frames; Target frame selection module 105 is used to select target frames from the multiple regularized video frames to obtain the target video frames corresponding to the initial video stream data.

[0087] In one embodiment, when the video frame normalization module 101 performs frame normalization processing on multiple initial video frames to generate multiple standard video frames, it is configured to: perform size normalization processing on the frame size of each initial video frame to obtain a size-normalized video frame; perform pixel linear mapping processing on the pixel values ​​of each size-normalized video frame to obtain a pixel-normalized video frame; convert the color space of each pixel-normalized video frame to a standard color space to obtain a color-normalized video frame; and perform image enhancement processing on the color-normalized video frames to obtain multiple standard video frames.

[0088] In one embodiment, when the spatiotemporal feature extraction module 102 extracts the spatial and temporal features of the standard video frame, it performs the following steps: performs inter-frame alignment processing on the standard video frame to obtain aligned video frames; extracts spatial feature maps of each aligned video frame using a preset spatial feature extraction network; performs multi-scale pooling processing on the spatial feature maps to generate spatial features of the standard video frame; performs cyclic convolution processing on the spatial feature maps according to a preset temporal dimension to generate inter-frame motion trajectories and pixel change trends of the standard video frame; performs temporal pooling processing on the inter-frame motion trajectories and pixel change trends to obtain target inter-frame motion trajectories and target pixel change trends; and uses the target inter-frame motion trajectories and target pixel change trends as the temporal features of the standard video frame.

[0089] In one embodiment, when the spatiotemporal feature extraction module 102 constructs a multi-resolution video feature map of the standard video frame based on the spatial features and the temporal features, it is configured to: perform multi-scale spatial downsampling on the spatial features to generate multiple spatial resolution feature maps; perform multi-granularity temporal pooling on the temporal features to generate multiple temporal resolution feature maps; identify the spatial resolution levels of the multiple spatial resolution feature maps and the temporal resolution level of the temporal resolution feature maps; perform feature fusion on the corresponding spatial resolution feature maps and temporal resolution feature maps according to the spatial resolution level and the temporal resolution level to obtain a primary multi-resolution fusion feature map; perform cross-resolution feature enhancement on the primary multi-resolution fusion feature map to obtain multiple enhanced resolution video feature maps; and concatenate the multiple enhanced resolution video feature maps along a preset channel dimension to obtain the multi-resolution video feature map of the standard video frame.

[0090] In one embodiment, when the attention calculation module 103 performs multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients, it is configured to: perform multi-head self-attention calculation on the multi-resolution video feature map to generate a query matrix, a key matrix, and a value matrix corresponding to each attention head; calculate the multi-head attention distribution matrix of the multi-resolution video feature map based on the query matrix and the key matrix; perform matrix normalization on the multi-head attention distribution matrix to obtain the attention weight matrix of each attention head; multiply the attention weight matrix of each attention head with the corresponding value matrix to obtain the multi-head attention feature map of each attention head; perform global average pooling on each of the multi-head attention feature maps to obtain pooled feature maps; and perform linear fully connected computation on the pooled feature maps to obtain multiple graph regularization coefficients.

[0091] In one embodiment, the regularization constraint module 104 performs adaptive regularization constraint processing on the standard video frames according to the graph regularization coefficients to generate multiple regularized video frames, including: determining a regularization intensity factor corresponding to each standard video frame according to the multiple graph regularization coefficients; generating an adaptive regularization constraint function for each standard video frame based on the regularization intensity factor; performing local structure regularization processing on each standard video frame using the adaptive regularization constraint function to obtain a locally regularized video frame; applying inter-frame consistency constraints to the locally regularized video frames to obtain constrained video frames; calculating the pixel gradient change of the constrained video frames, and determining the regularized video frames in the constrained video frames based on the pixel gradient change.

[0092] In one embodiment, when the target frame selection module 105 performs target frame selection on multiple regularized video frames to obtain the target video frame corresponding to the initial video stream data, it is configured to: calculate the image clarity feature value and content clarity feature value of each regularized video frame; calculate the inter-frame difference feature value between the regularized video frames according to the temporal position of each regularized video frame; perform weighted feature value fusion on the image clarity feature value, the content clarity feature value and the inter-frame difference feature value to obtain the target feature value; select frames for the regularized video frames according to a preset adaptive feature threshold and the target feature value to obtain multiple candidate keyframes; and perform content redundancy removal processing on the multiple candidate keyframes to obtain the target video frame.

[0093] In this invention, specific limitations regarding the adaptive regularization-based video frame extraction device can be found in the above-described limitations of the adaptive regularization-based video frame extraction method, and will not be repeated here. Each module in the aforementioned adaptive regularization-based video frame extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0094] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 5. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the adaptive regularization-based video frame extraction method on the server side.

[0095] In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram is shown in Figure 6. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client-side video frame extraction method based on adaptive regularization.

[0096] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: decomposing initial video stream data into multiple consecutive initial video frames, and performing frame normalization processing on the multiple initial video frames to generate multiple standard video frames; extracting spatial and temporal features of the standard video frames, and constructing a multi-resolution video feature map of the standard video frames based on the spatial and temporal features; performing multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients; performing adaptive regularization constraint processing on the standard video frames based on the graph regularization coefficients to generate multiple regularized video frames; and selecting target frames from the multiple regularized video frames to obtain target video frames corresponding to the initial video stream data.

[0097] In the several embodiments provided by this invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0098] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0099] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0100] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0101] The readable storage medium of the present invention stores a computer program, which, when executed by a processor of an electronic device, can: decompose initial video stream data into multiple consecutive initial video frames, and perform frame normalization processing on the multiple initial video frames to generate multiple standard video frames; extract spatial and temporal features of the standard video frames, and construct a multi-resolution video feature map of the standard video frames based on the spatial and temporal features; perform multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients; perform adaptive regularization constraint processing on the standard video frames based on the graph regularization coefficients to generate multiple regularized video frames; and select target frames from the multiple regularized video frames to obtain target video frames corresponding to the initial video stream data.

[0102] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0103] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0104] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0105] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0106] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Furthermore, any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0108] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0109] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0110] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A video frame extraction method based on adaptive regularization, characterized in that, The method includes: decomposing initial video stream data into multiple consecutive initial video frames, and performing frame normalization processing on the multiple initial video frames to generate multiple standard video frames; extracting spatial and temporal features of the standard video frames, and constructing a multi-resolution video feature map of the standard video frames based on the spatial and temporal features; performing multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients; performing adaptive regularization constraint processing on the standard video frames based on the graph regularization coefficients to generate multiple regularized video frames; and selecting target frames from the multiple regularized video frames to obtain the target video frames corresponding to the initial video stream data.

2. The video frame extraction method based on adaptive regularization as described in claim 1, characterized in that, The step of performing frame normalization processing on multiple initial video frames to generate multiple standard video frames includes: normalizing the frame size of each initial video frame to obtain a size-normalized video frame; performing pixel linear mapping processing on the pixel values ​​of each size-normalized video frame to obtain a pixel-normalized video frame; converting the color space of each pixel-normalized video frame to a standard color space to obtain a color-normalized video frame; and performing image enhancement processing on the color-normalized video frames to obtain multiple standard video frames.

3. The video frame extraction method based on adaptive regularization as described in claim 1, characterized in that, The step of extracting the spatial and temporal features of the standard video frame includes: performing inter-frame alignment processing on the standard video frame to obtain aligned video frames; using a preset spatial feature extraction network to extract spatial feature maps of each aligned video frame; performing multi-scale pooling processing on the spatial feature maps to generate spatial features of the standard video frame; performing cyclic convolution processing on the spatial feature maps according to a preset temporal dimension to generate inter-frame motion trajectories and pixel change trends of the standard video frame; performing temporal pooling processing on the inter-frame motion trajectories and pixel change trends to obtain target inter-frame motion trajectories and target pixel change trends; and using the target inter-frame motion trajectories and target pixel change trends as the temporal features of the standard video frame.

4. The video frame extraction method based on adaptive regularization as described in claim 1, characterized in that, The step of constructing a multi-resolution video feature map of the standard video frame based on the spatial features and the temporal features includes: performing multi-scale spatial downsampling on the spatial features to generate multiple spatial resolution feature maps; performing multi-granularity temporal pooling on the temporal features to generate multiple temporal resolution feature maps; identifying the spatial resolution levels of the multiple spatial resolution feature maps and the temporal resolution levels of the temporal resolution feature maps; fusing the corresponding spatial resolution feature maps with the temporal resolution feature maps according to the spatial resolution levels and the temporal resolution levels to obtain a primary multi-resolution fused feature map; performing cross-resolution feature enhancement on the primary multi-resolution fused feature map to obtain multiple enhanced resolution video feature maps; and concatenating the multiple enhanced resolution video feature maps along a preset channel dimension to obtain the multi-resolution video feature map of the standard video frame.

5. The video frame extraction method based on adaptive regularization as described in claim 1, characterized in that, The step of performing multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients includes: performing multi-head self-attention calculation on the multi-resolution video feature map to generate a query matrix, key matrix, and value matrix corresponding to each attention head; calculating the multi-head attention distribution matrix of the multi-resolution video feature map based on the query matrix and the key matrix; performing matrix normalization on the multi-head attention distribution matrix to obtain the attention weight matrix of each attention head; multiplying the attention weight matrix of each attention head with the corresponding value matrix to obtain the multi-head attention feature map of each attention head; performing global average pooling on each of the multi-head attention feature maps to obtain pooled feature maps; and performing linear fully connected computation on the pooled feature maps to obtain multiple graph regularization coefficients.

6. The video frame extraction method based on adaptive regularization as described in claim 1, characterized in that, The step of performing adaptive regularization constraint processing on the standard video frames based on the graph regularization coefficients to generate multiple regularized video frames includes: determining a regularization intensity factor corresponding to each standard video frame based on the multiple graph regularization coefficients; generating an adaptive regularization constraint function for each standard video frame based on the regularization intensity factor; performing local structure regularization processing on each standard video frame using the adaptive regularization constraint function to obtain locally regularized video frames; applying inter-frame consistency constraints to the locally regularized video frames to obtain constrained video frames; calculating the pixel gradient change of the constrained video frames, and determining the regularized video frames in the constrained video frames based on the pixel gradient change.

7. The video frame extraction method based on adaptive regularization as described in claim 1, characterized in that, The step of selecting a target video frame from multiple regularized video frames to obtain the target video frame corresponding to the initial video stream data includes: calculating the image clarity feature value and content clarity feature value for each regularized video frame; calculating the inter-frame difference feature value between the regularized video frames based on the temporal position of each regularized video frame; performing weighted feature value fusion on the image clarity feature value, the content clarity feature value, and the inter-frame difference feature value to obtain a target feature value; selecting frames from the regularized video frames based on a preset adaptive feature threshold and the target feature value to obtain multiple candidate keyframes; and performing content redundancy removal processing on the multiple candidate keyframes to obtain the target video frame.

8. A video frame extraction device based on adaptive regularization, characterized in that, The apparatus includes: a video frame normalization module, used to decompose initial video stream data into multiple consecutive initial video frames, and perform frame normalization processing on the multiple initial video frames to generate multiple standard video frames; a spatiotemporal feature extraction module, used to extract spatial and temporal features of the standard video frames, and construct a multi-resolution video feature map of the standard video frames based on the spatial and temporal features; an attention calculation module, used to perform multi-head self-attention regularization calculation on the multi-resolution video feature map to obtain multiple graph regularization coefficients; a regularization constraint module, used to perform adaptive regularization constraint processing on the standard video frames based on the graph regularization coefficients to generate multiple regularized video frames; and a target frame selection module, used to select target frames from the multiple regularized video frames to obtain the target video frames corresponding to the initial video stream data.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the adaptive regularization-based video frame extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video frame extraction method based on adaptive regularization as described in any one of claims 1 to 7.

Images