Video frame extraction method and device based on double-flow decoupling architecture, equipment and medium

Abstract

The invention relates to the technical field of image processing, can be applied to the field of medical health and vehicles, and discloses a video frame extraction method and device based on a double-flow decoupling architecture, equipment and a medium, and the method comprises the steps: extracting a spatial static feature from a single-frame sample image of a video sample through a spatial content stream encoder, extracting time dynamic features from the sequence of continuous frame samples of the video sample through a time dynamic stream encoder; performing adversarial decoupling on the space static characteristics and the time dynamic characteristics through an adversarial loss function by taking the minimization of mutual information between the space static characteristics of the single-frame sample image and the out-of-order time dynamic characteristics of the continuous frame sample sequence to which the single-frame sample image belongs as a target; fusing the space content features after adversarial decoupling with the time dynamic features to generate fusion features of the single-frame sample image; and training a key frame extraction model based on the fusion feature and the key frame label of the single-frame sample image, so as to extract a key frame image in the target video through the key frame extraction model.

Description

Technical Field

[0001] This application relates to the field of video processing technology and can be applied to the fields of medical health and vehicles. In particular, it relates to a video frame extraction method, apparatus, device and medium based on a dual-stream decoupling architecture. Background Technology

[0002] Video keyframe extraction is one of the core tasks of video analysis and retrieval. Its goal is to select the most representative and informative frames from a video sequence to reduce storage costs and improve subsequent processing efficiency. Traditional methods mainly rely on manually designed features (such as optical flow and color histograms) or clustering / compressed sensing-based algorithms, but these methods struggle to capture high-level semantic information and have poor adaptability to dynamic scenes. With the development of deep learning, end-to-end methods based on convolutional neural networks (CNNs) have made significant progress by modeling spatiotemporal features, but they still require a large amount of labeled data for training, and their performance drops sharply in scenarios with few samples. Summary of the Invention

[0003] In view of this, this application provides a video frame extraction method, apparatus, device and medium based on a dual-stream decoupling architecture to address the semantic confusion problem caused by spatiotemporal feature coupling in traditional methods.

[0004] Firstly, a video frame extraction method based on a dual-stream decoupling architecture is provided, including: Spatial static features are extracted from single-frame sample images of video samples by a spatial content stream encoder, and temporal dynamic features are extracted from a continuous frame sample sequence of the video samples by a temporal dynamic stream encoder, wherein the continuous frame sample sequence includes the single-frame sample images. With the goal of minimizing the mutual information between the spatial static features of the single-frame sample image and the disordered temporal dynamic features of the continuous frame sample sequence to which the single-frame sample image belongs, the spatial static features and the temporal dynamic features are adversarially decoupled through an adversarial loss function. The spatial content features after adversarial decoupling are fused with the temporal dynamic features to generate the fused features of the single-frame sample image; A keyframe extraction model is trained based on the fusion features and the keyframe labels of the single-frame sample images to extract keyframe images from the target video.

[0005] Secondly, a video frame extraction device based on a dual-stream decoupling architecture is provided, comprising: The feature extraction module is used to extract spatial static features from a single frame sample image of a video sample using a spatial content stream encoder, and to extract temporal dynamic features from a continuous frame sample sequence of the video sample using a temporal dynamic stream encoder, wherein the continuous frame sample sequence includes the single frame sample image. The decoupling module is used to decouple the spatial static features and the temporal dynamic features of the continuous frame sample sequence to which the single frame sample image belongs from the goal of minimizing the mutual information between the spatial static features and the temporal dynamic features through an adversarial loss function. The feature fusion module is used to fuse the spatial content features after adversarial decoupling with the temporal dynamic features to generate the fused features of the single-frame sample image; The training module is used to train a keyframe extraction model based on the fused features and the keyframe labels of the single-frame sample images, so as to extract keyframe images from the target video through the keyframe extraction model.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video frame extraction method based on the dual-stream decoupling architecture described above.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the video frame extraction method based on the dual-stream decoupling architecture described above.

[0008] The aforementioned video frame extraction method, apparatus, device, and medium based on a dual-stream decoupling architecture removes shared or redundant correlation information between spatial static features and temporal dynamic features through adversarial decoupling. This allows spatial features to focus on the essential content of a single frame sample image, while temporal features accurately reflect the dynamic change patterns of the frame sequence. By training the keyframe extraction model through feature fusion after decoupling, the trained model accurately captures the core discriminative attributes of keyframes, significantly reducing the amount of data required by the model. Thus, even with limited labeled data, the keyframe extraction model can efficiently and accurately identify and extract keyframes with the highest action semantic value, significantly improving the accuracy and robustness of keyframe extraction from target videos.

[0009] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the application environment for the video frame extraction method based on the dual-stream decoupling architecture in this application; Figure 2 This is a flowchart illustrating the video frame extraction method based on a dual-stream decoupling architecture in this application. Figure 3 This is a schematic diagram of the video frame extraction device based on the dual-stream decoupling architecture in this application; Figure 4 This is a schematic diagram of the computer device in this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The video frame extraction method based on a dual-stream decoupling architecture provided in this application relates to the fields of video processing technology and medical and health technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing a video search method, but is not limited to the above forms.

[0014] For example, the video frame extraction method based on a dual-stream decoupling architecture provided in this embodiment of the invention can be applied to, for example, Figure 1In this application environment, the camera device and the client communicate with the server via a network. The server can acquire video samples or target videos through the camera device. The server extracts spatial static features from single-frame sample images of the video samples using a spatial content stream encoder, and extracts temporal dynamic features from continuous frame sample sequences of the video samples using a temporal dynamic stream encoder. Aiming to minimize the mutual information between spatial static features and disordered temporal dynamic features, an adversarial loss function is used to decouple the spatial static features and temporal dynamic features. The adversarially decoupled spatial content features and temporal dynamic features are then fused to generate fused features for a single-frame sample image. A keyframe extraction model is trained based on the fused features and keyframe labels of the single-frame sample image. The server extracts keyframe images from the target video using the keyframe extraction model and returns them to the client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates this invention.

[0015] The following is a breakdown of several terms used in this application: Keyframe: A computer animation term referring to the frame containing a key action in the movement of a character or object, equivalent to a keyframe in 2D animation. Animation between keyframes can be created and added by software, called transition frames or in-between frames. A "frame" is the smallest unit of image in animation, equivalent to a single frame on film. On the timeline of animation software, a frame is represented by a single frame or a marker.

[0016] Encoder: An encoder is used to transform an input sequence into a fixed-length vector.

[0017] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the video frame extraction method based on a dual-stream decoupling architecture provided in this application embodiment includes the following steps: S110: Extract spatial static features from single-frame sample images of video samples using a spatial content stream encoder, and extract temporal dynamic features from a sequence of consecutive frame samples of video samples using a temporal dynamic stream encoder.

[0018] Specifically, the Spatial Stream Encoder is used to extract static visual features within a single frame, such as environmental features where the action occurs, participant attribute features, and spatial context features. The Temporal Stream Encoder is used to extract motion patterns and temporal evolution information from a sequence of consecutive frame samples, such as motion trajectories, human / object pose sequences, and state changes.

[0019] For example, the ViT-B / 16 model is used to process single-frame sample images. Output a 768-dimensional feature vector : ; ViT consists of 12 Transformer encoding layers, with an input image block size of 16×16. This feature captures the composition of objects in the scene (e.g., "medical equipment + patient + chair") and their spatial relationships.

[0020] Processing continuous frame sample sequences based on TSM-3D ResNet Output a 768-dimensional feature vector : ; The TSM module introduces temporal modeling into 2D convolutions through channel shifting operations, while the 3D ResNet-18 contains four 3D convolutional blocks. This feature describes motion patterns (such as "movement from left to right").

[0021] S120: With the goal of minimizing the mutual information between the spatial static features of a single-frame sample image and the disordered temporal dynamic features of the continuous frame sample sequence to which the single-frame sample image belongs, the spatial static features and temporal dynamic features are adversarially decoupled through an adversarial loss function.

[0022] In this embodiment, under the constraint of minimizing mutual information, the adversarial loss is minimized to force the two-stream network to eliminate mutual information between features for a single-frame sample image and its corresponding consecutive frame sample sequence, thus achieving decoupling. This removes redundant associations between the two types of features, allowing spatial static features to focus on representing the content attributes of a single-frame sample image, and temporal dynamic features to focus on representing the motion attributes of the frame sequence. This results in the two types of features having independent semantic orientations after decoupling, avoiding information interference between feature dimensions and improving the ability to describe the core information of the training data. Furthermore, during the structural process, out-of-order operations eliminate the interference of temporal correlation between consecutive frames, preventing temporal dynamic features from establishing associations with spatial static features through single-frame content. This forces the adversarial loss to rely solely on the motion differences between frames to form an effective representation, thereby more accurately driving the decoupling of the two types of features in the content dimension and ensuring the independence of temporal dynamic features.

[0023] Specifically, the adversarial loss function is expressed as: ; In the formula, It is a spatial static feature. For time-dynamic features, This refers to the out-of-order temporal dynamics. It is a one-way discriminator with the goal of minimizing mutual information.

[0024] In one embodiment, considering that the discriminator's task is relatively easy in one-way adversarial scenarios, the discriminator only focuses on one feature but does not impose constraints on the other feature, resulting in asymmetric information redundancy between spatial static features and temporal dynamic features, and not truly achieving complete decoupling. Therefore, before step S120, the video frame extraction method based on the dual-stream decoupling architecture further includes: performing high-order semantic mapping on spatial static features and temporal dynamic features through spatial feature refinement networks and temporal feature refinement networks respectively, obtaining high-order spatial features and high-order temporal features; training the first discriminator using high-order temporal features as positive samples and high-order spatial features as negative samples, and optimizing the parameters of the spatial feature refinement network with the goal of maximizing the discrimination rate of the first discriminator for high-order spatial features; training the second discriminator using high-order spatial features as positive samples and high-order temporal features as negative samples, and optimizing the parameters of the temporal feature refinement network with the goal of maximizing the discrimination rate of the second discriminator for high-order temporal features; and constructing a bidirectional discriminator with an adversarial loss function based on the first and second discriminators.

[0025] In this embodiment, two discriminators establish two independent adversarial processes, decoupling the adversarial loss function from two directions. This results in spatial static features that more purely represent objects and scenes, while temporal dynamic features more purely represent actions and events, avoiding low-level noise interference. This achieves symmetrical deep decoupling of spatial static and temporal dynamic features, facilitating the generation of higher-quality, more expressive decoupled features. Simultaneously, a higher-order semantic mapping layer is introduced, allowing the adversarial game to take place in a more aligned feature space. This effectively alleviates training instability caused by differences in the original feature distribution, improving decoupling efficiency and model convergence robustness. Furthermore, targeted optimization objectives guide the refinement of the network to remove irrelevant cross-modal information from each feature, further enhancing the decoupling effect.

[0026] S130: The spatial content features and temporal dynamic features after adversarial decoupling are fused to generate fused features for a single frame sample image.

[0027] S140: Train a keyframe extraction model based on fused features and keyframe labels of single-frame sample images to extract keyframe images from the target video.

[0028] The keyframe label includes a classification label indicating whether a single-frame sample image is a keyframe and / or a similarity label between the single-frame sample image and the keyframe prototype.

[0029] The video frame extraction method based on a dual-stream decoupling architecture provided in this application removes shared or redundant correlation information between spatial static features and temporal dynamic features through adversarial decoupling. This allows spatial features to focus on the essential content of a single frame sample image, while temporal features accurately reflect the dynamic change patterns of the frame sequence. By training the keyframe extraction model through feature fusion after decoupling, the trained keyframe extraction model accurately captures the core discriminative attributes of keyframes, significantly reducing the amount of data required by the model. Thus, even with limited labeled data, the keyframe extraction model can efficiently and accurately identify and extract keyframes with the highest action semantic value, significantly improving the accuracy and robustness of keyframe extraction from target videos.

[0030] In some embodiments of this application, step S140, which trains a keyframe extraction model based on fused features and keyframe labels of single-frame sample images, specifically includes the following steps: S141: Input the fused features into the preset extraction model, and determine the binary classification probability value and prototype similarity probability value corresponding to the fused features through the preset extraction model.

[0031] S142: Based on the binary classification probability value and the prototype similarity probability value, calculate the total loss value of the preset extraction model through the target loss function.

[0032] The objective loss function is a weighted sum of the keyframe classification loss function and the keyframe similarity loss function. The keyframe classification loss function is a cross-entropy loss function that predicts whether a single-frame sample image is a keyframe based on fused features. The keyframe similarity function is a cross-entropy loss function that predicts the similarity between a single-frame sample image and the keyframe set based on the Euclidean distance between the fused features and the prototype features in the keyframe prototype library. The keyframe set includes multiple single-frame sample images labeled as keyframes, and the prototype features are the average features of the keyframe set.

[0033] For example, a lightweight multilayer perceptron (MLP) is used as the preset extraction model. After obtaining the fused features, the fused features are first projected onto a low-dimensional space. The last layer of the MLP uses the sigmoid function to determine whether it is a keyframe. The FC layer of the MLP reduces the dimensionality of the fused dual-stream features from 2d (2x768 dimensions) to d (768 dimensions), and then performs a normalization operation. The similarity with the keyframe is determined using the normalized vector features.

[0034] The normalization operation can be represented as: ; The prototype feature can be represented as: ; The keyframe similarity function can be expressed as: ; In the formula, Let be the fusion feature of the q-th single-frame sample image. The fusion features of the single-frame sample image labeled as the i-th keyframe are: The number of frames in the keyframe set. These are prototype features.

[0035] S143: Optimize the preset extraction model based on the total loss value until the loss function converges, and obtain the trained keyframe extraction model.

[0036] In this embodiment, a preset extraction model synchronously outputs a binary classification probability value for whether the fused features are keyframes, and a similarity probability value with a representative keyframe prototype. A weighted sum of keyframe classification loss and similarity loss is used to calculate the total loss, and this total loss is used to optimize the model parameters. This allows for simultaneous constraints on model training from two dimensions: keyframe classification and keyframe feature similarity. This ensures the model accurately distinguishes between keyframes and non-keyframes while also encouraging the model to learn prototype representations that better match the core features of keyframes. This allows the model to infer new, unseen keyframes by calculating similarity with the prototype. This effectively avoids the model falling into overfitting or class imbalance-induced discrimination bias and significantly reduces reliance on large-scale labeled data, better adapting to small-sample application scenarios.

[0037] In some embodiments of this application, after step S110, the video frame extraction method based on the dual-stream decoupling architecture further includes the following steps: S210: Obtain the feature memory of the current feature.

[0038] The current feature can be either a spatial content feature or a temporal dynamic feature. The feature memories for spatial content features and temporal dynamic features can be independent of each other, or they can be stored in a single feature memory.

[0039] It should be noted that the feature memory is dynamically updated using an exponential moving average method. This ensures that the stored features slowly and steadily reflect the latest learning state of the encoder, without being affected by drastic fluctuations in noise from individual batches. This guarantees a high-quality and diverse set of negative samples used for contrastive learning, which is crucial for effective contrastive learning.

[0040] For example, feature memory Regarding the spatial content stream sample vector type The update rule can be expressed as: ; Feature memory Medium-time dynamic stream sample vector type The update rule can be expressed as: ; In the formula, For spatial content feature samples in the feature memory, These are time-dynamic feature samples in the feature memory.

[0041] S220: Based on the positive and negative samples in the feature memory, the current feature is evaluated using a contrastive loss function to determine the contrastive loss value.

[0042] In this context, positive samples are features belonging to the same semantic object as the current feature, while negative samples are features belonging to the same semantic object as the current feature. The contrastive loss aims to maximize the similarity between positive samples and minimize the similarity between negative samples. For example, if the current feature is a spatial content feature describing human appearance, then positive samples are spatial content features describing human appearance in the feature memory, and negative samples are spatial content features describing the scene environment in the feature memory.

[0043] The contrastive loss function is expressed as: ; In the formula, For the current sample features, As a positive sample, For negative samples, For temperature hyperparameters, This is a vector dot product operation.

[0044] S230: Based on the contrastive loss value, optimize the parameters of the spatial content stream encoder or the temporal dynamic stream encoder corresponding to the current feature.

[0045] In this embodiment, contrastive loss gradient backpropagation is utilized through contrastive learning to maximize the similarity between positive samples (the same semantic object / event) and minimize the similarity between negative samples (different semantic objects / events). This allows the encoder to learn more discriminative features and further optimizes the parameters of the spatial content flow encoder and the temporal dynamic flow encoder. Therefore, samples with different semantics can be distinguished in the feature space without providing explicit semantic labels for each sample. This makes the representation capabilities of the spatial content flow encoder and the temporal dynamic flow encoder more accurate, effectively mitigating prototype drift in low-sample scenarios and providing high-quality input features for downstream model learning tasks.

[0046] In some embodiments of this application, keyframe images are extracted from a target video using a keyframe extraction model. Specifically, this includes: obtaining the fusion features of each frame image in the target video; inputting the fusion features of each frame image in the target video into the keyframe extraction model to obtain the keyframe classification value and keyframe similarity value of each frame image; and extracting keyframe images from each frame image in the target video based on the keyframe classification value and keyframe similarity value.

[0047] It is understood that the fusion features of each frame in the target video can be obtained by using the same data extraction and fusion method as the single frame sample image of the video sample (steps S110~S120 above), which will not be repeated in this application embodiment.

[0048] In this embodiment, a comprehensive judgment of frame images is made using both classification and similarity dimensions. Classification values ​​are used to quickly filter candidate frames that match the keyframe category attributes, while similarity values ​​are used to further verify the fit between candidate frames and keyframe prototype features. This effectively avoids the problems of misjudgment and missed detection caused by relying on only a single indicator. Thus, it is possible to effectively identify and extract the most representative image frames from complex videos, greatly improving the efficiency of video content summarization and key point extraction.

[0049] It should be noted that keyframe classification values ​​and keyframe similarity values ​​can be compared with their corresponding thresholds. A frame is considered a keyframe if either the keyframe classification value or the keyframe similarity value meets the criteria. Alternatively, keyframe classification values ​​and keyframe similarity values ​​can be used as parallel criteria; a frame is considered a keyframe if both the keyframe classification value and the keyframe similarity value meet the criteria.

[0050] In some embodiments of this application, after extracting keyframe images from the target video using a keyframe extraction model, the following steps are further included: S310: Extract a video segment containing the keyframe image from the target video.

[0051] S320: Extract the human body region of each frame in the video segment using an object detection algorithm.

[0052] S330: Based on the temporal sequence of each frame in the video segment, the spatial static features of the keyframe image located in the human body region are correlated to generate a human body pose sequence of the video segment.

[0053] S340: Match the human posture sequence with the high-risk posture sequences stored in the medical event database.

[0054] S350: If the similarity between the human posture sequence and the high-risk posture sequence is greater than the similarity threshold, a medical event warning is triggered.

[0055] In this embodiment, a human posture sequence is generated by extracting video clips containing keyframes, extracting human body regions, and temporally associating the spatial static features of human body regions. This sequence is then matched with high-risk posture sequences in a medical event database. An alert is triggered when the similarity exceeds a threshold. This approach can focus on critical action periods, accurately locate human targets, and reconstruct continuous posture change trajectories, avoiding background interference and ineffective analysis of non-critical periods. Furthermore, relying on standardized high-risk posture matching logic, it enables automated, accurate identification and timely alerts for medical risk events, effectively improving the efficiency and reliability of risk monitoring in medical scenarios and reducing the risk of medical delays caused by human oversight.

[0056] For example, in a healthcare and elderly care scenario, historical video streams of elderly people's activities are collected by cameras deployed in the nursing home environment. The system extracts spatial static features from single-frame images of the historical video using a spatial content stream encoder, such as the limb outline of the elderly person when standing, the torso shape when sitting, and the relative position of the body to the ground when falling. Simultaneously, a temporal dynamic stream encoder extracts temporal dynamic features from continuous frame sequences, such as the inter-frame motion trajectories of actions such as the elderly person getting up, walking, accelerating from top to bottom, and falling due to imbalance. Subsequently, the system decouples the two types of features using an adversarial loss function and then fuses the decoupled features into a single-frame fused feature. Combined with annotation training of a keyframe extraction model for single-frame content, after the model is trained, a 10-minute video of the lobby captured in real time is input into the keyframe extraction model. The model accurately extracts 5 labeled fall events (e.g., falls, slips) and 3 unlabeled keyframe images that match the fall characteristics from the real-time video stream, indicating abnormal elderly person movements (e.g., falls, slips). The system extracts short video clips containing the aforementioned keyframes (e.g., a continuous sequence of 3 frames before and after each keyframe). It then uses a target detection algorithm to locate the human body region in each frame of the clip, eliminating irrelevant interference such as background furniture and clutter. Based on the temporal relationship between the frames of the short video clip, the system correlates the spatial static features of the keyframes corresponding to the human body region to generate a complete human posture sequence, such as a continuous posture change trajectory of "standing → leaning forward → shifting center of gravity → losing balance → falling." The system performs similarity matching between this real-time posture sequence and pre-stored high-risk posture sequences in a medical event database. If the similarity exceeds a preset threshold (e.g., any value between 80% and 95%), a medical event warning is immediately triggered. This is achieved by pushing warning information to family members' mobile phones or medical staff's monitoring computers, enabling timely intervention in fall risk and reducing the risk of secondary injury to the elderly due to delayed treatment.

[0057] It is worth mentioning that the video frame extraction method of the dual-stream decoupled architecture provided in this application can not only be applied to medical scenarios, but also to scenarios that require key frame extraction, such as transportation, industrial quality inspection, and bank monitoring.

[0058] For example, in a vehicle-assisted driving scenario, the vehicle's front-facing camera captures driving video at a rate of 30 frames per second. The system extracts spatial static features reflecting static elements of the road scene from the continuous video stream captured by the vehicle's camera using a spatial content stream encoder, and extracts temporal dynamic features reflecting the motion and change patterns of objects and the scene from the continuous video stream captured by the vehicle's camera using a temporal dynamic stream encoder. The system decouples the two types of features using an adversarial loss function, and then fuses the two types of features into a single-frame fused feature. The system uses a large amount of labeled driving video data, in which experts have labeled key frames of key events (such as the vehicle in front starting to brake, a pedestrian appearing in front of the vehicle, traffic light changes, etc.). Combining the fused features and their binary classification labels, a keyframe extraction classification model is trained. After the model is trained, the real-time captured continuous video stream is input into the keyframe extraction model, which accurately extracts driving keyframes from the real-time video stream for immediate use in triggering warnings from the Advanced Driver Assistance System (ADAS).

[0059] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0060] In some embodiments of this application, a video frame extraction device based on a dual-stream decoupling architecture is provided, which corresponds one-to-one with the video frame extraction method based on a dual-stream decoupling architecture described above. For example... Figure 3 As shown, the video frame extraction device based on a dual-stream decoupling architecture includes a feature extraction module 301, a decoupling module 302, a feature fusion module 303, and a training module 304. Detailed descriptions of each functional module are as follows: The feature extraction module 301 is used to extract spatial static features from a single frame sample image of a video sample through a spatial content stream encoder, and to extract temporal dynamic features from a continuous frame sample sequence of a video sample through a temporal dynamic stream encoder, wherein the continuous frame sample sequence contains single frame sample images. Decoupling module 302 is used to decouple spatial static features and temporal dynamic features adversarially by using an adversarial loss function, with the goal of minimizing the mutual information between the spatial static features of a single frame sample image and the disordered temporal dynamic features of the continuous frame sample sequence to which the single frame sample image belongs. The feature fusion module 303 is used to fuse the spatial content features and temporal dynamic features after adversarial decoupling to generate fused features of a single-frame sample image. Training module 304 is used to train a keyframe extraction model based on fused features and keyframe labels of single-frame sample images, so as to extract keyframe images from the target video through the keyframe extraction model.

[0061] In one embodiment, the adversarial loss function is expressed as: ; In the formula, It is a spatial static feature. For time-dynamic features, This refers to the out-of-order temporal dynamics. It is a discriminator that aims to minimize mutual information.

[0062] In one embodiment, the training module 304 is further configured to perform high-order semantic mapping on spatial static features and temporal dynamic features through a spatial feature refinement network and a temporal feature refinement network, respectively, to obtain high-order spatial features and high-order temporal features; train a first discriminator using high-order temporal features as positive samples and high-order spatial features as negative samples, and optimize the parameters of the spatial feature refinement network with the goal of maximizing the discrimination rate of the first discriminator on the high-order spatial features; train a second discriminator using high-order spatial features as positive samples and high-order temporal features as negative samples, and optimize the parameters of the temporal feature refinement network with the goal of maximizing the discrimination rate of the second discriminator on the high-order temporal features; and construct a bidirectional discriminator with an adversarial loss function based on the first discriminator and the second discriminator.

[0063] In one embodiment, the video frame extraction device based on the dual-stream decoupling architecture further includes: The meta-learning module (not shown in the figure) is used to acquire the feature memory of the current feature, where the current feature is either a spatial content feature or a temporal dynamic feature. The feature memory is dynamically updated using an exponential moving average. Based on the positive and negative samples in the feature memory, the current feature is evaluated using a contrastive loss function to determine the contrastive loss value. Positive samples are similar features belonging to the same semantic object as the current feature, while negative samples are similar features belonging to different semantic objects. The contrastive loss aims to maximize the similarity between positive samples and minimize the similarity between negative samples. Based on the contrastive loss value, the parameters of the spatial content stream encoder or the temporal dynamic stream encoder corresponding to the current feature are optimized. The contrastive loss function is expressed as: ; In the formula, For the current sample features, As a positive sample, For negative samples, For temperature hyperparameters, This is a vector dot product operation.

[0064] In one embodiment, the training module 304 is specifically used to input the fused features into a preset extraction model, determine the binary classification probability value and prototype similarity probability value corresponding to the fused features through the preset extraction model; calculate the total loss value of the preset extraction model based on the binary classification probability value and prototype similarity probability value through a target loss function, wherein the target loss function is a weighted sum of the keyframe classification loss function and the keyframe similarity loss function; optimize the preset extraction model based on the total loss value until the loss function converges, thereby obtaining the trained keyframe extraction model.

[0065] In one embodiment, the keyframe classification loss function is the cross-entropy loss function, which predicts whether a single-frame sample image is a keyframe based on fused features. The keyframe similarity function is a cross-entropy loss function that predicts the similarity between a single-frame sample image and the keyframe set based on the Euclidean distance between the fused features and the prototype features in the keyframe prototype library. The keyframe set includes multiple single-frame sample images labeled as keyframes, and the prototype features are the average features of the keyframe set.

[0066] In one embodiment, the feature extraction module 301 is further configured to acquire fusion features of each frame of the target video; The video frame extraction device based on the dual-stream decoupling architecture also includes: The keyframe extraction module (not shown in the figure) is used to input the fusion features of each frame image in the target video into the keyframe extraction model to obtain the keyframe classification value and keyframe similarity value of each frame image; and, based on the keyframe classification value and keyframe similarity value, to extract keyframe images from each frame image of the target video.

[0067] In one embodiment, the video frame extraction device based on the dual-stream decoupling architecture further includes: The video extraction module (not shown in the figure) is used to extract video segments containing keyframe images from the target video; The human pose recognition module (not shown in the figure) is used to extract the human body region of each frame in the video clip through the target detection algorithm; and to associate the spatial static features of the key frame images located in the human body region based on the temporal sequence of each frame in the video clip to generate the human pose sequence of the video clip; and to match the human pose sequence with the high-risk pose sequence stored in the medical event database. The early warning module (not shown in the figure) is used to trigger a medical event warning if the similarity between a human posture sequence and a high-risk posture sequence is greater than a similarity threshold.

[0068] This application provides a video frame extraction device based on a dual-stream decoupling architecture. Through adversarial decoupling, it removes shared or redundant correlation information between spatial static features and temporal dynamic features, allowing spatial features to focus on the essential content of a single frame sample image, and temporal features to accurately reflect the dynamic change patterns of the frame sequence. By fusing the decoupled features to train a keyframe extraction model, the trained model accurately captures the core discriminative attributes of keyframes, significantly reducing the amount of data required by the model. Thus, even with limited labeled data, the keyframe extraction model can efficiently and accurately identify and extract keyframes with the highest action semantic value, significantly improving the accuracy and robustness of keyframe extraction from target videos.

[0069] Specific limitations regarding the video frame extraction device based on a two-stream decoupling architecture can be found in the limitations of the video frame extraction method based on a two-stream decoupling architecture described above, and will not be repeated here. Each module in the aforementioned video frame extraction device based on a two-stream decoupling architecture can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0070] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: extracting spatial static features from a single-frame sample image of a video sample using a spatial content stream encoder, and extracting temporal dynamic features from a continuous frame sample sequence of video samples using a temporal dynamic stream encoder; decoupling the spatial static features and temporal dynamic features adversarially using an adversarial loss function, with the goal of minimizing the mutual information between the spatial static features of the single-frame sample image and the disordered temporal dynamic features of the continuous frame sample sequence to which the single-frame sample image belongs; fusing the adversarially decoupled spatial content features and temporal dynamic features to generate fused features of the single-frame sample image; and training a keyframe extraction model based on the fused features and keyframe labels of the single-frame sample image to extract keyframe images from the target video using the keyframe extraction model.

[0071] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon. When executed by a processor, the computer program performs the following steps: extracting spatial static features from a single-frame sample image of a video sample using a spatial content stream encoder, and extracting temporal dynamic features from a continuous frame sample sequence of the video sample using a temporal dynamic stream encoder; decoupling the spatial static features and temporal dynamic features adversarially using an adversarial loss function, with the goal of minimizing the mutual information between the spatial static features of the single-frame sample image and the disordered temporal dynamic features of the continuous frame sample sequence to which the single-frame sample image belongs; fusing the adversarially decoupled spatial content features and temporal dynamic features to generate fused features of the single-frame sample image; and training a keyframe extraction model based on the fused features and keyframe labels of the single-frame sample image to extract keyframe images from the target video using the keyframe extraction model.

[0072] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram is shown below. Figure 4 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a video frame extraction method based on a dual-stream decoupling architecture.

[0073] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant description of the video frame extraction method based on the dual-stream decoupling architecture in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0074] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0076] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A video frame extraction method based on a two-stream decoupling architecture, characterized in that, The method includes: Spatial static features are extracted from single-frame sample images of video samples by a spatial content stream encoder, and temporal dynamic features are extracted from a sequence of consecutive frame samples of the video samples by a temporal dynamic stream encoder. With the goal of minimizing the mutual information between the spatial static features of the single-frame sample image and the disordered temporal dynamic features of the continuous frame sample sequence to which the single-frame sample image belongs, the spatial static features and the temporal dynamic features are adversarially decoupled through an adversarial loss function. The spatial content features after adversarial decoupling are fused with the temporal dynamic features to generate the fused features of the single-frame sample image; A keyframe extraction model is trained based on the fusion features and the keyframe labels of the single-frame sample images to extract keyframe images from the target video.

2. The video frame extraction method based on a dual-stream decoupling architecture according to claim 1, characterized in that, The method further includes: By using spatial feature refinement networks and temporal feature refinement networks, high-order semantic mapping is performed on the spatial static features and the temporal dynamic features respectively to obtain high-order spatial features and high-order temporal features. The first discriminator is trained using the higher-order temporal features as positive samples and the higher-order spatial features as negative samples. The parameters of the spatial feature refinement network are optimized with the goal of maximizing the discrimination rate of the first discriminator for the higher-order spatial features. The second discriminator is trained using the higher-order spatial features as positive samples and the higher-order temporal features as negative samples. The parameters of the temporal feature refinement network are optimized with the goal of maximizing the discrimination rate of the second discriminator for the higher-order temporal features. A bidirectional discriminator for the adversarial loss function is constructed based on the first discriminator and the second discriminator.

3. The video frame extraction method based on a dual-stream decoupling architecture according to claim 1, characterized in that, The method further includes: Obtain the feature memory of the current feature, wherein the current feature is the spatial content feature or the temporal dynamic feature, and the feature memory is dynamically updated using an exponential moving average method; Based on the positive and negative samples in the feature memory, the current feature is evaluated by a contrastive loss function to determine the contrastive loss value. The positive samples are features of the same semantic object as the current feature, and the negative samples are features of the same semantic object as the current feature. The contrastive loss aims to maximize the similarity between positive samples and minimize the similarity between negative samples. Based on the contrast loss value, optimize the parameters of the spatial content stream encoder or the temporal dynamic stream encoder corresponding to the current feature.

4. The video frame extraction method based on a dual-stream decoupling architecture according to claim 1, characterized in that, The training of the keyframe extraction model based on the fused features and the keyframe labels of the single-frame sample image includes: The fused features are input into a preset extraction model, and the binary classification probability value and prototype similarity probability value corresponding to the fused features are determined by the preset extraction model. Based on the binary classification probability value and the prototype similarity probability value, the total loss value of the preset extraction model is calculated through the target loss function, wherein the target loss function is a weighted sum of the keyframe classification loss function and the keyframe similarity loss function; The preset extraction model is optimized based on the total loss value until the loss function converges, thus obtaining the trained keyframe extraction model.

5. The video frame extraction method based on a dual-stream decoupling architecture according to claim 4, characterized in that, The keyframe classification loss function is the cross-entropy loss function, which predicts whether a single-frame sample image is a keyframe based on the fusion features. The keyframe similarity function is a cross-entropy loss function that predicts the similarity between a single-frame sample image and the keyframe set based on the Euclidean distance between the fused features and the prototype features in the keyframe prototype library. The keyframe set includes multiple single-frame sample images labeled as keyframes, and the prototype features are the average features of the keyframe set.

6. The video frame extraction method based on a dual-stream decoupling architecture according to any one of claims 1 to 5, characterized in that, The step of extracting keyframe images from the target video using the keyframe extraction model includes: Obtain the fusion features of each frame in the target video; The fusion features of each frame in the target video are input into the keyframe extraction model to obtain the keyframe classification value and keyframe similarity value of each frame. Based on the keyframe classification value and the keyframe similarity value, the keyframe images are extracted from each frame of the target video.

7. The video frame extraction method based on a dual-stream decoupling architecture according to any one of claims 1 to 5, characterized in that, After extracting keyframe images from the target video using the keyframe extraction model, the method further includes: Extract a video segment containing the keyframe image from the target video; Human body regions in each frame of the video segment are extracted using an object detection algorithm; Based on the temporal sequence of each frame in the video clip, the spatial static features of the keyframe images located in the human body region are correlated to generate a human pose sequence of the video clip; The human posture sequence is matched with high-risk posture sequences stored in the medical event database; If the similarity between the human posture sequence and the high-risk posture sequence is greater than a similarity threshold, a medical event warning is triggered.

8. A video frame extraction device based on a dual-stream decoupling architecture, characterized in that, The device includes: The feature extraction module is used to extract spatial static features from a single frame sample image of a video sample using a spatial content stream encoder, and to extract temporal dynamic features from a continuous frame sample sequence of the video sample using a temporal dynamic stream encoder, wherein the continuous frame sample sequence includes the single frame sample image. The decoupling module is used to decouple the spatial static features and the temporal dynamic features of the continuous frame sample sequence to which the single frame sample image belongs from the goal of minimizing the mutual information between the spatial static features and the temporal dynamic features through an adversarial loss function. The feature fusion module is used to fuse the spatial content features after adversarial decoupling with the temporal dynamic features to generate the fused features of the single-frame sample image; The training module is used to train a keyframe extraction model based on the fused features and the keyframe labels of the single-frame sample images, so as to extract keyframe images from the target video through the keyframe extraction model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the video frame extraction method based on a dual-stream decoupling architecture as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the video frame extraction method based on a dual-stream decoupling architecture as described in any one of claims 1 to 7.

Images