Video frame extraction method and device, equipment and medium

Abstract

The invention relates to the technical field of finance, medical health and artificial intelligence, and provides a video frame extraction method, device and equipment and a medium, which can perform self-supervised end-to-end training based on a cross-domain enhanced consistency mechanism to obtain a video frame extraction model, and improve the video frame extraction efficiency by combining cross-domain enhanced consistency regularization and self-supervised learning. The method improves the adaptability and robustness of key frame selection under variable conditions, does not need target domain annotation data, and achieves cross-domain robust key frame extraction while maintaining space-time coherence. A preprocessing module in the video frame extraction model is utilized to obtain a target space-time cube, and time sequence context information is introduced, so that an inter-frame dynamic relationship can be conveniently captured; time sequence enhancement features are obtained by using a domain adaptive feature encoder, multi-scale time sequence modeling is realized, and inter-frame time coherence is kept; the target frame is screened based on the dynamic threshold, and the screening granularity can be automatically optimized according to the scene complexity, so that the accuracy and adaptability of key frame extraction are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of finance, medical health and artificial intelligence, and particularly relates to a video frame extraction method and device, equipment and medium. BACKGROUND

[0002] Key frame extraction is a basic task in video processing, which aims to select the most representative frames from a video sequence to reduce data redundancy and retain the core content. For example, in the financial field, key frames can be extracted from automatic teller machine monitoring videos to identify abnormal behavior and ensure the safety of funds; in the medical health field, key frames can be extracted from operating room monitoring videos to assist medical teaching.

[0003] In the prior art, the following methods are mainly used for key frame extraction:

[0004] (1) Depend on manually designed features (such as color histogram, optical flow, etc.) and heuristic rules (such as shot boundary detection, inter-frame difference threshold, etc.).

[0005] This method has limitations in cross-scene generalization ability, especially when the video content or shooting conditions change greatly, the performance will decrease significantly.

[0006] (2) Deep learning technology, such as frame-level representation learning based on spatial feature extraction using convolutional neural networks, and capturing inter-frame dynamic relationship based on long short-term memory network and other time series models.

[0007] This method usually requires a large amount of labeled data and is sensitive to domain shift (such as videos from different light, angle or device sources). SUMMARY

[0008] In view of the above, it is necessary to provide a video frame extraction method, device, equipment and medium, which aims to solve the problems of poor generalization ability, low accuracy and the need for a large amount of labeled data in video frame extraction.

[0009] A video frame extraction method, the video frame extraction method comprising:

[0010] In response to a video frame extraction instruction for a target video in a target domain, a video frame extraction model obtained by self-supervised end-to-end training based on a cross-domain enhanced consistency mechanism is called;

[0011] The pre-processing module in the video frame extraction model is used to pre-process the target video to obtain a target spatio-temporal cube;

[0012] The domain adaptive feature encoder in the video frame extraction model is used to extract features from the target video and the target spatio-temporal cube to obtain time series enhanced features;

[0013] screen the time sequence enhanced features based on a dynamic threshold value to obtain target frames in the target video.

[0014] A video frame extraction device, comprising:

[0015] a calling unit configured to, in response to a video frame extraction instruction for a target video in a target domain, call a video frame extraction model obtained through self-supervised end-to-end training based on a cross-domain enhanced consistency mechanism;

[0016] a preprocessing unit configured to preprocess the target video by using a preprocessing module in the video frame extraction model to obtain a target spatio-temporal cube;

[0017] a feature extraction unit configured to extract features of the target video and the target spatio-temporal cube by using a domain adaptive feature encoder in the video frame extraction model to obtain time sequence enhanced features;

[0018] a screening unit configured to screen the time sequence enhanced features based on a dynamic threshold value to obtain target frames in the target video.

[0019] A computer device, comprising:

[0020] a memory configured to store at least one instruction; and

[0021] a processor configured to execute the instruction stored in the memory to implement the video frame extraction method.

[0022] A computer readable storage medium, wherein at least one instruction is stored in the computer readable storage medium, and the at least one instruction is executed by a processor in a computer device to implement the video frame extraction method.

[0023] As can be seen from the above technical solutions, the video frame extraction model is obtained through self-supervised end-to-end training based on a cross-domain enhanced consistency mechanism, and the adaptability and robustness of key frame selection under variable conditions are improved by combining cross-domain enhanced consistency regularization and self-supervised learning, without the need for target domain labeled data, while maintaining spatio-temporal coherence, the cross-domain robust key frame extraction is realized; the target spatio-temporal cube is obtained by using the preprocessing module in the video frame extraction model, the temporal context information is introduced, and the inter-frame dynamic relationship is easily captured; the time sequence enhanced features are obtained by using the domain adaptive feature encoder, multi-scale time sequence modeling is realized, and the inter-frame temporal coherence is maintained; the target frames are screened based on a dynamic threshold value, the screening granularity can be automatically optimized according to the scene complexity, thereby improving the accuracy and adaptability of key frame extraction. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1is a flow chart of a preferred embodiment of the video frame extraction method of the present application.

[0025] Figure 2 is a function module diagram of a preferred embodiment of the video frame extraction device of the present application.

[0026] Figure 3 is a structural schematic diagram of a computer device implementing a preferred embodiment of the video frame extraction method of the present application. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] As shown in Figure 1 is a flow chart of a preferred embodiment of the video frame extraction method of the present application. The order of the steps in the flow chart can be changed according to different needs, and some steps can be omitted.

[0029] The video frame extraction method is applied in one or more computer devices, and the computer device is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and the hardware thereof includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0030] The computer device can be any electronic product capable of human-computer interaction with the user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive Internet protocol television (IPTV), a smart wearable device, etc.

[0031] The computer device can also include a network device and / or a user device. The network device includes but is not limited to a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing.

[0032] The server can be a standalone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0033] Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0034] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. Several major directions.

[0035] The network in which the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN), etc.

[0036] S10, in response to a video frame extraction instruction for a target video in a target domain, calling a video frame extraction model obtained by self-supervised end-to-end training based on a cross-domain enhanced consistency mechanism.

[0037] In this embodiment, the video frame extraction instruction can be triggered according to different task requirements in different fields.

[0038] For example: when it is necessary to process the monitoring video of an automatic teller machine in the financial field, or to process the monitoring video of an operating room in the medical and health field, the video frame extraction instruction can be triggered to extract key frames from the corresponding monitoring video, reduce data redundancy and retain core content, and provide a data basis for subsequent behavior recognition, content summary generation, video retrieval, etc. Task.

[0039] In this embodiment, before the video frame extraction model obtained by self-supervised end-to-end training based on the cross-domain enhanced consistency mechanism is called, the method further comprises:

[0040] Construct a contrast loss: Wherein, The contrast loss is represented as L; t represents a time sequence enhanced feature z tcosine similarity between the enhanced feature of the positive sample corresponding to the t-th time point and the enhanced feature of the negative sample corresponding to the t-th time point cosine similarity between the enhanced feature of the positive sample corresponding to the t-th time point and the enhanced feature of the negative sample corresponding to the t-th time point is obtained after domain-specific enhancement is applied to the spatio-temporal cube; represents a temperature coefficient; z - represents a negative sample feature; s(z t ,z - represents a temporal enhanced feature z t at the t-th time point; represents a cosine similarity between the enhanced feature z - at the t-th time point and the enhanced feature z

[0041] The reconstruction loss is constructed as follows: wherein, represents the reconstruction loss; represents a true enhanced parameter; and MLP τ (z t ) represents a temporal enhanced feature z t corresponding to the t-th time point; represents a multi-layer perceptron processing of the predicted enhanced parameter;

[0042] The sum of the contrast loss and the reconstruction loss is calculated to obtain an enhanced perception consistency loss.

[0043] In the process of training the video frame extraction model, the feature difference between the source domain and the target domain is minimized based on the minimum moving distance (Wasserstein distance) of the bulldozer by using the adversarial domain discriminator in the domain adaptive feature encoder, and the video frame extraction model is jointly optimized by using the enhanced perception consistency loss.

[0044] The domain-specific enhancement can include specific transformation operations such as temporal jitter, spatial deformation, spatial blur, brightness attenuation, etc. Through these targeted enhancements, the complex situations of different domains can be better simulated, and the model can learn features that are robust to these changes. Taking cross-scene video monitoring as an example, when processing videos of an outdoor parking lot with rain and fog interference, spatial blur and brightness attenuation and other rain and fog simulation enhancements are performed on the video frames, so that the model can adapt to this harsh environmental condition and thus more accurately extract key frames.

[0045] Based on the contrast loss, a positive-negative sample pair can be constructed in the feature space, and a cosine similarity can be calculated, and the contrast difficulty can be adjusted by a temperature coefficient.

[0046] Based on the reconstruction loss, an enhanced parameter can be predicted by an auxiliary head (such as a multi-layer perceptron (MLP, Multilayer Perceptron)), and an L2 distance between the predicted value and the true value can be calculated.

[0047] By combining domain-specific enhancement with self-supervised agent tasks, using a combination of contrastive loss and adversarial loss to optimize the feature encoder, the collaborative learning of domain invariance and spatio-temporal consistency can help the model maintain accurate perception of key information when facing data from different domains, avoid errors in key frame extraction due to domain differences, and improve the adaptability of key frame extraction under variable conditions.

[0048] Among them, the adversarial domain discriminator classifies the source domain and the target domain for the time series enhanced features, aligns the feature distribution through the gradient reversal layer, minimizes the feature difference between the source domain and the target domain, can promote the model to generate domain-invariant representation, and improve the cross-domain adaptability while reducing the influence of domain bias.

[0049] In the above embodiment, domain-specific enhancement (such as time series jitter, spatial deformation, etc.) is applied to the spatio-temporal cube to generate enhanced features, and the model is driven to learn enhancement invariance by calculating contrastive loss and reconstruction loss. The robustness of the model to input perturbations is improved, and the continuity of the key frame in the time dimension is ensured, while the discrimination ability of the features and the understanding ability of the image are enhanced.

[0050] S11, using a preprocessing module in the video frame extraction model to preprocess the target video to obtain a target spatio-temporal cube.

[0051] In this embodiment, in order to ensure the consistency of the data, the data needs to be standardized first to facilitate the execution of subsequent tasks.

[0052] Specifically, the preprocessing of the target video using the preprocessing module in the video frame extraction model to obtain a target spatio-temporal cube includes:

[0053] Using the preprocessing module, the target video is normalized and blocked by a sliding window mechanism with a preset size to obtain the target spatio-temporal cube;

[0054] Among them, the preset size is determined according to the time series context radius, and each sub-cube in the target spatio-temporal cube includes the content of the frames within the time series context radius before and after the current frame.

[0055] For example: when the original video sequence of the target video is V = {f1,...,f T}, the normalization and blocking of V = {f1,...,f T} can be performed first to generate a fixed-size spatio-temporal cube C t = [f t-k ,...,f t+k ], where k represents the time series context radius, T represents the total number of frames of the target video, and t represents the time.

[0056] The temporal context radius can be a hyperparameter determined during training based on task requirements and video characteristics. The temporal context radius determines the number of preceding and following frames contained in each spatiotemporal cube, enabling the model to capture the temporal information of the k frames surrounding the current frame, which helps in understanding the dynamic changes and motion relationships between frames.

[0057] Furthermore, a larger k value means the input spatiotemporal cube contains more frames, resulting in greater computational complexity, but potentially yielding more comprehensive temporal information; a smaller k value results in higher computational efficiency, but may lose some long-term temporal dependencies. In other words, a larger k value can capture longer temporal dependencies, making it suitable for scenarios that require attention to long-term motion trajectories; a smaller k value is more suitable for handling short-term, rapid motion changes.

[0058] The above preprocessing not only provides a standardized input format for subsequent processing, but also introduces temporal context information, making it easier to capture dynamic relationships between frames.

[0059] S12, the domain adaptive feature encoder in the video frame extraction model is used to extract features from the target video and the target spatiotemporal cube to obtain temporal enhancement features.

[0060] In this embodiment, the domain adaptive feature encoder may include a spatial coding subnetwork, a temporal coding subnetwork, and an adversarial domain discriminator.

[0061] In this embodiment, the step of using the domain adaptive feature encoder in the video frame extraction model to extract features from the target video and the target spatiotemporal cube to obtain temporal enhancement features includes:

[0062] In the spatial coding sub-network of the domain adaptive feature encoder, the target video is spatially encoded using the following formula to obtain spatial features: l = 1,...,4; where, represents the spatial features at time t obtained after processing by the l-th level window attention block; Window-MSA represents the multi-head self-attention mechanism of the window; LN represents the layer normalization operation. Let l represent the spatial features at time t obtained after processing by the (l-1)th level window attention block; l represents the level of the window attention block; wherein, the spatial coding sub-network uses a 4-level window attention block of SwinTransformer (Shifted Window Transformer, a visual model based on Transformer) to generate multi-scale spatial features.

[0063] In the temporal coding subnetwork of the domain adaptive feature encoder, the target spatiotemporal cube is temporally coded to obtain the temporal enhanced feature using the following formula: z t = ReLU(W d *{h t-k ,...,h t+k}+b d ); wherein ReLU represents a linear rectifier function; W d represents a learnable weight matrix; {h t-k ,...,h t+k} represents a spatial feature sequence of k frames before and after t; k represents the temporal context radius; b d represents a learnable bias vector; and d represents the layer depth.

[0064] The spatial coding subnetwork can include: an inflation rate 2^0 convolution, that is, a basic temporal convolution layer; an inflation rate 2^1 convolution, that is, an expanded temporal receptive field; and an inflation rate 2^2 convolution, that is, a long-range dependency relationship.

[0065] Specifically, the spatial coding subnetwork can use the hierarchical window attention mechanism of the Swin Transformer, process a single frame through 4-level window attention blocks, perform layer normalization and residual connection after each level of processing, and finally obtain a frame-level spatial feature through global average pooling.

[0066] Specifically, the temporal coding subnetwork can use a TCN (Temporal Convolutional Network) with an increasing inflation rate to convolve the spatial feature sequence, and the inflation rate can be 2d-1. Long-term temporal dependencies are captured through causal convolution.

[0067] In the above embodiment, the spatial coding subnetwork can extract spatial features with local-global modeling capability, which has certain robustness to changes in illumination, viewing angle, etc. The temporal coding subnetwork can capture cross-frame dynamic features, realize multi-scale temporal modeling, and maintain temporal coherence between frames.

[0068] S13, screening the temporal enhanced feature based on a dynamic threshold to obtain target frames in the target video.

[0069] In this embodiment, before the temporal enhanced feature is screened based on the dynamic threshold, the method further includes:

[0070] The domain difference score is calculated using the following formula: wherein δ d represents the domain difference score; T represents the total number of frames of the target video; and D represents the adversarial domain discriminator. represents a source domain time-series enhanced feature at time t; represents a target domain time-series enhanced feature at time t;

[0071] The dynamic threshold is calculated according to the domain difference score by using the following formula: λ d = λ0(1+αtanh(βδ d )); wherein λ d represents the dynamic threshold; λ0represents a basic threshold; tanh represents a hyperbolic tangent activation function; α represents an adjustment amplitude control parameter, α = 0.5; β represents a sensitivity control parameter, β = 2.

[0072] Through the above embodiment, the screening granularity can be automatically optimized according to the scene complexity, manual parameter optimization is avoided, and the accuracy and adaptability of key frame extraction are improved.

[0073] In the embodiment, the time-series enhanced feature is screened based on the dynamic threshold to obtain the target frame in the target video, which includes:

[0074] The length of the time-series enhanced feature is obtained.

[0075] The frame with a length greater than the dynamic threshold is obtained from the target video as the target frame.

[0076] The core of key frame extraction is to select the most representative frame from the video, and the time-series enhanced feature as a feature vector that fuses space-time information, the numerical attribute thereof reflects the semantic importance and time-series uniqueness of the frame. For example, in a video monitoring scene, the frame in which a pedestrian enters the picture may have a feature value of the time-series enhanced feature significantly higher than that of a background frame because it contains a new semantic object, and the action turning point in time series (such as the pedestrian turning from walking to staying) may cause the dynamic feature of the time-series enhanced feature to change, so that it exceeds the threshold.

[0077] Therefore, unlike the traditional method that relies on manually designed features (such as color histogram) and fixed thresholds, which cannot adapt to domain shift. For example, in an outdoor rain and fog scene, the color feature is invalid due to changes in light, and a fixed threshold will cause a large number of key frames to be missed. The embodiment combines the dynamic threshold with the adaptive feature, so that even when the feature value fluctuates as a whole due to environmental changes, the screening accuracy can still be maintained through threshold adjustment.

[0078] The domain adaptive feature encoder of the embodiment extracts spatial features and inter-frame dynamic features by mixing a visual Transformer (such as a Vision Transformer (ViT)) with a temporal convolutional network, enhances the perceptual consistency loss by using domain-specific enhancement (such as temporal jittering and spatial warping) to constrain feature consistency. The domain adaptive feature encoder minimizes the feature difference between the source domain and the target domain by adversarial learning, and the perceptual consistency loss enhances the parameter prediction and feature similarity by jointly optimizing the contrast loss and the reconstruction loss. The synergistic effect of the domain adaptive feature encoder and the perceptual consistency loss realizes the dual domain adaptability of feature expression and decision-making process. At the same time, the embodiment dynamically adjusts the key frame screening threshold to adapt to the domain shift, and uses Swin Transformer and dilated convolution to realize efficient calculation, without the need for target domain labeled data, while maintaining spatiotemporal coherence, the cross-domain robust key frame extraction is realized.

[0079] In the embodiment, after obtaining the target frame in the target video, the method further comprises:

[0080] Obtaining a to-be-processed task;

[0081] Accessing a target model corresponding to the to-be-processed task;

[0082] Inputting the target frame into the target model to obtain a processing result of the to-be-processed task.

[0083] The to-be-processed task can include video processing tasks in different fields.

[0084] For example, in the financial field, the source domain can be a bank monitoring video (resolution 1920x1080, 30fps), and the target domain can be an outdoor parking lot monitoring video (containing rain and fog interference, resolution 1280x720). The domain adaptive feature encoder can generate temporal enhancement features for rain and fog frames of the outdoor video that align with the feature distribution of the indoor video, and based on the enhanced perceptual consistency loss, the feature consistency is constrained by rain and fog simulation enhancement (spatial blur + brightness attenuation), and the threshold is automatically reduced to adapt to the high dynamic range of the outdoor scene, so that the extracted key frames completely cover the whole process of the pedestrian from entering to leaving (F1-score is improved by 42%). The extracted key frames are input into a behavior recognition model to identify whether there is abnormal behavior.

[0085] For example, in the medical health field, the source domain can be a hospital operating room monitoring video (resolution 1280x720, 25fps), and the target domain can be an ambulance transfer monitoring video (containing vibration interference, resolution 1080x720). Using the domain adaptive feature encoder, the vibration frames of the ambulance video can be generated with time sequence enhancement features aligned with the feature distribution of the operating room video, and based on the enhanced perception consistency loss, the feature consistency is constrained through vibration simulation enhancement (spatial distortion + dynamic blur), and the threshold is automatically reduced to adapt to the high dynamic range of the ambulance scene, so that the extracted key frames completely cover the whole process of the medical staff's first aid operation (F1-score is improved by 38%). The extracted key frames are input into a behavior recognition model to identify whether there is a standard first aid behavior.

[0086] From the above technical solutions, it can be seen that the application can perform self-supervised end-to-end training based on the cross-domain enhanced consistency mechanism to obtain a video frame extraction model, which combines cross-domain enhanced consistency regularization and self-supervised learning to improve the adaptability and robustness of key frame selection under variable conditions, without target domain labeled data, while maintaining spatio-temporal coherence, realizing cross-domain robust key frame extraction; the preprocessing module in the video frame extraction model is used to obtain a target spatio-temporal cube, which introduces temporal context information, facilitating the capture of inter-frame dynamic relationships; the domain adaptive feature encoder is used to obtain time sequence enhancement features, realizing multi-scale time sequence modeling and maintaining inter-frame temporal coherence; the target frames are selected based on a dynamic threshold, which can automatically optimize the selection granularity according to the scene complexity, thereby improving the accuracy and adaptability of key frame extraction.

[0087] As shown in Figure 2 Fig. 1 is a functional module diagram of a preferred embodiment of a video frame extraction device of the application. The video frame extraction device 11 comprises a calling unit 110, a preprocessing unit 111, a feature extraction unit 112, and a screening unit 113. The module / unit referred to by the application refers to a series of computer program segments that can be executed by a processor and can complete a fixed function, which are stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0088] The calling unit 110 is configured to call a video frame extraction model obtained by performing self-supervised end-to-end training based on a cross-domain enhanced consistency mechanism in response to a video frame extraction instruction for a target video in a target domain.

[0089] In this embodiment, the video frame extraction instruction can be triggered according to different task requirements in different fields.

[0090] For example, when it is necessary to process surveillance videos of ATMs in the financial sector or operating rooms in the medical and health sector, the video frame extraction instruction can be triggered to extract key frames from the corresponding surveillance videos, reduce data redundancy and retain core content, and provide a data foundation for subsequent tasks such as behavior recognition, content summary generation, and video retrieval.

[0091] In this embodiment, before retrieving the video frame extraction model obtained through self-supervised end-to-end training based on the cross-domain enhanced consistency mechanism, a contrastive loss is constructed: in, This represents the contrast loss; The time-series enhancement feature z represents time t. t Enhanced features of the positive sample at time t Cosine similarity between them This is obtained by applying a domain-specific enhancement to the spacetime cube; θ represents the temperature coefficient; z - Represents negative sample features; s(z) t ,z - ) represents the temporal enhancement feature z at time t. t With the corresponding negative sample feature z - Cosine similarity between them;

[0092] Constructing the reconstruction loss: in, The reconstruction loss is represented by τ; the true augmentation parameter is represented by MLP. τ (z t ) represents the temporal enhancement feature z at time t. t Multilayer perceptron processing of the corresponding prediction enhancement parameters;

[0093] The sum of the contrast loss and the reconstruction loss is calculated to obtain the enhanced perceptual consistency loss;

[0094] During the training of the video frame extraction model, the adversarial domain discriminator in the domain adaptive feature encoder is used to minimize the feature difference between the source domain and the target domain based on the bulldozer movement distance (Wasserstein distance), and the enhanced perceptual consistency loss is used to jointly optimize the video frame extraction model.

[0095] The domain-specific enhancement can include specific transformation operations such as timing jitter, spatial deformation, spatial blur, brightness attenuation, etc. Through these targeted enhancements, the complexity of different domains can be better simulated, enabling the model to learn features that are robust to these changes. Taking cross-scene video monitoring as an example, when processing videos of outdoor parking lots with rain and fog interference, spatial blur and brightness attenuation are performed on the video frames to simulate rain and fog interference, so that the model can adapt to this harsh environment and accurately extract key frames.

[0096] Based on the contrast loss, a positive and negative sample pair can be constructed in the feature space, and the cosine similarity can be calculated, and the contrast difficulty can be adjusted by a temperature coefficient.

[0097] Based on the reconstruction loss, the enhancement parameters can be predicted by an auxiliary head (such as a multilayer perceptron (MLP, Multilayer Perceptron)), and the L2 distance between the predicted value and the true value can be calculated.

[0098] By combining domain-specific enhancement with self-supervised proxy tasks, using contrast loss and adversarial loss to jointly optimize the feature encoder, the collaborative learning of domain invariance and spatio-temporal consistency can be achieved, which can help the model to accurately perceive key information when facing different domains, avoid errors in key frame extraction due to domain differences, and improve the adaptability of key frame extraction under variable conditions.

[0099] The adversarial domain discriminator classifies the timing enhancement features into source domain and target domain, aligns the feature distribution through a gradient reversal layer, minimizes the feature difference between the source domain and the target domain, and can promote the model to generate domain-invariant representations, improve cross-domain adaptability, and reduce the influence of domain bias.

[0100] In the above embodiment, domain-specific enhancement (such as timing jitter and spatial deformation) is applied to the spatio-temporal cube to generate enhanced features, and the model is driven to learn enhancement invariance by calculating contrast loss and reconstruction loss. The robustness of the model to input perturbations is improved, the continuity of the key frame in the time dimension is ensured, and the discrimination ability and understanding ability of the features are enhanced.

[0101] The preprocessing unit 111 is configured to preprocess the target video by using a preprocessing module in the video frame extraction model to obtain a target spatio-temporal cube.

[0102] In this embodiment, in order to ensure the consistency of the data, the data needs to be standardized first to facilitate the subsequent tasks.

[0103] Specifically, the preprocessing unit 111 preprocesses the target video by using a preprocessing module in the video frame extraction model to obtain a target spatio-temporal cube, which includes:

[0104] Using the preprocessing module, the target video is normalized and segmented into blocks at a preset size using a sliding window mechanism to obtain the target spatiotemporal cube;

[0105] The preset size is determined based on the temporal context radius, and each sub-cube in the target spatiotemporal cube includes the content of frames within the temporal context radius before and after the current frame.

[0106] For example: when the original video sequence of the target video is V={f1,...,f T When}, we can first set V={f1,...,f T Normalization and block division are performed to generate a fixed-size spacetime cube C. t =[f t-k ,...,f t+k ], where k represents the temporal context radius, T represents the total number of frames in the target video, and t represents the time.

[0107] The temporal context radius can be a hyperparameter determined during training based on task requirements and video characteristics. The temporal context radius determines the number of preceding and following frames contained in each spatiotemporal cube, enabling the model to capture the temporal information of the k frames surrounding the current frame, which helps in understanding the dynamic changes and motion relationships between frames.

[0108] Furthermore, a larger k value means the input spatiotemporal cube contains more frames, resulting in greater computational complexity, but potentially yielding more comprehensive temporal information; a smaller k value results in higher computational efficiency, but may lose some long-term temporal dependencies. In other words, a larger k value can capture longer temporal dependencies, making it suitable for scenarios that require attention to long-term motion trajectories; a smaller k value is more suitable for handling short-term, rapid motion changes.

[0109] The above preprocessing not only provides a standardized input format for subsequent processing, but also introduces temporal context information, making it easier to capture dynamic relationships between frames.

[0110] The feature extraction unit 112 is used to extract features from the target video and the target spatiotemporal cube using the domain adaptive feature encoder in the video frame extraction model to obtain temporal enhancement features.

[0111] In this embodiment, the domain adaptive feature encoder may include a spatial coding subnetwork, a temporal coding subnetwork, and an adversarial domain discriminator.

[0112] In the embodiment, the feature extraction unit 112 extracts features of the target video and the target spatio-temporal cube by using a domain adaptive feature encoder in the video frame extraction model, to obtain time series enhanced features, including:

[0113] In the spatial encoding sub-network of the domain adaptive feature encoder, the target video is spatially encoded by using the following formula to obtain spatial features: l = 1,..., 4; wherein, denotes the spatial feature of t time obtained after processing by the lth window attention block; Window-MSA denotes window multi-head self-attention mechanism; LN denotes layer normalization operation; denotes the spatial feature of t time obtained after processing by the (l-1)th window attention block; l denotes the number of window attention blocks; wherein, the spatial encoding sub-network uses a 4-level window attention block of SwinTransformer (Shifted Window Transformer, a visual model based on Transformer) to generate multi-scale spatial features;

[0114] In the time series encoding sub-network of the domain adaptive feature encoder, the target spatio-temporal cube is time series encoded by using the following formula to obtain the time series enhanced features: t = ReLU(W d *{h t-k ,...,h t+k}+b d ); wherein, ReLU denotes linear rectifier function; W d denotes a learnable weight matrix; {h t-k ,...,h t+k} denotes a sequence of spatial features of k frames before and after t time, and k denotes the time series context radius; b d denotes a learnable bias vector; d denotes layer depth.

[0115] In the spatial encoding sub-network, the following can be included: dilated rate 2^0 convolution, i.e. basic time series convolution layer; dilated rate 2^1 convolution, i.e. expanded time series receptive field; dilated rate 2^2 convolution, i.e. capturing long-range dependencies.

[0116] Specifically, the spatial encoding sub-network can use the hierarchical window attention mechanism of Swin Transformer, process a single frame by 4-level window attention blocks, perform layer normalization and residual connection after each level processing, and finally obtain frame-level spatial features by global average pooling.

[0117] Specifically, the temporal encoding subnetwork can use a TCN (Temporal Convolutional Network) with an increasing dilation rate, which can be 2d-1, to perform convolutional processing on the spatial feature sequence, and capture long temporal dependencies through causal convolution.

[0118] In the above embodiment, the spatial encoding subnetwork can extract spatial features with local-global modeling capability, which has certain robustness to changes in illumination, viewing angle, etc.; the temporal encoding subnetwork can capture cross-frame dynamic features, realize multi-scale temporal modeling, and maintain inter-frame temporal coherence.

[0119] The screening unit 113 is configured to screen the temporal enhancement features based on a dynamic threshold to obtain target frames in the target video.

[0120] In this embodiment, before screening the temporal enhancement features based on the dynamic threshold, the domain difference score is calculated using the following formula: Wherein, δ d represents the domain difference score; T represents the total number of frames of the target video; D represents the adversarial domain discriminator; represents the source domain temporal enhancement feature at time t; represents the target domain temporal enhancement feature at time t;

[0121] The dynamic threshold is calculated according to the domain difference score using the following formula: λ d = λ0(1+αtanh(βδ d )); wherein λ d represents the dynamic threshold; λ0 represents the basic threshold; tanh represents the hyperbolic tangent activation function; α represents the adjustment amplitude control parameter, α = 0.5; β represents the sensitivity control parameter, β = 2.

[0122] Through the above embodiment, the screening granularity can be automatically optimized according to the scene complexity, manual parameter tuning is avoided, and the accuracy and adaptability of key frame extraction are improved.

[0123] In this embodiment, the screening unit 113 screens the temporal enhancement features based on a dynamic threshold to obtain target frames in the target video, which includes:

[0124] Obtaining the length of the temporal enhancement features;

[0125] Obtaining frames with a length greater than the dynamic threshold from the target video as the target frames.

[0126] The core of key frame extraction is to select the most representative frames from the video, and the temporal enhancement feature is a feature vector that fuses space-time information, and its numerical property reflects the semantic importance and temporal uniqueness of the frame. For example, in a video monitoring scene, the frame in which a pedestrian enters the picture may have a feature value of the temporal enhancement feature significantly higher than that of the background frame because it contains a new semantic object. The turning point of the action in time (such as the pedestrian changing from walking to staying) will cause the dynamic feature of the temporal enhancement feature to change, making it exceed the threshold.

[0127] Therefore, unlike traditional methods that rely on manually designed features (such as color histograms) and fixed thresholds, which cannot adapt to domain shifts. For example, in an outdoor rain and fog scene, color features fail due to changes in lighting, and fixed thresholds will result in a large number of key frames being missed. The embodiment combines dynamic thresholds with adaptive features, so that even when the feature values fluctuate as a whole due to environmental changes, the screening accuracy can still be maintained through threshold adjustment.

[0128] The domain adaptive feature encoder of the embodiment extracts spatial features and inter-frame dynamic features by mixing visual Transformers (such as Vision Transformer, ViT) and temporal convolutional networks, respectively, and enhances the perceptual consistency loss by using domain-specific enhancement (such as temporal jitter and spatial deformation) to constrain feature consistency. The domain adaptive feature encoder minimizes the feature difference between the source domain and the target domain through adversarial learning, and the perceptual consistency loss enhances the similarity between the parameters and the features by jointly optimizing the contrast loss and the reconstruction loss. The synergistic effect of the domain adaptive feature encoder and the perceptual consistency loss achieves dual domain adaptability of feature expression and decision-making process. At the same time, the embodiment dynamically adjusts the key frame screening threshold to adapt to domain shifts, and uses Swin Transformer and dilated convolution to achieve efficient computation, without the need for target domain labeled data, while maintaining spatio-temporal coherence and achieving cross-domain robust key frame extraction.

[0129] In the embodiment, after obtaining the target frame in the target video, a to-be-processed task is acquired;

[0130] The target model corresponding to the to-be-processed task is called;

[0131] The target frame is input into the target model to obtain a processing result of the to-be-processed task.

[0132] The to-be-processed task can include video processing tasks in different domains.

[0133] For example, in the financial field, the source domain can be a bank monitoring video (resolution 1920x1080, 30fps), and the target domain can be an outdoor parking lot monitoring video (containing rain and fog interference, resolution 1280x720). The domain adaptive feature encoder can generate time sequence enhanced features for the rain and fog frames of the outdoor video that align with the feature distribution of the indoor video, and based on the enhanced perception consistency loss, constrain the feature consistency through rain and fog simulation enhancement (spatial blur + brightness attenuation), and automatically reduce the threshold to adapt to the high dynamic range of the outdoor scene, so that the extracted key frames completely cover the whole process of the pedestrian from entering to leaving (F1-score is improved by 42%). The extracted key frames are input into a behavior recognition model to identify whether there is abnormal behavior.

[0134] For another example, in the medical health field, the source domain can be a hospital operating room monitoring video (resolution 1280x720, 25fps), and the target domain can be an ambulance transfer monitoring video (containing vibration interference, resolution 1080x720). The domain adaptive feature encoder can generate time sequence enhanced features for the vibration frames of the ambulance video that align with the feature distribution of the operating room video, and based on the enhanced perception consistency loss, constrain the feature consistency through vibration simulation enhancement (spatial distortion + dynamic blur), and automatically reduce the threshold to adapt to the high dynamic range of the ambulance scene, so that the extracted key frames completely cover the whole process of the medical staff's first aid operation (F1-score is improved by 38%). The extracted key frames are input into a behavior recognition model to identify whether there is a standard first aid behavior.

[0135] As can be seen from the above technical solutions, the application can perform self-supervised end-to-end training based on the cross-domain enhanced consistency mechanism to obtain a video frame extraction model, which combines cross-domain enhanced consistency regularization and self-supervised learning to improve the adaptability and robustness of key frame selection under variable conditions, and realizes cross-domain robust key frame extraction while maintaining spatio-temporal coherence without target domain labeled data; the preprocessing module in the video frame extraction model obtains a target spatio-temporal cube, which introduces time sequence context information to facilitate the capture of inter-frame dynamic relationship; the domain adaptive feature encoder obtains time sequence enhanced features, which realizes multi-scale time sequence modeling and maintains inter-frame temporal coherence; the target frames are selected based on a dynamic threshold, which can automatically optimize the selection granularity according to the scene complexity, thereby improving the accuracy and adaptability of key frame extraction.

[0136] As shown in Figure 3 , it is a structural schematic diagram of a computer device of a preferred embodiment of the application for realizing a video frame extraction method.

[0137] The computer device 1 can include a memory 12, a processor 13 and a bus (the arrow in the figure is the bus), and can further include a computer program, such as a video frame extraction program, stored in the memory 12 and executable on the processor 13.

[0138] Those skilled in the art can understand that the schematic diagram is only an example of the computer device 1 and does not constitute a limitation on the computer device 1, which can be a bus type structure or a star type structure, and can further include more or less other hardware or software or different component arrangement, such as an input / output device, a network access device, etc.

[0139] It should be noted that the computer device 1 is only an example, and other existing or future electronic products, such as those adaptable to the present application, should also be included in the protection scope of the present application and included by reference.

[0140] The memory 12 includes at least one type of readable storage medium, such as a flash memory, a mobile hard disk, a multimedia card, a card type memory (such as an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 12 can be an internal storage unit of the computer device 1 in some embodiments, such as a mobile hard disk of the computer device 1. The memory 12 can also be an external storage device of the computer device 1 in other embodiments, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 12 can include both an internal storage unit and an external storage device of the computer device 1. The memory 12 can be used not only to store application software and various data installed in the computer device 1, such as the code of the video frame extraction program, but also to temporarily store data that has been output or will be output.

[0141] The processor 13 can be composed of integrated circuits in some embodiments, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control core of the computer device 1, and connects various components of the computer device 1 through various interfaces and lines, and performs various functions and processes data of the computer device 1 by running or executing programs or modules stored in the memory 12 (such as executing a video frame extraction program), and calling data stored in the memory 12.

[0142] The processor 13 executes the operating system of the computer device 1 and various installed application programs. The processor 13 executes the application programs to implement the steps in the above-mentioned various video frame extraction method embodiments, such as Figure 1 The steps shown.

[0143] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units can be a series of computer-readable instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device 1. For example, the computer program can be divided into a calling unit 110, a preprocessing unit 111, a feature extraction unit 112, and a screening unit 113.

[0144] The integrated units implemented in the form of software function modules described above can be stored in a computer-readable storage medium. The software function modules described above are stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute part of the video frame extraction method described in each embodiment of the present application.

[0145] The modules / units integrated in the computer device 1, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiments can also be implemented by a computer program to instruct related hardware devices to complete, and the computer program can be stored in a computer-readable storage medium. The computer program is executed by the processor to implement the steps of each method embodiment described above.

[0146] The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0147] Further, the computer readable storage medium can mainly include a storage program area and a storage data area, wherein the storage program area can store an operating system, application programs required by at least one function, etc. The storage data area can store data created according to the use of the blockchain node, etc.

[0148] The blockchain referred to in the application is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. The blockchain is essentially a decentralized database, which is a series of data blocks associated by using cryptographic methods. Each data block contains the information of a batch of network transactions, and is used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0149] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, only one straight line is used in the Figure 3 , but it does not mean that there is only one bus or one type of bus. The bus is arranged to realize the connection and communication between the memory 12, the at least one processor 13, etc.

[0150] Although not shown, the computer device 1 can also include a power supply (such as a battery) for powering various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, so as to realize functions such as charge management, discharge management, and power consumption management through the power management device. The power supply can also include one or more direct current or alternating current power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The computer device 1 can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described here.

[0151] Further, the computer device 1 can further comprise a network interface, which can optionally comprise a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is generally used to establish a communication connection between the computer device 1 and other computer devices.

[0152] Optionally, the computer device 1 can further comprise a user interface, which can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally can be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. The display can also be appropriately referred to as a display screen or a display unit, and is used to display information processed in the computer device 1 and to display a visualized user interface.

[0153] It should be understood that the embodiments are only for illustration and are not limited in the scope of the patent application by the structure.

[0154] Those skilled in the art can understand that, Figure 3 The structure shown does not constitute a limitation on the computer device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0155] In combination Figure 1 The memory 12 in the computer device 1 stores a plurality of instructions to implement a video frame extraction method, and the processor 13 can execute the plurality of instructions to implement:

[0156] In response to a video frame extraction instruction for a target video in a target domain, a video frame extraction model obtained based on a cross-domain enhanced consistency mechanism for self-supervised end-to-end training is invoked;

[0157] A preprocessing module in the video frame extraction model is used to preprocess the target video to obtain a target spatio-temporal cube;

[0158] A domain adaptive feature encoder in the video frame extraction model is used to extract features from the target video and the target spatio-temporal cube to obtain time series enhanced features;

[0159] The time series enhanced features are filtered based on a dynamic threshold to obtain target frames in the target video.

[0160] Specifically, the processor 13 can refer to the description of the related steps in the corresponding embodiments for the specific implementation method of the above instructions, which will not be described herein. Figure 1 The specific implementation method of the processor 13 can refer to the description of the related steps in the corresponding embodiments, which will not be described herein.

[0161] Need to explain, the data involved in the case are all legal acquisition. The non-company software tools or components appearing in the embodiments of the application are only illustrative and do not represent actual use.

[0162] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented by other manners. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and another division manner can be used in actual implementation.

[0163] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0164] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical units, i.e. they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs.

[0165] In addition, the functional modules in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional modules.

[0166] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0167] Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are intended to be embraced therein. No reference is intended to be made to any disclaimer or disclaimer to limit the claims notwithstanding anything to the contrary contained in this patent or any patent accompanying drawing.

[0168] Furthermore, the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural, and vice-versa, unless the context clearly requires that. The indefinite articles "a" or "an" do not exclude a plurality, and the definite article "the" preceding an element does not exclude the presence of a plurality of these elements, unless the context clearly requires that. The word "about" preceding a numerical value denotes a range of values around and including the value itself, unless the context clearly requires that.

[0169] Finally, it should be noted that the above-mentioned embodiments illustrate rather than limit the application, since the scope of the application is indicated by the appended claims rather than by the foregoing description. Although the application has been described in detail with reference to particular implementations, it will be apparent to those skilled in the art that many modifications can be made, and that the application extends beyond the specifically described implementations to other alternatives having equivalent technical characteristics. Accordingly, the application is not limited to the specific implementations described in the foregoing description except insofar as it is limited in the claims.

Claims

1. A video frame extraction method, characterized in that, The video frame extraction method includes: In response to the instruction to extract video frames of the target video within the target domain, the video frame extraction model obtained by self-supervised end-to-end training based on the cross-domain enhanced consistency mechanism is invoked. The target video is preprocessed using the preprocessing module in the video frame extraction model to obtain the target spatiotemporal cube; The target video and the target spatiotemporal cube are used to extract features by the domain adaptive feature encoder in the video frame extraction model to obtain temporal enhancement features; The temporal enhancement features are filtered based on a dynamic threshold to obtain the target frames within the target video.

2. The video frame extraction method as described in claim 1, characterized in that, Before retrieving the video frame extraction model obtained through self-supervised end-to-end training based on the cross-domain enhanced consistency mechanism, the method further includes: Constructing the contrastive loss: in, This represents the contrast loss; The time-series enhancement feature z represents time t. t Enhanced features of the positive sample at time t Cosine similarity between them This is obtained by applying a domain-specific enhancement to the spacetime cube; θ represents the temperature coefficient; z - Represents negative sample features; s(z) t ,z - ) represents the temporal enhancement feature z at time t. t With the corresponding negative sample feature z - Cosine similarity between them; Constructing the reconstruction loss: in, The reconstruction loss is represented by τ; the true augmentation parameter is represented by MLP. τ (z t ) represents the temporal enhancement feature z at time t. t Multilayer perceptron processing of the corresponding prediction enhancement parameters; The sum of the contrast loss and the reconstruction loss is calculated to obtain the enhanced perceptual consistency loss; During the training of the video frame extraction model, the adversarial domain discriminator in the domain adaptive feature encoder is used to minimize the feature difference between the source domain and the target domain based on the bulldozer's movement distance, and the enhanced perceptual consistency loss is used to jointly optimize the video frame extraction model.

3. The video frame extraction method as described in claim 2, characterized in that, The step of preprocessing the target video using the preprocessing module in the video frame extraction model to obtain the target spatiotemporal cube includes: Using the preprocessing module, the target video is normalized and segmented into blocks at a preset size using a sliding window mechanism to obtain the target spatiotemporal cube; The preset size is determined based on the temporal context radius, and each sub-cube in the target spatiotemporal cube includes the content of frames within the temporal context radius before and after the current frame.

4. The video frame extraction method as described in claim 3, characterized in that, The step of using the domain-adaptive feature encoder in the video frame extraction model to extract features from the target video and the target spatiotemporal cube to obtain temporal enhancement features includes: In the spatial coding sub-network of the domain adaptive feature encoder, the target video is spatially encoded using the following formula to obtain spatial features: l = 1,...,4; where, represents the spatial features at time t obtained after processing by the l-th level window attention block; Window-MSA represents the multi-head self-attention mechanism of the window; LN represents the layer normalization operation. Let l represent the spatial features at time t obtained after processing by the (l-1)th level window attention block; l represents the level of the window attention block; wherein, the spatial coding sub-network uses the 4-level window attention block of the Swing Transformer to generate multi-scale spatial features; In the temporal coding sub-network of the domain adaptive feature encoder, the target spatiotemporal cube is temporally encoded using the following formula to obtain the temporal enhanced features: z t =ReLU(W d * {h t-k ,...,h t+k }+b d ); where ReLU represents the linear rectified function; W d Represents the learnable weight matrix; {h t-k ,...,h t+k } represents the spatial feature sequence of k frames before and after time t, where k represents the temporal context radius; b d d represents the learnable bias vector; d represents the layer depth.

5. The video frame extraction method as described in claim 1, characterized in that, The step of filtering the temporal enhancement features based on a dynamic threshold to obtain the target frames within the target video includes: Obtain the magnitude of the temporal enhancement feature; Frames whose size is greater than the dynamic threshold are obtained from the target video and are used as the target frames.

6. The video frame extraction method as described in claim 2, characterized in that, Before filtering the temporal enhancement features based on a dynamic threshold, the method further includes: The domain difference score is calculated using the following formula: Where, δ d The domain difference score represents the domain difference score; T represents the total number of frames in the target video; D represents the adversarial domain discriminator. This represents the source domain temporal enhancement features at time t; This represents the target domain temporal enhancement features at time t; The dynamic threshold is calculated based on the domain difference score using the following formula: λ d = λ0(1+αtanh(βδ d )); where λ d λ represents the dynamic threshold; λ0 represents the base threshold; tanh represents the hyperbolic tangent activation function; α represents the adjustment amplitude control parameter, α = 0.5; β represents the sensitivity control parameter, β = 2.

7. The video frame extraction method as described in claim 1, characterized in that, After obtaining the target frame within the target video, the method further includes: Get tasks to be processed; Retrieve the target model corresponding to the task to be processed; The target frame is input into the target model to obtain the processing result of the task to be processed.

8. A video frame extraction device, characterized in that, The video frame extraction device includes: The retrieval unit is used to retrieve the video frame extraction model obtained by self-supervised end-to-end training based on the cross-domain enhanced consistency mechanism in response to the video frame extraction instruction of the target video in the target domain. The preprocessing unit is used to preprocess the target video using the preprocessing module in the video frame extraction model to obtain the target spatiotemporal cube; The feature extraction unit is used to extract features from the target video and the target spatiotemporal cube using the domain adaptive feature encoder in the video frame extraction model to obtain temporal enhancement features. The filtering unit is used to filter the temporal enhancement features based on a dynamic threshold to obtain the target frames within the target video.

9. A computer device, characterized in that, The computer device includes: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the video frame extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the video frame extraction method as described in any one of claims 1 to 7.

Images