Key frame feature extraction method based on dynamic sparse routing and related equipment

Abstract

The invention relates to the technical field of video analysis and processing, can be applied to the field of science and technology finance / digital medical treatment, and discloses a key frame feature extraction method based on dynamic sparse routing and related equipment. The method comprises the following steps: carrying out frame sampling and pixel standardization on an input video to obtain a preprocessed frame sequence; inputting a trunk feature extraction network, and generating a video frame initial feature; determining a path activation condition according to the initial feature, dynamically activating at least one path in the candidate network paths and inhibiting the non-activated path to obtain a frame feature of sparse routing activation; performing cross-frame association modeling on the activated frame features based on a frame-level attention mechanism to obtain semantic enhanced frame-level features; generating a feature screening mask based on a learnable threshold, performing selective retention or suppression on the semantically enhanced frame features; and fusing the screened frame features to obtain a target key frame feature representation. The method improves the key frame discrimination precision, reduces the calculation redundancy, and improves the feature expression efficiency.

Description

Technical Field

[0001] This application relates to the field of video analysis and processing technology, and can be applied to the fields of fintech / digital healthcare, particularly to a key frame feature extraction method and related equipment based on dynamic sparse routing. Background Technology

[0002] In the field of video analysis and processing, keyframe-related feature extraction is a crucial foundation for achieving video content understanding, retrieval, and decision-making. Existing technologies mainly fall into two categories: First, methods based on manual rules or statistical features, such as using color histograms, optical flow, inter-frame differencing, or shot segmentation rules for feature description or keyframe determination. However, this approach relies on fixed rules and low-level features, resulting in insufficient robustness in dynamic backgrounds, complex motion, or cross-scene transitions. Second, dense feature extraction methods based on deep networks, such as using convolutional networks, temporal networks, or attention mechanisms to model video representations. However, these methods generally rely on fixed network structures for full computation, lacking the ability to adaptively adjust the computation path based on input content. This leads to high computational redundancy, limited model generalization ability to new scenes, and insufficient discriminative power of keyframe-related features. The aforementioned problems are particularly prominent in business scenarios. For example, in the fintech sector, bank branch surveillance or business transaction videos are characterized by diverse camera perspectives, significant differences in branch scenarios, and scarce samples of abnormal behavior. Existing technologies struggle to extract stable and discriminative keyframe-related features across different institutions and scenarios. In the digital healthcare sector, endoscopic, ultrasound, or surgical process videos exhibit high foreground structural similarity, sparse occurrence of key actions, and rapid field-of-view switching. Existing methods struggle to extract highly sensitive feature representations of key processes under limited sample conditions, easily leading to the weakening or loss of crucial diagnostic and treatment information. Therefore, existing technologies still have significant shortcomings in terms of the ability to discriminate video keyframe-related features, cross-scenario generalization, limited sample adaptability, and computational efficiency, making it difficult to meet the application needs of complex real-world businesses. Summary of the Invention

[0003] The main technical problem addressed by the implementation method of this application is that existing video key feature extraction methods have weak discriminative power, poor generalization ability, and high computational redundancy across different scenarios.

[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is: providing a keyframe feature extraction method based on dynamic sparse routing, comprising: performing frame sampling and pixel normalization on the input video to obtain a preprocessed video frame sequence; inputting the preprocessed video frame sequence into a preset backbone feature extraction network to obtain initial video frame features; dynamically selecting and activating at least one candidate path in a candidate network path set according to the path activation conditions determined by the initial video frame features, and suppressing the calculation of inactive paths to obtain a sparse route activated frame feature representation; performing cross-frame association modeling on the sparse route activated frame features through a frame-level attention mechanism to obtain a semantically enhanced frame-level feature representation; generating a feature filtering mask for the semantically enhanced frame-level feature representation based on a learnable threshold, and selectively retaining or suppressing frame-level features according to the feature filtering mask; and fusing the frame-level features processed by the feature filtering mask to obtain a target keyframe feature representation.

[0005] Optionally, the step of dynamically selecting and activating at least one candidate path in the candidate network path set according to the path activation conditions determined by the initial features of the video frame, and suppressing the calculation of inactive paths to obtain the frame feature representation of sparse route activation, includes: mapping the initial features of the video frame to the path selection representation space to obtain a path selection representation vector; calculating the activation score of each candidate path in the candidate network path set according to the path selection representation vector; determining the active path set according to the activation score and preset activation conditions, and masking the calculation of inactive paths; and aggregating the features output by each path in the active path set to obtain the frame feature representation of sparse route activation.

[0006] Optionally, the step of performing cross-frame association modeling on the sparse route-activated frame features through a frame-level attention mechanism to obtain a semantically enhanced frame-level feature representation includes: linearly mapping the sparse route-activated frame features to generate an attention query vector and an attention key vector, respectively; calculating the correlation weight between the attention query vector and the attention key vector, normalizing the correlation weight to obtain a cross-frame attention weight matrix; and weightedly fusing the sparse route-activated frame features through the cross-frame attention weight matrix to obtain the semantically enhanced frame-level feature representation.

[0007] Optionally, the step of generating a feature filtering mask for the semantically enhanced frame-level feature representation based on a learnable threshold, and selectively retaining or suppressing frame-level features according to the feature filtering mask, includes: constructing a threshold generation function or mapping network containing learnable threshold parameters; inputting the semantically enhanced frame-level feature representation into the threshold generation function or mapping network to obtain an adaptive feature filtering threshold; comparing the adaptive feature filtering threshold with the semantically enhanced frame-level feature representation, and generating the feature filtering mask based on the comparison result; and performing element-wise or channel-wise retention or suppression on the semantically enhanced frame-level feature representation according to the feature filtering mask to obtain the filtered frame-level feature representation.

[0008] Optionally, the step of inputting the preprocessed video frame sequence into a preset backbone feature extraction network to obtain initial features of the video frames includes: converting the preprocessed video frame sequence into a batch frame input tensor in chronological order; inputting the batch frame input tensor into the forward propagation path of the backbone feature extraction network to extract intermediate feature representations at different levels of the network; and performing channel alignment or dimensional transformation on the intermediate feature representations to generate the initial features of the video frames under a unified feature dimension.

[0009] Optionally, the step of performing frame sampling and pixel normalization on the input video to obtain a preprocessed video frame sequence includes: performing inter-frame sampling on the input video according to an adaptive sampling strategy to generate a discrete video frame sequence; performing pixel value scaling and / or numerical normalization on the discrete video frame sequence to unify the numerical distribution of the frame data; and performing dimensionality adjustment on the processed frame sequence to generate the preprocessed video frame sequence that can be received by the backbone feature extraction network.

[0010] Optionally, the step of fusing the frame-level features processed by the feature filtering mask to obtain the target keyframe feature representation includes: performing temporal alignment and / or dimensional transformation on the frame-level features processed by the feature filtering mask to obtain an aggregatable feature sequence; performing weighted summation, and / or concatenation, and / or pooling operations on the aggregatable feature sequence to obtain a fused feature representation; and performing linear or nonlinear transformation and / or projection processing on the fused feature representation to obtain the target keyframe feature representation.

[0011] To address the aforementioned technical problems, the second technical solution adopted in this application is: providing a keyframe feature extraction device based on dynamic sparse routing, comprising: a video frame preprocessing module for performing frame sampling and pixel normalization on the input video to obtain a preprocessed video frame sequence; an initial feature extraction module for inputting the preprocessed video frame sequence into a preset backbone feature extraction network to obtain initial video frame features; a sparse route activation module for dynamically selecting and activating at least one candidate path from the candidate network path set according to the path activation conditions determined by the initial video frame features, and suppressing the calculation of inactive paths to obtain a sparse route activated frame feature representation; a frame semantic enhancement module for performing cross-frame association modeling on the sparse route activated frame features through a frame-level attention mechanism to obtain a semantically enhanced frame-level feature representation; a feature filtering mask module for generating a feature filtering mask for the semantically enhanced frame-level feature representation based on a learnable threshold, and selectively retaining or suppressing frame-level features according to the feature filtering mask; and a frame feature representation module for fusing the frame-level features processed by the feature filtering mask to obtain a target keyframe feature representation.

[0012] To solve the above-mentioned technical problems, the third technical solution adopted in the embodiments of this application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the key frame feature extraction method based on dynamic sparse routing as described above.

[0013] To solve the above-mentioned technical problems, the fourth technical solution adopted in the embodiments of this application is: to provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device performs the key frame feature extraction method based on dynamic sparse routing as described above.

[0014] Unlike related technologies, this application performs frame sampling, pixel standardization, and dimension adjustment at the video input end, ensuring that video data from different sources maintains consistency in terms of time dimension, pixel distribution, and tensor structure. This solves the problems of unstable feature extraction and decreased model inference accuracy caused by uneven frame distribution and inconsistent pixel dynamic range in existing technologies. By unifying the data format and numerical domain in the preprocessing stage, the subsequent backbone network can directly perform feature extraction, improving the comparability and robustness of feature representation, reducing the computational overhead of redundant data, and improving overall processing efficiency. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0016] Figure 1 This is a schematic diagram of the operating environment of the key frame feature extraction method based on dynamic sparse routing provided in the embodiments of this application.

[0017] Figure 2 This is a schematic diagram of the execution flow of the key frame feature extraction method based on dynamic sparse routing provided in the embodiments of this application.

[0018] Figure 3 This is a schematic diagram of the execution flow of obtaining the frame feature representation of sparse route activation in the key frame feature extraction method based on dynamic sparse routing provided in the embodiments of this application.

[0019] Figure 4 This is a schematic diagram of the execution flow of frame-level feature processing in the key frame feature extraction method based on dynamic sparse routing provided in the embodiments of this application.

[0020] Figure 5 This is a schematic diagram of the data processing flow of an example of the key frame feature extraction method based on dynamic sparse routing provided in the embodiments of this application.

[0021] Figure 6 This is a schematic diagram of the system structure of the key frame feature extraction device based on dynamic sparse routing provided in the embodiments of this application.

[0022] Figure 7 This is a schematic diagram of the hardware structure of an electronic device that performs a key frame feature extraction method based on dynamic sparse routing, as provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Software tools, components, or servers not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0024] It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device schematic diagram or the order in the flowchart.

[0025] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0026] To facilitate understanding of this embodiment, a keyframe feature extraction method based on dynamic sparse routing disclosed in this application will first be described in detail. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the operating environment of the key frame feature extraction method based on dynamic sparse routing provided in the embodiments of this application, such as... Figure 1 As shown, the execution subject of the keyframe feature extraction method based on dynamic sparse routing provided in this application embodiment is generally an electronic device with a certain computing power, such as a computer device. In some possible implementations, this keyframe feature extraction method based on dynamic sparse routing can be implemented by the processor calling computer-readable instructions stored in memory. Figure 1 The computer equipment mentioned can be a server. A server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This can be understood as... Figure 1 The number of computer devices shown is merely illustrative and can be expanded in any number according to actual needs.

[0027] Please continue reading. Figure 2 , Figure 2 This is a schematic diagram of the execution flow of the keyframe feature extraction method based on dynamic sparse routing provided in the embodiments of this application, as shown below. Figure 2 As shown, it includes the following steps: S1. Perform frame sampling and pixel normalization on the input video to obtain a preprocessed video frame sequence.

[0028] The process involves several key steps. First, by sampling the input video, a continuous video stream is transformed into a sparse frame sequence for subsequent model processing. This effectively removes redundant and invalid frames, reducing data size and computational load while maintaining the integrity of critical temporal information. Second, pixel normalization is applied to the sampled target frame sequence to ensure consistency in pixel value range, brightness, contrast, and color space across different video sources, eliminating biases caused by differences in shooting equipment, ambient lighting, and encoding methods. In fintech scenarios, such as smart bank teller monitoring, the source video typically comes from a high-definition visible light camera, while the target video may come from a low-light infrared camera. The brightness and pixel dynamic range of different video frames vary significantly. Pixel normalization in this step makes it easier to identify and extract key behavioral segments (such as abnormal lingering or unauthorized counter operations). In digital healthcare scenarios, such as infusion monitoring or ward behavior recognition, the resolution of video frames from different devices varies. Frame sampling and normalization improve inter-frame consistency, making it easier to extract effective behavioral features (such as key actions like patients leaving the bed).

[0029] As an optional implementation, step S1 may specifically include steps S11 to S13.

[0030] S11. Perform inter-frame sampling on the input video according to the adaptive sampling strategy to generate a discrete video frame sequence.

[0031] Among these strategies, the adaptive sampling strategy can selectively extract representative video frames based on the degree of change in video content or temporal distribution characteristics. For example, it can increase the sampling frequency in intervals of drastic action changes and decrease the sampling frequency in stable intervals, thereby reducing redundant computation and storage pressure while retaining the temporal information most meaningful for subsequent feature extraction.

[0032] S12. Perform pixel value scaling and / or numerical normalization on the discrete video frame sequence to unify the numerical distribution of the frame data.

[0033] In particular, by transforming the pixel scale (such as scaling to 0~1 or normalizing to a standard normal distribution), the differences in pixel values ​​caused by different device parameters (brightness, lighting conditions, exposure time, etc.) can be reduced, avoiding model training instability or feature extraction deviation due to inconsistent feature value ranges, and improving the consistency and comparability of feature representation.

[0034] S13. Perform dimensionality adjustments on the processed frame sequence to generate a preprocessed video frame sequence that can be received by the backbone feature extraction network.

[0035] Dimensional adjustment is used to adjust the sampled and normalized video frames according to the input dimensions required by the model, such as adjusting them to a standard tensor format, so as to meet the input specifications of the feature extraction network and thus ensure that subsequent feature extraction steps can be executed smoothly.

[0036] Through steps S11 to S13, adaptive frame sampling, pixel normalization, and dimensionality adjustment are performed on the input video data to ensure consistency in temporal dimension and pixel distribution across video data from different sources. On one hand, this reduces the unnecessary computational burden caused by redundant frames, improving overall processing efficiency. On the other hand, it ensures the uniformity of input features in terms of dimensions, numerical range, and tensor structure, enabling subsequent feature extraction and model inference to be performed on a stable data basis, thus improving the accuracy and robustness of keyframe feature extraction.

[0037] S2. Input the preprocessed video frame sequence into the preset backbone feature extraction network to obtain the initial features of the video frames.

[0038] In step S2, the preprocessed video frame sequence is input into a pre-defined backbone feature extraction network, such as a convolutional neural network (CNN), a visual Transformer network (ViT), or other lightweight visual encoders. The backbone network extracts feature vectors representing object actions, scene changes, and inter-frame differences from consecutive video frames through hierarchical convolution, global feature aggregation, and self-attention computation. In fintech scenarios, such as bank branch monitoring and analysis, the backbone network can extract spatial and temporal features related to changes in actions such as "customer queue fluctuations, abnormal behavior detection, and window service congestion" from the video. In digital healthcare scenarios, such as analyzing operating room monitoring videos or remote rehabilitation videos, the backbone network can extract information containing key dynamics such as "patient limb movement amplitude, entering and leaving the room, and operating equipment."

[0039] As an optional implementation, step S2 may specifically include steps S21 to S23.

[0040] S21. Convert the preprocessed video frame sequence into batch frame input tensors in chronological order.

[0041] This process involves transforming the chronologically ordered sequence of video frames into a tensor structure suitable for the backbone network's input format. This structure typically has dimensional definitions of [batch, frames, height, width, channels] or [frames, channels, height, width], enabling the backbone network to perform forward computation on multiple video frames in batches, ensuring temporal consistency and efficient memory allocation.

[0042] S22. Input batch frame input tensors to the forward propagation path of the backbone feature extraction network to extract intermediate feature representations at different levels of the network.

[0043] In this process, batch frame tensors propagate layer by layer within the network through convolutional layers, attention layers, or Transformer layers. The network extracts features of different granularities at different levels, such as local edge textures, motion contours, character behavior, and scene changes. The intermediate feature representations output contain spatial features and preliminary temporal feature information, which are used for subsequent path selection and attention modeling.

[0044] S23. Perform channel alignment or dimension transformation on the intermediate feature representation to generate initial features of the video frame under a unified feature dimension.

[0045] Intermediate features typically originate from different layers or network structures. To ensure a unified feature dimension for subsequent modules, the number of channels and dimensions need to be transformed, such as through linear projection, convolution, or pooling. This step ensures that all frame features have the same dimension, enabling subsequent dynamic sparse routing and frame-level attention mechanisms to be computed in a unified feature space.

[0046] As an example, when performing frame-level feature extraction, a multi-scale feature extraction backbone can be used as the backbone feature extraction network. For instance, a multi-scale convolutional block group containing five parallel convolutional sub-networks can be constructed based on ResNet-50, each sub-network having a different receptive field (denoted as ). arrive The convolutional kernel sizes are configured as 3×3, 5×5, 7×7, 9×9, and 11×11, respectively. The low-level parameters of each sub-network are shared, while the high-level structure remains independent. This allows the network to obtain feature representation capabilities at different scales while maintaining controllable overall parameter scale, thereby enhancing its ability to capture action details, local changes, and scene background information, and improving the diversity and robustness of frame-level features.

[0047] Through steps S21 to S23, structured feature encoding of the input video frame sequence can be achieved, transforming the original pixel data into a unified feature representation suitable for deep network processing. On one hand, by extracting multi-level visual features through batch frame input and forward propagation, the model can understand the content in the video frames from multiple scales and semantic levels. On the other hand, dimensionality transformation ensures the consistency of feature representation, enabling subsequent dynamic sparse routing and frame-level attention mechanisms to be computed within a unified feature space. This process not only preserves the semantic and action information in the video but also significantly reduces subsequent computational complexity, improving the efficiency and accuracy of keyframe feature extraction.

[0048] S3. In the candidate network path set, based on the path activation conditions determined by the initial features of the video frame, dynamically select and activate at least one candidate path, and suppress the calculation of inactive paths to obtain the frame feature representation of sparse route activation.

[0049] Step S3 involves dynamic decision-making among multiple alternative feature extraction paths. First, the initial features of the video frames are analyzed. Based on the importance of different frames, it is determined which feature processing sub-networks to activate. For example, frames containing obvious actions or changes trigger more complex paths, while frames with minor changes take lighter paths, thus reducing unnecessary computation. For instance, in intelligent monitoring of bank branches, frames exhibiting "abnormal behavior" (such as a customer repeatedly attempting to fill out different forms at the counter) show high semantic fluctuations, automatically activating complex paths to enhance analysis; while static frames during normal queuing use lighter paths to reduce computational consumption. As another example, in surgical teaching videos, when a doctor makes a crucial hand movement, the system automatically selects a deep path to extract rich details; while when the doctor's hand remains still for a long time or the image remains unchanged, a lighter path is activated to save computational resources.

[0050] As an alternative implementation method, please continue reading. Figure 3 , Figure 3 This is a schematic diagram illustrating the execution flow of obtaining the frame feature representation of sparse route activation in the key frame feature extraction method based on dynamic sparse routing provided in this application embodiment, as shown below. Figure 3 As shown, the process can specifically include the following steps S31 to S34.

[0051] S31. Map the initial features of the video frame to the path selection representation space to obtain the path selection representation vector.

[0052] The initial features of the video frames output from the backbone network are input into a feature mapping network or linear projection module to extract feature representations related to path selection and generate selection vectors that describe the degree of adaptation of frame features in different candidate paths, thus distinguishing the processing requirements of different frames.

[0053] S32. Calculate the activation score of each candidate path in the candidate network path set based on the path selection representation vector.

[0054] The path selection vector is multiplied or mapped with the scoring parameters related to the candidate paths to obtain the importance score of each path to the current video frame. This scoring process allows for comparison of the processing benefits of different paths on the frame.

[0055] S33. Determine the set of activation paths based on the activation score and preset activation conditions, and shield the calculation of inactive paths.

[0056] By comparing the activation score with a set threshold, selecting paths with scores exceeding the threshold as the set of activation paths, and pausing the execution of low-scoring paths, unnecessary computation can be reduced, thus achieving dynamic sparse execution.

[0057] S34. Aggregate the features of the output corresponding to each path in the set of activated paths to obtain the frame feature representation of sparse route activation.

[0058] By comparing the activation score with a set threshold, selecting paths with scores exceeding the threshold as the set of activation paths, and pausing the execution of low-scoring paths, unnecessary computation can be reduced, thus achieving dynamic sparse execution.

[0059] As an example, in the path selection and sparse activation process, the following dynamic sparse routing network can be used to achieve candidate path activation and feature fusion: The multi-scale feature backbone first processes the input frame Perform feature extraction to generate a set of feature maps output from 5 parallel convolutional sub-networks: ,in First, the current video frame is concatenated with the mean of the supporting sample features and input into a routing network based on a 4-layer Transformer encoder to jointly encode the frame content and task semantics. The mean of the supporting sample features (task embedding) can be represented as: Subsequently, Input the routing network and obtain the output routes corresponding to each candidate path. By employing Gumbel-Softmax or Gumbel-TopK sparse activation strategies, continuous logits are transformed into sparse discrete activation vectors, selecting only the top K most contributing paths so that computation is performed solely on these paths: Finally, the features output by each candidate path are weighted and summed based on the sparse activation vectors to obtain the frame-level feature representation after dynamic sparse route activation: ,in, Indicates the number of samples in the support set; This represents the height, width, and number of channels of the feature map; This represents Gumbel noise, used to maintain exploratory behavior; This indicates a hard thresholding operation that retains only the top three maximum values. For example, in fintech intelligent review scenarios, different convolutional paths can focus on risk points with significant scale differences, such as "form text areas," "face areas," and "handwritten signature areas." The routing network will select the most suitable path based on the importance of features in the frame. In digital medical remote diagnosis videos, the routing network can automatically identify key parts in the frame (such as the area where a scalpel contacts an organ) and activate network paths more suitable for processing detailed textures, thereby reducing irrelevant computations and improving the ability to extract effective frame features.

[0060] Through steps S31 to S34, the system can autonomously determine which candidate network processing path(s) to execute based on the differences in the initial features of video frames, achieving dynamic allocation of computing resources. On the one hand, frames with high information content or significant changes are assigned to more complex processing paths, allowing these frames to obtain more complete feature representation. On the other hand, frames with redundant information or minimal changes are assigned to lightweight paths or skipped directly, thereby reducing unnecessary computational overhead. While maintaining the feature representation capability of keyframes, the system achieves dynamic sparsity of model computation, avoids computational redundancy caused by the full-path approach, makes frame-level feature extraction more efficient, and provides structured and more discriminative feature inputs for subsequent keyframe determination.

[0061] S4. A frame-level attention mechanism is used to model the cross-frame association of frame features activated by sparse routes, so as to obtain a semantically enhanced frame-level feature representation.

[0062] Specifically, the frame-level attention mechanism models the feature relationships between different frames, enabling it to identify and prioritize frames related to key behaviors or information while reducing the influence of irrelevant frames. In fintech scenarios, such as video surveillance at branch counters, the attention mechanism automatically assigns higher weights to frames containing abnormal behaviors for events like "abnormal lingering" or "suspicious interactions." In digital healthcare scenarios, such as surgical video analysis, the attention mechanism can focus on key operation frames (such as the moment of incision or suturing), reducing interference from irrelevant preparatory actions and thus improving the distinguishability of key frame features.

[0063] As an optional implementation, step S4 may specifically include steps S41 to S43.

[0064] S41. Perform linear mapping on the frame features activated by sparse routes to generate attention query vector and attention key vector respectively.

[0065] In step S41, the frame features are linearly transformed and projected onto the query vector space and the key vector space, enabling the system to represent the query and matching capabilities of each frame in the attention mechanism with low-dimensional vectors, thus providing comparable feature representations for subsequent cross-frame correlation calculations.

[0066] S42. Calculate the correlation weight between the attention query vector and the attention key vector, normalize the correlation weight, and obtain the cross-frame attention weight matrix.

[0067] In step S42, the relevance weights are calculated based on the semantic similarity between different frames. The sum of each weight value is kept stable through normalization, thereby constructing an attention weight matrix that reflects the degree of attention between frames.

[0068] S43. The frame features activated by sparse routes are weighted and fused by the cross-frame attention weight matrix to obtain a semantically enhanced frame-level feature representation.

[0069] In step S43, the frame features are weighted according to the attention weight matrix, so that frames with higher semantic importance receive more feature contributions, thereby enhancing the feature expression ability of key frames.

[0070] As an example, semantic enhancement of frame-level features can be performed based on a cross-attention modulation mechanism during keyframe feature extraction. Specifically, the feature vectors extracted from support set samples are... Query features of the current frame Projecting to the shared query-key-value space is accomplished through the following linear mapping: ,in, , , This is a learnable linear transformation matrix. Then, the attention weights between the query features and the support set features are calculated using the scaled dot product attention formula: The attention weights obtained Used for support set features Weighted combinations are performed to obtain the dynamically modulated frame-level features: ,in, The query vector dimension is specified. LayerNorm is used to maintain numerical stability and avoid modulation failures caused by inconsistent feature scales. Furthermore, to achieve keyframe identification, a sigmoid operation can be performed on the modulated features to estimate whether it is a keyframe. .

[0071] Through steps S41 to S43 above, the frame-level attention mechanism realizes semantic interaction modeling between different frames, enabling the system to automatically identify the contribution differences between key frames and non-key frames, enhance the features of key frames, weaken the features of redundant frames, improve the discriminativeness and semantic integrity of key frame feature representation, and provide high-quality feature input for subsequent key frame extraction.

[0072] S5. Generate a feature filtering mask for semantically enhanced frame-level feature representation based on a learnable threshold, and selectively retain or suppress frame-level features according to the feature filtering mask.

[0073] Step S5 introduces a learnable threshold mechanism to assess and filter the importance of frame-level features. Instead of using fixed thresholds, the system dynamically generates feature filtering strategies based on the model, retaining only key semantic information and suppressing noise and redundant data. For example, in fintech scenarios, during intelligent risk control or counter service video analysis, the system automatically identifies key frames of customer actions, such as "signing," "submitting ID," and "confirming transaction," while automatically suppressing background actions (such as casual conversation or irrelevant movement), reducing the risk of misidentification and false triggering. Similarly, in digital healthcare scenarios, during intraoperative video or remote consultation video analysis, the system can learn feature frames of "key operations performed by the doctor," such as "suturing," "resection," and "sampling," while suppressing irrelevant frames (such as instrument adjustments or background occlusion), reducing redundant feature interference and improving the accuracy of extracting core diagnostic frames.

[0074] As an alternative implementation method, please continue reading. Figure 4 , Figure 4 This is a schematic diagram illustrating the execution flow of frame-level feature processing in the key frame feature extraction method based on dynamic sparse routing provided in this application embodiment, as shown below. Figure 4 As shown, it can specifically include the following steps S51 to S53.

[0075] S51. Construct a threshold generation function or mapping network containing learnable threshold parameters, input semantically enhanced frame-level feature representations into the threshold generation function or mapping network, and obtain an adaptive feature selection threshold.

[0076] In step S51, frame-level features are analyzed using a network with learnable parameters, and an adaptive threshold reflecting the importance of the current frame is output for subsequent filtering logic. Since the threshold is learned by the network, it can be continuously optimized during the training process, ensuring that the filtering criteria match the actual target task, rather than relying on human experience.

[0077] S52. Compare the adaptive feature filtering threshold with the semantically enhanced frame-level feature representation, and generate a feature filtering mask based on the comparison result.

[0078] In step S52, each feature dimension or channel is compared with a threshold one by one, and a binary or soft mask matrix is ​​generated based on the comparison results to identify the effective features that need to be retained and the non-important features that need to be suppressed. In this way, keyframes can be screened directly at the feature level, rather than relying solely on frame-level scores for coarse-grained screening.

[0079] S53. Based on the feature filtering mask, perform element-wise or channel-wise retention or suppression on the semantically enhanced frame-level feature representation to obtain the filtered frame-level feature representation.

[0080] In step S53, selective filtering of frame-level features is performed using a filtering mask. This mask can mark and control features at an element-wise or channel-wise granularity. When a mask marks an element or channel as valid, the original value of that feature is retained for subsequent modeling. When a mask marks an element or channel as invalid, operations such as zeroing, attenuation using a fixed or learnable attenuation coefficient, or ignoring during feature aggregation can be performed on that element or channel to prevent interference (i.e., suppression) in the final feature representation. Through these element-wise or channel-wise differentiated suppression strategies, noise, redundancy, and ambiguity can be removed from the original frame-level features, resulting in a concise feature representation containing only valid information. This provides a cleaner and more discriminative input basis for keyframe extraction.

[0081] Through steps S51 to S53, the adaptive threshold mechanism can dynamically generate a filtering strategy based on the semantic importance of the frame-level features, thereby achieving refined filtering at the feature level. This makes the key frame features more prominent, while irrelevant and redundant features are effectively suppressed, thus improving the stability and accuracy of subsequent key frame judgments and avoiding the problem of incorrect filtering caused by fixed thresholds.

[0082] S6. The frame-level features after feature filtering and masking are fused to obtain the target keyframe feature representation.

[0083] In step S6, the effective frame features filtered by the feature selection mask are fused to generate a key frame feature representation with the strongest discriminative power. After the previous feature selection, the feature data has been filtered from redundancy, high dimensionality, and irrelevant noise into compressed features that retain only key semantics. Therefore, the fusion process can achieve high information density expression with a small number of features. For example, in fintech, in bank counters or intelligent risk control scenarios, the selected important frames (such as customer submitting ID cards, signing, confirming transactions, etc.) are fused into a single high-semantic key frame feature. This feature is used for action flow recognition or anomaly detection, avoiding the input of a large number of invalid frames into subsequent models and saving computing power. As another example, in digital healthcare, in intraoperative video analysis and remote diagnosis scenarios, the selected key frames (such as resection / suture / sampling steps) are fused into frame key features for disease recognition and surgical quality assessment, improving the stability of capturing key steps and reducing noise interference.

[0084] As an optional implementation, step S6 may specifically include steps S61 to S63.

[0085] S61. Perform temporal alignment and / or dimensional transformation on the frame-level features after feature filtering and masking to obtain an aggregatable feature sequence.

[0086] For example, in step S61, for frame-level features after feature filtering and masking, it is first checked whether they are consistent in terms of time sequence, feature dimension, and channel distribution. If there are misalignments in time span, mismatches in dimensions, or inconsistencies in the number of channels, then time alignment and dimension transformation are performed on the feature to map all frame-level features to a unified feature space and structural layout, thereby organizing frame features from different sources or at different levels into an ordered feature sequence that can participate in aggregation operations.

[0087] S62. Perform weighted summation, concatenation, and / or pooling operations on the aggregateable feature sequences to obtain the fused feature representation.

[0088] After alignment and dimension specification are completed, the aggregatable feature sequences are input into the fusion logic, and operations such as weighted summation, feature concatenation, or feature pooling are performed according to the fusion strategy: weighted summation is used to emphasize the differences in contribution of different frames to the final features; concatenation is used to retain complementary semantic information from different frames; and pooling is used to compress information and focus on the main feature directions.

[0089] S63. Perform linear or nonlinear transformations and / or projection processing on the fused feature representation to obtain the target keyframe feature representation.

[0090] Specifically, based on the fused feature representation, the fused features are further compressed into structured, discriminative target keyframe features through linear mapping, nonlinear transformation, or feature projection operations. This transformation can be used to reduce feature dimensionality, improve the separability of feature distribution, and enhance feature expressive power, making keyframe features more suitable as input for subsequent recognition, classification, or inference models. The final output target keyframe features have high information density and good discriminative ability.

[0091] Through steps S61 to S63, the frame-level features that have undergone feature selection can be further structured and aggregated into a keyframe feature representation with discriminative capabilities. Temporal alignment and dimensional normalization ensure consistency of frame features from different time locations, feature dimensions, or network paths within the same feature space. Fusion operations integrate effective information from different frames and highlight key features while suppressing redundant or invalid features. Further linear or nonlinear feature transformations make the fused feature representation more compact and improve feature discriminative power. The resulting target keyframe feature structure is clear and highly discriminative, and can be directly used for subsequent recognition, classification, or reasoning tasks, improving the effectiveness and stability of the keyframe output.

[0092] As an example, in keyframe extraction or frame-level feature matching, a temporal consistency optimizer can be introduced to constrain the temporal alignment between the query sequence and the supporting sequences, preventing the model from focusing only on local frame features and ignoring temporal dynamics. This optimizer incorporates a Soft-DTW (Soft Dynamic Time Warping) term into the loss function, automatically finding the optimal alignment path between the two sequences through dynamic programming, thus making the matching process temporally flexible. First, the Soft-DTW distance is defined as: ,in, For smoothing coefficients, It is a set of time frames. i and j Represents the first time frame in the set of time frames. i and the j Frames. A smooth approximation of the Soft-DTW is performed using dynamic programming, making the loss function differentiable and allowing the gradient to be backpropagated to the front-end feature extraction network. To further improve alignment efficiency, the video can be divided into a fixed number of frame segments (e.g., 16 segments). Segment-level Soft-DTW is calculated first, followed by fine-grained alignment of frames within each segment, thus reducing the computational complexity of the alignment process. Finally, the temporal alignment loss and the classification loss of the keyframe recognition task are jointly optimized to form a joint loss function: ,in This is the standard cross-entropy loss (which calculates the error between the model output value Y and the manually annotated keyframes). This represents a sparse regularization term used to limit the number of activated keyframes, thereby promoting sparse activation in the model. By minimizing this joint loss, the model can simultaneously optimize classification accuracy and temporal structure consistency, ultimately achieving stable frame-level feature extraction performance.

[0093] As another example, please refer to Figure 5 , Figure 5This is a schematic diagram of the data processing flow of an example of the keyframe feature extraction method based on dynamic sparse routing provided in this application embodiment. The input video frames first sequentially enter the multi-scale convolutional backbone (Parallel Convolutional Blocks) to extract spatial semantics and detail information from different receptive fields and scales. Then, a feature projector maps the multi-branch outputs to a unified dimension, generating initial frame-level features for subsequent processing. Next, a task embedding representation is generated based on the current video's business scenario. For example, in a fintech scenario, a task description vector can be generated based on the type of bank counter service (transfer, account opening, etc.) to highlight personnel interaction or abnormal behavior; in a digital medical surgery scenario, a stage vector can be generated based on the progress of surgical steps to emphasize key operational segments. The frame features and task embeddings are jointly input into the sparse activation vector generation module, and under the guidance of a reinforcement learning policy, at least one optimal path is dynamically selected from candidate paths for inference. Unactivated paths do not participate in computation, achieving dynamic sparse routing allocation of computing resources. Subsequently, the cross-attention modulation section first extracts support set features, then uses attention weight computation to determine their relevance to frame features, resulting in an attention weight matrix. Next, feature modulation is used to enhance the frame features, highlighting features highly relevant to the target task while suppressing irrelevant or noisy information. Finally, the obtained frame-level features are output as query features and input into a few-shot learning algorithm to complete keyframe feature extraction and output. This method, through joint optimization of dynamic sparse routing and cross-attention modulation, enables the system to focus only on keyframes in long videos, improving the accuracy of keyframe extraction while reducing overall computational cost.

[0094] As a more concrete example, product shelf monitoring requires quickly learning and identifying abnormal keyframes such as "shelf out of stock" from real-time video. Specifically, the input includes: five labeled frames (containing close-ups of empty shelves) and a 30-second query video (containing three instances of product retrieval and placement). During processing, the routing network first detects the shelf area and prioritizes activating the subnetwork with the larger receptive field. This allows for the capture of overall layout information; when a user's hand is detected approaching the shelf, the cross-attention modulation module enhances local features and weights them accordingly. Dynamically weighted feature contribution is applied; subsequently, DDTW (Soft-DTW based on smoothed dynamic time warping) loss ensures that all output keyframes cover the complete "picking-out-replenishment" process. The system ultimately outputs three keyframes (e.g., frames 12 / 19 / 27), accurately corresponding to the start, duration, and end times of product shortages. Compared to traditional methods that typically output seven redundant frames, this embodiment reduces the number of redundant frames by approximately 57% while maintaining 100% anomaly identification accuracy.

[0095] The keyframe feature extraction method based on dynamic sparse routing provided in this application unifies input frame quality through adaptive sampling and pixel normalization; it extracts frame features at different semantic levels using a multi-scale feature backbone; based on the dynamic sparse routing mechanism, it only activates high-complexity paths for high-value frames, while inactive paths do not participate in computation, achieving on-demand inference and computational burden reduction; it enhances keyframe features and suppresses redundant features by using cross-frame attention and learnable threshold filtering masks; and it combines Soft-DTW to maintain inter-frame temporal consistency, ensuring that the output keyframes completely cover the entire event process. This application significantly reduces redundant frames and improves keyframe extraction efficiency and stability while maintaining the accuracy of key event identification.

[0096] Please continue reading. Figure 6 , Figure 6 This is a schematic diagram of the system structure of the keyframe feature extraction device based on dynamic sparse routing provided in the embodiments of this application, as shown below. Figure 6 As shown, the key frame feature extraction device 60 based on dynamic sparse routing includes: a video frame preprocessing module 61, an initial feature extraction module 62, a sparse routing activation module 63, a frame semantic enhancement module 64, a feature filtering mask module 65, and a frame feature representation module 66.

[0097] The video frame preprocessing module 61 is specifically used to perform frame sampling and pixel normalization processing on the input video to obtain a preprocessed video frame sequence.

[0098] The initial feature extraction module 62 is specifically used to input the preprocessed video frame sequence into a preset backbone feature extraction network to obtain the initial features of the video frames.

[0099] The sparse route activation module 63 is specifically used to dynamically select and activate at least one candidate path from the candidate network path set according to the path activation conditions determined by the initial features of the video frame, and suppress the calculation of inactive paths, so as to obtain the frame feature representation of sparse route activation.

[0100] The frame semantic enhancement module 64 is specifically used to perform cross-frame association modeling on the frame features activated by the sparse route through a frame-level attention mechanism, so as to obtain a semantically enhanced frame-level feature representation.

[0101] The feature filtering mask module 65 is specifically used to generate a feature filtering mask for the semantically enhanced frame-level feature representation based on a learnable threshold, and to selectively retain or suppress frame-level features according to the feature filtering mask.

[0102] The frame feature representation module 66 is specifically used to fuse the frame-level features after feature filtering and masking to obtain the target keyframe feature representation.

[0103] As an optional implementation, the video frame preprocessing module 61 is further configured to perform inter-frame sampling on the input video according to an adaptive sampling strategy to generate a discrete video frame sequence; perform pixel value scaling and / or numerical normalization on the discrete video frame sequence to unify the numerical distribution of the frame data; and perform dimensionality adjustment on the processed frame sequence to generate the preprocessed video frame sequence that can be received by the backbone feature extraction network.

[0104] As an optional implementation, the initial feature extraction module 62 is further configured to convert the preprocessed video frame sequence into a batch frame input tensor in chronological order; input the batch frame input tensor into the forward propagation path of the backbone feature extraction network to extract intermediate feature representations at different levels of the network; and perform channel alignment or dimensional transformation on the intermediate feature representations to generate the initial features of the video frames under a unified feature dimension.

[0105] As an optional implementation, the sparse route activation module 63 is further configured to map the initial features of the video frame to the path selection representation space to obtain a path selection representation vector; calculate the activation score of each candidate path in the candidate network path set based on the path selection representation vector; determine the activation path set based on the activation score and preset activation conditions, and mask the calculation of inactive paths; and aggregate the features output by each path in the activation path set to obtain the frame feature representation of the sparse route activation.

[0106] As an optional implementation, the frame semantic enhancement module 64 is further specifically used to perform linear mapping on the frame features activated by the sparse route, generating attention query vectors and attention key vectors respectively; calculate the correlation weight between the attention query vector and the attention key vector, normalize the correlation weight to obtain a cross-frame attention weight matrix; and perform weighted fusion on the frame features activated by the sparse route through the cross-frame attention weight matrix to obtain the semantically enhanced frame-level feature representation.

[0107] As an optional implementation, the feature filtering mask module 65 is further configured to construct a threshold generation function or mapping network containing learnable threshold parameters, input the semantically enhanced frame-level feature representation into the threshold generation function or mapping network to obtain an adaptive feature filtering threshold; compare the adaptive feature filtering threshold with the semantically enhanced frame-level feature representation, generate the feature filtering mask based on the comparison result; and perform element-wise or channel-wise retention or suppression on the semantically enhanced frame-level feature representation according to the feature filtering mask to obtain the filtered frame-level feature representation.

[0108] As an optional implementation, the frame feature representation module 66 is further specifically used to perform temporal alignment and / or dimensional transformation on the frame-level features after the feature filtering masking process to obtain an aggregatable feature sequence; to perform weighted summation, and / or concatenation, and / or pooling operations on the aggregatable feature sequence to obtain a fused feature representation; and to perform linear or nonlinear transformation and / or projection processing on the fused feature representation to obtain the target keyframe feature representation.

[0109] It should be noted that the aforementioned keyframe feature extraction device based on dynamic sparse routing can execute the keyframe feature extraction method based on dynamic sparse routing provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the keyframe feature extraction device based on dynamic sparse routing can be found in the keyframe feature extraction method based on dynamic sparse routing provided in the embodiments of this application.

[0110] Figure 7 This is a schematic diagram of the hardware structure of an electronic device that performs a keyframe feature extraction method based on dynamic sparse routing, as provided in an embodiment of this application. Figure 7 As shown, the electronic device 700 includes: One or more processors 710 and memory 720, Figure 7 Take the 710 processor as an example.

[0111] The processor 710 and memory 720 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0112] The memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the keyframe feature extraction method based on dynamic sparse routing in the embodiments of this application. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the keyframe feature extraction method based on dynamic sparse routing in the above-described method embodiments.

[0113] The memory 720 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the keyframe feature extraction device based on dynamic sparse routing. Furthermore, the memory 720 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include memory remotely located relative to the processor 710, and these remote memories may be connected to the keyframe feature extraction device based on dynamic sparse routing via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The one or more modules are stored in the memory 720. When executed by the one or more processors 710, they perform the key frame feature extraction method based on dynamic sparse routing in any of the above method embodiments. For example, they perform the methods described above. Figure 2 Method steps S1 to S6, Figure 3 Method steps S31 to S34, Figure 4 Steps S51 to S53 in the method are implemented. Figure 6 The functions of modules 61-66 in the document.

[0115] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0116] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 7 One of the processors 710 can enable the above-described one or more processors to execute the key frame feature extraction method based on dynamic sparse routing in any of the above method embodiments, for example, to execute the above-described... Figure 2 Method steps S1 to S6, Figure 3 Method steps S31 to S34, Figure 4 Steps S51 to S53 in the method are implemented. Figure 6 The functions of modules 61-66 in the document.

[0117] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by the electronic device, enable the electronic device to perform the keyframe feature extraction method based on dynamic sparse routing in any of the above method embodiments. For example, it can execute the method described above. Figure 2 Method steps S1 to S6, Figure 3 Method steps S31 to S34, Figure 4 Steps S51 to S53 in the method are implemented. Figure 6 The functions of modules 61-66 in the document.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A keyframe feature extraction method based on dynamic sparse routing, characterized in that, include: The input video is subjected to frame sampling and pixel normalization to obtain a preprocessed video frame sequence; The preprocessed video frame sequence is input into a preset backbone feature extraction network to obtain the initial features of the video frames; In the candidate network path set, at least one candidate path is dynamically selected and activated based on the path activation conditions determined by the initial features of the video frame, and the calculation of inactive paths is suppressed to obtain the frame feature representation of sparse route activation. The frame features activated by the sparse route are modeled across frames using a frame-level attention mechanism to obtain a semantically enhanced frame-level feature representation. A feature filtering mask is generated based on a learnable threshold to represent the semantically enhanced frame-level features, and the frame-level features are selectively retained or suppressed according to the feature filtering mask. By fusing the frame-level features after feature filtering and masking, the target keyframe feature representation is obtained.

2. The keyframe feature extraction method based on dynamic sparse routing according to claim 1, characterized in that, The step of dynamically selecting and activating at least one candidate path from the candidate network path set based on the path activation conditions determined by the initial features of the video frame, and suppressing the calculation of inactive paths to obtain the frame feature representation of sparse route activation, includes: Map the initial features of the video frame to the path selection representation space to obtain the path selection representation vector; The activation score of each candidate path in the candidate network path set is calculated based on the path selection representation vector. The set of activation paths is determined based on the activation score and preset activation conditions, and the calculation of inactive paths is disabled. The features of the output corresponding to each path in the set of activated paths are aggregated to obtain the frame feature representation of the sparse route activation.

3. The keyframe feature extraction method based on dynamic sparse routing according to claim 1, characterized in that, The step of performing cross-frame association modeling on the frame features activated by the sparse route through a frame-level attention mechanism to obtain semantically enhanced frame-level feature representations includes: Linear mapping is performed on the frame features activated by the sparse route to generate attention query vectors and attention key vectors, respectively; Calculate the correlation weight between the attention query vector and the attention key vector, and normalize the correlation weight to obtain a cross-frame attention weight matrix; The frame features activated by the sparse route are weighted and fused using the cross-frame attention weight matrix to obtain the semantically enhanced frame-level feature representation.

4. The keyframe feature extraction method based on dynamic sparse routing according to claim 1, characterized in that, The step of generating a feature filtering mask for the semantically enhanced frame-level feature representation based on a learnable threshold, and selectively retaining or suppressing frame-level features according to the feature filtering mask, includes: Construct a threshold generation function or mapping network containing learnable threshold parameters, input the semantically enhanced frame-level feature representation into the threshold generation function or mapping network, and obtain an adaptive feature filtering threshold; The adaptive feature filtering threshold is compared with the semantically enhanced frame-level feature representation, and the feature filtering mask is generated based on the comparison result; The semantically enhanced frame-level feature representation is subjected to element-wise or channel-wise retention or suppression based on the feature filtering mask to obtain the filtered frame-level feature representation.

5. The keyframe feature extraction method based on dynamic sparse routing according to claim 1, characterized in that, The step of inputting the preprocessed video frame sequence into a preset backbone feature extraction network to obtain the initial features of the video frames includes: The preprocessed video frame sequence is converted into a batch frame input tensor in chronological order. The batch frame input tensor is input into the forward propagation path of the backbone feature extraction network to extract intermediate feature representations at different levels of the network; The intermediate feature representation is subjected to channel alignment or dimensional transformation to generate the initial features of the video frame under a unified feature dimension.

6. The keyframe feature extraction method based on dynamic sparse routing according to claim 1, characterized in that, The step of performing frame sampling and pixel normalization on the input video to obtain a preprocessed video frame sequence includes: Inter-frame sampling is performed on the input video according to an adaptive sampling strategy to generate a discrete video frame sequence; Perform pixel value scaling and / or numerical normalization on the discrete video frame sequence to unify the numerical distribution of the frame data; The processed frame sequence is dimensionally rearranged to generate the preprocessed video frame sequence that can be received by the backbone feature extraction network.

7. The keyframe feature extraction method based on dynamic sparse routing according to claim 1, characterized in that, The step of fusing the frame-level features after feature filtering and masking to obtain the target keyframe feature representation includes: Perform temporal alignment and / or dimensional transformation on the frame-level features after the feature filtering masking process to obtain an aggregatable feature sequence; The aggregateable feature sequences are subjected to weighted summation, and / or concatenation, and / or pooling operations to obtain a fused feature representation; Perform linear or nonlinear transformations and / or projection processing on the fused feature representation to obtain the target keyframe feature representation.

8. A keyframe feature extraction device based on dynamic sparse routing, characterized in that, include: The video frame preprocessing module is used to perform frame sampling and pixel normalization on the input video to obtain a preprocessed video frame sequence; An initial feature extraction module is used to input the preprocessed video frame sequence into a preset backbone feature extraction network to obtain the initial features of the video frames; The sparse route activation module is used to dynamically select and activate at least one candidate path from the candidate network path set based on the path activation conditions determined by the initial features of the video frame, and suppress the calculation of inactive paths to obtain the frame feature representation of sparse route activation. The frame semantic enhancement module is used to perform cross-frame association modeling on the frame features activated by the sparse route through a frame-level attention mechanism to obtain a semantically enhanced frame-level feature representation. The feature filtering mask module is used to generate a feature filtering mask for the semantically enhanced frame-level feature representation based on a learnable threshold, and to selectively retain or suppress frame-level features according to the feature filtering mask. The frame feature representation module is used to fuse the frame-level features after feature filtering and masking to obtain the target keyframe feature representation.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the keyframe feature extraction method based on dynamic sparse routing as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the key frame feature extraction method based on dynamic sparse routing as described in any one of claims 1-7.

Images