Key frame extraction method based on decoupling multi-task feature learning and related equipment

Abstract

The invention relates to the technical field of video analysis and processing, and discloses a key frame extraction method based on decoupling multi-task feature learning and related equipment. The method comprises the following steps: performing frame sampling, size normalization and brightness standardization processing on an input video to obtain a standardized frame sequence; inputting the data into a time sequence convolution backbone network, and extracting multi-scale spatial-temporal features based on expansion causal convolution; splitting the features through decoupling branches of motion dynamics, scene composition and object semantics to obtain initial feature factors, and performing independence optimization through a factor separation discriminator to obtain independent motion feature factors, independent scene feature factors and independent object feature factors; encoding the scene description text into a scene vector, calculating the weight of each feature factor through an adaptive attention mechanism, and carrying out weighted fusion; and outputting the importance score of each frame by a multi-layer perceptron, and screening out a target key frame through time sequence non-maximum suppression. According to the method, the key frame extraction efficiency and accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of video analysis and processing technology, and in particular to a keyframe extraction method and related equipment based on decoupled multi-task feature learning. Background Technology

[0002] Keyframe extraction from videos is a core task in video analytics. The technical goal is to select representative frames from lengthy videos to support applications such as video summarization and surveillance analysis. Existing technologies fall into two categories: traditional methods rely on manually designed features (e.g., color histograms, optical flow features) and heuristic rules (e.g., inter-frame difference thresholds) to determine frame importance; deep learning methods use Temporal Convolutional Networks (TCNs) to model temporal dependencies, and some attempt to separate video factors or apply scene-aware weighting through adversarial training. However, existing technologies also have significant technical shortcomings: First, feature entanglement occurs, with semantic factors such as motion, scene, and object resulting in mixed feature representations, making it difficult to flexibly adjust the weights of each factor. For example, in financial risk control scenarios, high-risk actions such as customers blocking cameras or replacing bank cards are highly intertwined with normal interaction features, and the model easily submerges abnormal behavior in background features. In medical endoscopic surgery, subtle interactions between instruments and tissues are mixed with the overall surgical field morphology, making it difficult to achieve specific capture of key risk actions. Second, there is poor scene adaptability. Existing technologies mostly use fixed strategies to evaluate frame importance, which is difficult to adapt to the needs of different video types such as movies, surveillance, and sports. For example, financial counters focus more on identity-risk actions, while medical care focuses more on bleeding or accidental contact with key tissues, but fixed weight strategies cannot distinguish the focus. Third, existing technologies rely on heuristic rules, and post-processing parameters (e.g., non-maximum suppression parameters) are mostly set based on experience. In complex scenarios, redundant frames or key frames are easily missed. For example, in finance, continuous suspicious actions are mistakenly suppressed, and in medicine, bleeding signs are missed by empirical thresholds. Summary of the Invention

[0003] The main technical problems addressed by the embodiments of this application are feature entanglement, insufficient scene adaptability, and reliance on heuristic rules in existing video keyframe extraction technologies.

[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is: providing a keyframe extraction method based on decoupled multi-task feature learning, comprising: performing frame sampling, size normalization, and brightness normalization preprocessing on the input video to output a normalized frame sequence; inputting the normalized frame sequence into the temporal convolutional backbone network of the decoupled spatiotemporal feature extractor, and extracting the spatiotemporal features of the video through the stacked dilated causal convolutional layers of the temporal convolutional backbone network; decoupling and decomposing the spatiotemporal features according to the parallel motion dynamic factor decoupling branch, scene composition factor decoupling branch, and object semantic factor decoupling branch in the decoupled spatiotemporal feature extractor to obtain initial motion feature factors, initial scene feature factors, and initial object feature factors; and using a factor separation discriminator to analyze the initial motion features. The initial scene feature factor and initial object feature factor are subjected to independence verification and optimization processing to obtain independent motion feature factor, independent scene feature factor, and independent object feature factor. The scene description text of the input video is converted into a scene vector. Based on the scene vector, the independent weights of the independent motion feature factor, independent scene feature factor, and independent object feature factor are calculated using an adaptive attention mechanism. The independent motion feature factor, independent scene feature factor, and independent object feature factor are weighted and fused according to the independent weights. The importance score of each video frame is output by a multilayer perceptron based on the weighted fusion result. Temporal non-maximum suppression processing is applied to the importance score, and video frames with an importance score greater than a preset importance threshold are selected as target video keyframes.

[0005] Optionally, the construction process of the stacked dilated causal convolutional layer of the temporal convolutional backbone network includes: setting at least two cascaded dilated convolutional units, each of which is sequentially connected to a causal convolutional layer, a batch normalization layer, and an activation function layer; setting the dilation coefficients of adjacent dilated convolutional units in a preset ascending order, such that the dilation coefficients increase layer by layer to expand the temporal receptive domain; connecting the output features of each dilated convolutional unit across channels through the temporal constraints of the causal convolutional layer and the progressive setting of the dilation coefficients; and compressing the input dimension of the connected features using a convolutional layer to compress the feature dimension, thereby obtaining the constructed stacked dilated causal convolutional layer.

[0006] Optionally, the step of decoupling and splitting the spatiotemporal features according to the parallel motion dynamic factor decoupling branch, scene composition factor decoupling branch, and object semantic factor decoupling branch in the decoupled spatiotemporal feature extractor to obtain initial motion feature factors, initial scene feature factors, and initial object feature factors includes: configuring an independent feature processing unit for each decoupling branch; the motion dynamic factor decoupling branch adopts a temporal difference convolutional layer; the scene composition factor decoupling branch is configured with a spatial attention module; and the object semantic factor decoupling branch is connected to a pre-trained object detection feature extraction sub-network; the feature processing units of each branch process the input spatiotemporal features separately, controlling the temporal difference convolutional layer to capture inter-frame motion changes, the spatial attention module to enhance the image structure features, and the object detection feature extraction sub-network to extract object semantic information; and weakening the correlation of output features from different branches through an inter-branch feature suppression mechanism to obtain the initial motion feature factors, initial scene feature factors, and initial object feature factors.

[0007] Optionally, the step of performing independence verification and optimization processing on the initial motion feature factors, initial scene feature factors, and initial object feature factors through a factor separation discriminator to obtain independent motion feature factors, independent scene feature factors, and independent object feature factors includes: inputting the initial motion feature factors, initial scene feature factors, and initial object feature factors to the factor separation discriminator; calculating the cross-correlation between different factors through the feature association detection module of the factor separation discriminator; generating an association penalty signal based on the cross-correlation calculation result; and adjusting the parameters of the feature processing units of each decoupled branch in reverse to weaken the redundant information of different types of feature factors; repeating the steps from calculating the cross-correlation between different factors through the feature association detection module of the factor separation discriminator to adjusting the parameters of the feature processing units of each decoupled branch in reverse, until the cross-correlation result between different factors is less than a preset cross-correlation threshold, thereby obtaining the independent motion feature factors, independent scene feature factors, and independent object feature factors.

[0008] Optionally, the step of converting the scene description text of the input video into a scene vector, and calculating the independent weights of the independent motion feature factor, independent scene feature factor, and independent object feature factor based on the scene vector using an adaptive attention mechanism, includes: performing word segmentation and part-of-speech tagging on the input scene description text, and filtering out related words that are associated with motion state, scene structure, and object attributes; mapping the related words into word vectors, and performing sequence encoding on the word vectors to generate a scene text feature vector; calculating the different similarities between the scene text feature vector and the independent motion feature factor, independent scene feature factor, and independent object feature factor, respectively, to obtain three sets of feature similarity values; converting the three sets of feature similarity values ​​into corresponding weight coefficients through an adaptive attention mechanism, and setting the weight coefficients as the independent weights of each independent feature factor.

[0009] Optionally, the step of weightedly fusing the independent motion feature factors, independent scene feature factors, and independent object feature factors according to the independent weights, and outputting the importance score of each video frame through a multilayer perceptron based on the weighted fusion result, includes: multiplying the independent motion feature factors, independent scene feature factors, and independent object feature factors element-wise with their corresponding independent weights to obtain weighted motion features, weighted scene features, and weighted object features; concatenating the weighted motion features, weighted scene features, and weighted object features along the channel dimension to generate a fused feature matrix; inputting the fused feature matrix into a preset multilayer perceptron, and performing nonlinear transformation and dimensionality compression on the fused features through the fully connected layer and activation function layer of the multilayer perceptron to obtain a frame-level feature vector; and outputting a one-dimensional score vector matching the number of video frames based on the frame-level feature vector, where each element in the vector corresponds to the importance score of each video frame.

[0010] Optionally, the step of performing temporal non-maximum suppression processing on the importance scores and selecting video frames with importance scores greater than a preset importance threshold as target video keyframes includes: arranging the importance scores according to the temporal order of the video frames to obtain a score sequence with temporal labels; traversing the score sequence according to a preset suppression window size, retaining the video frame with the highest importance score in each window as a candidate keyframe; calculating the difference between the importance scores of all candidate keyframes and the preset importance threshold, and selecting candidate keyframes with positive differences; sorting the selected candidate keyframes according to the original video temporal order, and outputting the sorted frame sequence as the target video keyframe.

[0011] To address the aforementioned technical problems, the second technical solution adopted in this application is: providing a keyframe extraction device based on decoupled multi-task feature learning, comprising: an input video preprocessing module, used to perform frame sampling, size normalization, and brightness normalization preprocessing on the input video, and output a normalized frame sequence; a video spatiotemporal feature module, used to input the normalized frame sequence to the temporal convolutional backbone network of the decoupled spatiotemporal feature extractor, and extract the spatiotemporal features of the video through the stacked dilated causal convolutional layers of the temporal convolutional backbone network; a feature decoupling and splitting module, used to decouple and split the spatiotemporal features according to the parallel motion dynamic factor decoupling branch, scene composition factor decoupling branch, and object semantic factor decoupling branch in the decoupled spatiotemporal feature extractor, to obtain initial motion feature factors, initial scene feature factors, and initial object feature factors; and a factor separation discriminator module, used to pass the factor separation discriminator... The initial motion feature factors, initial scene feature factors, and initial object feature factors undergo independence verification and optimization to obtain independent motion feature factors, independent scene feature factors, and independent object feature factors. A factor independent weighting module is used to convert the scene description text of the input video into a scene vector, and calculate the independent weights of each of the independent motion feature factors, independent scene feature factors, and independent object feature factors based on the scene vector using an adaptive attention mechanism. A factor weighted fusion module is used to perform weighted fusion of the independent motion feature factors, independent scene feature factors, and independent object feature factors based on the independent weights, and output the importance score of each video frame based on the weighted fusion result using a multilayer perceptron. A video keyframe filtering module is used to perform temporal non-maximum suppression processing on the importance scores, and filter out video frames with importance scores greater than a preset importance threshold as target video keyframes.

[0012] To solve the above-mentioned technical problems, the third technical solution adopted in the embodiments of this application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the keyframe extraction method based on decoupled multi-task feature learning as described above.

[0013] To solve the above-mentioned technical problems, the fourth technical solution adopted in the embodiments of this application is: to provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device executes the keyframe extraction method based on decoupled multi-task feature learning as described above.

[0014] Unlike related technologies, this application improves the accuracy of feature representation by separating different features such as motion, scene, and object, and using a factor separation discriminator to ensure the independence of these features. Simultaneously, based on a scene-adaptive attention mechanism, the weights of various features are automatically adjusted according to different scenes, optimizing the feature fusion process and significantly improving the keyframe extraction accuracy. Finally, a scoring network is used to select the most representative keyframes, reducing the generation of redundant frames and improving the efficiency and accuracy of keyframe extraction. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0016] Figure 1 This is a schematic diagram of the operating environment of the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application.

[0017] Figure 2 This is a schematic diagram of the execution flow of the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application.

[0018] Figure 3 This is a schematic diagram of the execution flow of decoupling and splitting in the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application.

[0019] Figure 4 This is a schematic diagram of the execution flow of decoupling and splitting in the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application.

[0020] Figure 5 This is a data flow example diagram of the video keyframe extraction process based on a decoupled spatiotemporal feature extractor provided in the embodiments of this application.

[0021] Figure 6 This is a data flow example diagram of the decoupled spatiotemporal feature extractor and scene adaptive attention mechanism synchronously processing data provided in the embodiments of this application.

[0022] Figure 7 This is a schematic diagram of the system structure of the keyframe extraction device based on decoupled multi-task feature learning provided in the embodiments of this application.

[0023] Figure 8 This is a schematic diagram of the hardware structure of an electronic device that performs a keyframe extraction method based on decoupled multi-task feature learning, as provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Software tools, components, or servers not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0025] It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device schematic diagram or the order in the flowchart.

[0026] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0027] To facilitate understanding of this embodiment, a keyframe extraction method based on decoupled multi-task feature learning disclosed in this application will first be described in detail. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the operating environment of the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application, such as... Figure 1 As shown, the execution subject of the keyframe extraction method based on decoupled multi-task feature learning provided in this application embodiment is generally an electronic device with a certain computing power, such as a computer device. In some possible implementations, this keyframe extraction method based on decoupled multi-task feature learning can be implemented by the processor calling computer-readable instructions stored in the memory. Figure 1 The computer equipment mentioned can be a server. A server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This can be understood as... Figure 1 The number of computer devices shown is merely illustrative and can be expanded in any number according to actual needs.

[0028] Please continue reading. Figure 2 , Figure 2This is a schematic diagram of the execution flow of the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application, as shown below. Figure 2 As shown, it includes the following steps: S1. Perform frame sampling, size normalization, and brightness normalization preprocessing on the input video to output a normalized frame sequence.

[0029] For example, in step S1, the format of the original video stream is first parsed, converting it into a sequence of frames. Then, each frame is extracted from the original video stream, and its resolution is adjusted to a size suitable for subsequent processing (e.g., from 1920×1080 to 224×224) to reduce computation and meet the input requirements of later processing. Subsequently, luminance normalization is performed, normalizing the pixel values ​​of the video frames from 0 to 255 to between 0 and 1, thereby improving processing stability and efficiency. After the aforementioned series of processes, the normalized frame sequence is used as input to subsequent video analysis processes for further feature extraction and keyframe selection.

[0030] S2. Input the normalized frame sequence to the temporal convolutional backbone network of the decoupled spatiotemporal feature extractor, and extract the spatiotemporal features of the video through the stacked dilated causal convolutional layers of the temporal convolutional backbone network.

[0031] As an optional implementation, the construction process of the stacked dilated causal convolutional layers of the temporal convolutional backbone network in step S2 above may specifically include the following steps S21 to S24.

[0032] S21. Set up at least two cascaded dilated convolutional units, each of which is connected to a causal convolutional layer, a batch normalization layer, and an activation function layer in sequence.

[0033] Dilated convolutions expand the receptive field of the convolutional kernel, capturing a wider range of spatiotemporal information while avoiding a dramatic increase in computational cost. Causal convolutional layers ensure the causality of time-series data, considering only the current time step and previous time steps. Batch normalization layers normalize the output after each convolutional unit, improving training stability and accelerating convergence. Activation function layers introduce non-linearity into the convolutional operation, enhancing the model's ability to express complex patterns.

[0034] S22. Set the dilation coefficients of adjacent dilated convolution units in a preset increasing order, so that the dilation coefficients increase layer by layer to expand the temporal receiving domain.

[0035] The progressively increasing dilation coefficient expands the receptive field of each convolutional unit, capturing temporal features across different time ranges. This incremental dilation coefficient design allows each convolutional layer to focus on dependencies over longer time spans, increasing the network's ability to process long sequences while avoiding the high computational overhead associated with fully connected layers or long-term sequence convolutions.

[0036] S23. By using the temporal constraints of the causal convolutional layer and the progressive setting of the dilation coefficient, the output features of each dilated convolutional unit are connected across channels.

[0037] Temporal constraints ensure that future information is not involved when processing temporal data, thus guaranteeing the causality of the model. Through progressively increasing dilation coefficients, convolutional layers can handle temporal dependencies ranging from short-term to long-term. Cross-channel connections fuse the output features of multiple convolutional units, enhancing the fusion of spatiotemporal information and enriching the output features of each layer.

[0038] S24. Compress the convolutional layers of the connected feature input dimensions to compress the feature dimensions, resulting in the constructed stacked dilated causal convolutional layers.

[0039] For example, by using 1×1 convolutional layers, features concatenated across channels are compressed to reduce computational cost and feature dimensionality. 1×1 convolutional layers adjust the feature mapping for each channel, preserving the most important spatiotemporal information while removing redundant features, thus optimizing the use of computational resources. This process effectively reduces feature dimensionality, making subsequent processing more efficient and providing a concise and meaningful representation for further feature analysis.

[0040] Through steps S21 to S24, a temporal convolutional backbone network capable of effectively extracting spatiotemporal features from videos was successfully constructed. The combination of dilated convolutional units with causal convolutional layers, batch normalization layers, and activation function layers ensures that the network relies only on current and previous temporal data, thus avoiding interference from future information and improving the model's nonlinear expressive power. The incremental design of the dilation coefficient expands the receptive field of the convolution, enabling the network to capture temporal dependencies over longer time spans. The implementation of cross-channel connections further enhances the fusion of multi-level features and optimizes information transmission. Finally, dimensionality compression reduces feature redundancy, ensuring feature compactness and computational efficiency.

[0041] S3. Based on the parallel motion dynamic factor decoupling branch, scene composition factor decoupling branch, and object semantic factor decoupling branch in the decoupled spatiotemporal feature extractor, the spatiotemporal features are decoupled and decomposed to obtain the initial motion feature factor, initial scene feature factor, and initial object feature factor.

[0042] Step S3 involves using a feature suppression mechanism to reduce the correlation between features across different decoupled branches. The feature suppression mechanism weakens interference between different feature factors, allowing each feature factor to more independently reflect its corresponding motion, scene, and object features. Through this technique, the features of each factor can be expressed more purely in terms of physical or semantic meaning, thus providing a more accurate data foundation for subsequent feature fusion and keyframe selection. This operation ensures that the information after feature decoupling is not excessively interfered with or mixed up in subsequent processing.

[0043] As an example, the temporal convolutional network (TCN) backbone in the decoupled spatiotemporal feature extractor (DSFE) uses stacked dilated causal convolutional layers to process frame sequences and output spatiotemporal features. ,in, D For feature dimension, T For each frame, the dilated convolution coefficients grow exponentially (e.g., 1, 2, 4, ...) to balance local details and long-range dependencies. Furthermore, the decoupled spatiotemporal feature extractor is designed with a dual-path approach: the first path is a temporal convolution feature extraction path: the input frame sequence V first passes through L layers of dilated causal convolutions, with each layer's operation defined as: in, This represents the convolution operation. For the first l The expansion coefficient of the layer, It is the ReLU activation function. For the first l Layer bias terms. Final spatiotemporal features. It contains multi-scale information from local motion to the global scene. The second path is the factor decoupling path: for each target factor... Dedicated encoder Map F to a factor representation: in, The mask vector M is a learnable factor mask used to highlight relevant feature channels. Each factor has a unique mask vector M (randomly initialized). GAP represents global average pooling, Conv1D is a 1×1 convolution, and ⊙ represents the dot product operation. A factor-separating discriminator is then used. Adversarial training is implemented, and the loss function for adversarial training is: Among them, ensuring Factors Only kThe specific information, such as motion factors, should not include scene background features. The loss function for this adversarial training is the adversarial loss in traditional GANs.

[0044] As an alternative implementation method, please continue reading. Figure 3 , Figure 3 This is a schematic diagram of the execution flow of decoupling and splitting in the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application, as shown below. Figure 3 As shown, it can specifically include the following steps S31 to S33.

[0045] S31. Configure an independent feature processing unit for each decoupling branch. The motion dynamic factor decoupling branch adopts a temporal difference convolutional layer, the scene composition factor decoupling branch is configured with a spatial attention module, and the object semantic factor decoupling branch is connected to a pre-trained object detection feature extraction sub-network.

[0046] The motion dynamics factor decoupling branch employs temporal difference convolutional layers to capture motion changes between frames in the video. These layers extract motion features by calculating the differences between adjacent frames, enhancing the dynamic parts of the video. The scene composition factor decoupling branch incorporates a spatial attention module, which enhances the feature representation of important regions through a spatial attention mechanism, emphasizing the scene structure and background information in the video. The object semantic factor decoupling branch connects to a pre-trained object detection feature extraction subnetwork. This subnetwork uses a deep learning model to identify target objects in the video and extract semantic information, aiding in subsequent analysis of specific objects.

[0047] S32. The feature processing units of each branch process the spatiotemporal features of the input separately, controlling the temporal difference convolutional layer to capture inter-frame motion changes, the spatial attention module to enhance the structural features of the image, and the object detection feature extraction sub-network to extract the semantic information of the object.

[0048] The temporal difference convolutional layer focuses on motion changes between frames, capturing dynamic parts. The spatial attention module highlights key structural features in the image, enhancing background and scene details, especially visual information in important areas. The object detection feature extraction subnetwork focuses on extracting semantic information related to objects, thus supporting subsequent target analysis and recognition.

[0049] S33. By weakening the correlation of output features of different branches through the inter-branch feature suppression mechanism, the initial motion feature factor, initial scene feature factor and initial object feature factor are obtained.

[0050] The introduction of the inter-branch feature suppression mechanism makes the features of each branch more independent during subsequent processing, avoiding unnecessary interference between different features. This allows for a clearer extraction of the initial motion feature factors, scene feature factors, and object feature factors. This separation step lays a solid foundation for subsequent feature integration and processing, ensuring the purity and independence of each feature factor.

[0051] As an example, in bank counter or ATM risk control scenarios, during the decoupling spatiotemporal feature extraction stage, independent feature processing branches are configured for "motion behavior," "environmental scene," and "sensitive objects." Specifically, the motion dynamics factor decoupling branch uses a temporal difference convolutional layer to capture inter-frame motion abrupt changes such as "a hand suddenly obstructing the camera" or "suspicious card replacement actions." The scene composition factor decoupling branch uses a spatial attention module to enhance environmental structural information such as "counter window layout" and "whether the camera's visible area is obstructed." The object semantic factor decoupling branch connects to a pre-trained object detection network to extract semantic objects such as "bank cards, ID cards, and suspicious external devices." Then, through an inter-branch feature suppression mechanism, the coupling relationship between these features is explicitly weakened, ensuring that the "motion features of abnormal customer behavior" are not misled by background lighting fluctuations or device position shifts. This yields initial motion feature factors, initial scene feature factors, and initial object feature factors, providing a structured decoupling representation for subsequent risk weighting and financial keyframe screening.

[0052] As another example, in the monitoring of minimally invasive surgery, during the decoupling spatiotemporal feature extraction stage, independent feature branches are constructed for "intraoperative manipulation actions," "surgical field structural environment," and "key organs / instruments." Specifically, the motion dynamics factor decoupling branch uses a temporal difference convolutional layer to specifically capture minute changes in motion with potential risks, such as "instrument tip approaching a blood vessel" or "sudden force applied to traction tissue." The scene composition factor decoupling branch is equipped with a spatial attention module to enhance the structural sensitivity information of "hierarchical positional relationships within the surgical field" and "areas containing important tissues." The object semantic factor decoupling branch connects to a pre-trained medical object detection network to extract semantically key entities such as "blood vessels, nerves, and instrument ports." Then, through an inter-branch feature inhibition mechanism, interference between branches is explicitly reduced, ensuring that "high-risk operation signals" are not misled by fluctuations in surgical field lighting, camera shake, or smoke obstruction. This ultimately yields initial motion feature factors, initial scene feature factors, and initial object feature factors, providing high-resolution and independent semantic support for subsequent intraoperative risk dynamic assessment and keyframe selection.

[0053] Through steps S31 to S33, the decoupling and decomposition of video spatiotemporal features were successfully achieved, allowing motion, scene, and object features to be processed independently. By configuring an independent feature processing unit for each decoupled branch, dynamic changes, scene structure, and object semantic information in the video can be captured separately, thereby accurately extracting various features. The design of the feature processing unit ensures the independence of each feature and avoids cross-interference between features. Under the action of the subsequent feature suppression mechanism, the correlation between different feature factors is further weakened, allowing each factor to express its corresponding information more clearly and purely. The aforementioned decoupling and independent processing method significantly improves the accuracy and efficiency of feature extraction, providing more accurate and pure feature input for subsequent feature fusion and keyframe selection.

[0054] S4. The initial motion feature factor, initial scene feature factor, and initial object feature factor are subjected to independence verification and optimization processing by the factor separation discriminator to obtain independent motion feature factor, independent scene feature factor, and independent object feature factor.

[0055] The role of the factor separation discriminator is to ensure that each feature factor remains independent during processing, avoiding redundancy and interference between features. Through verification and optimization, the discriminator can determine the correlation between different feature factors and adjust parameters based on the calculation results, ensuring that each feature factor can accurately and independently express its corresponding spatiotemporal information in subsequent tasks. This process helps improve the quality of decoupled features and ensures that feature factors can clearly reflect the motion, scene, and object information of the video, thereby enhancing the accuracy and effectiveness of video analysis.

[0056] As an optional implementation, the independence verification and optimization process in step S4 above may specifically include the following steps S41 to S43.

[0057] S41. Input the initial motion feature factor, initial scene feature factor, and initial object feature factor into the factor separation discriminator, and calculate the cross-correlation between different factors through the feature association detection module of the factor separation discriminator.

[0058] The feature association detection module is used to detect whether there is excessive similarity or redundancy among various feature factors, thereby assessing the degree of information overlap between different factors. The calculated cross-correlation value can reflect whether the feature factors have been sufficiently decoupled, or whether there is still significant information overlap, requiring further optimization and adjustment.

[0059] S42. Generate a correlation penalty signal based on the cross-correlation calculation results, and adjust the parameters of the feature processing units of each decoupled branch in reverse order to weaken the redundant information of different types of feature factors.

[0060] The penalty signal serves to reduce redundant information among different types of feature factors and guide the network to optimize model parameters through backpropagation, further weakening the dependencies between feature factors. Through this optimization process, the parameters of each feature processing unit are adjusted, making each feature factor more independent, reducing cross-interference, and improving decoupling effectiveness.

[0061] S43. Repeat the steps from calculating the cross-correlation between different factors through the feature association detection module of the factor separation discriminator to adjusting the parameters of the feature processing unit of each decoupled branch in reverse, until the cross-correlation result between different factors is less than the preset cross-correlation threshold, and obtain independent motion feature factors, independent scene feature factors and independent object feature factors.

[0062] In step S43, the iterative process continuously optimizes the network parameters, enabling each feature factor to become more independent of other feature factors during the gradual adjustment process. Ultimately, through optimization, independent motion feature factors, independent scene feature factors, and independent object feature factors are obtained, ensuring that the features of each factor can more accurately express unique information in subsequent processing and analysis.

[0063] As an example, in the anti-fraud video surveillance scenario of bank self-service terminals, the initial motion feature factors, scene feature factors, and object semantic feature factors corresponding to abnormal behaviors such as "customer covering the camera," "rapidly exchanging bank cards," and "abnormal hand lingering on the keypad" are first input into the factor separation discriminator. The feature association detection module calculates the cross-correlation between these feature factors. When it is detected that "abnormal behavior features are still highly dependent on background features" (e.g., the customer's action of covering the camera is interfered with by changes in scene lighting), a penalty signal is generated to reverse the feature processing parameters of the motion and scene branches, reducing feature redundancy and interference. The above cycle continues until the "covering the camera" behavior is purely identified as a suspicious action factor, without being mixed with interference information such as background lighting or the presence of irrelevant people, thereby obtaining independent motion feature factors, independent scene feature factors, and independent object feature factors, providing a high-purity structured risk feature representation for subsequent risk weighting and keyframe screening.

[0064] As another example, in the scenario of minimally invasive surgery video analysis, the initial motion feature factors, scene feature factors, and object semantic feature factors corresponding to potential risk actions such as "instrument tip approaching blood vessel during surgery," "tissue tension change due to traction," and "sudden appearance of abnormal bleeding color in the surgical field" are input into the factor separation discriminator. The cross-correlation between them is calculated by the feature association detection module. When "critical medical risk actions are still interfered with by surgical field lighting, smoke, or non-critical background structures," a penalty signal is generated to inversely optimize the feature extraction parameters of different decoupling branches, forcibly decoupling "surgeon instrument actions" from interfering factors such as "surgical environment lighting changes or pneumoperitoneum reflections." This process will continue to iterate until "the interaction behavior between instruments and critical tissues" is purely identified as a risk action signal without mixing in low-value information unrelated to the background, thereby obtaining independent motion feature factors, independent scene feature factors, and independent object feature factors, providing highly clinically relevant feature representations for subsequent keyframe recognition.

[0065] Through steps S41 to S43, the independence optimization of different feature factors was successfully achieved. The factor separation discriminator identifies redundant information between different factors by calculating the cross-correlation between each feature factor, and optimizes the model parameters by generating a correlation penalty signal to reduce unnecessary correlations between feature factors. Through repeated adjustments and optimizations, the cross-correlation between motion, scene, and object feature factors is significantly reduced, ensuring that each feature factor can independently express its specific information. This optimization process effectively improves the quality of feature decoupling, providing a more accurate and independent feature representation for subsequent feature fusion and keyframe selection.

[0066] S5. Convert the scene description text of the input video into a scene vector. Based on the scene vector, calculate the independent weights of the independent motion feature factor, independent scene feature factor, and independent object feature factor through an adaptive attention mechanism.

[0067] As an example, the Scene-Adaptive Attention Mechanism (SAAM) in this embodiment receives a scene description vector. (For example, semantic embedding of "sports video"), factor weights are computed using a lightweight Transformer. Dynamic fusion of scores from each factor: Here, MLP stands for Multilayer Perceptron. In addition, the scene-adaptive attention mechanism achieves dynamic weighting through two steps: the first step is scene encoding, which encodes the scene description text (e.g., "basketball game") into vectors using a pre-trained language model (e.g., BERT model). The second step is attention calculation, which is processed using two cascaded Transformer blocks. : in, For learnable projection matrices, d Scaling factor T For the transpose operation, the final weight Determine the contribution ratio of each factor in the score calculation. Then, combine the attention weights with... Perform weighted summation and layer normalization, i.e.: in, Layer normalization processing. For example, for surveillance scenarios, the scene adaptive attention mechanism in this embodiment may assign higher weights to scene composition factors (detecting background anomalies), while for sports scenarios, it focuses on motion dynamic factors. After weighted summation of the scene description text by each factor, the feature factors are then weighted and fused again, using a dynamic weighting method to replace the traditional hyperparameters: Finally, Post-processing is performed, and keyframe selection based on temporal non-maximum suppression (NMS) is implemented to ensure that the output frames are reasonably distributed on the time axis.

[0068] As an alternative implementation method, please continue reading. Figure 4 , Figure 4 This is a schematic diagram of the execution flow of decoupling and splitting in the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application, as shown below. Figure 4 As shown, the process can specifically include the following steps S51 to S54.

[0069] S51. Perform word segmentation and part-of-speech tagging on the input scene description text, and filter out related words that are associated with motion state, scene structure, and object attributes.

[0070] In step S51, the scene description text is preprocessed using word segmentation and part-of-speech tagging techniques to filter out words related to motion states, scene structure, and object attributes in the video content. This step extracts key information from the text that is closely related to video features, thereby ensuring that the main semantics of the video content can be captured during subsequent feature processing.

[0071] S52. Map the associated words into word vectors, and perform sequence encoding on the word vectors to generate scene text feature vectors.

[0072] In step S52, word vector mapping and sequence encoding are used to convert the selected related words into numerical representations, generating scene text feature vectors. This process transforms text information into a form that computers can understand, providing a data foundation for subsequent similarity calculations and feature matching.

[0073] S53. Calculate the different similarities between the scene text feature vector and the independent motion feature factor, independent scene feature factor, and independent object feature factor, respectively, to obtain three sets of feature similarity values.

[0074] In step S53, the correlation between the text description and each feature factor is quantified to determine the degree of matching between different feature factors and the scene text, thereby providing data support for subsequent weighted processing.

[0075] S54. The three sets of feature similarity values ​​are converted into corresponding weight coefficients through an adaptive attention mechanism, and the weight coefficients are set as independent weights for each independent feature factor.

[0076] In step S54, an adaptive attention mechanism is used to assign independent weight coefficients to each feature factor based on the similarity value. This adaptive attention mechanism dynamically adjusts the weight of each feature factor according to the needs of different scenarios, making the model more accurate in handling different scenarios and optimizing the accuracy of keyframe extraction.

[0077] As an example, in bank branch anti-fraud monitoring, after receiving scenario description text such as "a customer is blocking the camera in front of an ATM and is suspected of changing their bank card," the system first performs word segmentation and part-of-speech tagging on the text to extract semantic keywords directly related to risky behavior, such as "blocking," "camera," "abnormal action," and "bank card." These keywords are then mapped to word vectors and sequence-encoded to generate scenario semantic vectors that can be used for feature matching. Next, the similarity between this scenario semantic vector and the decoupled motion feature factors, scenario feature factors, and object semantic feature factors is calculated to determine if the scenario is more likely to be related to "suspicious motion behavior" or "sensitive object interaction." Finally, through an adaptive attention mechanism, the similarity is dynamically converted into a weight allocation result. In such risky scenarios, the weights of motion features and object semantic features are automatically increased, while background scene features are weakened, thus focusing more on key risk signals such as "blocking behavior," "suspicious gestures," and "bank card replacement trajectory," achieving semantic adaptive focus on financial anti-fraud behavior.

[0078] As another example, in the monitoring of minimally invasive surgery, if a scenario description text such as "the instrument is approaching a major blood vessel, with a suspected risk of bleeding" is received, the system will first perform word segmentation and part-of-speech tagging on the text, extracting keywords with strong clinical semantic relevance such as "instrument," "blood vessel," "approaching," and "risk of bleeding." Then, the aforementioned keywords are mapped to word vectors and sequence encoded to generate a semantic scene feature representation, which is used to match with the decoupled features on the video side. Further, the semantic similarity between this scene semantic vector and independent motion feature factors, scene structure feature factors, and object semantic feature factors is calculated to determine whether the instruction leans more towards "minor continuous motion changes" or "risk of contact between the instrument and critical tissues." Finally, through an adaptive attention mechanism, the above similarities are mapped to dynamic weights, enabling the model to automatically strengthen the feature responses corresponding to "motion factors" and "object factors," while weakening the weight of background tissue texture or overall surgical field layout. This allows for more accurate focus on high-risk actions such as "whether the instrument tip is touching a blood vessel" and "whether there are subtle signs of bleeding," achieving precise semantic perception and focused monitoring of key intraoperative nodes.

[0079] Through steps S51 to S54, the effective matching and optimization of scene description text and video spatiotemporal feature factors were successfully achieved. First, keywords related to the video content were extracted through word segmentation and part-of-speech tagging, providing high-quality semantic information for subsequent feature mapping and calculation. Then, the scene description text was transformed into a numerical scene text feature vector, which provided a unified representation for subsequent similarity calculation. During similarity calculation, the degree of matching between the scene text and each feature factor was quantified, providing a basis for subsequent weight adjustments. Finally, through an adaptive attention mechanism, corresponding weight coefficients were dynamically assigned to each feature factor, enabling the model to optimize the independence and expressive power of motion, scene, and object feature factors according to different scene requirements, thereby improving the accuracy and efficiency of keyframe extraction.

[0080] S6. The independent motion feature factors, independent scene feature factors, and independent object feature factors are weighted and fused according to independent weights. The importance score of each video frame is output by a multilayer perceptron based on the weighted fusion result.

[0081] In this process, independent motion feature factors, scene feature factors, and object feature factors are weighted and fused according to their respective independent weights to generate a comprehensive video frame feature representation. Element-wise multiplication ensures that the contribution of each feature factor matches its actual importance. Through a multilayer perceptron, the fused features undergo further nonlinear transformation and dimensionality compression to extract deep features and generate frame-level feature vectors. Finally, the model outputs an importance score for each frame using these feature vectors, providing a basis for keyframe selection. This process accurately quantifies the importance of each frame, making video analysis more accurate.

[0082] As an optional implementation, the process of outputting importance scores in step S6 above may specifically include the following steps S61 to S64.

[0083] S61. Multiply the independent motion feature factor, independent scene feature factor, and independent object feature factor with their respective independent weights element by element to obtain the weighted motion feature, weighted scene feature, and weighted object feature.

[0084] Channel concatenation integrates weighted motion, scene, and object features into a unified feature representation, forming a multi-channel fused feature matrix. This fused feature matrix not only contains feature information from different decoupled branches but also preserves the independence of each feature factor, providing richer input for subsequent nonlinear transformations.

[0085] S62. The weighted motion features, weighted scene features, and weighted object features are concatenated along the channel dimension to generate a fused feature matrix.

[0086] Channel concatenation integrates weighted motion, scene, and object features into a unified feature representation, forming a multi-channel fused feature matrix. This fused feature matrix not only contains feature information from different decoupled branches but also preserves the independence of each feature factor, providing richer input for subsequent nonlinear transformations.

[0087] S63. Input the fused feature matrix into a preset multilayer perceptron. The fused features are nonlinearly transformed and dimensionally compressed through the fully connected layer and activation function layer of the multilayer perceptron to obtain the frame-level feature vector.

[0088] The multilayer perceptron processes fused features through multiple fully connected layers and activation function layers to capture the complex relationships between features and improves the model's expressive power through nonlinear transformations. Dimensionality compression reduces the dimensionality of the feature space and removes redundant information, making the final frame-level feature vector more compact and efficient.

[0089] S64. Based on the frame-level feature vector, output a one-dimensional score vector that matches the number of video frames. Each element in the vector corresponds to the importance score of each video frame.

[0090] The model assigns an importance score to each video frame using the output one-dimensional score vector. This score reflects the relative importance of each frame in the video. Frames with high scores indicate that the content of that frame in the video is more critical and can be used for subsequent keyframe selection or video summarization.

[0091] Through steps S61 to S64, weighted fusion of independent feature factors was achieved, and the importance score of each video frame was output through a multilayer perceptron. First, each independent feature factor was weighted with its corresponding weight using element-wise multiplication to ensure that the contribution of each feature matched its actual importance. Then, the weighted features were concatenated along the channel dimension to form a unified fusion feature matrix, providing a multi-dimensional feature representation for subsequent processing. Based on the aforementioned processing, deep feature information of the video frames was further extracted through nonlinear transformation and dimensionality compression of the multilayer perceptron, generating a refined frame-level feature vector. Finally, an importance score vector matching the number of video frames was generated using the calculated frame-level feature vector, providing a clear basis for keyframe selection. The aforementioned process effectively improves the accuracy of video analysis, enabling the model to accurately identify and select keyframes in the video.

[0092] S7. Perform temporal non-maximum suppression processing on the importance score, and select video frames with an importance score greater than the preset importance threshold as target video keyframes.

[0093] Temporal Non-Maximum Suppression (NMS) eliminates redundancy between adjacent frames, ensuring that each selected keyframe has independent importance in the video. In this process, if adjacent video frames have similar scores and exceed a preset importance threshold, only one frame is retained, while other redundant frames are suppressed. This ensures that the selected keyframes are representative while avoiding excessive redundant information. Ultimately, the selected target video keyframes more accurately reflect the core parts of the video content and provide effective support for video summarization or analysis tasks.

[0094] As an example, the problem of non-differentiability in traditional temporal nonmaximum suppression is solved by the following improvement: First, replace hard suppression with soft masking: in, for t Frame neighborhood window, The first step is to use the learnable inhibition strength parameter. The second step is to use Gumbel-Softmax (gradient estimation in deep learning, which solves the problem of "gradient non-differentiability" in discrete selection tasks by introducing the Gumbel distribution to continuously relax the discrete distribution, so that the discrete decision-making process can be optimized through backpropagation) to approximate the gradient of the Top-k frames (in time series data processing, the "top k most important" frames are selected from the sequence).

[0095] As an example, please continue reading Figure 5 , Figure 5 This is a data flow example diagram of the video keyframe extraction process based on a decoupled spatiotemporal feature extractor provided in the embodiments of this application, such as... Figure 5 As shown, the video keyframe extraction method in this example mainly includes four stages: data preprocessing, decoupled spatiotemporal feature extraction, scene-adaptive attention mechanism, and post-processing. First, in the data preprocessing stage, the system receives the raw video stream and performs video normalization and format conversion sequentially, unifying videos from different sources and resolutions into a standardized sequence suitable for model input. Then, in the decoupled spatiotemporal feature extraction stage, the video sequence is first input to the temporal convolutional network backbone (TCN Backbone) to extract preliminary temporal features. Next, decoupled branches are used to model independent semantics such as motion factors and scene factors, and factor-separation discriminators further enhance the mutual exclusivity and discriminability of each factor. In the scene-adaptive attention mechanism stage, the system introduces a Transformer structure as an attention weight calculation module, dynamically adjusting the importance of each feature factor according to scene semantics, and using Softmax feature weighting to adaptively allocate the weights of different spatial and temporal features. Subsequently, in the post-processing stage, redundant frames are first removed using temporal non-maximum suppression to avoid generating too many similar outputs in continuous actions. Finally, key frame selection is performed to obtain target key frames that are semantically complete and have the optimal information density, and the results are displayed and interacted with through the user interface module.

[0096] As another example, please continue reading Figure 6 , Figure 6 This is a data flow example diagram illustrating the synchronous data processing of the decoupled spatiotemporal feature extractor and the scene adaptive attention mechanism provided in the embodiments of this application, as shown below. Figure 6 As shown, in the keyframe extraction part of this example, the temporal convolutional backbone network first performs multi-scale feature representation learning on the input video frame sequence. The input video frame sequence is processed by dilated convolution to extract long-term dependency information, generating multi-scale spatiotemporal features containing different temporal receptive fields. Based on the above processing, the feature decoupling stage begins. The multi-scale spatiotemporal features are input to three independent decoupling coding branches, including a motion encoder for capturing motion dynamics, a scene encoder for extracting scene structural semantics, and an object encoder for recognizing target entity semantics. To further improve the discriminativeness of the decoupling features, corresponding factor discriminators are also set, including a motion discriminator, a scene discriminator, and an object discriminator, to constrain the features generated by each coding branch to be semantically independent and distinguishable. Subsequently, the scene adaptive attention mechanism stage begins. Task semantics or application instructions are provided through the scene description input module. This description is converted into a high-dimensional scene semantic vector by the Transformer Encoder and participates in the attention weight calculation together with the aforementioned decoupled features, thereby dynamically adjusting the importance of different feature subspaces. Finally, the optimal spatiotemporal feature representation is generated through the weighted feature fusion module, and keyframe importance scores are output for subsequent keyframe selection.

[0097] As another example, this embodiment extracts keyframes from a football match video. First, a 90-minute match video (1080p, 30fps) is processed to extract key actions from the game. Through dynamic spatial feature extraction, motion factors, scene factors, and object factors are extracted from each frame, specifically including player running and ball trajectory, the spatial division of the football field, and the interaction between the ball and players. Then, a spatial-temporal attention mechanism is used to dynamically adjust the weight of scene descriptions, focusing on key actions such as goals and passes. Finally, 15 keyframes are selected from the video to accurately capture crucial moments and actions of the match. Compared to traditional methods, this approach reduces redundant frames by at least 50%, removing irrelevant backgrounds or spectator activities, thus extracting the core content of the match more efficiently and accurately, providing a more focused and concise version of the game.

[0098] The keyframe extraction method based on decoupled multi-task feature learning provided in this application improves the accuracy and efficiency of video analysis by effectively decoupling motion, scene, and object features in videos. Through normalized preprocessing of video frames, decoupling and splitting spatiotemporal features, and weighted fusion, it effectively integrates information at different levels while maintaining the independence of each feature, generating more representative and informative keyframes. Adaptive attention mechanisms and factor separation discriminators further optimize the independence of each feature factor, ensuring accurate representation between features and reducing redundant information. Temporal non-maximum suppression processing also eliminates redundancy between adjacent frames, retaining the most important frames, thereby generating concise and accurate video summaries. The decoupled multi-task feature learning method in this application not only improves the model's adaptability to complex video content but also enhances the robustness and accuracy of keyframe selection in different scenarios.

[0099] Please continue reading. Figure 7 , Figure 7 This is a schematic diagram of the system structure of the keyframe extraction device based on decoupled multi-task feature learning provided in the embodiments of this application, as shown below. Figure 7 As shown, the keyframe extraction device 70 based on decoupled multi-task feature learning includes: an input video preprocessing module 71, a video spatiotemporal feature module 72, a feature decoupling and splitting module 73, a factor separation discriminator module 74, a factor independent weighting module 75, a factor weighted fusion module 76, and a video keyframe filtering module 77.

[0100] The input video preprocessing module 71 is specifically used to perform frame sampling, size normalization and brightness normalization preprocessing on the input video, and output a normalized frame sequence.

[0101] The video spatiotemporal feature module 72 is specifically used to input the normalized frame sequence into the temporal convolutional backbone network of the decoupled spatiotemporal feature extractor, and extract the spatiotemporal features of the video through the stacked dilated causal convolutional layers of the temporal convolutional backbone network.

[0102] The feature decoupling and splitting module 73 is specifically used to decouple and split the spatiotemporal features according to the parallel motion dynamic factor decoupling branch, scene composition factor decoupling branch and object semantic factor decoupling branch in the decoupled spatiotemporal feature extractor, to obtain initial motion feature factors, initial scene feature factors and initial object feature factors.

[0103] The factor separation discriminator module 74 is specifically used to perform independence verification and optimization processing on the initial motion feature factor, initial scene feature factor and initial object feature factor through the factor separation discriminator to obtain independent motion feature factor, independent scene feature factor and independent object feature factor.

[0104] The factor independent weight module 75 is specifically used to convert the scene description text of the input video into a scene vector, and calculate the independent weights of the independent motion feature factor, independent scene feature factor and independent object feature factor according to the scene vector through an adaptive attention mechanism. The factor weighted fusion module 76 is specifically used to perform weighted fusion of the independent motion feature factor, independent scene feature factor and independent object feature factor according to the independent weights, and output the importance score of each video frame through the multilayer perceptron based on the weighted fusion result. The video keyframe filtering module 77 is specifically used to perform temporal non-maximum suppression processing on the importance score, and filter out video frames with an importance score greater than a preset importance threshold as target video keyframes.

[0105] As an optional implementation, the video spatiotemporal feature module 72 is further configured to set at least two cascaded dilated convolutional units, each of which is sequentially connected to a causal convolutional layer, a batch normalization layer, and an activation function layer; set the dilation coefficients of adjacent dilated convolutional units in a preset ascending order, so that the dilation coefficients increase layer by layer to expand the temporal receptive domain; and connect the output features of each dilated convolutional unit across channels through the temporal constraints of the causal convolutional layer and the progressive setting of the dilation coefficients; and compress the input dimension of the connected features into a convolutional layer to compress the feature dimension, thereby obtaining the constructed stacked dilated causal convolutional layer.

[0106] As an optional implementation, the feature decoupling and splitting module 73 is further configured to configure an independent feature processing unit for each decoupling branch. The motion dynamic factor decoupling branch adopts a temporal differential convolutional layer, the scene composition factor decoupling branch is configured with a spatial attention module, and the object semantic factor decoupling branch is connected to a pre-trained object detection feature extraction sub-network. The feature processing units of each branch process the input spatiotemporal features separately, controlling the temporal differential convolutional layer to capture inter-frame motion changes, the spatial attention module to enhance the image structure features, and the object detection feature extraction sub-network to extract object semantic information. The correlation between the output features of different branches is weakened through the inter-branch feature suppression mechanism to obtain the initial motion feature factor, the initial scene feature factor, and the initial object feature factor.

[0107] As an optional implementation, the factor separation discriminator module 74 is further specifically used to input the initial motion feature factor, initial scene feature factor, and initial object feature factor to the factor separation discriminator, calculate the cross-correlation between different factors through the feature association detection module of the factor separation discriminator, generate an association penalty signal based on the cross-correlation calculation result, and reverse adjust the parameters of the feature processing units of each decoupled branch to weaken the redundant information of different types of feature factors; repeat the steps from calculating the cross-correlation between different factors through the feature association detection module of the factor separation discriminator to reverse adjusting the parameters of the feature processing units of each decoupled branch until the cross-correlation result between different factors is less than a preset cross-correlation threshold, thereby obtaining the independent motion feature factor, independent scene feature factor, and independent object feature factor.

[0108] As an optional implementation, the factor independent weight module 75 is further specifically used to perform word segmentation and part-of-speech tagging on the input scene description text, and filter out related words that are associated with motion state, scene structure, and object attributes; map the related words into word vectors, and perform sequence encoding on the word vectors to generate scene text feature vectors; calculate the different similarities between the scene text feature vectors and the independent motion feature factors, independent scene feature factors, and independent object feature factors respectively, to obtain three sets of feature similarity values; and convert the three sets of feature similarity values ​​into corresponding weight coefficients through an adaptive attention mechanism, and set the weight coefficients as the independent weights of each independent feature factor.

[0109] As an optional implementation, the factor-weighted fusion module 76 is further specifically used to multiply the independent motion feature factor, independent scene feature factor, and independent object feature factor with their respective independent weights element-wise to obtain weighted motion features, weighted scene features, and weighted object features; to concatenate the weighted motion features, weighted scene features, and weighted object features along the channel dimension to generate a fusion feature matrix; to input the fusion feature matrix into a preset multilayer perceptron, and to perform nonlinear transformation and dimensionality compression on the fusion features through the fully connected layer and activation function layer of the multilayer perceptron to obtain a frame-level feature vector; and to output a one-dimensional score vector matching the number of video frames based on the frame-level feature vector, where each element in the vector corresponds to the importance score of each video frame.

[0110] As an optional implementation, the video keyframe filtering module 77 is further configured to: arrange the importance scores according to the temporal order of the video frames to obtain a score sequence with temporal labels; traverse the score sequence according to a preset suppression window size, retaining the video frame with the highest importance score in each window as a candidate keyframe; calculate the difference between the importance scores of all candidate keyframes and a preset importance threshold, and filter out the candidate keyframes with positive differences; sort the filtered candidate keyframes according to the original video temporal order, and output the sorted frame sequence as the target video keyframe.

[0111] It should be noted that the aforementioned keyframe extraction device based on decoupled multi-task feature learning can execute the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the keyframe extraction device based on decoupled multi-task feature learning can be found in the keyframe extraction method based on decoupled multi-task feature learning provided in the embodiments of this application.

[0112] Figure 8 This is a schematic diagram of the hardware structure of an electronic device that performs a keyframe extraction method based on decoupled multi-task feature learning, as provided in an embodiment of this application. Figure 8 As shown, the electronic device 800 includes: One or more processors 810 and memory 820, Figure 8 Take the 810 processor as an example.

[0113] The processor 810 and the memory 820 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0114] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the keyframe extraction method based on decoupled multi-task feature learning in the embodiments of this application. The processor 810 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the keyframe extraction method based on decoupled multi-task feature learning described in the above embodiments.

[0115] The memory 820 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the keyframe extraction device based on decoupled multi-task feature learning. Furthermore, the memory 820 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include memory remotely located relative to the processor 810, and these remote memories may be connected to the keyframe extraction device based on decoupled multi-task feature learning via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0116] The one or more modules are stored in the memory 820. When executed by the one or more processors 810, they perform the keyframe extraction method based on decoupled multi-task feature learning in any of the above method embodiments. For example, they perform the methods described above. Figure 2 Method steps S1 to S7, Figure 3 Method steps S31 to S33, Figure 4 The method steps S51 to S54 are implemented. Figure 5 The functions of modules 71-77 in the document.

[0117] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0118] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 8 One of the processors 810 can enable the above-described one or more processors to execute the keyframe extraction method based on decoupled multi-task feature learning in any of the above method embodiments, for example, to execute the method described above. Figure 2Method steps S1 to S7, Figure 3 Method steps S31 to S33, Figure 4 The method steps S51 to S54 are implemented. Figure 5 The functions of modules 71-77 in the document.

[0119] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by an electronic device, enable the electronic device to perform the keyframe extraction method based on decoupled multi-task feature learning in any of the above method embodiments. For example, it can execute the method described above. Figure 2 Method steps S1 to S7, Figure 3 Method steps S31 to S33, Figure 4 The method steps S51 to S54 are implemented. Figure 5 The functions of modules 71-77 in the document.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A keyframe extraction method based on decoupled multi-task feature learning, characterized in that, include: Perform frame sampling, size normalization, and luminance normalization preprocessing on the input video, and output a normalized frame sequence; The normalized frame sequence is input into the temporal convolutional backbone network of the decoupled spatiotemporal feature extractor, and the spatiotemporal features of the video are extracted through the stacked dilated causal convolutional layers of the temporal convolutional backbone network. Based on the parallel motion dynamic factor decoupling branch, scene composition factor decoupling branch, and object semantic factor decoupling branch in the decoupled spatiotemporal feature extractor, the spatiotemporal features are decoupled and decomposed to obtain initial motion feature factors, initial scene feature factors, and initial object feature factors. The initial motion feature factor, initial scene feature factor, and initial object feature factor are subjected to independence verification and optimization processing by a factor separation discriminator to obtain independent motion feature factor, independent scene feature factor, and independent object feature factor. The scene description text of the input video is converted into a scene vector, and the independent weights of the independent motion feature factor, independent scene feature factor and independent object feature factor are calculated based on the scene vector using an adaptive attention mechanism. The independent motion feature factors, independent scene feature factors, and independent object feature factors are weighted and fused according to the independent weights, and the importance score of each video frame is output by a multilayer perceptron based on the weighted fusion result. Temporal nonmaximum suppression processing is applied to the importance scores to select video frames with importance scores greater than a preset importance threshold as target video keyframes.

2. The keyframe extraction method based on decoupled multi-task feature learning according to claim 1, characterized in that, The construction process of the stacked dilated causal convolutional layers of the temporal convolutional backbone network includes: Set up at least two cascaded dilated convolutional units, with each dilated convolutional unit connected sequentially to a causal convolutional layer, a batch normalization layer, and an activation function layer; The dilation coefficients of adjacent dilated convolutional units are set in a preset increasing order, so that the dilation coefficients increase layer by layer to expand the temporal receiving domain. By using the temporal constraints of the causal convolutional layer and the progressive setting of the dilation coefficient, the output features of each dilated convolutional unit are connected across channels. The convolutional layers with concatenated feature input dimensions are compressed to obtain the stacked dilated causal convolutional layers.

3. The keyframe extraction method based on decoupled multi-task feature learning according to claim 1, characterized in that, The step of decoupling and decomposing the spatiotemporal features according to the parallel motion dynamic factor decoupling branch, scene composition factor decoupling branch, and object semantic factor decoupling branch in the decoupled spatiotemporal feature extractor to obtain initial motion feature factors, initial scene feature factors, and initial object feature factors includes: Each decoupling branch is configured with an independent feature processing unit. The motion dynamic factor decoupling branch adopts a temporal difference convolutional layer, the scene composition factor decoupling branch is configured with a spatial attention module, and the object semantic factor decoupling branch is connected to a pre-trained object detection feature extraction sub-network. The feature processing units of each branch process the input spatiotemporal features separately, controlling the temporal difference convolutional layer to capture inter-frame motion changes, the spatial attention module to enhance the image structure features, and the object detection feature extraction sub-network to extract object semantic information. By weakening the correlation between the output features of different branches through the inter-branch feature suppression mechanism, the initial motion feature factor, initial scene feature factor, and initial object feature factor are obtained.

4. The keyframe extraction method based on decoupled multi-task feature learning according to claim 1, characterized in that, The step of performing independence verification and optimization processing on the initial motion feature factors, initial scene feature factors, and initial object feature factors through a factor separation discriminator to obtain independent motion feature factors, independent scene feature factors, and independent object feature factors includes: Input the initial motion feature factor, initial scene feature factor, and initial object feature factor into the factor separation discriminator, and calculate the cross-correlation between different factors through the feature association detection module of the factor separation discriminator; Based on the cross-correlation calculation results, a correlation penalty signal is generated, and the parameters of the feature processing units of each decoupled branch are adjusted in reverse to weaken the redundant information of different types of feature factors. Repeat the steps from calculating the cross-correlation between different factors through the feature association detection module of the factor separation discriminator to adjusting the parameters of the feature processing unit of each decoupled branch in reverse, until the cross-correlation result between different factors is less than the preset cross-correlation threshold, to obtain the independent motion feature factor, independent scene feature factor and independent object feature factor.

5. The keyframe extraction method based on decoupled multi-task feature learning according to claim 1, characterized in that, The scene description text of the input video is a scene vector. The step of calculating the independent weights of the independent motion feature factor, independent scene feature factor, and independent object feature factor based on the scene vector using an adaptive attention mechanism includes: The input scene description text is segmented and part-of-speech tagged to filter out related words that are associated with motion state, scene structure, and object attributes; The associated words are mapped to word vectors, and the word vectors are sequence encoded to generate scene text feature vectors; Calculate the different similarities between the scene text feature vector and the independent motion feature factor, independent scene feature factor, and independent object feature factor to obtain three sets of feature similarity values; The three sets of feature similarity values ​​are converted into corresponding weight coefficients through an adaptive attention mechanism, and the weight coefficients are set as the independent weights of each independent feature factor.

6. The keyframe extraction method based on decoupled multi-task feature learning according to claim 1, characterized in that, The step of weightedly fusing the independent motion feature factors, independent scene feature factors, and independent object feature factors according to the independent weights, and outputting the importance score of each video frame through a multilayer perceptron based on the weighted fusion result, includes: The independent motion feature factor, independent scene feature factor, and independent object feature factor are multiplied element-wise with their respective independent weights to obtain weighted motion features, weighted scene features, and weighted object features. The weighted motion features, weighted scene features, and weighted object features are concatenated along the channel dimension to generate a fused feature matrix; The fused feature matrix is ​​input into a preset multilayer perceptron, and the fused features are subjected to nonlinear transformation and dimensionality compression through the fully connected layer and activation function layer of the multilayer perceptron to obtain a frame-level feature vector; Based on the frame-level feature vector, a one-dimensional score vector matching the number of video frames is output, where each element in the vector corresponds to the importance score of each video frame.

7. The keyframe extraction method based on decoupled multi-task feature learning according to claim 1, characterized in that, The step of performing temporal non-maximum suppression processing on the importance score and selecting video frames with an importance score greater than a preset importance threshold as target video keyframes includes: The importance scores are arranged in temporal order according to the video frames to obtain a score sequence with temporal labels; The score sequence is traversed according to a preset suppression window size, and the video frame with the highest importance score is retained as a candidate keyframe within each window. Calculate the difference between the importance score of all the candidate keyframes and the preset importance threshold, and filter out the candidate keyframes with a positive difference. The selected candidate keyframes are sorted according to the original video time sequence, and the sorted frame sequence is output as the target video keyframes.

8. A keyframe extraction device based on decoupled multi-task feature learning, characterized in that, include: The input video preprocessing module is used to perform frame sampling, size normalization and brightness normalization preprocessing on the input video and output a normalized frame sequence; The video spatiotemporal feature module is used to input the normalized frame sequence into the temporal convolutional backbone network of the decoupled spatiotemporal feature extractor, and extract the spatiotemporal features of the video through the stacked dilated causal convolutional layers of the temporal convolutional backbone network. The feature decoupling and splitting module is used to decouple and split the spatiotemporal features according to the parallel motion dynamic factor decoupling branch, scene composition factor decoupling branch and object semantic factor decoupling branch in the decoupled spatiotemporal feature extractor, to obtain initial motion feature factors, initial scene feature factors and initial object feature factors; The factor separation discriminator module is used to perform independence verification and optimization processing on the initial motion feature factors, initial scene feature factors and initial object feature factors through the factor separation discriminator to obtain independent motion feature factors, independent scene feature factors and independent object feature factors; The factor-independent weighting module is used to convert the scene description text of the input video into a scene vector, and calculate the independent weights of the independent motion feature factor, independent scene feature factor and independent object feature factor based on the scene vector through an adaptive attention mechanism. The factor weighted fusion module is used to weight and fuse the independent motion feature factors, independent scene feature factors, and independent object feature factors according to the independent weights, and output the importance score of each video frame through a multilayer perceptron based on the weighted fusion result; The video keyframe filtering module is used to perform temporal non-maximum suppression processing on the importance score and filter out video frames with an importance score greater than a preset importance threshold as target video keyframes.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the keyframe extraction method based on decoupled multi-task feature learning as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the keyframe extraction method based on decoupled multi-task feature learning as described in any one of claims 1-7.

Images