Video classification method and system based on multi-fragment sampling and cross-modal matching

Abstract

A video classification method and system based on multi-fragment sampling and cross-modal matching. The method first performs adaptive multi-fragment division and inter-frame motion difference boundary adjustment according to the input video to generate semantically coherent video fragments; then performs key frame sampling on each fragment based on the cumulative information density function-based equal-order inverse mapping, and performs saliency weighted fusion on the sampled frames to obtain fragment visual features; then uses global semantic anchor points to weight and fuse each fragment feature to obtain a global video feature; and finally generates a classification result through cross-modal semantic matching. The present application can solve the problem of long video action truncation, reduce the amount of calculation while retaining the core semantic information, effectively suppress background noise, and significantly improve the semantic consistency and cross-modal robustness of video classification.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a video classification method and system based on multi-segment sampling and cross-modal matching. Background Technology

[0002] With the rapid development of internet technology and mobile terminals, video data is experiencing explosive growth, and video content analysis has become one of the core research directions in the field of artificial intelligence. Video classification, as a fundamental task of video understanding, is a key technological support for numerous applications such as content retrieval, intelligent recommendation, security monitoring, ad filtering, and copyright protection. Its accuracy and efficiency directly determine the performance of upper-layer applications.

[0003] Existing video classification methods are mainly divided into two categories: one is frame-level classification methods based on two-dimensional convolutional neural networks. This method extracts visual features frame by frame and then fuses temporal information through average pooling or recurrent neural networks. However, it cannot effectively capture long-term dependencies and is difficult to handle video content containing complex actions or rapid scene changes. The other category is end-to-end methods based on three-dimensional convolutional or Transformer architectures, such as C3D, I3D, and ViViT. Although these methods can model spatiotemporal relationships, they suffer from huge computational costs, high memory usage, and strong dependence on training data. They are difficult to adapt to long video processing and edge deployment scenarios, and lack effective semantic alignment capabilities, resulting in poor interpretability of classification results.

[0004] In recent years, cross-modal alignment models such as CLIP and VideoCLIP have demonstrated significant advantages in video understanding tasks due to their unified visual-text semantic space modeling capabilities. However, existing cross-modal video classification methods still have many shortcomings: they lack reasonable multi-segment sampling strategies for long videos, which can easily lead to the loss of key semantic information; the features of each segment are mostly fused using simple averaging methods, which cannot suppress irrelevant background noise and are difficult to stably represent the overall semantics of the video; the use of text prompts is limited in its approach, resulting in limited semantic coverage, and the accuracy of cross-modal alignment still needs to be improved. These problems prevent existing technologies from simultaneously meeting the high-precision and high-efficiency classification requirements of long videos, complex scene videos, and open-domain videos. Summary of the Invention

[0005] The purpose of this application is to provide a video classification method and system based on multi-segment sampling and cross-modal matching, so as to solve the problems in the prior art, such as insufficient sampling of long videos leading to feature loss, high video frame redundancy leading to feature instability, insufficient alignment between visual and text semantic spaces leading to reduced classification accuracy, and excessively high model computation cost making it unsuitable for practical deployment.

[0006] To achieve the above objectives, this application provides a video classification method based on multi-segment sampling and cross-modal matching, specifically comprising: generating multiple video segments that meet preset semantic conditions based on the input video through adaptive multi-segment division and inter-frame motion difference boundary adjustment; sampling keyframes based on the video segments through equal-order inverse mapping based on the cumulative information density function, and performing saliency-weighted fusion on the sampled keyframes to generate visual feature vectors for each segment; obtaining global video features by weighted fusion of the visual feature vectors of each segment using global semantic anchors; and generating video classification results through cross-modal semantic matching based on the global video features.

[0007] In the above video classification method, optionally, generating multiple video segments that meet preset semantic conditions based on the input video through adaptive multi-segment segmentation and inter-frame motion difference boundary adjustment includes: obtaining the total number of valid frames based on the input video; generating an initial segment boundary set based on the total number of valid frames through dynamic mixed segment segmentation; generating a final segment boundary set based on the initial segment boundary set through motion saliency boundary fine-tuning; and generating multiple video segments that meet preset semantic conditions based on the final segment boundary set.

[0008] In the above video classification method, optionally, generating the final segment boundary set by fine-tuning the motion saliency boundary based on the initial segment boundary set includes: calculating the inter-frame difference of all adjacent frames within a local time window of each initial boundary frame; generating a dynamic scene change threshold based on all inter-frame differences within the local time window; when the inter-frame difference of the initial boundary frame is greater than or equal to the dynamic scene change threshold, finding the frame that minimizes the cost function within the local time window as the updated boundary frame; verifying whether all updated boundary frames meet the minimum segment length limit, and generating the final segment boundary set.

[0009] In the above video classification method, optionally, the process of sampling keyframes based on the video segments through equal-order inverse mapping based on the cumulative information density function, and generating visual feature vectors for each segment by saliency-weighted fusion of the sampled keyframes, includes: generating a keyframe sequence for each video segment whose semantics meet preset conditions through equal-order inverse mapping of the information density axis; extracting visual features of each keyframe in the keyframe sequence; and generating a visual feature vector for the segment by saliency-weighted fusion based on the visual features.

[0010] In the above video classification method, optionally, the saliency-weighted fusion of the sampled keyframes to generate visual feature vectors for each segment includes: calculating the arithmetic mean of all visual features in the keyframe sequence as the segment baseline feature; calculating the L2 distance between each keyframe visual feature and the segment baseline feature as a saliency score; converting the saliency score into normalized weights using a Softmax function with a preset temperature parameter; and weighting and summing all keyframe visual features according to the normalized weights to generate the visual feature vector of the segment.

[0011] In the above video classification method, optionally, generating a keyframe sequence for each video segment that meets the preset semantic conditions through inverse mapping of information density axis order includes: calculating the transient motion saliency of adjacent frames within each video segment that meets the preset semantic conditions; constructing a cumulative information density function based on the transient motion saliency; dividing the value range of the cumulative information density function into order intervals equal to the standard target sampling frame number minus one; determining the target energy level corresponding to each order interval, and finding the frame with the cumulative information density closest to the target energy level on the time axis as the keyframe; performing temporal hard constraint verification on all keyframes to generate a keyframe sequence that is strictly increasing in time and has no repetition.

[0012] In the above video classification method, optionally, the video global features are obtained by weighted fusion of the visual feature vectors of each segment using global semantic anchors, and the video classification results are generated by cross-modal semantic matching based on the video global features, including: extracting text features of all preset classification categories; calculating the mean vector of the text features of all preset classification categories as global semantic anchors; performing semantic weighted fusion of the global semantic anchors and the visual feature vectors of each segment to generate video global features; calculating the cross-modal similarity between the video global features and the text features of each preset classification category; and generating video classification results based on the cross-modal similarity.

[0013] In the above video classification method, optionally, the process of generating global video features by semantically weighting and fusing the visual feature vectors of all segments based on the global semantic anchor point includes: calculating the cosine similarity between the visual feature vector of each segment and the global semantic anchor point; converting the cosine similarity into segment normalization weights using a Softmax function with a preset temperature parameter; and weighting and summing the visual feature vectors of all segments based on the segment normalization weights to generate global video features.

[0014] In the above video classification method, optionally, extracting text features of all preset classification categories includes: generating basic prompt words for each preset classification category; semantically expanding the basic prompt words to generate multiple extended prompt words; extracting text features of the basic prompt words and all extended prompt words respectively; and arithmetically averaging all extracted text features to generate the final text features of the preset classification category.

[0015] This application also provides a video classification system based on multi-segment sampling and cross-modal matching. The system includes a preprocessing module, a visual feature extraction module, and a classification module. The preprocessing module generates multiple video segments with semantically satisfied preset conditions based on the input video through adaptive multi-segment division and inter-frame motion difference boundary adjustment. The visual feature extraction module samples keyframes from the video segments using equal-order inverse mapping based on the cumulative information density function, and performs saliency-weighted fusion on the sampled keyframes to generate visual feature vectors for each segment. The classification module obtains global video features by weighted fusion of the visual feature vectors of each segment using global semantic anchors, and generates video classification results through cross-modal semantic matching based on the global video features.

[0016] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0017] This application also provides a computer-readable storage medium storing a computer program that performs the above-described methods.

[0018] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0019] The beneficial technical effects of this application are as follows: By adaptively dividing multiple segments and adjusting the boundary of inter-frame motion differences, long videos can be divided into multiple semantically complete segments, avoiding the truncation of continuous actions caused by mechanical division and ensuring that the content of each time period of the video is effectively captured; by sampling keyframes through equal-order inverse mapping based on the cumulative information density function, the sampled frames adaptively gather towards high information density regions, maximizing the preservation of the core semantics of the segments while significantly reducing the amount of computation; through a two-level fusion strategy of intra-segment saliency-weighted fusion and segment-level global semantic anchor weighted fusion, irrelevant background noise can be effectively suppressed, key semantic information related to the classification task can be highlighted, and the stability and representativeness of global video features can be improved; through cross-modal semantic matching based on global semantic anchors, the global features of the video naturally have textual orientation, significantly improving the semantic consistency between vision and text, while using cue word expansion and ensemble techniques to enhance the model's semantic coverage of categories.

[0020] This application can quickly expand to new classification categories without retraining the model, has strong open-domain dynamic expansion capabilities, and exhibits good robustness to anomalies such as missing frames, video noise, and decoding failures. The classification results can be interpreted through semantic similarity. It can be widely applied to automatic classification tasks of various long videos, complex scene videos, and open-domain videos, significantly reducing computational costs while ensuring classification accuracy, and facilitating engineering deployment and practical application. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a video classification method based on multi-segment sampling and cross-modal matching provided in an embodiment of this application. Figure 2 This is a schematic diagram illustrating the process of adaptive multi-segment segmentation and inter-frame motion difference boundary adjustment provided in an embodiment of this application; Figure 3 This is a schematic diagram of the process for fine-tuning motion saliency boundaries according to an embodiment of this application; Figure 4 This is a schematic diagram of the process of equal-order inverse mapping and saliency-weighted fusion based on the cumulative information density function provided in an embodiment of this application; Figure 5 This is a schematic diagram of a saliency-weighted fusion process for keyframes provided in an embodiment of this application; Figure 6 This is a schematic diagram of the process for generating a keyframe sequence through inverse mapping of information density axis according to an embodiment of this application; Figure 7This is a flowchart illustrating the global semantic anchor weighted fusion and cross-modal semantic matching classification provided in an embodiment of this application. Figure 8 This is a schematic diagram of a semantic weighted fusion process based on global semantic anchors, provided in an embodiment of this application. Figure 9 This is a schematic diagram of a process for extracting text features of a preset category according to an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The following will describe in detail the implementation methods of this application with reference to the accompanying drawings and embodiments, so as to fully understand how this application uses technical means to solve technical problems and achieve technical effects, and to implement it accordingly. It should be noted that, as long as there is no conflict, the various embodiments and features in each embodiment of this application can be combined with each other, and the resulting technical solutions are all within the protection scope of this application.

[0023] Furthermore, the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0024] In the description of this application, it should be understood that, unless otherwise specified, the same or similar terms appearing in different embodiments or different drawings all have the same or similar technical meanings. The terms are explained as follows: In the description of this application, it should be understood that, unless otherwise specified, the same or similar terms appearing in different embodiments or different figures have the same or similar technical meanings; "inter-frame motion difference boundary adjustment" and "boundary fine-tuning based on motion saliency" have the same meaning; "cumulative information density function", "cumulative information density distribution function" and "cumulative distribution function (CDF)" have the same meaning; "information density axis equal-order inverse mapping" and "inverse mapping to the time axis after equal-order division on the cumulative information density function value range" have the same meaning; "saliency weighted fusion" and "weighted fusion based on saliency score" have the same meaning; "global semantic anchor point" and "mean vector of text features of all preset classification categories" have the same meaning; "preset temperature parameter" and "temperature coefficient" have the same meaning, including frame-level temperature parameter, temporal temperature parameter and classification temperature parameter; "transient motion saliency" and "instantaneous motion saliency" have the same meaning.

[0025] Please refer to Figure 1As shown, the video classification method based on multi-segment sampling and cross-modal matching provided in this application specifically includes: S101 generates multiple video segments that meet preset semantic conditions by adaptively dividing the input video into multiple segments and adjusting the boundaries of inter-frame motion differences. Specifically, this step first acquires the input video to be classified and analyzes its total number of valid frames. Then, an initial set of segment boundaries is generated based on a dynamic hybrid segmentation strategy. Next, a boundary fine-tuning algorithm based on motion saliency is used to locally optimize the initial boundaries, ultimately outputting multiple semantically coherent video segments with untrunculated actions. This step addresses the technical challenges of sparse key features and the potential for mechanical segmentation to destroy the integrity of actions in long videos.

[0026] S102 performs keyframe sampling based on the video segment using equal-order inverse mapping based on the cumulative information density function, and performs saliency-weighted fusion on the sampled keyframes to generate visual feature vectors for each segment. Specifically, for each video segment generated in step S101, this step first calculates the transient motion saliency of adjacent frames within the segment and constructs a cumulative information density function based on this. Then, the value range of this function (i.e., the information energy axis) is divided into several intervals of equal order, and the keyframe index is determined through inverse mapping, so that the sampled frames adaptively cluster towards the region of intense motion. Next, the visual encoder of the pre-trained VideoCLIP-XL model is used to extract the visual features of each keyframe, and the extracted feature vector is L2 normalized. Finally, the features of all keyframes within the segment are saliency-weighted and fused to generate a segment visual feature vector that can represent the core semantics of the segment.

[0027] S103 uses global semantic anchors to weighted fuse the visual feature vectors of each segment to obtain global video features, and generates video classification results through cross-modal semantic matching based on the global video features.

[0028] Specifically, this step first extracts text features from all preset classification categories and calculates the mean vector of these text features as global semantic anchors. Then, it calculates the cosine similarity between the visual features of each segment and the global semantic anchors, normalizes it using a Softmax function with a temperature parameter to obtain segment-level weights, and performs a weighted sum of all segment features to obtain the video global features (i.e., weighted fusion of global semantic anchors). Next, it uses the text encoder of the pre-trained VideoCLIP-XL model to encode the text description (which can be expanded with prompt words) corresponding to each category label and performs L2 normalization to obtain text semantic features for each category. Finally, it calculates the cosine similarity between the video global features and the text semantic features of each category, normalizes it using a Softmax function with a temperature parameter to generate classification probabilities, and outputs the category corresponding to the highest probability as the video classification result.

[0029] Please refer to Figure 2 As shown, in one embodiment of this application, generating multiple video segments that semantically meet preset conditions based on the input video through adaptive multi-segment segmentation and inter-frame motion difference boundary adjustment includes: S201 Obtains the total number of valid frames based on the input video; generates an initial segment boundary set based on the total number of valid frames through dynamic segment blending. Specifically, let the total number of valid frames in the input video be... In this embodiment, the standard target number of sampling frames for each segment is preset. (e.g., 8 frames) and the maximum number of segments supported by the model threshold. (In this embodiment, 32 is optional); this value is determined based on the engineering optimality that the memory usage of Transformer architectures such as VideoCLIP-XL increases non-linearly with sequence length, while also taking into account GPU memory limitations and inference latency. First, calculate the theoretical number of segments that satisfy the minimum sampling density: Then determine the actual number of segments: Next, calculate the average segment length (in frames): Finally, based on the above parameters, generate K initial fragment boundary frame index sets, denoted as... , If the total number of video frames At that time, the video is not divided into segments; the entire video is treated as a single, complete segment.

[0030] S202 generates the final fragment boundary set by fine-tuning the motion saliency boundary based on the initial fragment boundary set; This step introduces an adaptive boundary fine-tuning mechanism based on a local cost function to avoid mechanically truncating continuous actions (e.g., a complete punching motion being split into two segments). For each initial boundary frame, fine-tuning is performed within its local search window.

[0031] S203 generates multiple video segments whose semantics meet preset conditions based on the final segment boundary set.

[0032] Using the final boundary set obtained after optimization in step S202 as the basis for segmentation, K video segments are cut from the original video. Each segment contains multiple consecutive frame sequences. Because these segments have undergone boundary fine-tuning, their internal actions or scenes have better coherence, avoiding semantic damage to key actions.

[0033] After obtaining the initial boundary set, to avoid mechanically truncating continuous actions (e.g., a complete punching motion being divided into two segments), this embodiment introduces an adaptive boundary fine-tuning mechanism based on a local cost function. Please refer to... Figure 3 As shown, in one embodiment of this application, generating a final fragment boundary set by fine-tuning the motion saliency boundary based on the initial fragment boundary set includes: S301 calculates the inter-frame differences of all adjacent frames within the local time window of each initial boundary frame; Specifically, the inter-frame motion saliency metric is applied to any given initial boundary frame. Within its local search window, the motion saliency of adjacent frames is calculated. To ensure computational efficiency, this application uses the sum of absolute differences (SAD) of adjacent frames (after grayscale conversion), denoted as... , indicating the first Frame and the The degree of difference between frames.

[0034] S302 generates a dynamic scene change threshold based on all inter-frame differences within the local time window; Specifically, it is possible Centered on, with radius Local time window Inside (e.g.) (corresponding to 0.5 seconds of the video frame), calculate the mean of all inter-frame differences. with standard deviation Define the threshold for dynamic scene changes. (in (This is the sensitivity coefficient, for example, 1.5). This indicates that the boundary is in a relatively static and stable area of ​​the image, and no adjustment is needed.

[0035] S303 When the inter-frame difference of the initial boundary frame is greater than or equal to the dynamic scene change threshold, find the frame that minimizes the cost function within the local time window as the updated boundary frame. Specifically, if This indicates the initial boundary. The video is currently in a high dynamic range (e.g., during rapid movement or camera transitions), posing a risk of motion cutoff and triggering boundary fine-tuning. If This indicates that the boundary is in a relatively static and stable area of ​​the image, and no adjustment is needed.

[0036] In practice, firstly, the saliency of all motions within the local search window is subjected to minimax local normalization: Local time window for search Significance of all motions within The local normalization process for maximum and minimum values ​​is performed, and its calculation formula is as follows: ; in: Indicates the normalized i-th The motion saliency of the frame, with a value range of 100. ; Indicates the first before normalization The original motion saliency of the frame (i.e., absolute difference and SAD); Indicates the current local search window Within, the minimum value of motion saliency across all frames, i.e. ; Indicates the current local search window Within, the maximum value of motion saliency across all frames, i.e. ; To prevent extremely small constants with a denominator of zero (e.g., taking...) ).

[0037] When boundary fine-tuning is triggered, in the aforementioned local search window Find an optimal candidate boundary frame within the frame. To prevent excessive boundary adjustments from causing severe imbalances in the lengths of adjacent segments, this invention constructs the following cost function. : ; in: Candidate Frame The motion saliency normalization results are intended to guide the segmentation boundary to slide into a stable area where the image is still and the action is paused; This is an offset penalty term designed to constrain the distance the boundary deviates from its initial position; These are the balancing coefficients and weights.

[0038] Balance coefficient The physical meaning lies in defining the relative saliency cost required for each frame offset of the allowed segmentation boundary. If... The offset penalty is too weak. During cost optimization, the algorithm searches for absolutely still frames without considering distance costs, causing the segmentation boundary to easily slip to the very edge of the search window. This leads to severe distortion and imbalance in the length of adjacent video segments, disrupting the uniform distribution of global keyframe sampling. The offset penalty is too strong. This means that even if the segmentation boundary is only offset by two or three frames, the resulting penalty will quickly exceed the change in motion saliency between frames. This will cause the cost function to become overly rigid, and the boundary will be forced to remain at the initial high-dynamic position, unable to cross the peak, thus causing the anti-motion truncation protection mechanism to completely fail. This application will... Set as The range of values ​​can achieve optimal decoupling and balance between two interdependent indicators: minimizing local action peaks (protecting the semantic integrity of actions) and strongly constraining the overall offset distance (maintaining the balance of temporal segment division).

[0039] Search within the search window for the cost function The smallest frame position is updated to the final fragment boundary: ; S304 checks whether all updated boundary frames meet the minimum fragment length limit and generates the final fragment boundary set.

[0040] Specifically, it can be verified whether the updated adjacent boundaries satisfy the minimum fragment length constraint (i.e. If the conditions are met, fine-tuning is accepted; otherwise, the current fine-tuning is abandoned to ensure sampling uniformity. Minimum segment length limit. It is not a fixed constant, but an adaptive dynamic threshold based on the target sampling density and the global video segmentation scale. The specific calculation formula is as follows: ; in, (Physical sampling lower limit): Represents the standard target sampling frame number set within each segment (e.g., ... The algorithm then needs to extract from this fragment. If there are 1 keyframe, then the absolute physical length of that segment must not be less than 1. If the fragment length Subsequent algorithms will have to resort to frame padding to pad the data, resulting in a large amount of redundant temporal features being injected into the network, severely interfering with VideoCLIP-XL's attention mechanism. Therefore, It constitutes The absolute bottom line. (Global Scale Benchmark): Represents the initial average segment length calculated in step one (i.e., ). (Flexibility tolerance coefficient): This is a manually set equilibrium coefficient, with a preferred value range of [value range missing]. (This application is for) ).

[0041] Please refer to Figure 4 As shown, in one embodiment of this application, keyframe sampling is performed on the video segments using an equal-order inverse mapping based on a cumulative information density function, and the sampled keyframes are then saliency-weightedly fused to generate visual feature vectors for each segment, including: S401 generates a keyframe sequence for each video segment whose semantics meet the preset conditions by performing an inverse mapping of the information density axis. This step implements an equal-order inverse mapping algorithm based on the cumulative information density function, which transforms uniform sampling on the time axis into uniform sampling on the information energy axis, allowing sampled frames to adaptively cluster densely towards areas of intense motion. The specific implementation will be described in detail in Example Six. Finally, for each segment, a keyframe index sequence with a length strictly equal to the preset target number of sampled frames is output, and the first frame of this sequence must be the first frame of the segment, and the last frame must be the last frame of the segment.

[0042] S402 extracts the visual features of each keyframe in the keyframe sequence; For each segment's keyframe sequence, each frame image is normalized (e.g., size normalization, pixel value normalization), and then input into the pre-trained VideoCLIP-XL model's visual encoder. The VideoCLIP-XL model's visual encoder employs a Transformer architecture, capable of extracting visual features with spatiotemporal semantics. For each frame, a high-dimensional feature vector (e.g., 768-dimensional or 1024-dimensional) is output. After extraction, L2 normalization is applied to all frame features to ensure scale consistency across the feature space. This results in a vector containing the standard target sampled frames. The frame fragment will be obtained Each L2-normalized frame-level visual feature vector.

[0043] S403 generates the visual feature vector of the segment by saliency-weighted fusion based on the visual features.

[0044] This step abandons simple average pooling and adopts a saliency-guided weighting strategy. In a few-second clip, most frames are likely to have similar backgrounds or static action, with only a few frames showing drastic changes (i.e., frames far from the average feature). These changed frames contain the core action semantics of the clip. Therefore, frames far from the average feature should be assigned higher weights. Please refer to [reference needed]. Figure 5 As shown, in one embodiment of this application, the saliency-weighted fusion of the sampled keyframes to generate visual feature vectors for each segment includes: S501 calculates the arithmetic mean of all visual features in the keyframe sequence as the segment reference feature; Containing a fragment For example, a frame ( The eight frame feature vectors extracted from this segment are as follows: First, calculate this The arithmetic mean of the eigenvectors is used to obtain the global average feature value of the segment. : ; It represents the average state of the image over those few seconds (such as a background and people that remain still).

[0045] S502 calculates the L2 distance between the visual features of each keyframe and the reference features of the segment as a saliency score; Specifically, calculate each frame Compared with the average state The degree of difference between them is denoted as (Based on L2 distance): ; This distance reflects the degree of difference between the frame and the average state of the segment. The greater the distance, the more drastic the change in action in the frame, and the more important the semantic information it contains.

[0046] S503 converts the significance score into normalized weights using a Softmax function with a preset temperature parameter; The saliency scores of all calculated frames Then, the Softmax function with a temperature parameter is used: ; It is a frame-level temperature parameter used to control the polarization of the weights. The value range is set to .like If the value is too large (e.g., greater than 1), the weighted fusion will degenerate into mean pooling, failing to effectively filter out redundant background interference; if If the value is too small (e.g., close to 0), the model will evolve into absolute max pooling, which is prone to abrupt feature changes due to video compression artifacts or single-frame noise. This solution will... Limited to Within this range, when using Softmax for probability normalization, it can exponentially amplify the weights of highly significant frames while retaining the small weights of neighboring frames, achieving the best balance between achieving keyframe polarization focus and maintaining local temporal smoothness.

[0047] S504 generates the visual feature vector of the segment by weighted summation of the visual features of all keyframes according to the normalized weights.

[0048] Ultimately, the fusion features of this fragment It is no longer a simple average, but the sum of the products of each frame's features and their weights: ; This fusion feature That is, it serves as the visual feature vector of the segment and participates in subsequent segment-level fusion.

[0049] Please refer to Figure 6 As shown, in one embodiment of this application, generating a keyframe sequence for each video segment whose semantics meet preset conditions through information density axis equal-order inverse mapping includes: S601 calculates the transient motion saliency of adjacent frames within each video segment whose semantics meet the preset conditions; The currently segmented video clip contains the following frame sequence: First, calculate the feature difference between two adjacent frames, as the first... Frame transient motion saliency : ; For feature extraction operators, to ensure efficiency, it is preferable to convert the frame into a grayscale image and calculate the pixel-level absolute difference sum (SAD). As a smoothing constant (noise floor baseline), if a segment remains absolutely still for a long time, a difference of 0 will cause a gradient-less dead zone in the subsequent accumulation function. A minimum value is introduced. (like This ensures a basic increase in information density and prevents the algorithm from not sampling at all in the static region.

[0050] S602 constructs a cumulative information density function based on the transient motion saliency; Specifically, the saliency of transient motion can be transformed into a probability density distribution within that segment. , indicating the first The proportion of information contained in a frame to the total information in the entire segment: ; Subsequently, by integrating along the time axis, a monotonically increasing cumulative information density function is constructed. : ; The value range is strictly mapped to [0, 1]. When the image is still, The slope of the curve is extremely gentle (only by relying on) (Maintaining growth); when there is violent movement or scene change in the picture. The curve produces a steep step leap.

[0051] S603 divides the range of the cumulative information density function into equal-order intervals equal to one less than the standard target sampling frame number; Specifically, instead of performing equal-interval segmentation on the X-axis (time axis), equal-interval segmentation is performed on the Y-axis (cumulative information density axis). The information interval [0, 1] is strictly divided into... Each step length (due to sampling requirements) (Frames, therefore divided into 7 equal parts), step size .

[0052] S604 determines the target energy level corresponding to each grade interval, and finds the frame with the cumulative information density closest to the target energy level on the time axis as the key frame; For the Target sampling points ( ), calculate its target energy level on the Y-axis. By reverse mapping, the physical frame index on the time axis that makes the cumulative density closest to the water level is found. As a keyframe: ; This mapping allows for the absolutely stable output of lengths that are strictly equal to... Keyframe index sequence .in It must be the first frame of the clip. It must be the last frame of the segment, ensuring global coverage of the temporal receptive field.

[0053] S605 performs hard temporal constraint verification on all keyframes, generating a keyframe sequence that is strictly increasing in time and has no repetitions.

[0054] Specifically, in cases of extremely severe single-frame flicker, mapping may cause two sampling points to fall into the same frame (i.e., To ensure that the 8 input frames are strictly increasing and non-repeating in time series, a hard constraint anti-collision correction mechanism is introduced: if a collision is detected... If so, it will be forcibly postponed to the next position: .

[0055] Please refer to Figure 7 As shown, in one embodiment of this application, the video global features are obtained by weighted fusion using global semantic anchors based on the visual feature vectors of each segment, and the video classification results are generated by cross-modal semantic matching based on the video global features, including: S701 extracts text features from all preset classification categories; Specifically, the text encoder of the pre-trained VideoCLIP-XL model can be used to encode the text descriptions corresponding to each category label in the video classification task. To enhance semantic coverage, this embodiment employs cue word expansion and ensemble techniques, the specific implementation of which is described in Embodiment Nine. Finally, for each category label, an L2-normalized text semantic feature vector is obtained, denoted as the category feature vector, whose dimension is the same as the visual feature vector.

[0056] S702 calculates the mean vector based on the text features of all preset classification categories as a global semantic anchor point; Specifically, there can be C preset classification categories. The text feature vectors of each category (all L2 normalized) are summed, and then divided by the total number of categories C to obtain the arithmetic mean vector, which is the global semantic anchor. This global semantic anchor acts as the benchmark coordinate for task relevance in the cross-modal feature space. By calculating the similarity between a single segment and this anchor, the model can adaptively measure the information salience of each segment in the current classification task without introducing an additional attention network or knowing the true category label of the video.

[0057] S703 performs semantic weighted fusion based on the global semantic anchor points and the visual feature vectors of each segment to generate global video features; Specifically, the visual feature vectors (all L2 normalized) of the K adaptively segmented video segments can be semantically weighted and fused with global semantic anchors. First, the cosine similarity between each segment's features and the global semantic anchor is calculated, resulting in a similarity value for each segment. Then, these similarity values ​​are input into a Softmax function with a preset temporal temperature parameter. This function converts the similarity values ​​of all segments into normalized segment weights, giving higher weights to segments with higher similarity. Next, the visual feature vector of each segment is multiplied by its corresponding segment weight, and all weighted feature vectors are summed to obtain the original representation of the video's global features. Finally, this original representation is L2 normalized to obtain the final global video features.

[0058] S704 calculates the cross-modal similarity between the global features of the video and the text features of each preset classification category; Specifically, the global video features (already L2 normalized) obtained in step S703 can be defined as video feature vectors. For each category, the cosine similarity between the video feature vector and the text feature vector of that category is calculated. Since both have been L2 normalized, the cosine similarity is equivalent to calculating the dot product of the two vectors (i.e., multiplying corresponding positions and then summing). This similarity value reflects the degree of semantic matching between the video and the corresponding category.

[0059] S705 generates video classification results based on the cross-modal similarity.

[0060] Specifically, the calculated similarity scores for all categories can be input into a Softmax function with a classification temperature parameter to obtain the predicted probability for each category. The classification temperature parameter is a positive number ranging from (0, 1], for example, 0.5. This parameter adjusts the smoothness of the probability distribution output by the Softmax function: the lower the temperature, the sharper the probability distribution (the category with the highest probability is more prominent); the higher the temperature, the smoother the probability distribution (the difference in probability between categories is reduced). This embodiment uses a reasonable setting of the classification temperature parameter to ensure that the model maintains its discriminative power even when categories are similar. Finally, the category with the highest probability is selected as the video classification result output.

[0061] Please refer to Figure 8 As shown, in one embodiment of this application, generating global video features by semantically weighting and fusing the visual feature vectors of all segments based on the global semantic anchor points includes: S801 calculates the cosine similarity between the visual feature vector of each segment and the global semantic anchor point; For each segment i (i ranges from 1 to K), its segment visual feature vector has been L2 normalized, and its global semantic anchor point has also been L2 normalized. The cosine similarity between the two is calculated by multiplying corresponding positions of the two vectors and then summing the results. This value is denoted as similarity i.

[0062] S802 converts the cosine similarity into fragment normalization weights using a Softmax function with a preset temperature parameter; The similarity scores of all segments (similarity 1, similarity 2, …, similarity K) are input into a Softmax function with a temporal temperature parameter. The Softmax function works as follows: first, each similarity score is divided by the temporal temperature parameter; then, an exponential function with the natural constant e as its base is calculated; finally, each exponential value is divided by the sum of all exponential values ​​to obtain the normalized weight of each segment. The temporal temperature parameter is a positive number ranging from [0.01, 0.1]. When there are many video segments and the similarity scores are relatively concentrated, if the temporal temperature parameter is large, the weight distribution of each segment tends to be uniform after Softmax, failing to reflect the significant differences in segment-level semantics. If the temporal temperature parameter is between [0.01, 0.1], it amplifies the differences in similarity, giving highly relevant segments an absolute dominant weight, achieving a segment selection effect comparable to supervised attention mechanisms.

[0063] S803 performs a weighted summation of the visual feature vectors of all segments based on the segment normalization weights to generate global video features.

[0064] The original representation of the video's global features is obtained as follows: The visual feature vector of each segment is multiplied by its corresponding normalized weight, and then all weighted vectors are summed. This summation yields an original aggregated vector, which is then L2 normalized: the magnitude of the original aggregated vector is calculated (the square root of the sum of the squares of each component), and each component is divided by this magnitude to obtain the final video global feature vector. This feature integrates information from all video segments and is highly semantically relevant to the classification task, allowing it to be directly used for subsequent cross-modal matching with text features from various categories.

[0065] Please refer to Figure 9 As shown, in one embodiment of this application, extracting text features from all preset classification categories includes: S901 generates basic prompt words for each preset category; For each category label, build a basic prompt template. For example, for the "Education" category, the basic prompt could be designed as: "A video of an educational lecture or online course showing a teacher lecturing, writing on a blackboard, or explaining slides."

[0066] S902 performs semantic expansion on the basic prompt words to generate multiple extended prompt words; To enhance the model's semantic coverage of categories and address the semantic uniformity issue caused by single labels, this embodiment employs cue word expansion and ensemble techniques. Taking "educational video" as an example, it is automatically expanded into a set of phrases. Expansion methods can include synonym replacement, generation based on a large language model, or semantic augmentation using a pre-trained language model (such as WordNet). This embodiment does not limit the specific expansion algorithm, as long as it can generate multiple descriptive phrases semantically related to the original category.

[0067] S903 extracts the text features of the basic prompt words and all extended prompt words respectively; The base prompt and each extended prompt are input into the text encoder of the pre-trained VideoCLIP-XL model to extract their respective text feature vectors. L2 normalization is performed on each vector to obtain a set of normalized feature vectors, where the first vector represents the feature of the base prompt, and the rest represent the features of the extended prompts.

[0068] S904 performs an arithmetic average of all extracted text features to generate the final text features for the preset classification category.

[0069] The system sums all the normalized text feature vectors obtained from S903, then divides them by the total number of vectors (the number of basic prompts 1 plus the number of extended prompts Q) to obtain the original aggregated features for that category. Then, L2 normalization is applied to these original aggregated features: the magnitude is calculated, and each component is divided by this magnitude to obtain the final text semantic features. This integration method significantly broadens the open-domain recognition boundary of the model, enabling accurate classification even when videos present unseen specific teaching formats. When actual business requires adding new video classification categories (e.g., adding a "fire recognition" category to existing security monitoring), there is no need to re-collect video datasets or retrain the model. The system can seamlessly complete high-precision classification simply by directly inputting new text prompts and their extended phrases, giving the system strong open-domain dynamic expansion capabilities and agile deployment value.

[0070] This application also provides a video classification system based on multi-segment sampling and cross-modal matching. The system includes a preprocessing module, a visual feature extraction module, and a classification module. The preprocessing module generates multiple video segments with semantically satisfied preset conditions based on the input video through adaptive multi-segment division and inter-frame motion difference boundary adjustment. The visual feature extraction module samples keyframes from the video segments using equal-order inverse mapping based on the cumulative information density function, and performs saliency-weighted fusion on the sampled keyframes to generate visual feature vectors for each segment. The classification module obtains global video features by weighted fusion of the visual feature vectors of each segment using global semantic anchors, and generates video classification results through cross-modal semantic matching based on the global video features.

[0071] The beneficial technical effects of this application are as follows: By adaptively dividing multiple segments and adjusting the boundary of inter-frame motion differences, long videos can be divided into multiple semantically complete segments, avoiding the truncation of continuous actions caused by mechanical division and ensuring that the content of each time period of the video is effectively captured; by sampling keyframes through equal-order inverse mapping based on the cumulative information density function, the sampled frames adaptively gather towards high information density regions, maximizing the preservation of the core semantics of the segments while significantly reducing the amount of computation; through a two-level fusion strategy of intra-segment saliency-weighted fusion and segment-level global semantic anchor weighted fusion, irrelevant background noise can be effectively suppressed, key semantic information related to the classification task can be highlighted, and the stability and representativeness of global video features can be improved; through cross-modal semantic matching based on global semantic anchors, the global features of the video naturally have textual orientation, significantly improving the semantic consistency between vision and text, while using cue word expansion and ensemble techniques to enhance the model's semantic coverage of categories.

[0072] This application can quickly expand to new classification categories without retraining the model, has strong open-domain dynamic expansion capabilities, and exhibits good robustness to anomalies such as missing frames, video noise, and decoding failures. The classification results can be interpreted through semantic similarity. It can be widely applied to automatic classification tasks of various long videos, complex scene videos, and open-domain videos, significantly reducing computational costs while ensuring classification accuracy, and facilitating engineering deployment and practical application.

[0073] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0074] This application also provides a computer-readable storage medium storing a computer program that performs the above-described methods.

[0075] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0076] like Figure 10 As shown, the electronic device 600 may also include: a communication module 110, an input unit 120, an audio processor 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily need to include these components. Figure 10 All components shown; in addition, the electronic device 600 may also include Figure 10 For components not shown, please refer to existing technologies.

[0077] like Figure 10 As shown, the central processing unit 100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 100 receives inputs and controls the operation of various components of the electronic device 600.

[0078] The memory 140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 100 may execute the program stored in the memory 140 to perform information storage or processing, etc.

[0079] Input unit 120 provides input to central processing unit 100. Input unit 120 may be, for example, a keypad or touch input device. Power supply 170 provides power to electronic device 600. Display 160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0080] The memory 140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 140 can also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142 (application / function 142) for storing application programs and function programs or processes for executing the operation of the electronic device 600 via the central processing unit 100.

[0081] The memory 140 may also include a data storage unit (data 143) for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit (driver 144) of the memory 140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0082] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processing unit 100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0083] Based on different communication technologies, multiple communication modules 110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132, thereby enabling typical telecommunications functions. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 130 is coupled to a central processing unit 100, enabling on-device recording via the microphone 132 and on-device playback of stored audio via the speaker 131.

[0084] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments in this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0089] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video classification method based on multi-segment sampling and cross-modal matching, characterized in that, The method includes: Based on the input video, multiple video segments with semantically consistent preset conditions are generated through adaptive multi-segment segmentation and inter-frame motion difference boundary adjustment. Based on the video segments, keyframes are sampled using an equal-order inverse mapping based on the cumulative information density function, and the sampled keyframes are saliency-weighted and fused to generate visual feature vectors for each segment. Based on the visual feature vectors of each segment, global semantic anchors are used to weighted fuse the video global features. Based on the video global features, cross-modal semantic matching is used to generate video classification results.

2. The video classification method according to claim 1, characterized in that, Based on the input video, multiple video segments with semantically consistent preset conditions are generated through adaptive multi-segment segmentation and inter-frame motion difference boundary adjustment, including: Obtain the total number of valid frames based on the input video; An initial fragment boundary set is generated by dynamically mixing fragments based on the total number of valid frames. The final fragment boundary set is generated by fine-tuning the motion saliency boundary based on the initial fragment boundary set. Multiple video segments with semantically consistent preset conditions are generated based on the final segment boundary set.

3. The video classification method according to claim 2, characterized in that, The final fragment boundary set is generated by fine-tuning the motion saliency boundary based on the initial fragment boundary set, including: Calculate the inter-frame differences of all adjacent frames within the local time window of each initial boundary frame; A dynamic scene change threshold is generated based on all inter-frame differences within the local time window. When the inter-frame difference of the initial boundary frame is greater than or equal to the dynamic scene change threshold, the frame that minimizes the cost function within the local time window is selected as the updated boundary frame. Verify that all updated boundary frames meet the minimum fragment length limit, and generate the final fragment boundary set.

4. The video classification method according to claim 1, characterized in that, Based on the video segments, keyframes are sampled using an equal-order inverse mapping based on a cumulative information density function, and the sampled keyframes are then saliency-weightedly fused to generate visual feature vectors for each segment, including: For each video segment whose semantics meet the preset conditions, the keyframe sequence of the corresponding segment is generated by inverse mapping of the information density axis. Extract the visual features of each keyframe in the keyframe sequence; The visual feature vector of the segment is generated by saliency-weighted fusion based on the visual features.

5. The video classification method according to claim 4, characterized in that, The visual feature vectors of each segment are generated by saliency-weighted fusion of the sampled keyframes, including: The arithmetic mean of all visual features in the keyframe sequence is calculated as the segment baseline feature; The L2 distance between the visual features of each keyframe and the baseline features of the segment is calculated as a saliency score; The significance score is converted into normalized weights using a Softmax function with a preset temperature parameter; The visual feature vector of the segment is generated by weighted summation of the visual features of all keyframes based on the normalized weights.

6. The video classification method according to claim 4, characterized in that, For each video segment that meets the preset semantic conditions, the keyframe sequence of that segment is generated through an inverse mapping of the information density axis, including: Calculate the transient motion saliency of adjacent frames within each video segment whose semantics meet the preset conditions; Construct a cumulative information density function based on the transient motion saliency; The range of the cumulative information density function is divided into equal-order intervals equal to one less than the standard target sampling frame number; Determine the target energy level corresponding to each grade interval, and find the frame with the cumulative information density closest to the target energy level on the time axis as the key frame; Perform hard temporal constraint verification on all keyframes to generate a keyframe sequence that is strictly increasing in time and has no repetitions.

7. The video classification method according to claim 1, characterized in that, Based on the visual feature vectors of each segment, a global video feature is obtained by weighted fusion using global semantic anchors. Based on this global video feature, a video classification result is generated through cross-modal semantic matching, including: Extract text features from all preset classification categories; The mean vector is calculated based on the text features of all preset classification categories and used as a global semantic anchor. The global semantic anchors and the visual feature vectors of each segment are used to perform semantic weighted fusion to generate global video features; Calculate the cross-modal similarity between the global features of the video and the text features of each preset classification category; Video classification results are generated based on the cross-modal similarity.

8. The video classification method according to claim 7, characterized in that, Based on the global semantic anchors, the visual feature vectors of all segments are semantically weighted and fused to generate global video features, including: Calculate the cosine similarity between the visual feature vector of each segment and the global semantic anchor point; The cosine similarity is converted into fragment normalization weights using a Softmax function with a preset temperature parameter; The visual feature vectors of all segments are weighted and summed according to the segment normalization weights to generate global video features.

9. The video classification method according to claim 7, characterized in that, Extracting text features from all preset classification categories includes: Generate basic prompt words for each preset category; The basic prompt words are semantically expanded to generate multiple extended prompt words; Extract the text features of the basic prompt words and all extended prompt words respectively; The final text features for the preset classification category are generated by arithmetically averaging all extracted text features.

10. A video classification system based on multi-segment sampling and cross-modal matching, characterized in that, The system includes: a preprocessing module, a visual feature extraction module, and a classification module; The preprocessing module is used to generate multiple video segments that meet preset conditions based on the input video through adaptive multi-segment segmentation and inter-frame motion difference boundary adjustment. The visual feature extraction module is used to sample keyframes from the video segments using an equal-order inverse mapping based on a cumulative information density function, and to perform saliency-weighted fusion on the sampled keyframes to generate visual feature vectors for each segment. The classification module is used to obtain global video features by weighted fusion of the visual feature vectors of each segment using global semantic anchors, and to generate video classification results by cross-modal semantic matching based on the global video features.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

Images