A video multi-type boundary detection method and device and a storage medium

Abstract

The present application relates to a kind of video multi-type boundary detection method, device and storage medium.The method extracts uniform timing context representation by sharing coding module, to provide uniform feature modeling framework for video multi-type boundary detection.To effectively capture local details and global timing dependence in video data, a local-global context encoder is designed, which integrates local window self-attention mechanism and bidirectional state space modeling mechanism, and can provide uniform and discriminative timing feature representation for various boundary detection tasks.In addition, a two-stage incremental learning strategy is used: when the model continuously adapts to new boundary types, the catastrophic forgetting problem is alleviated through knowledge distillation, replay and adversarial data distribution alignment mechanism.The present application provides a scalable solution for video multi-type boundary detection tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and video understanding technology, and particularly relates to a method, device and storage medium for multi-type boundary detection in video. Background Technology

[0002] With the continuous development of video acquisition equipment, content production methods, and network transmission technologies, video data has become one of the most important forms of information carriers. Especially in scenarios such as security monitoring, film and television production, sports event analysis, and online media platforms, long videos have become a crucial data format. Unlike static media such as images, the semantic information of long videos is usually not uniformly distributed along the timeline, but rather naturally aggregates into several relatively stable semantic segments due to event evolution or scene transitions. Obtaining semantically coherent video segments is the first step in understanding the semantics of the entire long video. If the structural features embodied in the long video are ignored and the entire video is processed uniformly, the model often needs to handle a large amount of information and a long temporal input, thus increasing computational and storage overhead. To improve the efficiency of video processing, long videos usually need to be divided into meaningful time segments. How to identify structurally clear and semantically stable segment units from continuous video streams has become a fundamental problem in video understanding research.

[0003] Video temporal boundary detection is a key step in solving the aforementioned problems. From a semantic perspective, video temporal boundaries mainly encompass two categories: scene boundaries and event boundaries. In existing research, event boundary detection and scene boundary detection are typically conducted independently, with researchers building dedicated models for each task. This results in high model design and maintenance costs, and the low correlation between information acquired also affects the final detection accuracy. To improve detection accuracy and reduce costs, a general feature learning framework and feature representation method are urgently needed. Summary of the Invention

[0004] In view of the above-mentioned problems in the existing technology, the technical problem to be solved by the present invention is: how to improve the accuracy of video boundary detection in long videos, while reducing the model maintenance cost.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for detecting multiple types of boundaries in video includes the following steps:

[0007] Step 1: Obtain a public scene video sample set S1 and an event video sample set S2, wherein S1 includes... A lens unit with real-world scene boundary labels, wherein S2 includes A frame unit with real event boundary labels;

[0008] The lens unit is defined at time position t as The corresponding real-world boundary label is defined as ,by Define the context sequence for each shot unit, centered around [the concept of a central element]. It is a continuous sequence consisting of c lens units before and after it, i.e. Where c is the context radius, the sequence length is 2c+1, and Each camera unit in the video contains real-world scene boundary labels;

[0009] right Extracting a d-dimensional feature vector from each lens unit yields... Corresponding context feature sequence ;

[0010] The context feature sequence of each shot unit and its real-world boundary labels constitute a scene training sample. The expression is Similarly, the context feature sequence of each frame unit and its real event boundary labels constitute an event training sample. The expression is ;

[0011] The contextual feature sequences corresponding to all camera units constitute the scene training sample training set. The context feature sequences corresponding to all frame units constitute the event training sample training set. ;

[0012] Step 2: Construct a local-global context encoder The This includes local context encoders and global context encoders, among which, This represents the trainable parameters in the encoder;

[0013] The local context encoder is used to model the local dependencies of visual temporal features to obtain local context features;

[0014] The global context encoder is used to model the long-range temporal dependencies of the local context features to obtain global context features;

[0015] Step 3: Use right Perform pre-training to obtain a pre-trained context encoder. ,in, express Trainable parameters in;

[0016] Step 4: According to the sampling ratio from Randomly selected A sample, denoted as a replay sample. ; and then the above and All data are merged to form a mixed feature sequence set. , The event-type feature sequence in the middle is denoted as The scene-class feature sequence is denoted as , The expression is as follows:

[0017]

[0018] in, Indicates from dataset Random sampling One sample, for The number of samples, This indicates a round-down operation;

[0019] Step 5: As a teacher context encoder, and will As a student context encoder The initialization parameters, i.e. ;

[0020] Will The data is divided into several equal batches, each batch containing several event-type feature sequences. and several scene-type feature sequences ,and ;

[0021] Will and Enter them separately In this context, the corresponding student context features for event-type events are obtained. and contextual features of students in different scenarios ;

[0022] Then and Input discriminator respectively The predicted values ​​for the event video categories were obtained respectively. Scene video category prediction value ;

[0023] Will Input Teacher Context Encoder Obtain the contextual features of event-type teachers ;

[0024] Construct a discriminant loss function based on a binary classifier. Furthermore, it is used to train the discriminator D, and the distillation loss function is constructed using the L2 norm. Alignment loss function And constructing an event boundary detection loss function based on binary cross-entropy loss. ,use , and Composition of joint loss function Used for training student context encoders and event detection head Parameters;

[0025] Step 6: Let i = 1;

[0026] Step 6-1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The Middle Batch data as input, frozen The model parameters, using Train D to obtain the current trained discriminator D';

[0027] Then freeze the parameters of D', and again set the first... Batch data as input, using train and Updated via backpropagation and The parameters are used to obtain the current trained parameters. and ;

[0028] Step 6-2: Traversal For all batches of data, a preset loss threshold is used, and after each training iteration, the following judgments are made regarding the training iteration number and the loss function value:

[0029] If the current training iteration reaches its maximum value, or and Training stops when all values ​​are below a preset loss threshold, at which point the trained value is obtained. and If the condition is met, proceed to the next step; otherwise, return to step 6 and set i = i + 1.

[0030] Step 7: For each batch enter Obtain contextual features of students in different scenarios Then Input scene boundary detection head Obtain scene boundary prediction results ;

[0031] Based on the scene boundary prediction results Boundary labels of real-world scenes Calculate scene boundary detection loss ,freeze Parameters, and utilize train Updated via backpropagation The parameters are iteratively trained using the training method described in step 6-2 until... If it converges, then the trained result is obtained. ;

[0032] Step 8: Input the video Y to be tested Obtain context features Then Determine the video category of Y from input D'. Then based on Will Input to the corresponding detection head or In the process, the video boundary corresponding to Y is detected.

[0033] Preferably, the local-global context encoder in step 2 The calculation process is as follows:

[0034] The local-global context encoder It includes a local context encoder and a global context encoder;

[0035] context feature sequence The process of building an input local context encoder is as follows:

[0036] Will have The multi-head self-attention mechanism of the size, for each head Each head processes the normalized input feature sequence in parallel. Performing a linear mapping, we obtain:

[0037]

[0038] in, For learnable parameter matrix, These are the query matrix, key matrix, and value matrix of the h-th header, respectively;

[0039] Construct a mask matrix The The Middle OK Column elements are represented as follows:

[0040]

[0041] in, Indicates the window radius;

[0042] Calculate attention head The output, expressed as follows:

[0043]

[0044] in, This is for normalizing the exponential function operation. This represents the feature dimension in the h-th attention head.

[0045] The outputs of all attention heads are concatenated and mapped to the local context representation space through a linear transformation to obtain the final local context features. :

[0046]

[0047] in, To output the projection matrix, Indicates a splicing operation;

[0048] Local context features The global context encoder construction process is as follows:

[0049] Will After performing root mean square normalization and linear mapping operations in sequence, the input of the global context encoder is obtained:

[0050]

[0051]

[0052] in, For root mean square normalization, For linear mapping, This represents the local context features after root mean square normalization. This represents the features after a linear mapping.

[0053] Will Inputting a multi-scale convolutional module and a state-space model (SSM), global dependency modeling is performed along the original sequence direction and the reverse sequence direction, respectively, to obtain... Corresponding positive global features and reverse global features The expression is as follows:

[0054]

[0055]

[0056] in, This represents the state-space model, and MConv represents the multi-scale convolutional module.

[0057] Input Gated features are obtained after applying the SiLU activation function. The expression is as follows:

[0058]

[0059] in, Indicates the SiLU activation function;

[0060] use , and Computational fusion feature intermediate representation The expression is as follows:

[0061]

[0062] in, This is a dot product operation;

[0063] Finally, the final global context features are calculated.

[0064]

[0065] in, Represents global context features.

[0066] Preferably, the pre-trained context encoder is obtained in step 3. The steps are as follows:

[0067] from Choose any one of the context feature sequences ,Will Enter one by one ,get Then Input to dedicated Classification head, get each Predicted labels for corresponding lens units All Constructing a set of predicted labels ;in, Encoding representation;

[0068] Constructing a scene boundary detection loss function ,Will As input, using right Training is performed, and training stops when the maximum number of iterations is reached or the loss function converges, resulting in a pre-trained context encoder. , The expression is as follows:

[0069]

[0070] in, This represents the real-world scene boundary label for the j-th camera unit. This represents the predicted scene boundary label for the j-th camera unit. Indicates label The loss weight coefficient of the corresponding term, This represents the Sigmoid activation function. Indicates the distribution of training samples The expected value of the samples is used to calculate the average loss over all training samples.

[0071] Preferably, in step 5, the binary classification discriminator The expression is as follows:

[0072]

[0073]

[0074]

[0075]

[0076] in, This represents the output of the student encoder. Represents the ReLU activation function. This represents a one-dimensional convolution operation. This represents a one-dimensional adaptive mean pooling layer operation. Discriminator The output discrimination score, MLP Disc Learnable parameters in.

[0077] Preferably, in step 5, constructing and The process is as follows:

[0078] Construct an event boundary detection loss based on binary cross-entropy loss. The expression is as follows:

[0079]

[0080] in, Indicates the first The true boundary label of each frame unit Indicates the first The predicted label value for each frame unit. express Activation function Indicates label The loss weight coefficient of the corresponding item;

[0081] Constructing the distillation loss function using the L2 norm , The expression is as follows:

[0082]

[0083] in, express The number of samples in the sample;

[0084] Construct a discriminant loss based on a binary classifier. , The expression is as follows:

[0085]

[0086] in, Indicates the distribution of training samples The expected value of the samples is used to calculate the average loss over all training samples. Denotes the trainable parameters in D;

[0087] Training the student context encoder At that time, an adversarial data distribution alignment loss was adopted. constraint The distribution of output features; let Used to construct the joint loss function;

[0088] Constructing a joint loss function The expression is as follows:

[0089] = + +

[0090] in, , Each represents the weighting coefficient of the corresponding loss.

[0091] A video multi-type boundary detection device supporting an incremental learning strategy includes a video data acquisition unit, a feature extraction unit, an encoding unit, and a boundary determination unit;

[0092] The video acquisition unit is used to acquire input video data; wherein the input video data includes at least one of scene-type videos and event-type videos.

[0093] The feature extraction unit is used to preprocess and extract features from the input video data to obtain a video feature sequence.

[0094] The encoding unit is used to call a pre-trained local-global context encoder to encode the temporal feature sequence of the video to obtain a context feature representation; the pre-trained local-global context encoder obtains the joint detection capability of multiple types of boundaries in the video through sequential learning of scene boundary detection tasks and event boundary detection tasks;

[0095] The boundary determination unit is used to determine whether the target is a boundary based on the boundary confidence and boundary type information of the output time sequence position according to the context feature representation, and to determine whether the target is a boundary according to a preset threshold or post-processing rules; wherein, the boundary type includes scene boundary or event boundary.

[0096] A computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed, implements the method steps as described in any one of claims 1-5.

[0097] Compared with the prior art, the present invention has at least the following advantages:

[0098] At the feature perception level, this invention designs a local-global context encoder to provide a unified feature learning framework. At the feature representation level, in order to enable the model to detect two types of boundaries in the same feature representation space, this invention explores an incremental learning method that combines knowledge distillation and replay mechanisms with an adversarial data distribution alignment mechanism.

[0099] 1. This invention can simultaneously perceive local and global features in each video sequence of a long video, improving the model's ability to represent features of different types of videos, especially long videos, and can effectively extract features for event boundary and scene boundary detection.

[0100] 2. This invention eliminates the need for complete retraining of the model when performing two types of boundary detection tasks. It can learn new boundary detection tasks while maintaining a certain detection capability for the learned boundary detection tasks, thus achieving balance and stability between the two types of boundary detection tasks and reducing model maintenance costs. Attached Figure Description

[0101] Figure 1 This is a schematic diagram of the structure of the present invention.

[0102] Figure 2 This is a model diagram of the local temporal encoder in the local-global context encoder of this invention.

[0103] Figure 3This is a model diagram of the global module encoder in the local-global context encoder of this invention.

[0104] Figure 4 This is a schematic diagram of the structure of a video multi-type boundary detection device according to the present invention.

[0105] Figure 5 This is a schematic diagram of the structure of the storage medium of the present invention. Detailed Implementation

[0106] The present invention will now be described in further detail.

[0107] The core idea of ​​this invention is a multi-type boundary detection method for video. To effectively capture short-term local dynamics and long-range contextual dependencies in video data, a local-global context encoder is constructed. Furthermore, this invention proposes a two-stage incremental learning strategy. The first stage trains the boundary detection task, and the second stage utilizes replay mechanisms, knowledge distillation mechanisms, and adversarial data distribution alignment mechanisms to enable multi-type boundary detection tasks to be performed within the same model.

[0108] The current method selects to first train the scene boundary detection task using scene-type video sample sets, and then train the event boundary detection task using event-type video sample sets. However, in actual implementation, this order is not a necessary condition. Training the event boundary detection task first and then the scene boundary detection task is also valid.

[0109] See Figures 1-5 A method for detecting multiple types of boundaries in video, comprising the following steps:

[0110] Step 1: Obtain a public scene video sample set S1 and an event video sample set S2, wherein S1 includes... A lens unit with real-world scene boundary labels, wherein S2 includes S1 is a frame unit with real event boundary labels; S2 is used for scene boundary detection task and S1 is used for event boundary detection task; the shot unit in S1 is generally long video data of story context, and the frame unit in S2 is generally video data containing continuous action or behavior change process.

[0111] The lens unit is defined at time position t as The corresponding real-world boundary label is defined as ,by Define the context sequence for each shot unit, centered around [the concept of a central element]. It is a continuous sequence consisting of c lens units before and after it, i.e. Where c is the context radius, the sequence length is 2c+1, and Each camera unit in the video contains real-world scene boundary labels;

[0112] right Extracting a d-dimensional feature vector from each lens unit yields... Corresponding context feature sequence ;

[0113] The context feature sequence of each shot unit and its real-world boundary labels constitute a scene training sample. The expression is Similarly, the context feature sequence of each frame unit and its real event boundary labels constitute an event training sample. The expression is ;

[0114] For scene-based video samples, the input video is divided into a sequence of video shot units according to the shot segmentation method, and each video shot unit is extracted. Dimensional features are used to form a scene feature sequence; for event-type video samples, the input video is divided into a sequence of video frame units according to the frame-level segmentation method, and features are extracted from each video frame unit. 3D temporal visual features are used to form an event feature sequence;

[0115] The contextual feature sequences corresponding to all camera units constitute the scene training sample training set. The context feature sequences corresponding to all frame units constitute the event training sample training set. ;

[0116] Step 2: Construct a local-global context encoder The This includes local context encoders and global context encoders, among which, This represents the trainable parameters in the encoder;

[0117] The local context encoder is used to model local dependencies of visual temporal features to obtain local context features. Specifically, it includes: mapping the input visual temporal features to query matrices, key matrices, and value matrices corresponding to multiple attention heads through a linear mapping layer; inputting the query matrices, key matrices, and value matrices into the attention layer to obtain the local context output of each attention head; concatenating the outputs of each attention head and mapping them through a linear projection layer to output the local context features.

[0118] The global context encoder is used to model the long-range temporal dependencies of the local context features to obtain global context features. Specifically, it includes: inputting the local context features sequentially into a root mean square normalization layer and a linear mapping layer to obtain linear mapping features; inputting the linear mapping features into a multi-scale convolutional module and a state space model respectively, and performing bidirectional modeling along the original sequence direction and the reverse sequence direction to obtain forward global features and reverse global features; simultaneously, inputting the linear mapping features into a SiLU activation layer to generate gated features; inputting the forward global features, reverse global features, and gated features into a gated fusion layer for fusion, and performing residual connection between the fusion result and the features processed by the root mean square normalization layer to output global context features covering the entire long-range temporal video sequence.

[0119] Local-Global Context Encoder in Step 2 The calculation process is as follows:

[0120] The local-global context encoder It includes a local context encoder and a global context encoder; the local context encoder mainly captures short-term context dependencies, while the global context encoder captures long-term context dependencies;

[0121] context feature sequence The process of building an input local context encoder is as follows:

[0122] Will have The multi-head self-attention mechanism of the size, for each head Each head processes the normalized input feature sequence in parallel. Performing a linear mapping, we obtain:

[0123]

[0124] in, For learnable parameter matrix, These are the query matrix, key matrix, and value matrix of the h-th header, respectively;

[0125] Construct a mask matrix The The Middle OK Column elements are represented as follows:

[0126]

[0127] in, Indicates the window radius; to focus on local dependencies, attention is restricted to a fixed-size window around each column position, only when the position... and satisfy Attention weights are calculated in real time;

[0128] Calculate attention head The output, expressed as follows:

[0129]

[0130] in, This is for normalizing the exponential function operation. This represents the feature dimension in the h-th attention head. Softmax is performed row-wise, allowing positions with a mask value of 0 to effectively participate in the attention calculation process.

[0131] The outputs of all attention heads are concatenated and mapped to the local context representation space through a linear transformation to obtain the final local context features. :

[0132]

[0133] in, To output the projection matrix, Indicates a splicing operation;

[0134] Local context features The global context encoder construction process is as follows:

[0135] Will After performing root mean square normalization and linear mapping operations in sequence, the input of the global context encoder is obtained:

[0136]

[0137]

[0138] in, For root mean square normalization, For linear mapping, This represents the local context features after root mean square normalization. This represents the features after a linear mapping.

[0139] Will Inputting a multi-scale convolutional module and a state-space model (SSM), global dependency modeling is performed along the original sequence direction and the reverse sequence direction, respectively, to obtain... Corresponding positive global features and reverse global features The expression is as follows:

[0140]

[0141]

[0142] in, The state-space model is represented by MConv, which represents a multi-scale convolutional module. To simultaneously perceive the context of the current segment, a temporal reversal operation is introduced. Flip along the time dimension, that is A bidirectional state-space modeling branch is designed; the forward branch directly models the original sequential sequence to obtain the forward global features. The reverse branch first reverses the sequence, then... Then reverse the order back to the original order to obtain the reverse global features. This allows us to utilize future information at every moment;

[0143] Input Gated features are obtained after applying the SiLU activation function. The expression is as follows:

[0144]

[0145] in, Indicates the SiLU activation function;

[0146] use , and Computational fusion feature intermediate representation The expression is as follows:

[0147]

[0148] in, This is a dot product operation;

[0149] Finally, the final global context features are calculated.

[0150]

[0151] in, Represents global context features.

[0152] The multi-scale convolution module extracts features in parallel using multiple convolution kernels of different sizes, capturing multi-scale information from the input. After batch normalization and GELU activation, the features are concatenated to form the output. Let the set of convolution kernels be... The calculation process can be expressed as follows: ,in, Indicates that the convolution kernel is Convolution operation, For splicing operations, this bidirectional structure enables the model to integrate global context in both forward and reverse temporal order, significantly enhancing its ability to characterize semantic mutations and long-range dependencies near action boundaries.

[0153] Gating weights are generated for the forward and reverse global features using the SiLU activation function to achieve differentiated modulation of bidirectional information. Specifically, the SiLU activation function is used to generate gating weights for the forward and reverse global features. Gated features are obtained after applying the SiLU activation function. Subsequently, the bidirectional global features are modulated element-wise using gated features and then summed to obtain the fused intermediate representation. Finally, the fusion result is passed through a linear mapping layer to adjust the channels and then compared with the normalized result. Perform residual connections to obtain the final global context features. .

[0154] Step 3: Use right Perform pre-training to obtain a pre-trained context encoder. ,in, express Trainable parameters in;

[0155] In step 3, the pre-trained context encoder is obtained. The steps are as follows:

[0156] from Choose any one of the context feature sequences ,Will Enter one by one ,get Then Input to dedicated Classification head, get each Predicted labels for corresponding lens units All Constructing a set of predicted labels ;in, Encoding representation;

[0157] Constructing a scene boundary detection loss function ,Will As input, using right Training is performed, and training stops when the maximum number of iterations is reached or the loss function converges, resulting in a pre-trained context encoder. , The expression is as follows:

[0158]

[0159] in, This represents the real-world scene boundary label for the j-th camera unit. This represents the predicted scene boundary label for the j-th camera unit. Indicates label The loss weight coefficient of the corresponding term, This represents the Sigmoid activation function. Indicates the distribution of training samples The expected value of the samples is used to calculate the average loss over all training samples.

[0160] Step 4: According to the sampling ratio from Randomly selected A sample, denoted as a replay sample. ; and then the above and All data are merged to form a mixed feature sequence set. , The event-type feature sequence in the middle is denoted as The scene-class feature sequence is denoted as , The expression is as follows:

[0161]

[0162] in, Indicates from dataset Random sampling One sample, for The number of samples, This indicates a round-down operation;

[0163] Step 5: As a teacher context encoder, and will As a student context encoder The initialization parameters, i.e. ;

[0164] Will The data is divided into several equal batches, each batch containing several event-type feature sequences. and several scene-type feature sequences ,and ;

[0165] Will and Enter them separately In this context, the corresponding student context features for event-type events are obtained. and contextual features of students in different scenarios ;

[0166] Then and Input discriminator respectively The predicted values ​​for the event video categories were obtained respectively. Scene video category prediction value In actual calculations, event categories are labeled as 1, and scene categories are labeled as 0, based on video category tags.

[0167] Will Input Teacher Context Encoder Obtain the contextual features of event-type teachers ;

[0168] Construct a discriminant loss function based on a binary classifier. Furthermore, it is used to train the discriminator D, and the distillation loss function is constructed using the L2 norm. Alignment loss function And constructing an event boundary detection loss function based on binary cross-entropy loss. ,use , and Composition of joint loss function Used for training student context encoders and event detection head Parameters;

[0169] In step 5, the binary classification discriminator The expression is as follows:

[0170]

[0171]

[0172]

[0173]

[0174] in, This represents the output of the student encoder. Represents the ReLU activation function. This represents a one-dimensional convolution operation. This represents a one-dimensional adaptive mean pooling layer operation. Discriminator The output discrimination score, its value is determined by... The function can be interpreted as the probability that the input features belong to the event boundary task. MLP Disc Learnable parameters in.

[0175] In step 5, construct and The process is as follows:

[0176] Construct an event boundary detection loss based on binary cross-entropy loss. The expression is as follows:

[0177]

[0178] in, Indicates the first The true boundary label of each frame unit Indicates the first The predicted label value for each frame unit. express Activation function Indicates label The loss weight coefficient of the corresponding term; The output of arbitrary features from the event boundary detection task Input to dedicated The classification head obtains the predicted labels for the sequence. Through the loss function train Learning new task feature representations;

[0179] Constructing the distillation loss function using the L2 norm , The expression is as follows:

[0180]

[0181] in, express The number of samples in the sample; to guide consistency in the feature space at the current stage; among which As a frozen teacher context encoder, any feature sequence As input, the output is the teacher's code representation. Train a student context encoder and its parameters Receive input sequence any feature sequence Generate student code representation Loss through distillation train The knowledge accumulated in the previous stage is transferred to the student model, thereby keeping the feature spaces of the two tasks consistent.

[0182] Construct a discriminant loss based on a binary classifier. , The expression is as follows:

[0183]

[0184] in, Indicates the distribution of training samples The expected value of the samples is used to calculate the average loss over all training samples. Denotes the trainable parameters in D;

[0185] Training the student context encoder At that time, an adversarial data distribution alignment loss was adopted. constraint The distribution of output features; let Used to construct the joint loss function; With the discriminant loss Both are built on the same binary classification loss, but their optimization directions are opposite: that is, when training the discriminator D, the goal is to minimize... The goal is to train the discriminator D to distinguish between the two types of task features as much as possible, thereby improving task separability; training... At that time, gradient reversal makes along Update in the direction of increase, that is, let This forces the discriminator D to have difficulty distinguishing between event-type features and scene-type features, thereby making the distribution of features corresponding to the two types of tasks in the feature space gradually become consistent, achieving alignment of the feature spaces of the two tasks;

[0186] Constructing a joint loss function The expression is as follows:

[0187] = + +

[0188] in, , Each represents the weighting coefficient of the corresponding loss.

[0189] Step 6: Let i = 1;

[0190] Step 6-1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require The Middle Batch data as input, frozen The model parameters, using Training D yields the currently trained discriminator D', which allows the discriminator to... Distinguish between event-based features and scenario-based features as accurately as possible;

[0191] Then freeze the parameters of D', and again set the first... Batch data as input, using train and Updated via backpropagation and The parameters are used to obtain the current trained parameters. and ;

[0192] Step 6-2: Traversal For all batches of data, a preset loss threshold is used, and after each training iteration, the following judgments are made regarding the training iteration number and the loss function value:

[0193] If the current training iteration reaches its maximum value, or and Training stops when all values ​​are below a preset loss threshold, at which point the trained value is obtained. and If the training process continues, proceed to the next step; otherwise, return to step 6 and set i = i + 1. During training, the number of data batches is much smaller than the set number of iterations. Therefore, if all batches of data used for training are used up and the goal of stopping training is not achieved, the batches of data will continue to be used repeatedly in subsequent training iterations until the goal of stopping training is achieved.

[0194] Step 7: For each batch enter Obtain contextual features of students in different scenarios Then Input scene boundary detection head Obtain scene boundary prediction results ;

[0195] Based on the scene boundary prediction results Boundary labels of real-world scenes Calculate scene boundary detection loss ,freeze Parameters, and utilize train Updated via backpropagation The parameters are iteratively trained using the training method described in steps 6-6-2 until... If it converges, then the trained result is obtained. ;

[0196] Step 8: Input the video Y to be tested Obtain context features Then Determine the video category of Y from input D'. Then based on Will Input to the corresponding detection head or In the process, the video boundary corresponding to Y is detected.

[0197] Feature sequences of the video to be detected Input Student Context Encoder To obtain the context features to be detected ,Will Input discriminator To obtain the predicted probability of video category ;Will Compared with the preset discrimination threshold (like Comparison: When When the video to be detected is determined to be an event, it will be... Input event boundary detection head The event boundary prediction results are obtained. ;when When the video to be detected is determined to belong to the scene category, it will be... Input scene boundary detection head The scene boundary prediction results are obtained. The event boundary prediction results mentioned above With scene boundary prediction results All are frame-by-frame boundary probability sequences along the time dimension of the video to be detected, i.e., for a length of... Feature sequences, output by the detection head Each component Indicates the first Frame (or the first) The probability with a time position as the boundary.

[0198] A video multi-type boundary detection device supporting an incremental learning strategy includes a video data acquisition unit, a feature extraction unit, an encoding unit, and a boundary determination unit;

[0199] The video acquisition unit is used to acquire input video data; wherein the input video data includes at least one of scene-type videos and event-type videos.

[0200] The feature extraction unit is used to preprocess and extract features from the input video data to obtain a video feature sequence.

[0201] The encoding unit is used to call a pre-trained local-global context encoder to encode the temporal feature sequence of the video to obtain a context feature representation; the pre-trained local-global context encoder obtains the joint detection capability of multiple types of boundaries in the video through sequential learning of scene boundary detection tasks and event boundary detection tasks;

[0202] The boundary determination unit is used to determine whether the target is a boundary based on the boundary confidence and boundary type information of the output time sequence position according to the context feature representation, and to determine whether the target is a boundary according to a preset threshold or post-processing rules; wherein, the boundary type includes scene boundary or event boundary.

[0203] A computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed, implements the method steps as described in any one of claims 1-5.

[0204] Examples and Experimental Results

[0205] Example 1: The invention uses a context radius c=10 and a total of 21 shots (for the MovieNet dataset) or frames (for the Kinetics dataset) as input. In scene-level feature extraction, this invention simultaneously captures foreground and background information. Foreground features are extracted using a ResNet-50 pre-trained on ImageNet, while background features are obtained using a ResNet-50 pre-trained on Places365. This design is based on the observation that scene boundaries are often accompanied by changes in environment or layout, and background information can also affect the semantic interpretation of scene transitions. Conversely, event boundary detection mainly reflects changes in foreground motion or interaction, while the background is relatively stable; therefore, this invention only uses ImageNet pre-trained features for frame-level processing. In each learning stage, the local window radius r is set to 4, and the replay ratio... Set to 0.30, and Set to 0.5 and 0.1 respectively, positive sample parameters It is 5.

[0206] Table 1. Comparison of detection performance (%) of this invention with other methods in scene → event sequence.

[0207]

[0208] As shown in Table 1, under the scene → event sequence setting, in terms of scene-level data evaluation metrics, this invention outperforms EWC and BECAME in mAP (53.4%) and F1 (52.6%), respectively. Specifically, in event-level evaluation, the AvgF1 metric (86.7%) of this invention is similar to that of BECAME and PGM. Compared to the EWC method, which relies solely on incremental learning regularization, this invention improves task adaptability while maintaining event detection performance.

[0209] Table 2. Comparison of detection performance (%) of this invention with other methods in the event → scene sequence.

[0210]

[0211] As shown in Table 2, in the event-to-scene sequence setting, from the perspective of event-level evaluation metrics, the average F1 score (86.5%) of this invention is comparable to BECAME and PGM, and it also maintains competitiveness at F1@0.10 (82.1%). It is worth noting that this invention demonstrates unique advantages in scene-level metrics: mAP reaches 57.4%, and mIoU reaches 50.2%, both exceeding all other methods, indicating a more balanced understanding of the two types of tasks. This symmetry further verifies that the proposed method can achieve consistency between the two feature spaces.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for detecting multiple types of boundaries in video, characterized in that: Includes the following steps: Step 1: Obtain a public scene video sample set S1 and an event video sample set S2, wherein S1 includes... A lens unit with real-world scene boundary labels, wherein S2 includes A frame unit with real event boundary labels; The lens unit is defined at time position t as The corresponding real-world boundary label is defined as ,by Define the context sequence for each shot unit, centered around [the concept of a central element]. It is a continuous sequence consisting of c lens units before and after it, i.e. Where c is the context radius, the sequence length is 2c+1, and Each camera unit in the video contains real-world scene boundary labels; right Extracting a d-dimensional feature vector from each lens unit yields... Corresponding context feature sequence ; The context feature sequence of each shot unit and its real-world boundary labels constitute a scene training sample. The expression is ; Similarly, the context feature sequence of each frame unit and its real event boundary labels constitute an event training sample. The expression is ; The contextual feature sequences corresponding to all camera units constitute the scene training sample training set. The context feature sequences corresponding to all frame units constitute the event training sample training set. ; Step 2: Construct a local-global context encoder The This includes local context encoders and global context encoders, among which, This represents the trainable parameters in the encoder; The local context encoder is used to model the local dependencies of visual temporal features to obtain local context features; The global context encoder is used to model the long-range temporal dependencies of the local context features to obtain global context features; Step 3: Use right Perform pre-training to obtain a pre-trained context encoder. ,in, express Trainable parameters in; Step 4: According to the sampling ratio from Randomly selected from A sample, denoted as a replay sample. Then the above and All data are merged to form a mixed feature sequence set. , The event-type feature sequence in the middle is denoted as The scene-class feature sequence is denoted as , The expression is as follows: in, Indicates from dataset Random sampling One sample, for The number of samples, This indicates a round-down operation; Step 5: As a teacher context encoder, and will As a student context encoder The initialization parameters, i.e. ; Will The data is divided into several equal batches, each batch containing several event-type feature sequences. and several scene-type feature sequences ,and ; Will and Enter them separately In this context, the corresponding student context features for event-type students are obtained. and contextual features of students in different scenarios ; Then and Input discriminator respectively The predicted values ​​for the event video categories were obtained respectively. Scene video category prediction value ; Will Input Teacher Context Encoder Obtain the contextual features of event-type teachers ; Construct a discriminant loss function based on a binary classifier. Furthermore, it is used to train the discriminator D, and the distillation loss function is constructed using the L2 norm. Alignment loss function And constructing an event boundary detection loss function based on binary cross-entropy loss. ,use , and Composition of joint loss function Used for training student context encoders and event detection head Parameters; Step 6: Let i = 1; Step 6-1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require The Middle Batch data as input, frozen The model parameters, using Train D to obtain the current trained discriminator D'; Then freeze the parameters of D', and again set the first... Batch data as input, using train and Updated via backpropagation and The parameters are used to obtain the current trained parameters. and ; Step 6-2: Traversal For all batches of data, a preset loss threshold is used, and after each training iteration, the following judgments are made regarding the training iteration number and the loss function value: If the current training iteration reaches its maximum value, or and Training stops when all values ​​are below a preset loss threshold, at which point the trained value is obtained. and If the condition is met, proceed to the next step; otherwise, return to step 6 and set i = i + 1. Step 7: For each batch enter Obtain contextual features of students in different scenarios Then Input scene boundary detection head Obtain scene boundary prediction results ; Based on the scene boundary prediction results Boundary labels of real-world scenes Calculate scene boundary detection loss ,freeze Parameters, and utilize train Updated via backpropagation The parameters are iteratively trained using the training method described in step 6-2 until... If it converges, then the trained result is obtained. ; Step 8: Input the video Y to be tested Obtain context features Then Determine the video category of Y from input D'. Then based on Will Input to the corresponding detection head or In the process, the video boundary corresponding to Y is detected.

2. The video multi-type boundary detection method as described in claim 1, characterized in that: Local-Global Context Encoder in Step 2 The calculation process is as follows: The local-global context encoder It includes a local context encoder and a global context encoder; context feature sequence The process of building an input local context encoder is as follows: Will have The multi-head self-attention mechanism of the size, for each head Each head processes the normalized input feature sequence in parallel. Performing a linear mapping, we obtain: in, For learnable parameter matrix, These are the query matrix, key matrix, and value matrix of the h-th header, respectively; Construct a mask matrix The The Middle OK Column elements are represented as follows: in, Indicates the window radius; Calculate attention head The output, expressed as follows: in, This is for normalizing the exponential function operation. This represents the feature dimension in the h-th attention head. The outputs of all attention heads are concatenated and mapped to the local context representation space through a linear transformation to obtain the final local context features. : in, To output the projection matrix, Indicates a splicing operation; Local context features The global context encoder construction process is as follows: Will After performing root mean square normalization and linear mapping operations in sequence, the input of the global context encoder is obtained: in, For root mean square normalization, For linear mapping, This represents the local context features after root mean square normalization. This represents the features after a linear mapping. Will Inputting a multi-scale convolutional module and a state-space model (SSM), global dependency modeling is performed along the original sequence direction and the reverse sequence direction, respectively, to obtain... Corresponding positive global features and reverse global features The expression is as follows: in, This represents the state-space model, and MConv represents the multi-scale convolutional module. Input Gated features are obtained after applying the SiLU activation function. The expression is as follows: in, Indicates the SiLU activation function; use , and Computational fusion feature intermediate representation The expression is as follows: in, This is a dot product operation; Finally, the final global context features are calculated. in, Represents global context features.

3. The video multi-type boundary detection method as described in claim 2, characterized in that: In step 3, the pre-trained context encoder is obtained. The steps are as follows: from Choose any one of the context feature sequences ,Will Enter one by one ,get Then Input to dedicated Classification head, get each Predicted labels for corresponding lens units All Constructing a set of predicted labels ;in, This represents the encoded representation; Constructing a scene boundary detection loss function ,Will As input, using right Training is performed, and training stops when the maximum number of iterations is reached or the loss function converges, resulting in a pre-trained context encoder. , The expression is as follows: in, This represents the real-world scene boundary label for the j-th camera unit. This represents the predicted scene boundary label for the j-th camera unit. Indicates label The loss weight coefficient of the corresponding term, This represents the Sigmoid activation function. Indicates the distribution of training samples The expected value of the samples is used to calculate the average loss over all training samples.

4. The video multi-type boundary detection method as described in claim 3, characterized in that: In step 5, the binary classification discriminator The expression is as follows: in, This represents the output of the student encoder. Represents the ReLU activation function. This represents a one-dimensional convolution operation. This represents a one-dimensional adaptive mean pooling layer operation. Discriminator The output discrimination score, MLP Disc Learnable parameters in.

5. The video multi-type boundary detection method as described in claim 4, characterized in that: In step 5, construct and The process is as follows: Construct an event boundary detection loss based on binary cross-entropy loss. The expression is as follows: in, Indicates the first The true boundary label of each frame unit Indicates the first The predicted label value for each frame unit. express Activation function Indicates label The loss weight coefficient for the corresponding item; Constructing the distillation loss function using the L2 norm , The expression is as follows: in, express The number of samples in the sample; Construct a discriminant loss based on a binary classifier. , The expression is as follows: in, Indicates the distribution of training samples The expected value of the samples is used to calculate the average loss over all training samples. Denotes the trainable parameters in D; Training the student context encoder At that time, an adversarial data distribution alignment loss was adopted. constraint The distribution of output features; let Used to construct the joint loss function; Constructing a joint loss function The expression is as follows: = + + in, , Each represents the weighting coefficient of the corresponding loss.

6. A video multi-type boundary detection device supporting an incremental learning strategy, comprising a video data acquisition unit, a feature extraction unit, an encoding unit, and a boundary determination unit; The video acquisition unit is used to acquire input video data; wherein... The input video data includes at least one of scene-based videos and event-based videos; The feature extraction unit is used to preprocess and extract features from the input video data to obtain a video feature sequence. The encoding unit is used to call a pre-trained local-global context encoder to encode the temporal feature sequence of the video to obtain a context feature representation; the pre-trained local-global context encoder obtains the joint detection capability of multiple types of boundaries in the video through sequential learning of scene boundary detection tasks and event boundary detection tasks; The boundary determination unit is used to determine whether the target is a boundary based on the boundary confidence and boundary type information of the output time sequence position according to the context feature representation, and to determine whether the target is a boundary according to a preset threshold or post-processing rules; wherein, the boundary type includes scene boundary or event boundary.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed, implements the method steps as described in any one of claims 1-5.

Images