Video-level oriented behavior feature recognition method and system

Abstract

The application relates to a video-level behavior feature recognition method and system, which comprises the following steps: S10, twice sampling of an input original video to obtain a frame sequence with a specified length; S20, dividing the frame sequence into a plurality of local instances, extracting initial visual features of each local instance, and performing bidirectional time sequence modeling on an initial visual feature sequence composed of all the initial visual features to obtain a context instance feature sequence; S30, combining multi-head self-attention and dynamic multi-instance normalization to perform context enhancement on the context instance feature sequence to obtain an instance feature sequence after feature enhancement; and S40, constructing a key prototype and performing context aggregation based on the enhanced instance feature sequence to output a behavior feature recognition result. The application can improve the accuracy of the behavior feature recognition result.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method and system for video-level behavioral feature recognition. Background Technology

[0002] Depression is a common mental illness with a significant disease burden, typically accompanied by persistent negative mood, loss of interest, and abnormal cognitive and physical states. Existing research has shown that depression-related states can be reflected in an individual's facial behavior, and relevant cues can be obtained through non-invasive video equipment. Therefore, automated depression assessment based on facial videos has gradually become an important research direction in intelligent mental health analysis. Facial video depression assessment usually requires identifying effective cues related to the depressive state from a long temporal sequence. Because long videos often have characteristics such as long sequence length, high information redundancy, and high modeling complexity, existing methods typically divide the video into single frames or fixed-length short segments and perform local modeling based on these segments. According to different video-level information aggregation methods, related methods can be roughly divided into two categories: one is decision-level aggregation methods, which first predict single frames or local segments separately, and then obtain video-level results through averaging, weighting, or other fusion strategies; the other is feature-level aggregation methods, which first extract frame-level or segment-level local representations, and then form a video-level representation using temporal modeling and feature fusion modules for assessing the severity of depression. While existing methods have improved the performance of video-level depression assessment to some extent, their aggregation process generally implicitly assumes a strong correspondence between local units and overall labels: different frames or segments can reflect the overall depressive state of the video to varying degrees. For long videos, this assumption is insufficient. In reality, depression-related behavioral cues are often more pronounced only in a few local segments, and different segments show significant differences in diagnostic relevance and information contribution. If a uniform video-level label is applied to all local units, the model is easily affected by weakly correlated or noisy segments, making it difficult to accurately identify truly discriminative key content. Summary of the Invention

[0003] The purpose of this invention is to provide a video-level behavioral feature recognition method and system to improve the accuracy of behavioral feature recognition results such as depression.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A video-level behavior feature recognition method includes the following steps:

[0006] S10, sample the input raw video twice to obtain a frame sequence of a specified length;

[0007] S20, the frame sequence is divided into several local instances, the initial visual features of each local instance are extracted, and bidirectional temporal modeling is performed on the initial visual feature sequence composed of all the initial visual features to obtain the context instance feature sequence.

[0008] S30, combine multi-head self-attention and dynamic multi-instance normalization to perform context enhancement on the context instance feature sequence, and obtain the enhanced instance feature sequence.

[0009] S40: Construct key prototypes and perform context aggregation based on the enhanced instance feature sequences, and output behavioral feature recognition results.

[0010] In step S10, the original video is first downsampled in time, with one frame sampled every few frames. Then, a starting position is randomly selected from the downsampled video sequence, and sampling continues according to the specified sampling interval to obtain a frame sequence of a specified length.

[0011] In S20, the initial visual features of each local instance are extracted using a three-dimensional convolutional network R3 ResNet-18 with shared parameters; and / or, a two-layer bidirectional Bi-LSTM is used to perform bidirectional temporal modeling on the initial visual feature sequence composed of all initial visual features.

[0012] In the above scheme, the shared encoding method can enable different local instances to be mapped to a unified feature space, thereby forming an initial visual feature sequence with consistent structure; using the three-dimensional convolutional network R3D-18 as an instance feature extractor can accurately extract spatial appearance information and local temporal dynamic information.

[0013] The following processing operations are included in S30:

[0014] First, a linear mapping is performed on the context instance feature sequence to obtain query, key, and value representations;

[0015] Then, multi-head self-attention is used to model the dependencies between different local instances to obtain attention-enhanced representations;

[0016] Finally, the attention enhancement representation is dynamically normalized.

[0017] In the above scheme, the dependency relationship between different local instances is modeled and dynamically normalized by multi-head self-attention. The instance representation achieves stronger discriminativeness and stability while maintaining temporal correlation information.

[0018] S40 includes the following processing operations:

[0019] Generate a continuous weight distribution using instance importance scores. , ws and b s For learnable parameters, To adjust the parameters, N is the number of instance features;

[0020] Weighted aggregation of all instance features yields the key prototype. h i Features of the i-th instance;

[0021] L2 normalization is performed on both key prototype and instance features, and cosine similarity is used to calculate the relevance score between each instance feature and the key prototype. Then, the relevance scores of all instance features are normalized to obtain the context aggregation weights, and the aggregated context features c are constructed. , ;

[0022] The key prototype and context features are mapped separately to obtain their respective prediction results. Then, the two prediction results are fused with equal weights to obtain the final behavior feature recognition result. , , , and This represents two independent regression mapping functions.

[0023] A video-level behavior feature recognition system includes a preprocessing unit and a video-level weakly supervised multi-instance learning model, DepMIL-Net. DepMIL-Net includes an instance partitioning module, a feature extraction module, a feature enhancement module, and a dual-branch aggregation module.

[0024] The preprocessing unit is used to sample the input raw video twice to obtain a frame sequence of a specified length;

[0025] The instance partitioning module is used to divide the frame sequence into several local instances;

[0026] The feature extraction module is used to extract the initial visual features of each local instance and perform bidirectional temporal modeling on the initial visual feature sequence composed of all initial visual features to obtain the context instance feature sequence;

[0027] The feature enhancement module is used to combine multi-head self-attention and dynamic multi-instance normalization to enhance the context instance feature sequence, resulting in an enhanced instance feature sequence.

[0028] The dual-branch aggregation module is used to construct key prototypes and perform context aggregation based on the enhanced instance feature sequence, and output behavioral feature recognition results.

[0029] A computer program product includes computer-readable instructions that, when executed by a processor, implement the steps of the video-level behavior feature recognition method of the present invention.

[0030] A computer-readable storage medium comprising computer-readable instructions that, when executed by a processor, implement the steps of the video-level behavior feature recognition method of the present invention.

[0031] An electronic device is characterized by comprising: a memory for storing program instructions; and a processor connected to the memory for executing the program instructions in the memory to implement the steps in the video-level behavior feature recognition method of the present invention.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] We propose a multi-instance learning model, DepMIL-Net, for video-level weakly supervised depression recognition. This model effectively models scattered behavioral feature cues in long videos under conditions where instance-level annotations are lacking, thereby achieving video-level behavioral feature recognition and improving the accuracy of the recognition results.

[0034] A key prototype construction branch is proposed, which constructs key prototypes through a soft-weighted aggregation mechanism driven by instance importance scores, thereby highlighting high-contribution instances and focusing on representing the core local cues most relevant to the depressive state.

[0035] A prototype-guided context aggregation branch is proposed, which uses key prototypes as semantic anchors and performs weighted aggregation of relevant context instances based on the similarity between instances and prototypes, thereby supplementing the auxiliary discriminative information that is easily ignored in traditional single-stage aggregation.

[0036] Experiments were conducted on three publicly available depression recognition datasets: AVEC2013, AVEC2014, and E-DAIC, and the proposed method was compared with various basic aggregation methods and existing methods. Experimental results show that the proposed method can better distinguish local segment contributions and retain key discriminative information, thereby improving the performance of video-level weakly supervised depression prediction. Attached Figure Description

[0037] Figure 1 This is a flowchart of a video-level behavior feature recognition method provided in the embodiments;

[0038] Figure 2 This is a structural diagram of DepMIL-Net, a video-level weakly supervised multi-instance learning model.

[0039] Figure 3This is a structural diagram of the dual-branch aggregation module in the DepMIL-Net model;

[0040] Figure 4 A comparison of experimental results for different aggregation strategies on the AVEC2013 and AVEC2014 datasets;

[0041] Figure 5 A schematic diagram showing the weight distribution of each branch instance in the ProtoDual aggregation for samples with different levels of depression.

[0042] Figure 6 This is a graph showing the experimental results of Keep Top-K on the AVEC2013 dataset;

[0043] Figure 7 This is a graph showing the experimental results of Keep Top-K on the AVEC2014 dataset;

[0044] Figure 8 A scatter plot showing the distribution of true and predicted values ​​on the AVEC2013 dataset;

[0045] Figure 9 This is a scatter plot showing the distribution of the true and predicted values ​​on the AVEC2014 dataset. Detailed Implementation

[0046] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0047] You can refer to this. Figure 1 and Figure 2 This embodiment provides a video-level behavior feature recognition method, which is implemented based on the video-level weakly supervised multi-instance learning model DepMIL-Net, and includes the following steps:

[0048] S10, sample the input raw video twice to obtain a frame sequence of a specified length;

[0049] S20, the frame sequence is divided into several local instances, and the initial visual features of each local instance are extracted using a shared method such as 3D ResNet-18 (R3D-18). The initial visual feature sequence composed of all the initial visual features is bidirectionally time-series modeled using a method such as Bi-LSTM to obtain the context instance feature sequence.

[0050] S30, combine multi-head self-attention (MHSA) and dynamic multi-instance normalization (DMIN) to perform context enhancement on the context instance feature sequence, and obtain the enhanced instance feature sequence;

[0051] S40: Construct key prototypes and perform context aggregation based on the enhanced instance feature sequences, and output behavioral feature recognition results.

[0052] This paper uses depression recognition as an example, therefore the original video is a facial video. Since original facial videos typically have a long temporal length and significant inter-frame redundancy, directly inputting the complete video into the DepMIL-Net model would increase computational overhead and introduce more redundant information, thus affecting the modeling of effective dynamic cues. Therefore, we can first perform temporal sampling on the video samples and then construct a local instance set based on this, forming an input representation suitable for video-level weakly supervised multi-instance learning.

[0053] Specifically, in step S10 above, a two-stage temporal sampling strategy is used to construct the model input. First, in the data preprocessing stage, the original video is initially downsampled temporally, for example, one frame is sampled every 10 frames, to compress redundant frames and reduce the overall sequence length. Subsequently, in the sample construction stage, a starting position is randomly selected from the downsampled video sequence, and sampling continues according to a specified sampling interval to form a sequence of length T. S The frame sequence is denoted as: ;

[0054] Where, x t T represents the t-th frame in the frame sequence. S This represents the number of frames input to the DepMIL-Net model, i.e., the length of the frame sequence. During the training phase, samples are constructed using random starting positions to enhance the temporal diversity of the same video samples and reduce the DepMIL-Net model's dependence on fixed time positions.

[0055] After obtaining a fixed-length input frame sequence X, it is divided into several local instances, denoted as:

[0056] ; ;

[0057] in, Let N represent the i-th local instance, N represent the number of local instances, and L represent the number of frames contained in a single local instance. A non-overlapping partitioning method is adopted to reduce content duplication between adjacent local instances, allowing each local instance to correspond more to local dynamic changes at different time positions, thereby reducing redundant interference.

[0058] Through the aforementioned local instance construction process, the original video is transformed into an input set composed of multiple local segments, thereby reducing temporal redundancy while preserving fine-grained dynamic information, providing a foundation for subsequent instance representation learning and video-level depression prediction.

[0059] In step S20 above, after segmenting local instances, feature encoding is further performed on each local instance to obtain an initial visual feature representation at the instance level. Considering that a single local instance consists of multiple consecutive frames of images, containing both spatial appearance information and local temporal dynamics, a shared-parameter 3D convolutional network R3D-18 is used as the instance feature extractor. Let the i-th local instance... The input form can be represented as Where 3 represents the number of channels, and H' and W represent the height and width of a frame, respectively. This is achieved through a shared encoder. Each local instance is mapped to a corresponding initial visual feature: ;

[0060] Among them, u i Let represent the initial visual features of the i-th local instance, and N represent the number of local instances in the current video sample. The shared encoding method allows different local instances to be mapped to a unified feature space, thereby forming a consistent sequence of initial visual features.

[0061] After obtaining the initial visual features of each local instance, temporal modeling of the entire initial visual feature sequence is also required. Let the initial visual features of all local instances in the same video sample constitute the initial visual feature sequence in chronological order. The initial visual feature sequence U is input into the Bi-LSTM to model the temporal dependencies between different local instances.

[0062] In practical implementation, a two-layer bidirectional Bi-LSTM can be used, with a hidden state dimension of 512 in each direction. Therefore, after bidirectional concatenation, a context-enhanced instance representation can be obtained:

[0063] ;

[0064] Where, m i Let M represent the context instance feature corresponding to the i-th local instance. The context instance features of all local instances constitute the context instance feature sequence M. Through this process, the original local instance is mapped into a sequence representation containing preceding and following temporal context information.

[0065] In step S30 above, the context instance feature sequence M is first linearly mapped to obtain the query, key, and value representation: Q=MW Q K=MW K V=MW V Among them, W Q W K and W V All of these are learnable parameter matrices.

[0066] Subsequently, multi-head self-attention is used to model the dependencies between different local instances, resulting in an attention-enhanced representation: ;in, This is the learnable parameter matrix.

[0067] The h-th attention head can be represented as: ;

[0068] Among them, Q h K represents the query for the h-th attention head. h V represents the key of the h-th attention head. h d represents the value of the h-th attention head. h This represents the feature dimension of the query and key in the h-th attention head, used to scale the dot product attention result.

[0069] like Figure 2 As shown, in the Dynamic Multi-instance Normalization submodule, the attention-enhanced representation A is dynamically normalized. According to the definition of M3DFEL, DMIN combines both bag-level and instance-level statistics to adaptively correct the features, which can be expressed as: ;

[0070] in, This represents two types of normalizers: bag-level and instance-level. and These represent the corresponding statistics. and For dynamically learned weight parameters, and These are the learnable affine transformation parameters. Here, Attention enhancement indicates the value of the nth local instance in A on the cth channel. This represents the corresponding normalization result. For simplicity, the overall normalization result is represented in matrix form below, with the relevant statistics and weight constraints shown in the following:

[0071] , ;

[0072] , ;

[0073] ;

[0074] Where N represents the number of instances and C represents the number of channels. This represents element-wise multiplication, and σ(∙) represents the Sigmoid activation function, used to dynamically normalize the attention-enhanced representation A. The matrix is ​​mapped to the range of 0 to 1, thus forming the gating weights. H represents the enhanced instance feature sequence, h i This represents the enhanced instance features achieved through multi-head self-attention and dynamic normalization. After the above processing, the instance representations achieve stronger discriminativeness and stability while preserving temporal correlation information.

[0075] Instance feature sequence obtained after feature enhancement The data is input into the ProtoDual module. Considering that video-level depression labels are typically dominated by only a few discriminative local segments, directly aggregating all local instances by average is easily influenced by weakly related or even irrelevant instances, thus weakening the model's ability to represent key depression cues. Therefore, this embodiment first constructs key prototypes to centrally represent the most representative core instance information in the video samples. Under the weakly supervised video setting, a few key instances often play a dominant role in video-level semantics, which is consistent with the basic modeling motivation of multi-instance learning.

[0076] like Figure 3 As shown, the process first enters the Key Prototype Construction branch. Specifically, it uses instance importance scores to generate a continuous weight distribution and then aggregates instance features based on this weight distribution. Let the importance score of the i-th instance be... Then its normalized weight is defined as:

[0077] ;in, w s and b s For learnable parameters, These are adjustable parameters used to control the concentration of instance weight distribution.

[0078] Based on the above weights, the key prototype can be represented as: ;in, This represents the key prototype formed by the weighted aggregation of local instances, where d is the dimension of instance features.

[0079] Unlike extraction methods that directly select key instances, the construction method described in this embodiment employs a differentiable soft selection mechanism, which highlights high-response instances while preserving the continuity and stability of overall optimization. The resulting key prototypes can more effectively characterize the core local cues related to the depressive state in the video and serve as an important reference for subsequent prototype-guided contextual aggregation.

[0080] Considering that the determination of video-level depressive states relies not only on a few high-response local segments but also on a set of contextual instances semantically related to key prototypes, using only key prototypes for prediction may still overlook some weakly related segments that have auxiliary discriminative functions. Therefore, this paper uses key prototypes as a reference to perform prototype-guided context aggregation on instance feature sequences, thereby obtaining a more complete video-level semantic representation. For example... Figure 3 As shown, after completing the construction of the key prototype, the ProtoDual module further introduces the Prototype-guided Context branch to supplement semantically relevant contextual clues on the basis of highlighting key instance information.

[0081] Specifically, to measure the semantic relevance between each instance and the key prototype, the features of the key prototype and instances are first L2 normalized, resulting in: , ;

[0082] Subsequently, cosine similarity is used to calculate the relevance score between the i-th instance and the key prototype:

[0083] ;||.||2, represents the L2 norm.

[0084] Based on this, the relevance scores of all instances are normalized to obtain the context aggregation weight β. From this, the prototype-guided context aggregation representation can be constructed as follows:

[0085] ; ;

[0086] in, This represents the aggregated contextual features.

[0087] The above aggregation method uses key prototypes as semantic anchors and adaptively allocates contextual weights based on the similarity between instances and key prototypes. This allows instances that are semantically consistent with or highly related to the key prototypes to contribute more, while the interference from weakly related or irrelevant instances is suppressed accordingly. Compared with indiscriminate average aggregation, this strategy can selectively integrate auxiliary contextual cues that are semantically related to the key prototypes while retaining the dominant information of key instances, thereby forming a more discriminative contextual representation.

[0088] like Figure 3As shown, in the ProtoDual module, the key prototype construction branch and the prototype-guided context aggregation branch represent video samples from different perspectives. The former focuses on highlighting the core local cues most relevant to the depressive state, while the latter supplements contextual information consistent with the semantics of the key prototype. Considering the complementarity of these two types of information in video-level depression level prediction, after obtaining the key prototype representation p and the context aggregation representation c, a dual-branch prediction fusion strategy is further adopted to jointly utilize core discriminative information and auxiliary contextual information, thereby improving the robustness of the final prediction result.

[0089] Specifically, separate regression heads are set for the key prototype branch and the context aggregation branch to map the two types of representations and obtain the corresponding prediction results:

[0090] ; ;

[0091] in, and This represents two independent regression mapping functions. This represents the predicted value obtained based on the key prototype. This represents the predicted value obtained based on context aggregation. These two branches are mapped by two independent linear regression layers. Based on this, the prediction results from the two branches are combined using an equal-weighted fusion method to obtain the final video-level depression level prediction. .

[0092] This fusion method does not introduce new fusion parameters, and can achieve the synergistic utilization of key instance information and contextual auxiliary information while maintaining the simplicity of the model structure. Compared with relying solely on single-path prediction, dual-branch fusion can highlight core depressive cues while taking into account relevant contextual fragments, thus obtaining more stable video-level regression results. In the current implementation, the final output is obtained by directly performing an equal-weighted average of the two prediction results.

[0093] This embodiment models depression level prediction as a continuous value regression task. To reduce the impact of label scale differences on the training process, the mean value is first calculated based on the labels in the training set. with standard deviation And the true label y is standardized as: .

[0094] Let the standardized prediction value of the model output be Then, a weighted combination of Smooth L1 loss and mean squared error (MSE) loss is used as the final training objective: ;

[0095] in, and This is the loss weight coefficient. Based on the current implementation settings, take... =0.3、 =0.7. After training, the standardized predictions are then restored to the original label space: .

[0096] The above design can effectively constrain the overall regression error while reducing the interference of abnormal biases, thus balancing training stability and prediction accuracy.

[0097] In terms of training mechanism, a two-stage optimization approach can be adopted. The first stage is the regular training stage, in which the overall parameters of the model are jointly optimized, enabling the feature extraction, instance relationship modeling, and video-level aggregation modules to collaboratively learn task-related spatiotemporal representations. Subsequently, after the regular training, a Regression-Based Reduction (RRT) stage is introduced for later fine-tuning. Unlike the regular training stage, the RRT stage fixes the feature extraction backbone, retaining only the high-level aggregation and regression-related parameters for further updates, and rebuilding the optimizer with a smaller learning rate. The purpose of this design is to reduce the disturbance to the existing feature structure caused by the continued large-scale updates of the backbone, given that the low-level visual representation is relatively stable. This allows the training focus to shift from overall representation learning to the refined optimization of high-level aggregation and regression mappings, thereby further improving the model's adaptability to the target distribution. It should be noted that the RRT stage does not change the form of the loss function, still using the combined regression loss consistent with the regular training stage.

[0098] To evaluate the effectiveness of the above method in the depression recognition task, experiments were conducted on two public datasets, AVEC2013 and AVEC2014, and compared and validated on the E-DAIC dataset. It should be noted that the experiments on AVEC2013 and AVEC2014 used preprocessed frames as input, while the experiments on E-DAIC used pre-extracted visual features as input representation, but the subsequent modeling and prediction framework remained consistent.

[0099] The AVEC2013 dataset contains 150 video samples, each corresponding to one participant, involving a total of 82 different participants. It is divided into three subsets: training, development, and testing. Each video in this dataset provides a self-reported BDI-II depression scale label for the participant; the rating range and degree of depression are shown in Table 1. All videos are in color MP4 format with a resolution of 640×480 pixels and a frame rate of 30fps. Participants ranged in age from 18 to 63 years, with a mean age of approximately 31.5 years.

[0100] Table 1: BDI-II Score Ranges and Depression Severity

[0101]

[0102] The AVEC2014 dataset contains 300 video samples, divided into training, development, and testing sets. This dataset consists of two task types: Freeform (free-response task) and Northwind (specified text reading task). Since this paper focuses on video-level modeling of depressive states rather than the differences between different interview tasks, the two task types were not modeled separately in the experiments. Instead, they were merged and uniformly incorporated into the training, validation, and testing processes to ensure consistency in the experimental setup.

[0103] In addition to the two main datasets mentioned above, this paper also conducts comparative experiments on the E-DAIC dataset. E-DAIC is an extended version of DAIC-WOZ, containing 275 interview sessions with an average duration of approximately 16.2 minutes. It includes 209 control samples and 66 depressed samples, exhibiting a significant imbalance in class distribution. This dataset was divided into 163 training samples, 56 validation samples, and 56 test samples.

[0104] After clarifying the data sources, corresponding preprocessing and input construction strategies were adopted for different datasets. For AVEC2013, OpenFace2 was first used to extract face regions from video frames and uniformly adjusted to 224×224 pixels. For AVEC2014, the Dlib library was used for face detection and cropping, and the input size was also normalized to 224×224 pixels. For E-DAIC, the original video frames were not used directly, but visual features extracted by ResNet-50 pre-training were used as input representations.

[0105] Building upon this, this paper employs a two-stage temporal sampling strategy to construct the model input. First, in the preprocessing stage, the original video is initially downsampled, retaining one frame for every ten frames. Then, in the training stage, a starting position is randomly selected from the downsampled video sequence, and further sampling is performed at a rate of one frame for every three frames. Finally, 64 frames are selected from each video sample and organized into a bag for input into the DepMIL-Net model. To improve the model's generalization ability, data augmentation is applied to the 64-frame sequence during the training stage, primarily including random horizontal flipping and color jitter.

[0106] The experiments were implemented using PyTorch 2.8.0, with CUDA 12.6 and an NVIDIA GeForce RTX4090 GPU. The visual encoding part used pre-trained R3D-18, while the E-DAIC dataset used ResNet50 pre-extracted visual features as input. Input configurations for different datasets are shown in Table 2, and key training parameters are shown in Table 3.

[0107] Table 2: Input configurations for different datasets

[0108]

[0109] Table 3: Main Training Parameter Settings

[0110]

[0111] After the regular training is completed, a retraining phase (RRT) for the regressor is introduced. In this phase, the low-level feature extraction backbone is frozen, and only the high-level aggregation and regression-related parameters are optimized to improve the fitting ability of the high-level prediction module to the target distribution while maintaining the relative stability of the low-level representation.

[0112] Mean absolute error (MAE) and root mean square error (RMSE) were used as performance evaluation metrics for the model. MAE reflects the average absolute deviation between the predicted and actual values, while RMSE is more sensitive to larger errors. The smaller the values ​​of both metrics, the better the model's predictive performance.

[0113] To verify the effectiveness of the ProtoDual dual-branch aggregation module, module ablation experiments were conducted, and the results are shown in Table 4. It can be seen that the complete ProtoDual achieved state-of-the-art performance on both datasets, indicating that dual-branch collaborative modeling can effectively improve the prediction of video-level depression levels. When the key prototype construction branch (Key Branch) is removed, the model performance drops significantly, indicating that this branch helps highlight high-contribution instances and enhances the identification of key depressive cues. Similarly, when the prototype-based context aggregation branch (Context Branch) is removed, the model performance also decreases, indicating that relying solely on key instances is insufficient to fully represent the depressive state in videos, and supplementary modeling based on contextual information is necessary. In summary, the Key Branch and Context Branch have significant complementary roles in the video depression recognition task, and their collaborative modeling is a key reason why ProtoDual achieves superior performance.

[0114] Table 4: Ablation experimental results of the ProtoDual aggregation strategy on the AAVEC2013 and AAVEC2014 datasets

[0115]

[0116] To analyze the impact of different instance lengths on model performance, this paper compares the experimental results when instance_length is 2, 4, 8, and 16, as shown in Table 5. The results show that the performance is optimal when instance_length=4, while instance lengths that are too short or too long will lead to a decrease in performance, indicating that a reasonable instance partitioning granularity plays an important role in the effective modeling of depression-related cues.

[0117] Table 5. Experimental results under different instance_length settings

[0118]

[0119] To further analyze the role of the aggregation module, this paper compares ProtoDual with two basic aggregation methods: mean pooling and max pooling. The purpose of introducing these two baselines is not merely to provide numerical comparisons, but because they represent two of the most fundamental approaches in video-level aggregation: mean pooling emphasizes uniform fusion, assuming all instances contribute equally to the final prediction; max pooling emphasizes extreme value selection, retaining only the strongest local responses. Therefore, comparing ProtoDual with these two methods more clearly illustrates the advantages of the proposed aggregation module compared to basic aggregation strategies.

[0120] Figure 4 Experimental results of different aggregation strategies on the AAVEC2013 and AAVEC2014 datasets are presented. It can be seen that ProtoDual outperforms mean pooling and max pooling on both datasets. This is because mean pooling includes irrelevant or weakly related segments in the video-level representation, thus weakening key depressive cues; while max pooling can highlight local salient responses, it relies too heavily on single peak instances, easily ignoring the temporal dispersion of depressive cues and their contextual relevance. In contrast, ProtoDual can integrate contextual information while emphasizing high-contribution instances, making it more suitable for handling the characteristics of sparse, unevenly distributed, and context-dependent cues in video depression recognition. Experimental results show that this aggregation method of "key information extraction + contextual supplementation modeling" can construct more discriminative sample-level representations, thereby improving the performance of video-level depression prediction.

[0121] To verify the effectiveness of the behavioral feature method described in this embodiment, it was compared with existing methods on three publicly available depression recognition datasets: AVEC2013, AVEC2014, and E-DAIC. Furthermore, the performance difference between the proposed ProtoDual aggregation strategy and the basic aggregation method was analyzed. The former was used to evaluate the model's overall predictive ability on different datasets, while the latter was used to verify the role of the designed aggregation module in video-level representation construction.

[0122] Table 6: Performance of different methods on the AVEC2013 test dataset

[0123]

[0124] Table 7: Performance of different methods on the AVEC2014 test dataset

[0125]

[0126] Tables 6 and 7 present the comparative results of our proposed method on the AVEC2013 and AVEC2014 test sets, respectively. Overall, our proposed method demonstrates strong competitiveness on both benchmark datasets: achieving the best RMSE on AVEC2013 and the best MAE on AVEC2014, with a relatively good RMSE as well.

[0127] From a methodological perspective, early methods such as Baseline, MHH+PLS, LPQ+1-NN, LPQ-TOP+MFA, LGBP-TOP-SVM, and LPQ-DFT primarily rely on manually designed textures, motion, and spatiotemporal descriptors. While these methods offer some interpretability, they are heavily reliant on domain priors and struggle to fully represent high-level semantic patterns related to depression, thus limiting their overall performance. In contrast, our proposed method extracts local spatiotemporal features through shared R3D-18 and combines BiLSTM, MHSA, and DMIN for instance-level representation enhancement, enabling a more comprehensive modeling of local dynamic changes and inter-instance dependencies.

[0128] Deep learning methods such as Two-stream CNNs, ResNet-50, DLGA-CNN, DCNNs, RNN-C3D, and FDHH have been able to learn deep representations from static appearances, local region relationships, or short-term dynamics. However, most still use single frames, local regions, or fixed-length segments as the main modeling units, directly forming overall predictions under video-level label supervision, lacking explicit characterization of the differences between key segments and weakly related segments. For long video samples, this approach is easily affected by redundant content.

[0129] Newer methods such as DPFV, VLDN, DJ-LDML, MTB-DFE, and Zhang have further enhanced temporal modeling or video-level fusion, but most still adopt a single-stage aggregation approach, making it difficult to simultaneously address key instance extraction and relevant context integration. This paper proposes the ProtoDual aggregation strategy: First, the Key Branch constructs key prototypes based on instance scores, highlighting the core local cues most relevant to the depressive state; then, the Context Branch uses the key prototypes as references to perform similarity-weighted aggregation of relevant context instances. This approach preserves key discriminative information while supplementing auxiliary context, making it more suitable for handling the sparse and unevenly distributed depressive cues in long videos. Ultimately, it achieves superior or more competitive results on two AVEC datasets.

[0130] Table 8: Performance of different methods on the E-DAIC validation and test datasets

[0131]

[0132] Furthermore, Table 8 presents the comparative results of our proposed method on the E-DAIC dataset. E-DAIC, like AVEC2013 / 2014, is geared towards depression recognition, differing only in data organization and experimental setup. The results show that our proposed method achieves MAE / RMSE scores of 4.25 / 5.48 on the Dev set and 5.12 / 6.24 on the Test set under a single-modal visual setup, demonstrating strong competitiveness among visual methods. This indicates that our proposed method is not limited to a specific dataset but has good applicability across various forms of depression recognition datasets.

[0133] like Figure 5 As shown in the figure, the Baseline represents the aggregation result of M3DFEL as a control method. Compared with its relatively flat overall weight distribution, the Key Branch in ProtoDual exhibits more obvious local focusing characteristics. Specifically, high-scoring samples show a significant peak near the 8th–10th instance, while medium- and low-scoring samples show a relatively concentrated high-response interval. The Context Branch maintains a smoother and more continuous response near the corresponding region, indicating that it focuses more on integrating contextual information related to the semantics of the key prototype. This suggests that the Key Branch and Context Branch respectively play the roles of key prototype extraction and contextual information supplementation, and the two have good complementarity.

[0134] To verify whether the DepMIL-Net model primarily relies on a few high-weight instances for prediction, a Keep Top-K instances experiment was conducted. Specifically, instances within a bag were sorted according to the instance weights output by the Key Branch, and only the top K high-weight instances were retained for aggregation. The results were then compared with the original model. The experiments were conducted with K=2, 4, and 8 respectively. Figure 6 and Figure 7 As shown, the results indicate that the model performance gradually recovers as the number of retained instances increases: the performance decline is most significant at KeepTop2, there is some improvement at KeepTop4, and KeepTop8 is close to the original model. This result suggests that the model's effective discriminative information is mainly concentrated in a subset of high-weight instances, but relying solely on a very small number of peak instances is insufficient to support stable predictions; a certain number of key instances and their related context are still needed to participate in the discrimination.

[0135] like Figure 8 and Figure 9 As shown, the predicted scatter plots on the AVEC2013 and AVEC2014 datasets generally exhibit a trend similar to that of the true labels, indicating that the DepMIL-Net model can capture some discriminative information related to the degree of depression.

[0136] This paper proposes a video-level weakly supervised multi-instance learning model, DepMIL-Net, which uses the ProtoDual aggregation strategy as its core. It extracts core discriminative information through key prototype construction branches and integrates relevant contextual information by guiding context aggregation branches through prototypes, thereby improving the performance of video-level depression prediction. Experimental results show that this method achieves competitive results on the AVEC2013 and AVEC2014 datasets and demonstrates good applicability on the E-DAIC dataset. Further basic aggregation comparisons, ablation experiments, instance length analysis, and Keep Top-K experiments show that the KeyBranch and Context Branch in ProtoDual have a good complementary effect. Reasonable instance partitioning granularity and effective utilization of high-weight instances are crucial for improving model performance.

[0137] It should be noted that this embodiment only uses facial video depression level prediction as a specific application scenario. However, the method of this invention is not limited to depression recognition tasks; it addresses a more general class of video-level or sequence-level weakly supervised problems: different local segments do not contribute consistently to the overall prediction, averaging aggregation easily weakens key discriminative information, while relying solely on a few high-response segments may ignore their related contextual content. Based on this, the method of this invention is theoretically applicable to other tasks with similar modeling requirements, such as fatigue detection, pain assessment, behavioral anomaly recognition, and emotional state analysis. In other words, the method described in this embodiment is a video-level behavioral feature recognition method that can be used for various behavioral feature recognitions, improving the accuracy of the recognition results.

[0138] Based on the same inventive concept, this embodiment also provides a video-level behavior feature recognition system, including a preprocessing unit and a video-level weakly supervised multi-instance learning model, DepMIL-Net. See also... Figure 2 The DepMIL-Net includes an instance partitioning module, a feature extraction module, a feature enhancement module, and a dual-branch aggregation module.

[0139] The preprocessing unit is used to sample the input raw video twice to obtain a frame sequence of a specified length. For example, as one possible implementation, the raw video is first downsampled in time, sampling one frame every few frames; then, a starting position is randomly selected from the downsampled video sequence, and sampling continues according to a specified sampling interval to obtain a frame sequence of a specified length.

[0140] The instance segmentation module is used to divide the frame sequence into several local instances.

[0141] The feature extraction module is used to extract the initial visual features of each local instance and perform bidirectional temporal modeling on the initial visual feature sequence composed of all initial visual features to obtain the context instance feature sequence. For example, as an example of an implementation, a shared-parameter 3D convolutional network R3 ResNet-18 can be used to extract the initial visual features of each local instance; alternatively, a two-layer bidirectional Bi-LSTM can be used to perform bidirectional temporal modeling on the initial visual feature sequence composed of all initial visual features.

[0142] The feature enhancement module combines multi-head self-attention and dynamic multi-instance normalization to enhance the context instance feature sequence, resulting in an enhanced instance feature sequence. For example, first, a linear mapping is performed on the context instance feature sequence to obtain query, key, and value representations; then, multi-head self-attention is used to model the dependencies between different local instances to obtain attention-enhanced representations; finally, dynamic normalization is performed on the attention-enhanced representations.

[0143] The dual-branch aggregation module is used to construct key prototypes and perform context aggregation based on the enhanced instance feature sequence, and output behavioral feature recognition results.

[0144] The dual-branch aggregation module can include a key prototype construction branch, a prototype guiding context aggregation branch, and a fusion submodule; among which, the key prototype construction branch first uses the instance importance score to generate a continuous weight distribution, and then performs weighted aggregation on all instance features to obtain the key prototype;

[0145] The prototype-guided context aggregation branch first performs L2 normalization on the key prototype and instance features respectively, and uses cosine similarity to calculate the relevance score between each instance feature and the key prototype. Then, it normalizes the relevance scores of all instance features to obtain the context aggregation weights, and constructs the aggregated context features c.

[0146] The fusion submodule maps key prototypes and context features separately to obtain their respective prediction results, and then fuses the two prediction results with equal weights to obtain the final behavior feature recognition result. , , .

[0147] This embodiment also provides an electronic device that may include a processor and a memory, wherein the memory is coupled to the processor. It is worth noting that this figure is exemplary, and other types of structures can be used to supplement or replace this structure to achieve data extraction, report generation, communication, or other functions.

[0148] The electronic device may also include an input unit, a display unit, and a power supply. It is worth noting that the electronic device is not necessarily required to include all the components mentioned above; it may also include components not mentioned, as can be found in existing technologies.

[0149] A processor, sometimes also called a controller or operating control, may include a microprocessor or other processor device and / or logic device that receives input and controls the operation of various components of an electronic device.

[0150] The memory may be one or more of the following: a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store information such as the processor's configuration information and the instructions executed by the processor. The processor can execute programs stored in the memory to perform information storage or processing. In one embodiment, the memory also includes a buffer memory, or buffer, to store intermediate information.

[0151] This invention also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed in an electronic device, the program product causes the electronic device to perform the operation steps included in the method of this invention.

[0152] This invention also provides a storage medium storing computer-readable instructions that cause an electronic device to perform the operational steps included in the method of this invention. More specific forms of the readable storage medium may include, but are not limited to: portable disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0153] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0154] The above specific embodiments are merely several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant teachings of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A video-level behavior feature recognition method, characterized in that, Includes the following steps: S10, sample the input raw video twice to obtain a frame sequence of a specified length; S20, the frame sequence is divided into several local instances, the initial visual features of each local instance are extracted, and bidirectional temporal modeling is performed on the initial visual feature sequence composed of all the initial visual features to obtain the context instance feature sequence. S30, combine multi-head self-attention and dynamic multi-instance normalization to perform context enhancement on the context instance feature sequence, and obtain the enhanced instance feature sequence. S40: Construct key prototypes and perform context aggregation based on the enhanced instance feature sequences, and output behavioral feature recognition results.

2. The video-level behavior feature recognition method according to claim 1, characterized in that, In step S10, the original video is first downsampled in time, with one frame sampled every few frames. Then, a starting position is randomly selected from the downsampled video sequence, and sampling continues according to the specified sampling interval to obtain a frame sequence of a specified length.

3. The video-level behavior feature recognition method according to claim 1, characterized in that, In S20, the initial visual features of each local instance are extracted using a three-dimensional convolutional network R3 ResNet-18 with shared parameters; and / or, a two-layer bidirectional Bi-LSTM is used to perform bidirectional temporal modeling on the initial visual feature sequence composed of all initial visual features.

4. The video-level behavior feature recognition method according to claim 1, characterized in that, The following processing operations are included in S30: First, a linear mapping is performed on the context instance feature sequence to obtain query, key, and value representations; Then, multi-head self-attention is used to model the dependencies between different local instances to obtain attention-enhanced representations; Finally, the attention enhancement representation is dynamically normalized.

5. The video-level behavior feature recognition method according to claim 1, characterized in that, S40 includes the following processing operations: Generate a continuous weight distribution using instance importance scores. , w s and b s For learnable parameters, To adjust the parameters, N is the number of instance features; Weighted aggregation of all instance features yields the key prototype. h i Features of the i-th instance; L2 normalization is performed on both key prototype and instance features, and cosine similarity is used to calculate the relevance score between each instance feature and the key prototype. Then, the relevance scores of all instance features are normalized to obtain the context aggregation weights, and the aggregated context features c are constructed. , ; The key prototype and context features are mapped separately to obtain their respective prediction results. Then, the two prediction results are fused with equal weights to obtain the final behavior feature recognition result. , , , and This represents two independent regression mapping functions.

6. A video-level behavior feature recognition system, characterized in that, It includes a preprocessing unit and a video-level weakly supervised multi-instance learning model, DepMIL-Net, which comprises an instance partitioning module, a feature extraction module, a feature enhancement module, and a dual-branch aggregation module; wherein, The preprocessing unit is used to sample the input raw video twice to obtain a frame sequence of a specified length; The instance partitioning module is used to divide the frame sequence into several local instances; The feature extraction module is used to extract the initial visual features of each local instance and perform bidirectional temporal modeling on the initial visual feature sequence composed of all initial visual features to obtain the context instance feature sequence; The feature enhancement module is used to combine multi-head self-attention and dynamic multi-instance normalization to enhance the context instance feature sequence, resulting in an enhanced instance feature sequence. The dual-branch aggregation module is used to construct key prototypes and perform context aggregation based on the enhanced instance feature sequence, and output behavioral feature recognition results.

7. The video-level behavior feature recognition system according to claim 6, characterized in that, The dual-branch aggregation module includes a key prototype construction branch, a prototype guidance context aggregation branch, and a fusion submodule; wherein... The key prototype construction branch first uses instance importance scores to generate a continuous weight distribution. , w s and b s For learnable parameters, To adjust the parameters, N is the number of instance features; then, all instance features are weighted and aggregated to obtain the key prototype. h i Features of the i-th instance; The prototype-guided context aggregation branch first performs L2 normalization on the key prototype and instance features, and then uses cosine similarity to calculate the relevance score between each instance feature and the key prototype. Next, it normalizes the relevance scores of all instance features to obtain the context aggregation weights, and constructs the aggregated context features c. , ; The fusion submodule maps key prototypes and context features separately to obtain their respective prediction results, and then fuses the two prediction results with equal weights to obtain the final behavior feature recognition result. , , , and This represents two independent regression mapping functions.

8. A computer program product comprising computer-readable instructions, characterized in that, The computer-readable instructions, when executed by a processor, implement the steps of the video-level behavior feature recognition method according to any one of claims 1-5.

9. A computer-readable storage medium comprising computer-readable instructions, characterized in that, The computer-readable instructions, when executed by a processor, implement the steps of the video-level behavior feature recognition method according to any one of claims 1-5.

10. An electronic device, characterized in that, include: Memory, which stores program instructions; The processor, connected to the memory, executes program instructions in the memory to implement the steps of the video-level behavior feature recognition method according to any one of claims 1-5.

Images