An interview video behavior feature analysis and sorting method based on multi-modal fusion

Abstract

The application discloses an interview video behavior feature analysis and sorting method based on multi-modal fusion, comprising the following steps: step S1, processing input interview video data, and respectively extracting visual modal features, audio modal features and text modal features; step S2, inputting the extracted visual modal features, audio modal features and text modal features into a multi-modal fusion network model; step S3, the multi-modal fusion network model performs feature fusion according to the input features, and then outputs behavior trait scores of multiple preset dimensions; and step S4, sorting the behavior trait scores of the multiple preset dimensions to obtain a sorting result. The application can realize standardized collection of multi-modal data, comprehensive feature extraction and accurate correlation analysis, finally output an objective feature sorting result which can be directly used for assisting decision-making, and improves the efficiency and precision of online interview data processing.

Description

Technical Field

[0001] This invention relates to a method for analyzing and ranking behavioral features in interview videos based on multimodal fusion, belonging to the field of deep learning machine vision. Background Technology

[0002] In online video interviews, a crucial step is evaluating candidates based on a set of pre-defined behavioral dimensions to determine their suitability for the position. When deep learning models are used to automate this evaluation task, it essentially becomes a multi-output regression problem. Multi-output regression tasks encounter a common technical challenge: the "regression to the mean" phenomenon. When deep learning models handle such multi-objective prediction tasks, the scores across multiple dimensions tend to cluster closely around the average of the training data, resulting in minimal differences in scores between different dimensions. This convergence in scores makes it difficult to reliably distinguish the highest-scoring dimension, i.e., to accurately determine which is the candidate's most prominent behavioral trait (Top Trait). This problem typically stems from the loss functions commonly used in model training (such as minimizing mean squared error), which, in order to minimize overall error, tend to suppress predictions of extreme values, thus sacrificing the ability to distinguish individual salient features. Therefore, in the field of behavioral feature analysis in interview videos, a technical method is needed that can overcome the inherent limitation of deep learning models' "regression to the mean" and clearly reveal the relative importance of each behavioral trait to achieve accurate feature ranking. Summary of the Invention

[0003] The purpose of this invention is to provide a method for analyzing and ranking behavioral features in interview videos based on multimodal fusion. This invention addresses the problem of low scoring discrimination and difficulty in effective ranking caused by mean regression in existing deep learning models when processing multi-dimensional feature scores. It enables accurate identification of behavioral traits and offers the advantage of accurate results.

[0004] The technical solution of this invention: A method for analyzing and ranking behavioral features in interview videos based on multimodal fusion, comprising the following steps:

[0005] Step S1: Process the input interview video data and extract visual modal features, audio modal features and text modal features respectively;

[0006] Step S2: Input the extracted visual modal features, audio modal features, and text modal features into the multimodal fusion network model; wherein, the multimodal fusion network model is trained using a hybrid loss function that combines list sorting loss and pairwise sorting loss to optimize its sorting ability;

[0007] Step S3: The multimodal fusion network model performs feature fusion based on the input features and then outputs behavioral trait scores in multiple preset dimensions.

[0008] Step S4: Sort the behavioral trait scores of the multiple preset dimensions to obtain the sorting results.

[0009] The above-mentioned method for analyzing and ranking behavioral features of interview videos based on multimodal fusion includes a background branch and a face branch in the multimodal fusion network model for visual features. The background branch is used to extract individual images and then use a pre-trained neural network model for feature extraction. The face branch is used to extract sparse facial image sequences with fixed step sizes and then use a pre-trained neural network model to fuse the temporal features of the sequence frames.

[0010] The aforementioned method for analyzing and ranking interview video behavioral features based on multimodal fusion, specifically the process of extracting features from the background branch and the facial branch, is as follows:

[0011] The interview video was segmented into multiple time segments using a video processing library; frames from points in the video stream were extracted as background environment images to generate a background tensor; multiple frames were uniformly extracted from the middle of the segments, and face detection and face alignment were performed on these consecutive frames using a face detection model; the frames were filled and resized to obtain standard-sized RGB consecutive frames to generate a face tensor; features were extracted from the background tensor and the face tensor using a temporal neural network model respectively.

[0012] The aforementioned method for analyzing and ranking interview video behavioral features based on multimodal fusion includes an audio branch in the multimodal fusion network model for audio features. The audio branch uses a pre-trained VGGish embedding model to extract features, and then uses an LSTM model to fuse temporal features.

[0013] The aforementioned method for analyzing and ranking behavioral features in interview videos based on multimodal fusion, the process of extracting features from the audio branch is as follows:

[0014] The audio stream was resampled to 16 kHz mono; the signal was sampled using a sliding window with a step size of 10 ms; the sampled sliding window signal was processed using the Mel frequency method, and a Hanning window was applied for windowing; after windowing and short-time Fourier transform, a 64-dimensional Mel filter bank was used to map the spectrum to the Mel scale, covering a frequency range of 125-7500 Hz; a small offset was applied to the generated spectrogram, and then a logarithmic function was used for stabilization to obtain a stable log-Mel spectrogram; the processed signal was input into the VGGish embedding model, which outputs the audio feature tensor.

[0015] The aforementioned method for analyzing and ranking interview video behavioral features based on multimodal fusion includes a dialogue text branch in the multimodal fusion network model for text features. The dialogue text branch uses the BERT model as a feature extractor, and then uses two layers of bidirectional LSTM to fuse the features of the text sequence.

[0016] The aforementioned method for analyzing and ranking interview video behavioral features based on multimodal fusion, the process of extracting features from the dialogue text branches is as follows:

[0017] Acronyms were replaced with full spellings or standardized forms to ensure consistency and readability; special characters and numbers were removed from the text using regular expressions, and the text was converted to lowercase to standardize case and reduce word variation. Then, word tokenization was used to segment the text, stop words and punctuation were removed, and lemmatization was performed using the NLTK lemmatization package to restore the original form and semantics of words. Finally, the maximum length of each text sequence was standardized, and BERT was used as a feature extractor to extract features.

[0018] The aforementioned method for analyzing and ranking behavioral features in interview videos based on multimodal fusion uses a multilayer perceptron to fuse multimodal features in the multimodal fusion network model. Finally, a linear layer is used as the activation layer for the regression task to output the inference results. After the multimodal fusion network model outputs the inference results, a ranking algorithm is applied to obtain the predicted order of behavioral traits.

[0019] The aforementioned method for analyzing and ranking behavioral features of interview videos based on multimodal fusion uses a regression training method with a hybrid ranking loss for the multimodal fusion network model. The regression training with the hybrid ranking loss involves constructing an equalized loss function that carries both list-wise and pair-wise loss information, and using the equalized loss function to train the multimodal fusion network model.

[0020] The aforementioned method for analyzing and ranking interview video behavioral features based on multimodal fusion, the process of constructing an equalization loss function carrying list-wise loss information and pair-wise loss information is as follows:

[0021] We obtain the weighted mean squared error by introducing weights into the loss function. :

[0022] ;

[0023] ;

[0024] In the formula: It is the number of sampling points; , , , and These are training weights for different dimensions; A true label representing normalization; The prediction results of the representative model; It is a weight vector;

[0025] List-wise loss information uses standardized discounted cumulative gain as a performance evaluation metric for the ranking algorithm:

[0026] ;

[0027] In the formula: The order of prediction for behavioral characteristics;

[0028] The closer the predicted ranking of the multimodal fusion network model is to the actual ranking, the lower the ranking loss. The smaller:

[0029] ;

[0030] Pair-wise loss information uses Cosine Loss as the performance evaluation metric for the ranking algorithm:

[0031] ;

[0032] ;

[0033] In the formula: Used to measure a given pair of tensors and The loss generated by the input; when the flag is set to positive, it indicates that the cosine similarity of the pair of tensors is calculated; when it is set to negative, the dissimilarity is calculated. It represents a controllable hard boundary; It is the parameter of the vector pair;

[0034] Final training loss Combining the three losses, the expression is as follows:

[0035] ;

[0036] in: , and All of these are hyperparameters.

[0037] Compared to existing technologies, this invention focuses on evaluating and optimizing the relative order of multiple behavioral dimensions, rather than predicting their precise values, thereby reducing the tendency for the highest-scoring trait to regress to the mean. By optimizing the relative ranking based on predicted scores, this invention improves the ability to identify an individual's dominant behavioral traits, resulting in more stable and robust results. Furthermore, by setting dedicated branches to extract and fuse visual, audio, and textual features separately, this invention supplements more feature information compared to single-modal analysis, avoiding the analytical bias caused by the lack of information from a single modality. This invention improves the ability to identify highly correlated features through a hybrid ranking loss optimization model, making the correlation analysis results between features and reference labels more accurate, and providing objective data references for interview screening; at the same time, through lightweight processing and a simple fusion network structure, it shortens the analysis time for a single data point, meeting the rapid processing needs of online interviews. Furthermore, for online video interview scenarios, this invention allows for the rapid identification of the interviewee's dominant personality traits in just about 15 seconds of video, regardless of the content. This method can improve the accuracy of interview screening, help employers select candidates who are more suitable for the job requirements, quickly process large amounts of video data and provide feedback, and greatly improve the efficiency and accuracy of the early screening stage of the recruitment process. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the multimodal fusion network model of the present invention. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0040] Example: A method for analyzing and ranking behavioral features in interview videos based on multimodal fusion.

[0041] Step S1: Process the input interview video data and extract visual modal features, audio modal features and text modal features respectively;

[0042] Step S2: Input the extracted visual modal features, audio modal features, and text modal features into the multimodal fusion network model; wherein, the multimodal fusion network model is trained using a hybrid loss function that combines list sorting loss and pairwise sorting loss to optimize its sorting ability;

[0043] Step S3: The multimodal fusion network model performs feature fusion based on the input features and then outputs behavioral trait scores in multiple preset dimensions.

[0044] Step S4: Sort the behavioral trait scores of the multiple preset dimensions to obtain the sorting results.

[0045] In this embodiment, the interview video data is collected by a data acquisition module, which includes:

[0046] The video acquisition unit uses a camera with a resolution of no less than 1080P to acquire video data at a standard rate of 15 seconds per segment, and synchronously records the acquisition timestamp. Background features and facial features are extracted from the video data. Background features are extracted through video frame sampling (extracting 1 keyframe every 15-second segment) and image feature algorithms (such as MobileNet-v3). Facial features are extracted through continuous frame sampling (extracting 6 frames evenly every 15-second segment), face detection (BlazeFace model), and alignment processing, followed by temporal feature algorithms (such as LSTM).

[0047] The audio acquisition unit uses a microphone with a sampling rate of 16kHz to acquire mono audio data and filter background noise (noise threshold ≤30dB); the audio data is preprocessed (resampled to 16kHz mono, 10ms step sliding window sampling), and audio features are extracted through Mel spectrum analysis and VGGish embedding model.

[0048] The text generation unit converts audio data into text data using a speech-to-text algorithm (word error rate ≤5%), generating time-stamped text sequences. The text data undergoes standardization processing (abbreviation replacement, special character removal, and lemmatization), and text features are extracted using the BERT model and bidirectional LSTM.

[0049] like Figure 1 As shown, the multimodal fusion network model assigns a separate branch to process each modality, extracting features separately before using them in multimodal fusion. The multimodal fusion network model processes three main modalities:

[0050] Visual Features: Simultaneous processing of background and facial image sequences. For the background branch, a single image is extracted; for the face branch, a sparse, fixed-step sequence of facial images is extracted. In the background branch, a pre-trained model, MobileNet-v3-Large, is used for feature extraction. For the face branch, a single-frame spatial feature fusion is performed using the feature extraction model, followed by fusion of temporal features from the sequence frames using an LSTM model. Specifically, the interview video is segmented into 15-second segments using the ffmpeg video processing library. After segmentation, frames at points in the video stream are extracted as background environment images, generating a background tensor shape of (batch, 1, 224, 224, 3). Six frames are uniformly extracted from the 5th to the 10th second. Face detection is then performed on these consecutive frames using the BlazeFace model, and face alignment is performed using provided facial markers. The frames are then padded and resized to obtain 224×224 RGB consecutive frames, generating a tensor shape of (batch, 6, 224, 224, 3).

[0051] Audio Features: Audio features are extracted using a pre-trained VGGish embedding model and then fed into the audio branch. Considering the audio sample length is fixed at 30 seconds during video editing, the resulting feature shapes are consistent. The audio branch directly uses LSTM for temporal feature fusion. Specifically, all audio is first resampled to 16 kHz mono and sampled with a sliding window of 10 ms. The second step processes the sampled sliding window signal using the Mel frequency method, applying a Hanning window to limit energy leakage in the Fourier transform (extending the window to 25 ms). After windowing and short-time Fourier transform, a 64-dimensional Mel filter bank is used to map the spectrum, covering the range of 125-7500 Hz. This spectrogram is stabilized using a logarithmic function after an offset of 1e-6 (to prevent log0 errors), resulting in 64 stable log-Mel spectrograms. The third step involves assembling the input tensor required for the VGGish model, concatenating 960ms of signal, corresponding to 96 10ms sampling windows, to form a frame. For a 15-second video clip, a maximum of 15 frames can be extracted, and the last 600ms of audio needs to be discarded. The input tensor shape to the VGGish model is (batch, 15, 96, 64), and the VGGish model processes it into a tensor of (batch, 15, 128).

[0052] Text Features: Audio is transcribed into text files, and replacement rules for acronyms and stop words are defined as follows: Acronyms are replaced with full spellings or standardized forms to ensure consistency and readability. Special characters and numbers are removed from the text using regular expressions for subsequent processing. The text is converted to lowercase to standardize case and reduce word variation. Then, word tokenization is used to segment the text, and stop words and punctuation are removed for further analysis. Lexical reconstruction is performed using the NLTK lemma package to restore the original morphology and semantics of words. Finally, the maximum length of each text sequence is standardized. After processing, BERT is used as the feature extractor, followed by a two-layer bidirectional LSTM (Bi-LSTM) fusion of the text sequence features.

[0053] After extracting features from three modalities, the features are concatenated and fused using a multilayer perceptron (MLP). The MLP was chosen because it requires less data than transformer-based attention mechanisms, which helps improve recognition efficiency. Finally, a linear layer is used as the activation layer for the regression task. After inference of the multimodal fusion network model, a ranking algorithm is applied to obtain the predicted order of behavioral traits.

[0054] In this embodiment, the multimodal fusion network model is trained using regression with hybrid ranking loss. This invention constructs a balanced loss function carrying both list-wise and pair-wise loss information, which is suitable for small-scale, embeddable models.

[0055] Specifically, weights are introduced into the loss function to obtain the weighted mean squared error. :

[0056] ;

[0057] ;

[0058] In the formula: It is the number of sampling points, for the th One sampling point: , , , and These are training weights for different dimensions, for example, values ​​of 0.1, 0.3, 0.1, 0.2, and 0.3 respectively; A true label representing normalization; The prediction results of the representative model; It is a weight vector;

[0059] List-wise loss information uses standardized discounted cumulative gain as a performance evaluation metric for the ranking algorithm:

[0060] ;

[0061] In the formula: The order of prediction for behavioral traits;

[0062] For example, if the true rank of a behavioral trait label at a specific sampling point is OECAN, and the corresponding evaluation score is 5-4-3-2-1, the calculated ideal standardized discount cumulative gain is approximately 10.272. During model inference, if the predicted rank... It is ECOAN, with a correlation score of 4-3-5-2-1, resulting in a cumulative gain of approximately 9.641 from the discount.

[0063] The closer the predicted ranking of the multimodal fusion network model is to the actual ranking, the lower the ranking loss. The smaller:

[0064] ;

[0065] Pair-wise loss information uses Cosine Loss as the performance evaluation metric for the ranking algorithm:

[0066] ;

[0067] ;

[0068] In the formula: Used to measure a given pair of tensors and The loss generated by the input; when the flag is set to positive, it indicates that the cosine similarity of the pair of tensors is calculated; when it is set to negative, the dissimilarity is calculated. It represents a controllable hard boundary; It is the parameter of the vector pair;

[0069] Final training loss Combining the three losses, the expression is as follows:

[0070] ;

[0071] in: , and All are hyperparameters. In this embodiment, 0.01 represents the step, the optimal parameters obtained during training. , and The values ​​of 0.10, 0.32, and 0.58 respectively achieve the best behavioral trait recognition results.

[0072] In this embodiment, the training hyperparameters of the multimodal fusion network model are set as follows: learning rate 2e-5, batch size 64, training for 100 epochs, and using the Adam optimizer. Regarding model building techniques, a batch normalization (BN) layer is introduced to enhance training stability. Training is driven by a balanced loss function and a normalization factor... =0.7.

[0073] The multimodal fusion network model outputs behavioral trait scores across multiple preset dimensions. These scores (including five dimensions: extraversion, agreeableness, conscientiousness, neuroticism, and openness) are ranked to avoid direct judgment of "personality traits," providing only objective characteristic data and correlation analysis results. The specific design is as follows:

[0074] Output content: Centered on the "feature-relevance" correspondence, it includes: Multimodal feature list: clearly listing the specific features extracted from vision, audio, and text (such as "facial expression temporal changes", "voice intonation fluctuations", "text sentence coherence", etc.); Relevance analysis results: the correlation values ​​and rankings of features calculated by the model and preset reference labels (such as behavioral feature labels related to job requirements) (arranged from high to low correlation), only reflecting the objective degree of correlation between features and reference labels, without involving conclusive statements on "personality traits".

[0075] Output format: Standardized technical formats are adopted, including: Data tables: The feature names, correlation scores (e.g., 0-10 points), and ranking numbers are presented in structured tables for easy reading by the system or by humans; Visual charts: The distribution of feature correlation and ranking results are displayed in a graphical way, such as bar charts and line charts, to present the analysis data intuitively and avoid subjective descriptions.

[0076] This invention transforms the abstract feature data of the multimodal fusion analysis module into standardized results that can be used directly. This not only solves the problem of chaotic and unusable analysis results in the prior art, but also avoids the judgment of non-technical conclusions by designing to "output only objective data", thus meeting the objectivity requirements of the technical solution.

[0077] In this embodiment, the entire system is implemented on a 64-bit Windows system, using CUDA 12.1 and PyTorch 2.3.0, equipped with 32 GB of RAM, an Intel i7-6700 CPU, and an NVIDIA GeForce GTX 2060 for model training and parallel inference. Through overall ablation experiments, the loss function and hyperparameter combination achieve excellent behavioral feature recognition accuracy across the video, audio, and text modalities from high to low complexity. Furthermore, the balanced loss function of this invention can be extended to various information modalities of varying complexity, such as web text, voice dialogue audio, and facial video. The behavioral feature recognition accuracy of the model surpasses the performance of existing models trained using traditional loss functions such as MSE.

[0078] In summary, this invention can achieve standardized collection of multimodal data, comprehensive feature extraction, and accurate correlation analysis, and finally output objective feature ranking results that can be directly used to assist decision-making, thereby improving the efficiency and accuracy of online interview data processing.

Claims

1. A method for analyzing and ranking behavioral features in interview videos based on multimodal fusion, characterized in that: Includes the following steps: Step S1: Process the input interview video data and extract visual modal features, audio modal features and text modal features respectively; Step S2: Input the extracted visual modal features, audio modal features, and text modal features into the multimodal fusion network model; wherein, the multimodal fusion network model is trained using a hybrid loss function that combines list sorting loss and pairwise sorting loss to optimize its sorting ability; Step S3: The multimodal fusion network model performs feature fusion based on the input features and then outputs behavioral trait scores in multiple preset dimensions. Step S4: Sort the behavioral trait scores of the multiple preset dimensions to obtain the sorting results; The multimodal fusion network model is trained using regression with hybrid ranking loss. The regression training with hybrid ranking loss involves constructing an equalized loss function that carries both list-wise and pair-wise loss information, and using the equalized loss function to train the multimodal fusion network model. The process of constructing the balanced loss function carrying list-wise and pair-wise loss information is as follows: We obtain the weighted mean squared error by introducing weights into the loss function. : ； ； In the formula: It is the number of sampling points; , , , and These are training weights for different dimensions; A true label representing normalization; The prediction results of the representative model; It is a weight vector; List-wise loss information uses standardized discounted cumulative gain as a performance evaluation metric for the ranking algorithm: ； In the formula: The order of prediction for behavioral characteristics; The closer the predicted ranking of the multimodal fusion network model is to the actual ranking, the lower the ranking loss. The smaller: ； Pair-wise loss information uses Cosine Loss as the performance evaluation metric for the ranking algorithm: ； ； In the formula: Used to measure a given pair of tensors and The loss generated by the input; when the flag is set to positive, it indicates that the cosine similarity of the pair of tensors is calculated; when it is set to negative, the dissimilarity is calculated. It represents a controllable hard boundary; It is the parameter of the vector pair; Final training loss Combining the three losses, the expression is as follows: ； in: , and All of these are hyperparameters.

2. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 1, characterized in that: For visual features, the multimodal fusion network model includes a background branch and a face branch; the background branch is used to extract individual images and then use a pre-trained neural network model for feature extraction; the face branch is used to extract sparse, fixed-step facial image sequences and then use a pre-trained neural network model to fuse the temporal features of the sequence frames.

3. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 2, characterized in that: The process of extracting features from the background branch and the facial branch is as follows: The interview video was segmented into multiple time segments using a video processing library; frames from points in the video stream were extracted as background environment images to generate a background tensor; multiple frames were uniformly extracted from the middle of the segments, and face detection and face alignment were performed on these consecutive frames using a face detection model; the frames were filled and resized to obtain standard-sized RGB consecutive frames to generate a face tensor; features were extracted from the background tensor and the face tensor using a temporal neural network model respectively.

4. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 1, characterized in that: For audio features, the multimodal fusion network model includes an audio branch; the audio branch uses a pre-trained VGGish embedding model to extract features, and then uses an LSTM model to fuse temporal features.

5. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 4, characterized in that: The process of extracting features from the audio branch is as follows: The audio stream was resampled to a 16 kHz mono channel; a sliding window sampling with a 10 ms step was performed on the 16 kHz mono channel; the sampled sliding window signal was processed using the Mel frequency method, and a Hanning window was applied for windowing; after windowing and short-time Fourier transform, a 64-dimensional Mel filter bank was used to map the spectrum to the Mel scale, covering a frequency range of 125-7500 Hz; a small offset was applied to the generated spectrogram, and then a logarithmic function was used for stabilization to obtain a stable log-Mel spectrogram; the processed signal was input into the VGGish embedding model, which outputs the audio feature tensor.

6. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 1, characterized in that: For text features, the multimodal fusion network model includes a dialogue text branch; the dialogue text branch uses the BERT model as a feature extractor, and then uses two layers of bidirectional LSTM to fuse the features of the text sequence.

7. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 6, characterized in that: The process of extracting features from the dialogue text branches is as follows: Acronyms are replaced with full spellings or standardized forms; special characters and numbers are removed from the text using regular expressions, the text is converted to lowercase to standardize case and reduce word variation, then word tokenization is used to segment the text, stop words and punctuation are removed, and then NLTK lemmatization is used to restore the original form and semantics of words; finally, the maximum length of each text sequence is standardized, and BERT is used as a feature extractor to extract features.

8. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 1, characterized in that: The multimodal fusion network model uses a multilayer perceptron to fuse multimodal features, and finally uses a linear layer as the activation layer for the regression task to output the inference result. After the multimodal fusion network model outputs the inference result, a ranking algorithm is applied to obtain the prediction order of behavioral traits.

Images