Interview video behavior feature analysis and sorting method based on multi-modal fusion

Abstract

The invention discloses an interview video behavior feature analysis and sorting method based on multi-modal fusion, and the method comprises the following steps: S1, processing input interview video data, and respectively extracting visual modal features, audio modal features and text modal features; s2, inputting the extracted visual modal features, the extracted audio modal features and the extracted text modal features into a multi-modal fusion network model; s3, performing feature fusion by the multi-modal fusion network model according to the input features, and outputting behavior characteristic scores of a plurality of preset dimensions; and S4, sorting the behavior characteristic scores of the plurality of preset dimensions to obtain a sorting result. According to the method, standardized acquisition, comprehensive feature extraction and accurate correlation analysis of multi-modal data can be realized, an objective feature sorting result which can be directly used for auxiliary decision making is finally output, and the efficiency and precision of online interview data processing are improved.

Description

TECHNICAL FIELD

[0001] The application relates to an interview video behavior feature analysis and sorting method based on multi-modal fusion, and belongs to the field of deep learning machine vision. BACKGROUND

[0002] In an online video interview, a key link is to evaluate a candidate according to a set of preset behavior dimensions to determine the matching degree with a post. When a deep learning model is applied to automatically complete this evaluation task, it is essentially a multi-output regression problem. The multi-output regression task encounters a ubiquitous technical difficulty. The difficulty is the "regression to the mean" phenomenon: when processing such multi-target prediction tasks, the deep learning model tends to closely gather the multiple dimension scores of the output around the average value of the training data, resulting in small differences between different dimensions. The convergence of the scores makes it difficult to reliably distinguish the highest scoring dimension, that is, it is difficult to accurately determine which is the most prominent behavior trait (Top Trait) of the candidate. This problem is usually caused by the loss function (such as minimizing mean square error) commonly used in model training, which tends to suppress extreme value prediction in order to minimize the overall error, thereby sacrificing the ability to distinguish individual significant features. Therefore, in the field of interview video behavior feature analysis, a technical method is needed to overcome the inherent limitation of deep learning model "regression to the mean" and clearly reveal the relative importance of each behavior trait, so as to realize accurate feature sorting. SUMMARY

[0003] The purpose of the present application is to provide an interview video behavior feature analysis and sorting method based on multi-modal fusion. The present application can solve the problem of low score differentiation and difficulty in effective sorting caused by the "regression to the mean" phenomenon when existing deep learning models process multi-dimensional feature scores. It can accurately identify behavior traits and has the advantage of accurate results.

[0004] The technical solution of the present application: an interview video behavior feature analysis and sorting method based on multi-modal fusion, comprising the following steps:

[0005] Step S1, processing the input interview video data, and extracting visual modal features, audio modal features and text modal features respectively;

[0006] Step S2, inputting the extracted visual modal features, audio modal features and text modal features into a multi-modal fusion network model; wherein the multi-modal fusion network model is trained using a hybrid loss function combining list sorting loss and pairwise sorting loss to optimize its sorting ability;

[0007] Step S3, the multi-modal fusion network model performs feature fusion according to the input features, and outputs a plurality of preset dimension behavior trait scores;

[0008] Step S4, the plurality of preset dimension behavior trait scores are sorted to obtain a sorting result.

[0009] The interview video behavior feature analysis and sorting method based on multi-modal fusion described above, for visual features, the multi-modal fusion network model is provided with a background branch and a face branch; the background branch is used to extract a single image and then perform feature extraction using a pre-trained neural network model; the face branch is used to extract a sparse fixed step face image sequence, and then use a pre-trained neural network model to fuse the time features of the sequence frames.

[0010] The aforementioned interview video behavior feature analysis and sorting method based on multi-modal fusion, the feature extraction process of the background branch and the face branch is as follows:

[0011] The interview video is segmented into a plurality of time segments using a video processing library; the frame of the point in the video stream is extracted as a background environment image to generate a background tensor; a plurality of frames of images are uniformly extracted from the middle period of the segment, face detection and face alignment are performed on these continuous frames using a face detection model, the frames are padded and resized to obtain standard size RGB continuous frames to generate a face tensor; a time sequence neural network model is used to extract features from the background tensor and the face tensor respectively.

[0012] The aforementioned interview video behavior feature analysis and sorting method based on multi-modal fusion, for audio features, the multi-modal fusion network model is provided with an audio branch; the audio branch uses a pre-trained VGGish embedding model to extract features, and then uses an LSTM model to fuse time features.

[0013] The aforementioned interview video behavior feature analysis and sorting method based on multi-modal fusion, the feature extraction process of the audio branch is as follows:

[0014] The audio stream is resampled to 16 kHz monaural; the signal is sampled with a sliding window of 10 ms step; the sampled sliding window signal is processed using the Mel frequency method, and a Hanning window is applied for windowing; after windowing and short-time Fourier transform, the spectrum is mapped to the Mel scale using a 64-dimensional Mel filter bank, covering a frequency range of 125-7500 Hz; a slight offset is applied to the generated spectrum graph, and a logarithmic function is used for stabilization to obtain a stable log Mel spectrum graph; the processed signal is input into the VGGish embedding model, and the model outputs an audio feature tensor.

[0015] For the aforementioned interview video behavior feature analysis and sorting method based on multi-modal fusion, for the text features, a dialogue text branch is arranged in the multi-modal fusion network model; the dialogue text branch uses a BERT model as a feature extractor, and then uses two layers of bidirectional LSTM to fuse the features of the text sequence.

[0016] For the aforementioned interview video behavior feature analysis and sorting method based on multi-modal fusion, the process of feature extraction of the dialogue text branch is as follows:

[0017] The initial abbreviations are replaced with full spelling or standardized forms to ensure consistency and readability; special characters and numbers are removed from the text using regular expressions, the text is converted to lowercase to standardize the case and reduce word variations, and then the text is segmented using word tokenization, stop words and punctuation are removed, and word stemming is performed using the NLTK word stemming package to restore the original form and semantics of the words; finally, the maximum length of each text sequence is standardized, and the BERT is used as a feature extractor to extract features.

[0018] For the aforementioned interview video behavior feature analysis and sorting method based on multi-modal fusion, the multi-modal fusion network model uses a multi-layer perceptron for multi-modal feature fusion, and finally uses a linear layer as the activation layer of the regression task to output the inference result; after the multi-modal fusion network model outputs the inference result, a ranking algorithm is applied to obtain the predicted order of the behavior traits.

[0019] For the aforementioned interview video behavior feature analysis and sorting method based on multi-modal fusion, the multi-modal fusion network model uses a hybrid ranking loss regression training; the hybrid ranking loss regression training is to construct a balanced loss function carrying list-wise loss information and pair-wise loss information, and the balanced loss function is used to train the multi-modal fusion network model.

[0020] For the aforementioned interview video behavior feature analysis and sorting method based on multi-modal fusion, the process of constructing a balanced loss function carrying list-wise loss information and pair-wise loss information is as follows:

[0021] The loss function is introduced into the weighted mean square error :

[0022] ;

[0023] ;

[0024] In the formula: is the number of sampling points; , , , and are training weights of different dimensions, respectively; represents a normalized true label; represents a prediction result of the model; is a weight vector;

[0025] The list-wise loss information uses a normalized discount cumulative gain as an effectiveness evaluation index of the ranking algorithm:

[0026] ;

[0027] In the formula: is the predicted order of the behavior characteristics;

[0028] The closer the ranking predicted by the multi-modal fusion network model is to the real ranking, the smaller the ranking loss :

[0029] ;

[0030] The pair-wise loss information uses Cosine Loss as an effectiveness evaluation index of the ranking algorithm:

[0031] ;

[0032] ;

[0033] In the formula: is used to measure the loss generated by the input of a given pair of tensors and ; when the flag is set to positive, it represents the calculation of the cosine similarity of the pair of tensors; when it is set to negative, the calculation of dissimilarity is calculated; represents a controllable hard boundary; is a parameter of the vector pair;

[0034] The final training loss Combining the three losses, the expression is as follows:

[0035] ;

[0036] Wherein: , and are hyperparameters.

[0037] Compared with the prior art, the application focuses on evaluating and optimizing the relative order of multiple behavior dimensions instead of predicting their exact values, thereby reducing the tendency of the highest score trait to mean reversion. By optimizing the relative ranking based on the predicted scores, the application can improve the identification of the individual's dominant behavior traits, making the results more stable and robust. Further, the application sets up dedicated branches to extract visual, audio and text features respectively and fuse them, which supplements more feature information compared to single modal analysis and avoids one-sidedness caused by missing single modal information. The application optimizes the model through a hybrid ranking loss to improve the recognition of high correlation features, making the correlation analysis results of the features and reference labels more accurate and providing objective data reference for interview screening. At the same time, through lightweight processing and simple fusion network structure, the analysis time of a single data is shortened, meeting the online interview rapid processing demand. Further, the application is aimed at online video interview scenarios, and the interviewee answers the interview questions or introduces himself, only about 15s of video is needed to quickly identify the dominant personality traits of the interviewee regardless of the content. This method can improve the accuracy of interview screening, help the employer to select the candidate who meets the job requirements, quickly process a large amount of video data and provide feedback, and greatly improve the efficiency and accuracy of the early screening stage of the recruitment process. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 Figure 1 is a schematic diagram of a multi-modal fusion network model of the application. DETAILED DESCRIPTION

[0039] The application will be further described below in conjunction with the drawings and examples, but not as the basis for limiting the application.

[0040] Embodiment: An interview video behavior feature analysis and ranking method based on multi-modal fusion,

[0041] Step S1, process the input interview video data, and extract visual modal features, audio modal features and text modal features respectively;

[0042] Step S2, input the extracted visual modal features, audio modal features and text modal features into a multi-modal fusion network model; wherein the multi-modal fusion network model is trained by a hybrid loss function combining list ranking loss and pairwise ranking loss to optimize its ranking ability;

[0043] Step S3, the multi-modal fusion network model performs feature fusion according to the input features, and then outputs behavior trait scores of multiple preset dimensions;

[0044] Step S4, rank the behavior trait scores of the multiple preset dimensions to obtain a ranking result.

[0045] In this embodiment, the interview video data is collected by a data collection module, which includes:

[0046] The video collection unit uses a camera with a resolution not less than 1080P to collect video data according to the standard of 15 seconds per segment, and synchronously records the collection timestamp. The background features and facial features are extracted from the video data. The background features are extracted by video frame sampling (extracting 1 key frame per 15-second segment) and image feature algorithm (such as MobileNet-v3), and the facial features are extracted by continuous frame sampling (uniformly extracting 6 frames per 15-second segment), face detection (BlazeFace model) and alignment processing, and then extracted by time sequence feature algorithm (such as LSTM).

[0047] The audio collection unit uses a microphone with a sampling rate of 16kHz to collect monaural audio data and filter background noise (noise threshold ≤ 30dB). The audio data is preprocessed (resampling to 16kHz monaural, 10ms step sliding window sampling), and the audio features are extracted by Mel spectrum analysis and VGGish embedding model.

[0048] The text generation unit converts the audio data into text data by speech-to-text algorithm (word error rate ≤ 5%) to generate a text sequence with time mark. The text data is standardized (abbreviation replacement, special character removal, morphological restoration), and the text features are extracted by BERT model and bidirectional LSTM.

[0049] As shown in Figure 1 The multi-modal fusion network model allocates a separate branch for processing each modality, and extracts the features separately before using multi-modal fusion to fuse these features. The multi-modal fusion network model processes three main modalities:

[0050] Visual features: Both background image and face image sequence are processed. For the background branch, a single image is extracted; for the face branch, a sparse fixed-step face image sequence is extracted. In the background branch, a pre-trained model is used for feature extraction: MobileNet-v3-Large. For the face branch, a feature extraction model is used for single-frame spatial feature fusion, and then an LSTM model is used to fuse the temporal features of the sequence frames. Specifically, the interview video is segmented into 15-second segments using the video processing library ffmpeg. After segmenting the video, the frame at the midpoint of the video stream is extracted as the background environment image, and the generated background tensor has a shape of (batch, 1, 224, 224, 3). From the 5th second to the 10th second, 6 frames of images are uniformly extracted. Then the BlazeFace model is used to detect faces in these consecutive frames, and the provided face landmarks are used for face alignment. The frames are padded and resized to obtain 224x224 RGB consecutive frames, and the generated tensor has a shape of (batch, 6, 224, 224, 3).

[0051] Audio features: A pre-trained VGGish embedding model is used to extract audio features, which are then input into the audio branch. Considering that the audio sample length is fixed at 30 seconds during video clipping, the resulting feature shape is consistent. The audio branch directly uses LSTM for temporal feature fusion. Specifically, first, all audio is resampled to 16 kHz mono, and the signal is sampled with a sliding window of 10 ms. The second step uses the Mel frequency method to process the sampled sliding window signal, and applies a Hann window to limit the energy leakage of the Fourier transform (the window is expanded to 25 ms). After windowing and short-time Fourier transform, a 64-dimensional Mel filter bank is used to map the spectrum, covering the range of 125-7500 Hz. This spectrum is stabilized using a logarithmic function after a shift of 1e-6 (to prevent log0 errors), resulting in 64 stable log Mel spectrum. The third step involves assembling the input tensor required by the VGGish model, which concatenates 960 ms of signal corresponding to 96 10 ms sampling windows into a frame. For a 15-second video segment, up to 15 frames can be extracted, and the last 600 ms of audio needs to be discarded. The tensor shape input into the VGGish model is (batch, 15, 96, 64), and the VGGish model processes it into a tensor of shape (batch, 15, 128).

[0052] Text Feature: The audio is transcribed into a text file, and replacement rules for acronyms and stop words list are defined, the process is as follows: Acronyms are replaced with full spelling or standardized forms to ensure consistency and readability. Special characters and numbers are removed from the text using regular expressions to facilitate subsequent processing. The text is converted to lowercase to standardize capitalization and reduce word variations. Then the text is segmented using word tokenization, removing stop words and punctuation for further analysis. Lemmatization is performed using the NLTK Lemmatization package to restore the original form and semantics of the words. Finally, the maximum length of each text sequence is standardized. After processing, BERT is used as a feature extractor, followed by a two-layer bidirectional LSTM (Bi-LSTM) to fuse the features of the text sequence.

[0053] After extracting features from three modalities, the features are concatenated and a multi-layer perceptron (MLP) is used for multi-modal feature fusion. The reason for choosing MLP is that it requires less data compared to transformer-based attention mechanisms, which helps improve recognition efficiency. Finally, a linear layer is used as the activation layer for the regression task. After the multi-modal fusion network model is inferred, a ranking algorithm is applied to obtain the predicted order of behavioral traits.

[0054] In this embodiment, the multi-modal fusion network model uses regression training with hybrid ranking loss. The present application constructs a balanced loss function carrying list-wise loss information and pair-wise loss information, which is suitable for small-scale embeddable models.

[0055] Specifically, a weighted mean square error is introduced to the loss function

[0056]

[0057]

[0058] wherein: is the number of sampling points, for the th sampling point: , , , and are training weights of different dimensions, for example, the values are 0.1, 0.3, 0.1, 0.2 and 0.3, respectively; represents the normalized true label; represents the prediction result of the model; is the weight vector;

[0059] The list-wise loss information uses the standard discount cumulative gain as the evaluation index of the ranking algorithm: ​​​

[0060] ;

[0061] In the formula: is the prediction order of the behavior trait;

[0062] For example, if the true ranking of the behavior trait label of a certain sampling point is O-E-C-A-N, and the corresponding evaluation score is 5-4-3-2-1, the calculated ideal normalized discount cumulative gain is about 10.272. During model inference, if the predicted ranking is E-C-O-A-N, the relevant score is 4-3-5-2-1, and the obtained discount cumulative gain is about 9.641.

[0063] The closer the multi-modal fusion network model prediction ranking is to the true ranking, the smaller the ranking loss :

[0064] ;

[0065] The pair-wise loss information uses Cosine Loss as the evaluation index of the ranking algorithm:

[0066] ;

[0067] ;

[0068] In the formula: is used to measure the loss generated by the input of a given pair of tensors and ; when the flag is set to positive, it represents the calculation of the cosine similarity of the pair of tensors; when it is set to negative, the dissimilarity is calculated; represents a controllable hard boundary; is the parameter of the vector pair;

[0069] The final training loss Combining the three losses, the expression is as follows:

[0070] ;

[0071] Among them: , and are hyperparameters. In this embodiment, 0.01 is the step, and the optimal parameters , and obtained by training are 0.10 0.32 0.58 respectively, which can achieve the best behavior trait recognition effect.

[0072] ​In this embodiment, the training hyperparameters of the multi-modal fusion network model are set as follows: learning rate is 2e-5, batch size is 64, training is 100 cycles, and Adam optimizer is used. In terms of model construction technology, batch normalization (BN) layer is introduced to enhance training stability. Training is driven by balanced loss function, and the normalization factor = 0.7.

[0073] The multi-modal fusion network model outputs a plurality of preset dimension behavior trait scores, sorts the behavior trait scores of the plurality of preset dimensions (including five dimensions: extroversion, agreeableness, conscientiousness, neuroticism and openness), avoids direct determination of “personality traits”, and only provides objective feature data and correlation analysis results, and the specific design is as follows:

[0074] Output content: taking the “feature-correlation degree” correspondence as the core, including: multi-modal feature list: clearly listing specific features extracted from vision, audio and text (such as “facial expression time sequence change”, “voice tone fluctuation” and “text sentence coherence”); correlation analysis result: the correlation value and order (arranged from high to low according to correlation degree) of the features and the preset reference label (such as the behavior trait label related to the job requirement) calculated based on the model, only reflects the objective correlation degree of the features and the reference label, and does not involve the conclusion of “personality traits”.

[0075] Output form: using standardized technical format, including: data table: presenting feature name, correlation value (such as 0-10 points) and order number in a structured table, which is convenient for subsequent system or manual reading; visual chart: displaying feature correlation degree distribution and order result through column chart, line chart and other graphical methods, intuitively presenting analysis data and avoiding subjective description.

[0076] The present application converts the abstract feature data of the multi-modal fusion analysis module into standardized results that can be directly used, which not only solves the problem of chaotic analysis result format and difficult reuse in the prior art, but also avoids the determination of non-technical conclusions through the design of “only outputting objective data”, which meets the objectivity requirement of the technical scheme.

[0077] In this embodiment, the whole system is implemented on a 64-bit Windows system, uses CUDA 12.1 and Pytorch 2.3.0, is equipped with 32 GB RAM, an Intel i7-6700 CPU and a NVIDIA GeForce GTX 2060 for model training and parallel inference. Through overall ablation experiment verification, the ablation of the three modalities of video, audio and text from high to low, the loss function and the combination of hyperparameters can achieve good behavior trait recognition accuracy. Further, the balanced loss function of the application can be extended to various information modalities from low to high, for example: network text, speech dialogue audio and face video, and the behavior trait recognition accuracy of the model is higher than that of the existing model using traditional loss training such as MSE.

[0078] In summary, the application can realize standardized collection of multi-modal data, comprehensive feature extraction and accurate correlation analysis, finally output objective feature sorting results which can be directly used for assisting decision-making, and improve the efficiency and accuracy of online interview data processing.

Claims

1. A method for analyzing and ranking behavioral features in interview videos based on multimodal fusion, characterized in that: Includes the following steps: Step S1: Process the input interview video data and extract visual modal features, audio modal features and text modal features respectively; Step S2: Input the extracted visual modal features, audio modal features, and text modal features into the multimodal fusion network model; wherein, the multimodal fusion network model is trained using a hybrid loss function that combines list sorting loss and pairwise sorting loss to optimize its sorting ability; Step S3: The multimodal fusion network model performs feature fusion based on the input features and then outputs behavioral trait scores in multiple preset dimensions. Step S4: Sort the behavioral trait scores of the multiple preset dimensions to obtain the sorting results.

2. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 1, characterized in that: For visual features, the multimodal fusion network model includes a background branch and a face branch; the background branch is used to extract individual images and then use a pre-trained neural network model for feature extraction; the face branch is used to extract sparse, fixed-step facial image sequences and then use a pre-trained neural network model to fuse the temporal features of the sequence frames.

3. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 2, characterized in that: The process of extracting features from the background branch and the facial branch is as follows: The interview video was segmented into multiple time segments using a video processing library; frames from points in the video stream were extracted as background environment images to generate a background tensor; multiple frames were uniformly extracted from the middle of the segments, and face detection and face alignment were performed on these consecutive frames using a face detection model; the frames were filled and resized to obtain standard-sized RGB consecutive frames to generate a face tensor; features were extracted from the background tensor and the face tensor using a temporal neural network model respectively.

4. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 1, characterized in that: For audio features, the multimodal fusion network model includes an audio branch; the audio branch uses a pre-trained VGGish embedding model to extract features, and then uses an LSTM model to fuse temporal features.

5. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 4, characterized in that: The process of extracting features from the audio branch is as follows: The audio stream was resampled to 16 kHz mono; the signal was sampled using a sliding window with a step size of 10 ms; the sampled sliding window signal was processed using the Mel frequency method, and a Hanning window was applied for windowing; after windowing and short-time Fourier transform, a 64-dimensional Mel filter bank was used to map the spectrum to the Mel scale, covering a frequency range of 125-7500 Hz; a small offset was applied to the generated spectrogram, and then a logarithmic function was used for stabilization to obtain a stable log-Mel spectrogram; the processed signal was input into the VGGish embedding model, which outputs the audio feature tensor.

6. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 1, characterized in that: For text features, the multimodal fusion network model includes a dialogue text branch; the dialogue text branch uses the BERT model as a feature extractor, and then uses two layers of bidirectional LSTM to fuse the features of the text sequence.

7. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 6, characterized in that: The process of extracting features from the dialogue text branches is as follows: Acronyms are replaced with full spellings or standardized forms; special characters and numbers are removed from the text using regular expressions, the text is converted to lowercase to standardize case and reduce word variation, then word tokenization is used to segment the text, stop words and punctuation are removed, and then NLTK lemmatization is used to restore the original form and semantics of words; finally, the maximum length of each text sequence is standardized, and BERT is used as a feature extractor to extract features.

8. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 1, characterized in that: The multimodal fusion network model uses a multilayer perceptron to fuse multimodal features, and finally uses a linear layer as the activation layer for the regression task to output the inference result. After the multimodal fusion network model outputs the inference result, a ranking algorithm is applied to obtain the prediction order of behavioral traits.

9. The method for analyzing and ranking behavioral features of interview videos based on multimodal fusion according to claim 1, characterized in that: The multimodal fusion network model is trained using regression with hybrid ranking loss. The regression training with hybrid ranking loss involves constructing an balanced loss function that carries both list-wise and pair-wise loss information, and using the balanced loss function to train the multimodal fusion network model.

10. The method for analyzing and ranking interview video behavioral features based on multimodal fusion according to claim 9, characterized in that: The process of constructing the balanced loss function carrying list-wise and pair-wise loss information is as follows: We obtain the weighted mean squared error by introducing weights into the loss function. : ； ； In the formula: It is the number of sampling points; , , , and These are training weights for different dimensions; A true label representing normalization; The prediction results of the representative model; It is a weight vector; List-wise loss information uses standardized discounted cumulative gain as a performance evaluation metric for the ranking algorithm: ； In the formula: The order of prediction for behavioral characteristics; The closer the predicted ranking of the multimodal fusion network model is to the actual ranking, the lower the ranking loss. The smaller: ； Pair-wise loss information uses Cosine Loss as the performance evaluation metric for the ranking algorithm: ； ； In the formula: Used to measure a given pair of tensors and The loss generated by the input; when the flag is set to positive, it indicates that the cosine similarity of the pair of tensors is calculated; when it is set to negative, the dissimilarity is calculated. It represents a controllable hard boundary; It is the parameter of the vector pair; Final training loss Combining the three losses, the expression is as follows: ； in: , and All of these are hyperparameters.

Images