Interview evaluation method and system based on multi-modal information

Abstract

The invention discloses an interview evaluation method and system based on multi-modal information, and relates to the technical field of artificial intelligence, and the method comprises the following steps: starting an interview client, synchronously collecting a video stream and an audio stream of a candidate by the interview client, and generating a corresponding interactive text stream in real time; wherein the interactive text stream, the video stream and the audio stream corresponding to the same timestamp form multi-modal interview data; and locally caching and preprocessing an interactive text stream, a video stream and an audio stream in the multi-modal interview data to obtain multi-modal time sequence data. According to the method, by setting the adaptive buffer area and the priority, the contradiction between delay and packet loss can be intelligently balanced while the alignment and integrity of the multi-modal time sequence data can be ensured, high-quality and high-reliability input can be provided for the deep neural network model, and finally, accurate and steady remote automatic interview evaluation is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an interview assessment method and system based on multimodal information. Background Technology

[0002] With the development of artificial intelligence technology, automated interview assessment systems based on multimodal information (such as video, audio, and text) have become an important tool for talent screening and evaluation. These systems analyze multi-dimensional data such as candidates' facial expressions, tone of voice, and language content during the interview process to build algorithmic models to predict their job competency, aiming to improve recruitment efficiency. However, when facing remote online interview scenarios, there is a long-standing core technical bottleneck that seriously restricts the validity of the assessment: the instability of network transmission, especially network latency, jitter and packet loss. These problems cause serious timing misalignment and quality damage to multimodal data streams when they are transmitted to the server. Therefore, there is an urgent need to propose an interview assessment method and system based on multimodal information to solve the above problems. Summary of the Invention

[0003] The purpose of this invention is to provide an interview assessment method and system based on multimodal information to address the shortcomings in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: an interview assessment method based on multimodal information, comprising the following steps: The interview client is launched, and it simultaneously collects the candidate's video stream and audio stream, and generates corresponding interactive text streams in real time; among them, the interactive text stream, video stream and audio stream corresponding to the same timestamp constitute multimodal interview data; Local caching and preprocessing of interactive text streams, video streams, and audio streams in multimodal interview data yield multimodal time-series data; The multimodal time series data is transmitted to the server. The server aligns and synchronizes the received multimodal time series data, and compensates and corrects the multimodal time series data. The compensated and corrected multimodal time series data are input into the trained deep neural network model to obtain the comprehensive feature vector of the candidate. By inputting the comprehensive feature vector into the competency model of the target position, predicted values ​​of multiple assessment indicators are obtained; and a multi-dimensional assessment report containing assessment indicators, key multimodal time series data fragments, and interpretability analysis is generated.

[0005] In a preferred embodiment, the interview client is launched, and the interview client synchronously collects the candidate's video stream and audio stream, and generates corresponding interactive text streams in real time, including: The interview client device synchronously collects the candidate's video and audio streams, and uses the integrated interview client device to perform real-time speech recognition on the audio stream to generate an interactive text stream corresponding to the audio stream; at the start of the collection, a start timestamp based on the same clock source is assigned to the video stream, audio stream and interactive text stream.

[0006] In a preferred embodiment, the step of locally caching and preprocessing the interactive text stream, video stream, and audio stream in the multimodal interview data to obtain multimodal time-series data includes: Dynamically monitor the current network status and adjust the encoding resolution, frame rate, and audio bitrate of the video stream accordingly. Perform face detection and tracking on the video stream, and extract face region image sequences and head pose parameters as primary visual features; The audio stream is used to calculate the temporal statistics of volume, speech rate and fundamental frequency as primary audio features; the interactive text stream is used directly as primary text features; among them, the primary visual features, primary audio features and primary text features are all temporal sequences, and the three together constitute multimodal temporal data.

[0007] In a preferred embodiment, the step of transmitting the multimodal time-series data to the server, and the server aligning and synchronizing the received multimodal time-series data, and compensating and correcting the multimodal time-series data, includes: Transmitting multimodal time-series data; Set up an adaptive buffer on the server side to align data from different modalities to the same timeline based on timestamps; Forward error correction and packet loss concealment algorithms are used to repair the audio stream, while temporal error concealment algorithms are used to repair the video stream.

[0008] In a preferred embodiment, the establishment of the adaptive buffer includes: Throughout the interview process, real-time network performance metrics are continuously monitored, including instantaneous one-way latency, instantaneous jitter, and instantaneous packet loss rate. The client marks the transmitted multimodal time-series data and assigns a content priority tag to each multimodal time-series data; the content priority tags include high priority, standard priority, and low priority. The adaptive buffer depth is dynamically adjusted based on real-time network performance metrics and the priority distribution of current multimodal time-series data. The dynamic adjustment includes a first adjustment rule, a second adjustment rule, and a third adjustment rule. Record the alignment success rate and the added latency after each adaptive buffer depth adjustment; The alignment success rate and the new processing delay are used as feedback signals and input into a control algorithm to dynamically adjust the proportional coefficients and thresholds involved in the first and second adjustment rules online, thereby achieving adaptive optimization based on the current network environment and data flow characteristics.

[0009] In a preferred embodiment, the step of inputting the compensated and corrected multimodal time-series data into a trained deep neural network model to obtain the candidate's comprehensive feature vector includes: Collect and preprocess historical multimodal interview data and corresponding historical comprehensive feature vectors. Use the preprocessed historical multimodal interview data and corresponding historical comprehensive feature vectors to train a deep neural network model and obtain a trained deep neural network model. The compensated and corrected multimodal time series data is input into the trained deep neural network model, and the trained deep neural network model outputs the comprehensive feature vector of the candidate.

[0010] In a preferred embodiment, the comprehensive feature vector is input into the competency model of the target position to obtain predicted values ​​for multiple assessment indicators; and a multi-dimensional assessment report is generated, including assessment indicators, key multimodal time-series data segments, and interpretability analysis, comprising: The comprehensive feature vector is input into the configurable capability model corresponding to the target position, and the predicted values ​​of multiple assessment indicators are calculated through the mapping relationship defined by the capability model. Meanwhile, based on the internal mechanism and capability model parameters of the deep neural network that generates comprehensive feature vectors, the key multimodal time-series data segments that affect the score are located. The predicted values ​​of multiple evaluation indicators and key multimodal time series data segments are correlated and analyzed to generate a multidimensional evaluation report.

[0011] This invention also provides an interview assessment system based on multimodal information, comprising: The acquisition module is used to launch the interview client, which synchronously acquires the candidate's video stream and audio stream, and generates the corresponding interactive text stream in real time; the interactive text stream, video stream and audio stream corresponding to the same timestamp constitute multimodal interview data; The preprocessing module is used to locally cache and preprocess the interactive text stream, video stream, and audio stream in the multimodal interview data to obtain multimodal time series data. The transmission module is used to transmit multimodal time series data to the server. The server aligns and synchronizes the received multimodal time series data, and compensates and corrects the multimodal time series data. The model processing module is used to input the compensated and corrected multimodal time series data into the trained deep neural network model to obtain the comprehensive feature vector of the candidate. The assessment module is used to input the comprehensive feature vector into the competency model of the target position to obtain the predicted values ​​of multiple assessment indicators; and generate a multi-dimensional assessment report that includes assessment indicators, key multimodal time series data segments, and interpretable analysis.

[0012] The technical effects and advantages provided by the present invention in the above technical solution are as follows: 1. By setting adaptive buffers and priorities, this invention can intelligently balance the contradiction between latency and packet loss while ensuring the alignment and integrity of multimodal time-series data. It can provide high-quality and reliable input for deep neural network models, and ultimately achieve accurate and robust remote automated interview assessment. 2. This invention achieves efficient feature extraction and deep fusion of multimodal time-series data through a trained deep neural network model, a dedicated encoder, and a cross-modal attention mechanism, thereby improving the accuracy and objectivity of candidate comprehensive ability assessment. By setting up the trained deep neural network model and the ability model, the correspondence between comprehensive feature vectors and job ability indicators can be accurately mapped, enabling the assessment results to have both quantitative accuracy and interpretability, providing a reliable basis for recruitment decisions. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0014] Figure 1 This is a flowchart of the method of the present invention.

[0015] Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1, please refer to Figure 1 As shown in this embodiment, an interview assessment method based on multimodal information includes the following steps: S1. Start the interview client. The interview client synchronously collects the candidate's video stream and audio stream, and generates the corresponding interactive text stream in real time. The interactive text stream, video stream, and audio stream corresponding to the same timestamp constitute multimodal interview data. In one embodiment, the interview client is launched, and the interview client synchronously collects the candidate's video stream and audio stream, and generates a corresponding interactive text stream in real time, including: The interview client device synchronously collects the candidate's video and audio streams, and uses the integrated interview client device to perform real-time speech recognition on the audio stream to generate an interactive text stream corresponding to the audio stream; at the start of the collection, a start timestamp based on the same clock source is assigned to the video stream, audio stream, and interactive text stream. It should be noted that the interview client deployed on the candidate's terminal is launched. This client has audio and video capture and real-time speech recognition capabilities. The client simultaneously activates the camera and microphone to capture the candidate's video and audio streams, respectively. The client integrates a lightweight real-time speech recognition (such as ASR) module to transcribe the captured audio stream sentence by sentence, generating a corresponding interactive text stream. The content of the text stream completely corresponds to the speech content of the audio stream. Using the client's local high-precision clock source as a reference, the same starting timestamp is assigned to the video stream, audio stream, and interactive text stream at the start of the capture. During the capture process, continuous timestamps are appended to the three-modal data at fixed time intervals (such as 100ms). Finally, the video frame, audio segment, and text segment corresponding to the same timestamp constitute a complete set of multimodal interview data, ensuring that the three-modal data are accurately aligned in the time dimension. Specifically, by using a high-precision local clock source on the client as a reference, the same starting timestamp is assigned to the three modal data and continuous timestamps are added at fixed intervals. This ensures that video frames, audio segments, and text segments corresponding to the same timestamp are accurately associated, solving the problem of temporal misalignment of multimodal time series data from the source of data acquisition. This provides accurate time dimension support for subsequent multimodal time series data alignment and feature fusion. Lightweight speech recognition and timestamp labeling are completed locally on the client, without relying on real-time processing on the server side. This reduces network bandwidth pressure and server load in the early stages of data transmission, while ensuring the real-time nature of data acquisition, thus adapting to remote interviews.

[0018] S2. Local caching and preprocessing of the interactive text stream, video stream, and audio stream in the multimodal interview data to obtain multimodal time series data; In one embodiment, the local caching and preprocessing of the interactive text stream, video stream, and audio stream in the multimodal interview data to obtain multimodal time-series data includes: The system dynamically monitors the current network status and adjusts the encoding resolution, frame rate, and bitrate of the video stream and audio stream accordingly. Face detection and tracking are performed on the video stream, extracting face region image sequences and head pose parameters as primary visual features. Temporal statistics of volume, speech rate, and fundamental frequency are calculated on the audio stream as primary audio features. Interactive text streams are directly used as primary text features. All three primary features—visual, audio, and text—are temporal sequences, collectively forming multimodal temporal data. It should be noted that the client has a built-in network status monitoring module that monitors current network metrics such as bandwidth, latency, and packet loss rate in real time. It dynamically adjusts audio and video encoding parameters based on network conditions: when network bandwidth is sufficient, it uses high resolution (e.g., 1080P), high frame rate (e.g., 30fps) to encode the video stream and high bitrate to encode the audio stream; when network bandwidth is insufficient, it automatically reduces video resolution (e.g., 720P / 480P), frame rate (e.g., 15fps), and audio bitrate to prioritize data transmission stability. During preprocessing and transmission, the client temporarily caches the original audio and video streams and interactive text streams locally to prevent data loss due to network fluctuations; the cached data can be retransmitted after the network recovers. The acquisition of primary visual features, primary audio features, and primary text features is as follows: On the client side, a lightweight face detection and tracking algorithm is executed on the video stream to locate the candidate's face region and crop out the face region image sequence; simultaneously, through facial landmark detection technology, the head pose parameters corresponding to the head pitch angle, yaw angle, and roll angle are calculated, and the face image sequence and head pose parameters are used as primary visual features; on the client side, temporal analysis is performed on the audio stream to calculate the temporal statistical values ​​(such as mean, variance, and extreme values) of volume (decibels), speech rate (syllables per second), and fundamental frequency (the basic frequency of speech, reflecting pitch), and these statistical values ​​are used as primary audio features; the interactive text stream is directly used as primary text features. Specifically, by deploying a network status monitoring module locally on the client side, it can sense indicators such as bandwidth, latency, and packet loss rate in real time. By adaptively adjusting audio and video encoding parameters, it can ensure the quality of high-resolution, high-frame-rate multimodal time-series data when the network is sufficient, providing rich information for subsequent feature extraction. When the network is insufficient, it can prioritize reducing encoding complexity to ensure the stability of data transmission and avoid data transmission interruptions caused by network fluctuations, thus adapting to the network environment of remote interviews.

[0019] S3. Transmit the multimodal time series data to the server. The server aligns and synchronizes the received multimodal time series data, and compensates and corrects the multimodal time series data. In one embodiment, the step of transmitting multimodal time-series data to a server, where the server aligns and synchronizes the received multimodal time-series data, and compensates and corrects the multimodal time-series data, includes: Transmit multimodal time-series data; set up an adaptive buffer on the server side to align data from different modalities to the same timeline based on timestamps; use forward error correction and packet loss concealment algorithms to repair audio streams, and use temporal error concealment algorithms to repair video streams; In one embodiment, the establishment of the adaptive buffer includes: Throughout the entire interview process, real-time network performance metrics are continuously monitored, including instantaneous one-way latency, instantaneous jitter, and instantaneous packet loss rate. The transmitted multimodal time-series data is tagged on the client side, and each multimodal time-series data is assigned a content priority label. These content priority labels include high priority, standard priority, and low priority. Based on the real-time network performance metrics and the priority distribution of the current multimodal time-series data, the adaptive buffer depth is dynamically adjusted. The alignment success rate and the added latency after each adaptive buffer depth adjustment are recorded. The alignment success rate and the added processing latency are used as feedback signals and input into a control algorithm to dynamically adjust the proportional coefficients and thresholds involved in the first and second adjustment rules online, achieving adaptive optimization based on the current network environment and data flow characteristics. It should be noted that high priority corresponds to multimodal temporal data where the voice activity detection result is active and the text contains words from the preset keyword library, or the facial key point movement amplitude exceeds the first threshold; standard priority corresponds to multimodal temporal data where the voice activity detection result is active but the high priority condition is not triggered; low priority corresponds to multimodal temporal data where the voice activity detection result is inactive; the alignment success rate is the percentage of data packets with timestamp deviations within the preset tolerance window out of the total number of processed data packets; the additional processing delay is the average delay increment introduced due to buffer waiting. The first adjustment rule, the second adjustment rule, and the third adjustment rule specifically include: The first adjustment rule is: when the instantaneous jitter value Jcurrent is detected to be greater than the basic jitter threshold Jbase, the buffer depth is increased according to the formula ΔB = k1 * (Jcurrent - Jbase), where ΔB is the amount of depth increase and k1 is the first adjustment coefficient; The second adjustment rule is as follows: when the instantaneous one-way delay Lcurrent is continuously exceeded by the warning threshold Lwarning for M consecutive monitoring cycles, the buffer depth is reduced according to the formula ΔB` = -k2 * (Lcurrent / Lwarning), where ΔB` is the amount of depth reduction, k2 is the second adjustment coefficient, and Lwarning is set to η times the initial average round-trip delay, where η is a constant greater than 1. The third adjustment rule is: when all high-priority multimodal time series data with timestamps in the adaptive buffer have arrived within the alignment window Talign, and only low-priority multimodal time series data is missing, the buffer is immediately released, and the ready multimodal time series data is submitted to the subsequent processing stage. The alignment window Talign is dynamically calculated based on the current buffer depth and jitter value. The essence of an adaptive buffer is as follows: In a real-time stream processing system, the buffer is a temporary storage area used to balance the speed difference between data producers and consumers. In interview and assessment scenarios, the producer is the client's data collection and sending, and the consumer is the server-side processing module. Due to network latency, jitter, and out-of-order arrival of multimodal time-series data, the buffer's functions include: temporarily storing arriving multimodal time-series data until it can be processed in the correct chronological order; providing a certain time window for reordering multimodal time-series data, packet loss detection and retransmission requests (if a retransmission mechanism exists), or error hiding; and smoothing the data flow so that subsequent processing modules can consume data at a stable rate, avoiding processing interruptions or quality degradation due to short-term data loss. Primary Feature Transmission: The client transmits the extracted primary visual features, primary audio features, and primary text features to the server according to the agreed communication protocol; Adaptive Buffer Alignment: The server sets up an adaptive buffer. After receiving the multimodal primary features from the client, it maps the visual, audio, and text features onto the same timeline based on the unified timestamp carried by each feature, ensuring that the three modal features at the same time point correspond one-to-one, thus solving the feature misalignment problem caused by transmission delay differences; Audio Data Repair: To address the packet loss and distortion problems that may occur during the transmission of audio modal time-series data, forward error correction (FEC) and packet loss hiding algorithms are used for repair; the forward error correction algorithm automatically recovers a small number of lost packets by adding redundant information to the transmitted data; the packet loss hiding algorithm generates approximate signals to fill the gaps by predicting the features of lost audio segments, ensuring the continuity of audio; Video Data Repair: To address transmission impairments in video streams ( (For example, frame loss and block artifacts) employ a temporal error concealment algorithm. This algorithm uses the visual features of adjacent frames before and after the lost frame to predict and reconstruct the content of the lost frame, ensuring the smoothness of the video sequence. The forward error correction algorithm process involves: when the client transmits primary audio feature data, it simultaneously generates redundant check data with a preset ratio and encapsulates it with the original audio feature data for transmission; after receiving the data, the server checks for packet loss using a checksum. If the number of lost packets is within the redundancy tolerance range, the server directly uses the redundant check data to reverse-engineer and recover the lost original audio feature data without initiating a retransmission request. The packet loss concealment algorithm process involves: after the server detects audio data packet loss, it extracts effective audio features (including the temporal patterns of fundamental frequency, volume, and speech rate) adjacent to the lost packet period. Based on a temporal continuity model, it predicts the feature parameters of the lost segment, generates filler data that smoothly connects with the preceding and following audio, and replaces the lost segment to ensure the continuity and integrity of the audio stream. Specifically, by continuously monitoring network performance metrics and prioritizing multimodal time-series data (high priority focuses on key information such as voice activity and core keywords), the buffer depth is dynamically adjusted. This avoids data alignment failures caused by network jitter and prevents critical information from being blocked while waiting for low-priority data through a high-priority data release mechanism, thus balancing the integrity and real-time performance of data processing. A feedback optimization mechanism uses alignment success rate and new processing latency as inputs to dynamically fine-tune adjustment coefficients and thresholds, enabling the buffer strategy to adapt to fluctuations in different network environments and significantly improving robustness in complex network scenarios. The forward error correction algorithm achieves automatic recovery of minor packet loss through redundant verification data. Furthermore, the packet loss hiding algorithm fills in missing segments through temporal prediction, and the temporal error hiding algorithm repairs video transmission damage. The three work together to ensure the transmission integrity of multimodal temporal data and avoid the impact of data loss or distortion on the accuracy of subsequent feature extraction. The buffer alignment mechanism based on a unified timestamp can accurately solve the feature misalignment problem caused by the difference in multimodal data transmission latency, providing a high-quality data foundation for context-aware deep feature fusion. The buffer can avoid processing interruptions caused by short-term data loss. At the same time, the combination of client-side local tagging priority and server-side dynamic adjustment balances the terminal transmission pressure and server processing efficiency, and can adapt to the complex network environment and real-time interaction requirements of remote interviews.

[0020] S4. Input the compensated and corrected multimodal time series data into the trained deep neural network model to obtain the comprehensive feature vector of the candidate. In one embodiment, the step of inputting the compensated and corrected multimodal time-series data into a trained deep neural network model to obtain the candidate's comprehensive feature vector includes: Collect and preprocess historical multimodal interview data and corresponding historical comprehensive feature vectors. Use the preprocessed historical multimodal interview data and corresponding historical comprehensive feature vectors to train a deep neural network model and obtain a trained deep neural network model. The compensated and corrected multimodal time series data is input into the trained deep neural network model, and the trained deep neural network model outputs the comprehensive feature vector of the candidate. It should be noted that the process of building and training the deep neural network model is as follows: Collect historical multimodal interview data (historical interactive text streams, historical video streams, and historical audio streams), and label the corresponding candidate ability scores, including communication ability scores and logical thinking scores, to construct a historical dataset; preprocess the historical dataset (consistent with steps S2 and S3) to obtain standardized multimodal primary features; construct a deep neural network model containing a visual encoder, audio encoder, text encoder, and cross-modal fusion layer; train the deep neural network model using the historical dataset, and optimize the model parameters through backpropagation until the error between the predicted score of the deep neural network model and the manually labeled score reaches a preset threshold, thus obtaining a trained deep neural network model; for the multimodal temporal data, high-level deep features are extracted from the synchronized video, audio, and interactive text data using the visual encoder, audio encoder, and text encoder, respectively; a cross-modal attention mechanism is used to model the dynamic correlation weights between the high-level deep features of vision, audio, and text, identifying intermodal consistency and conflict information; the multimodal high-level features weighted by attention weights are then fused to generate a comprehensive feature vector; Simultaneously refined primary visual features are input into a visual encoder (such as CNN or Vision Transformer) to extract facial expressions and body movements as high-level visual semantic features; primary audio features are input into an audio encoder (such as CNN or LSTM) to extract tone, speech rate, and emotional tendency as high-level audio semantic features; primary text features are input into a text encoder (such as BERT or GPT). A pre-trained language model is used to extract the semantics, logic, and keywords of the text as high-level text semantic features. A cross-modal attention mechanism is used as a fusion layer to model the dynamic association weights of the three-modal high-level features (high-level visual semantic features, high-level audio semantic features, and high-level text semantic features). This cross-modal attention mechanism can automatically identify consistency information (such as the consistency between confidence in textual expression and confidence in facial expression) and conflict information (such as the conflict between affirmation in textual expression and hesitation in tone), and assign differentiated weights to different modal features (such as assigning high weights to consistency information and low weights to conflict information). For example, the word "challenge" in the text will be given higher attention weights for its corresponding tone and facial expression. This process dynamically models the associations, complementarities, and even contradictions between the modalities, achieving true context awareness. Finally, the high-level visual semantic features, high-level audio semantic features, and high-level text semantic features, after being weighted by attention weights, are weighted and summed to generate a comprehensive feature vector. Specifically, by constructing a historical dataset through labeled candidate competency scores, it is easier to build and train deep neural network models; a dedicated encoder is used to extract high-level semantic features from visual, audio, and textual data, accurately capturing core multimodal information; a cross-modal attention mechanism dynamically models intermodal relationships and conflicts, differentiates weights, strengthens effective information, weakens contradictory interference, and achieves context-aware fusion; the final generated comprehensive feature vector fully integrates multi-dimensional information, objectively reflects the candidate's comprehensive qualities, provides high-quality feature support for subsequent job competency assessments, and improves the accuracy and objectivity of assessment results.

[0021] S5. Input the comprehensive feature vector into the competency model of the target position to obtain the predicted values ​​of multiple assessment indicators; and generate a multi-dimensional assessment report containing assessment indicators, key multimodal time series data segments, and interpretability analysis. In one embodiment, the comprehensive feature vector is input into the competency model of the target position to obtain predicted values ​​for multiple assessment indicators; and a multi-dimensional assessment report containing assessment indicators, key multimodal time-series data fragments, and interpretability analysis is generated, including: The comprehensive feature vector is input into the configurable capability model corresponding to the target position. The predicted values ​​of multiple assessment indicators are calculated through the mapping relationship defined by the capability model. At the same time, based on the internal mechanism of the deep neural network that generates the comprehensive feature vector and the parameters of the capability model, the key multimodal time series data segments that affect the score are located. The predicted values ​​of multiple assessment indicators and key multimodal time series data segments are correlated and analyzed to generate a multidimensional assessment report. It should be noted that the target job competency model is configured as follows: Configurable competency models are built for different positions (such as technical R&D, sales, and management). The competency model defines the assessment indicators (competency indicators) required for the target job. For example, the assessment indicators include logical thinking and problem-solving ability for technical R&D positions, and communication and customer orientation ability for sales positions. A mapping relationship between comprehensive feature vectors and each competency indicator is established (the competency model realizes the conversion from feature vectors to indicator scores). The method for predicting assessment indicators is as follows: input the comprehensive feature vector into the competency model corresponding to the target position, and calculate the predicted value of each assessment indicator (such as logical thinking score of 85 points and communication expression score of 78 points) through the mapping relationship built into the competency model. The predicted value can quantify the matching degree between the candidate and the position. The key multimodal time-series data segment localization method is as follows: based on the distribution of cross-modal attention weights within the deep neural network along the time dimension, and the ability model parameters, the key time segments that have the greatest impact on the predicted values ​​of the final evaluation indicators are located back to the source, and the corresponding multimodal time-series data is extracted as the key multimodal time-series data segments; for example, a high score in logical thinking may correspond to a segment where the candidate's text is logically rigorous and the tone is firm when explaining technical solutions; a low score in communication expression may correspond to a segment where the candidate's body language is stiff and the speaking speed is too fast when answering questions. The multi-dimensional assessment report is generated by performing correlation analysis on the predicted values ​​of each competency indicator and key multimodal time-series data segments (such as video segments, audio segments, and text segments) to generate a multi-dimensional assessment report that includes assessment indicator scores, indicator interpretations, key data segment displays, interpretability analysis, and job matching suggestions. The report not only provides quantitative scores but also explains the basis for the scores, thereby improving the credibility of the assessment results. Specifically, by constructing configurable competency models for different positions, the core competency requirements of different positions are precisely matched (such as logical thinking for technical positions and communication skills for sales positions), achieving personalized adaptation of assessment indicators and improving the pertinence of job matching evaluation. Based on the clear mapping relationship between comprehensive feature vectors and competency indicators (such as fully connected layer transformation), quantitative assessment scores are output, shifting candidate competency assessment from qualitative to quantitative, enhancing the objectivity and comparability of assessment results. Based on the attention weights and competency model parameters of deep neural networks, key multimodal time-series data segments are back-tracked, providing intuitive audio-visual and textual evidence for each score, solving the "black box" problem of traditional AI assessments. The generated multi-dimensional report integrates quantitative scores, indicator interpretations, key segment displays, interpretability analysis, and job matching suggestions, providing both clear assessment conclusions and a clear explanation of the scoring logic, improving the credibility and persuasiveness of the assessment results, providing interviewers with comprehensive and accurate decision support, and helping candidates understand their strengths and weaknesses, balancing recruitment decision-making efficiency with candidate experience.

[0022] Example 2, please refer to Figure 2 As shown in this embodiment, an interview assessment system based on multimodal information includes: The acquisition module is used to launch the interview client, which synchronously acquires the candidate's video stream and audio stream, and generates the corresponding interactive text stream in real time; the interactive text stream, video stream and audio stream corresponding to the same timestamp constitute multimodal interview data; The preprocessing module is used to locally cache and preprocess the interactive text stream, video stream, and audio stream in the multimodal interview data to obtain multimodal time series data. The transmission module is used to transmit multimodal time series data to the server. The server aligns and synchronizes the received multimodal time series data, and compensates and corrects the multimodal time series data. The model processing module is used to input the compensated and corrected multimodal time series data into the trained deep neural network model to obtain the comprehensive feature vector of the candidate. The assessment module is used to input the comprehensive feature vector into the competency model of the target position to obtain the predicted values ​​of multiple assessment indicators; and generate a multi-dimensional assessment report that includes assessment indicators, key multimodal time series data segments, and interpretable analysis.

[0023] In this embodiment, by setting adaptive buffers and priorities, the system can intelligently balance latency and packet loss while ensuring the alignment and integrity of multimodal time-series data. This provides high-quality and reliable input for the deep neural network model, ultimately achieving accurate and robust remote automated interview assessment. The trained deep neural network model, through a dedicated encoder and cross-modal attention mechanism, achieves efficient feature extraction and deep fusion of multimodal time-series data, improving the accuracy and objectivity of candidate comprehensive ability assessment. Furthermore, the training deep neural network model and ability model settings accurately map the correspondence between comprehensive feature vectors and job ability indicators, enabling the assessment results to possess both quantitative accuracy and interpretability, providing a reliable basis for recruitment decisions.

[0024] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for interview assessment based on multimodal information, characterized in that, Includes the following steps: The interview client is launched, and it simultaneously collects the candidate's video stream and audio stream, and generates corresponding interactive text streams in real time; among them, the interactive text stream, video stream and audio stream corresponding to the same timestamp constitute multimodal interview data; Local caching and preprocessing of interactive text streams, video streams, and audio streams in multimodal interview data yield multimodal time-series data; The multimodal time series data is transmitted to the server. The server aligns and synchronizes the received multimodal time series data, and compensates and corrects the multimodal time series data. The compensated and corrected multimodal time series data are input into the trained deep neural network model to obtain the comprehensive feature vector of the candidate. By inputting the comprehensive feature vector into the competency model of the target position, predicted values ​​of multiple assessment indicators are obtained; and a multi-dimensional assessment report containing assessment indicators, key multimodal time series data fragments, and interpretability analysis is generated.

2. The interview assessment method based on multimodal information according to claim 1, characterized in that, The interview client is launched, and the interview client synchronously collects the candidate's video stream and audio stream, and generates corresponding interactive text streams in real time, including; The interview client device synchronously collects the candidate's video and audio streams, and uses the integrated interview client device to perform real-time speech recognition on the audio stream to generate an interactive text stream corresponding to the audio stream; at the start of the collection, a start timestamp based on the same clock source is assigned to the video stream, audio stream and interactive text stream.

3. The interview assessment method based on multimodal information according to claim 1, characterized in that, The process of locally caching and preprocessing the interactive text stream, video stream, and audio stream in the multimodal interview data to obtain multimodal time-series data includes: Dynamically monitor the current network status and adjust the encoding resolution, frame rate, and audio bitrate of the video stream accordingly. Perform face detection and tracking on the video stream, and extract face region image sequences and head pose parameters as primary visual features; The volume, speech rate, and fundamental frequency of the audio stream are calculated as primary audio features; the interactive text stream is directly used as primary text features; among them, the primary visual features, primary audio features, and primary text features are all time-series, and the three together constitute multimodal time-series data.

4. The interview assessment method based on multimodal information according to claim 1, characterized in that, The process of transmitting multimodal time-series data to the server, whereby the server aligns and synchronizes the received multimodal time-series data, and compensates and corrects the data, includes: Transmitting multimodal time-series data; Set up an adaptive buffer on the server side to align data from different modalities to the same timeline based on timestamps; Forward error correction and packet loss concealment algorithms are used to repair the audio stream, while temporal error concealment algorithms are used to repair the video stream.

5. The interview assessment method based on multimodal information according to claim 4, characterized in that, The establishment of the adaptive buffer includes: Throughout the interview process, real-time network performance metrics are continuously monitored, including instantaneous one-way latency, instantaneous jitter, and instantaneous packet loss rate. The client marks the transmitted multimodal time-series data and assigns a content priority tag to each multimodal time-series data; the content priority tags include high priority, standard priority, and low priority. The adaptive buffer depth is dynamically adjusted based on real-time network performance metrics and the priority distribution of current multimodal time-series data. The dynamic adjustment includes a first adjustment rule, a second adjustment rule, and a third adjustment rule. Record the alignment success rate and the added latency after each adaptive buffer depth adjustment; The alignment success rate and the new processing delay are used as feedback signals and input into a control algorithm to dynamically adjust the proportional coefficients and thresholds involved in the first and second adjustment rules online, thereby achieving adaptive optimization based on the current network environment and data flow characteristics.

6. The interview assessment method based on multimodal information according to claim 1, characterized in that, The step of inputting the compensated and corrected multimodal time-series data into the trained deep neural network model to obtain the candidate's comprehensive feature vector includes: Collect and preprocess historical multimodal interview data and corresponding historical comprehensive feature vectors. Use the preprocessed historical multimodal interview data and corresponding historical comprehensive feature vectors to train a deep neural network model and obtain a trained deep neural network model. The compensated and corrected multimodal time series data is input into the trained deep neural network model, and the trained deep neural network model outputs the comprehensive feature vector of the candidate.

7. The interview assessment method based on multimodal information according to claim 1, characterized in that, The process involves inputting a comprehensive feature vector into the competency model of the target position to obtain predicted values ​​for multiple assessment indicators; and generating a multi-dimensional assessment report that includes assessment indicators, key multimodal time-series data segments, and interpretable analysis, including: The comprehensive feature vector is input into the configurable capability model corresponding to the target position, and the predicted values ​​of multiple assessment indicators are calculated through the mapping relationship defined by the capability model. Meanwhile, based on the internal mechanism and capability model parameters of the deep neural network that generates comprehensive feature vectors, the key multimodal time-series data segments that affect the score are located. The predicted values ​​of multiple evaluation indicators and key multimodal time series data segments are correlated and analyzed to generate a multidimensional evaluation report.

8. A multimodal information-based interview assessment system, used to implement the multimodal information-based interview assessment method according to any one of claims 1-7, characterized in that, include: The acquisition module is used to launch the interview client, which synchronously acquires the candidate's video stream and audio stream, and generates the corresponding interactive text stream in real time; the interactive text stream, video stream and audio stream corresponding to the same timestamp constitute multimodal interview data; The preprocessing module is used to locally cache and preprocess the interactive text stream, video stream, and audio stream in the multimodal interview data to obtain multimodal time series data. The transmission module is used to transmit multimodal time series data to the server. The server aligns and synchronizes the received multimodal time series data, and compensates and corrects the multimodal time series data. The model processing module is used to input the compensated and corrected multimodal time series data into the trained deep neural network model to obtain the comprehensive feature vector of the candidate. The assessment module is used to input the comprehensive feature vector into the competency model of the target position to obtain the predicted values ​​of multiple assessment indicators; and generate a multi-dimensional assessment report that includes assessment indicators, key multimodal time series data segments, and interpretable analysis.

Images