Intelligent interview evaluation and feedback system based on multi-modal data fusion

Abstract

The present application relates to a kind of intelligent interview evaluation and feedback system based on multi-modal data fusion, specifically relates to data processing field, by meta-learning mechanism dynamic perception interview scene and generate initial fusion weight, subsequently utilize the complex interaction relationship between modalities modeled by graph neural network to carry out fine-grained correction to weight, so as to significantly improve the accuracy and scene adaptability of multi-modal evaluation, further introduce reinforcement learning, link evaluation decision and long-term performance of talents, continuously optimize weight generation strategy, ensure that evaluation standard and business goal are aligned, finally, through closed-loop iteration mechanism, make the whole scheme can be updated automatically according to new data and performance feedback, continuous evolution, with strong self-optimizing ability and long-term robustness, realize the fundamental change from static rule to dynamic intelligent decision.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to an intelligent interview assessment and feedback system based on multimodal data fusion. Background Technology

[0002] With the widespread adoption of remote work and the normalization of global recruitment, intelligent interview assessment systems based on video conferencing platforms have become an important tool for multinational companies to initially screen talent. These systems aim to assist human resources departments in conducting efficient and objective preliminary assessments by synchronously collecting and analyzing multi-dimensional behavioral data of candidates during the interview process. Typical systems comprehensively utilize speech recognition technology to analyze the text content and tone characteristics of candidates' answers, capture their facial micro-expressions, gaze direction, and body gestures through computer vision algorithms, and integrate their answer logic and response time on the interactive interface. However, in real cross-cultural remote interview scenarios, candidates' external behavior often produces complex and even contradictory signals due to differences in their cultural background, personal habits, and the nature of the interview scenario. For example, a senior technical expert from an East Asian cultural background may use concise language, restrained expressions, and few body movements when answering professional questions, but their textual logic is extremely meticulous; while another candidate may speak fluently and persuasively, but their technical solution explanation lacks depth. This common phenomenon of "confident in language but stiff in body" or "flat in expression but meticulous in logic" poses a serious challenge to the comprehensive judgment ability of assessment systems.

[0003] Currently, multimodal information fusion technologies for intelligent interview scenarios have significant shortcomings in addressing the aforementioned conflicts in evaluation conclusions. Mainstream solutions typically rely on pre-defined static weighting fusion strategies. For example, the system assigns a fixed 40% weight to the speech modality, 30% to the video modality, and 30% to the text-based logic modality, then linearly weights and sums the evaluation scores of each modality to obtain the final score. Another common method is to use shallow machine learning models such as support vector machines for simple fusion at the decision layer. The core flaw of these methods lies in their rigid and context-independent fusion mechanisms, failing to adaptively adjust to the dynamic situations during the interview process. Specifically, in stress interview scenarios, a candidate's stress-induced micro-expressions and tone changes may be more valuable for evaluation than their verbal content; while in technical defense scenarios… In this context, the logical rigor and depth of knowledge in answering questions should take precedence. Furthermore, existing fusion models struggle to effectively model and utilize deep, non-linear interactions between different modalities. For instance, how do emphatic gestures synergize with keywords in speech to enhance persuasiveness? More critically, due to the lack of large-scale, finely labeled cross-cultural interview datasets, existing models struggle to learn universal, culture-independent core competency representations. This leads to their evaluation criteria potentially biased towards the mainstream cultural behavioral patterns from which the training data originates, raising fairness concerns in global applications. Therefore, designing an adaptive fusion mechanism that can perceive interview context in real time, dynamically coordinate multimodal evidence, and intelligently resolve intermodal evaluation conflicts has become a key technical bottleneck for improving the accuracy, fairness, and scenario robustness of intelligent interview systems. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing an intelligent interview assessment and feedback system based on multimodal data fusion. It solves the problems mentioned in the background art through a meta-learning dynamic weight generation module, a graph neural network weight correction module, a reinforcement learning strategy optimization module, and a model closed-loop iterative update module.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: Specifically, it includes: a meta-learning dynamic weight generation module, a graph neural network weight correction module, a reinforcement learning strategy optimization module, and a model closed-loop iterative update module connected in sequence, wherein; Meta-learning dynamic weight generation module: In response to the start of the interview evaluation process for candidates, it obtains the interview context feature vectors that have been collected in real time and have undergone preliminary structuring processing from the upstream interface. The interview context feature vectors are then input into a meta-learning neural network model that has been pre-trained using the meta-learning paradigm. The meta-learning neural network model performs calculations and outputs a set of initial basic weights corresponding to the interview context feature vectors for multimodal fusion. The initial basic weight set contains weight values ​​corresponding to the speech modality, video modality, and text modality, respectively. The graph neural network weight correction module receives the initial basic weight set and the feature vectors corresponding to speech, video, and text from the multimodal feature extraction device. Based on the initial basic weight set, it constructs a heterogeneous graph of modal relationships with each modality as a node. It calls a pre-configured multi-layer graph attention network to perform node feature aggregation and edge weight update calculation on the heterogeneous graph of modal relationships. Based on the representation state of each node in the graph after the update calculation, it recalculates a set of target fusion weights adjusted by the intermodal interaction relationship. The reinforcement learning strategy optimization module, after completing the evaluation of a batch of candidates and obtaining their subsequent job performance data, is triggered to execute a strategy update operation. It uses multiple initial basic weight sets stored in the historical database, generated by multiple rounds of historical evaluations and output by the meta-learning dynamic weight generation module, as the action data set; multiple interview context feature vectors stored in the historical database, corresponding to the multiple initial basic weight sets, as the state data set; and the evaluation quality reward value calculated based on the job performance data as the reward data. It constructs a strategy gradient optimization objective and uses a proximal policy optimization algorithm to iteratively update the parameters of the meta-learning neural network model in the meta-learning dynamic weight generation module. Model closed-loop iterative update module: During operation, the interview context feature vector, modality feature vectors, target fusion weight set, and subsequently acquired job performance data generated from each completed interview evaluation process are continuously added to the historical database. When the preset trigger conditions are met, the model retraining process is automatically started, and the parameters of the meta-learning neural network model are updated using the interview context feature vectors, modality feature vectors, target fusion weight set, and job performance data accumulated in the historical database. In a preferred embodiment, the specific process of obtaining the interview context feature vector, which has been collected in real time and preliminarily structured, from the upstream interface in the meta-learning dynamic weight generation module includes: When the interview evaluation process for a single candidate is initiated, the system simultaneously receives speech recognition text and intonation analysis tags from the upstream speech processing unit, candidate facial micro-expression codes and body movement feature sequences from the upstream visual analysis unit, and answer timing and content logic tags from the upstream interaction analysis unit. The above speech recognition text and intonation analysis labels, candidate facial micro-expression codes and body movement feature sequences, answer timing and content logic tags are vectorized to form corresponding speech feature vectors, visual feature vectors and text interaction feature vectors. Subsequently, the aforementioned speech feature vector, visual feature vector, and text interaction feature vector are aligned and concatenated with the preset current interview question type code and interview stage identifier to generate a unified, high-dimensional interview context feature vector, which serves as the input to the meta-learning neural network model.

[0006] In a preferred embodiment, the specific process by which the meta-learning neural network model calculates and outputs the initial basic weight set is as follows: The meta-learning neural network model maintains a trainable scene prototype memory matrix, which stores K scene prototype vectors. The meta-learning neural network model first calculates the cosine similarity between the interview context feature vector and each scene prototype vector in the scene prototype memory matrix, and obtains K similarity scores. Next, the meta-learning neural network model performs a normalized exponential operation on the K similarity scores, thereby transforming the K similarity scores into a set of K attention weights, where each attention weight corresponds to a scene prototype vector; Subsequently, the meta-learning neural network model calculates a weighted sum of K scene prototype vectors, where the weighting coefficient of each scene prototype vector is its corresponding attention weight, and the result of the weighted sum is defined as the scene context encoding vector. Subsequently, the meta-learning neural network model simultaneously inputs the scene context encoding vector into a gated signal generation branch and a basic weight bias generation branch. The gated signal generation branch passes through a fully connected layer containing trainable weight and bias parameters, and applies a non-linear activation function to the output of this fully connected layer, outputting a three-dimensional gated signal vector. The basic weight bias generation branch passes through another fully connected layer containing trainable weight and bias parameters, outputting a three-dimensional basic weight bias vector. Then, the meta-learning neural network model performs element-wise multiplication of the gated signal vector and the basic weight bias vector to obtain a set of modulated original weight values ​​containing three values. Finally, the meta-learning neural network model performs a normalized exponential operation on a set of modulated raw weight values ​​containing three values, thereby outputting an initial set of basic weights containing three basic weight values ​​corresponding to the speech modality, video modality, and text modality, respectively, where the three basic weight values ​​are named wspeech, wvideo, and wtext.

[0007] In a preferred embodiment, the specific operation of constructing a heterogeneous graph of modal relationships with each modality as a node in the graph neural network weight correction module is as follows: The system receives an initial basic weight set from the meta-learning dynamic weight generation module, which includes three weight values: w_speech, w_video, and w_text. It also receives speech feature vectors, video feature vectors, and text feature vectors from the multimodal feature extraction device. Each of the three modalities (speech, video, and text) is treated as a node in the graph, with the initial node feature vector set to the corresponding modal feature vector. Two directed edges with opposite directions are established between any two nodes of different modalities, forming a fully connected directed graph structure. The initial edge weight calculation process for a directed edge from the first modal node to the second modal node is as follows: First, the natural logarithm of the weight value corresponding to the first modal node is calculated and summed with a preset positive protection constant to obtain the first intermediate value. Simultaneously, the cosine similarity between the feature vectors of the first modal node and the feature vectors of the second modal node is calculated to obtain the second intermediate value. Next, the first intermediate value is added to the result of multiplying the second intermediate value by a preset mixing coefficient to obtain the third intermediate value. Finally, using the second modal node as the normalization target, an exponential function with the natural constant e as the base is calculated for all third intermediate values ​​pointing to the second modal node. The result of the exponential function calculation for each third intermediate value is divided by the sum of the exponential function calculation results for all third intermediate values ​​pointing to the second modal node. The resulting normalized result is the initial edge weight of the directed connection edge.

[0008] In a preferred embodiment, the process of calling a pre-configured multi-layer graph attention network to perform node feature aggregation and edge weight update calculation on the heterogeneous graph of modal relationships, and recalculating the target fusion weight set based on the representation state of each node in the graph after the update calculation, specifically involves: The multi-layer graph attention network contains L computational layers. In each layer, for each target node in the graph, a message vector from each source node to that target node is calculated. This message vector is obtained by multiplying the current feature vector of the source node by a trainable message transformation matrix. Next, the attention coefficient from each source node to that target node is calculated. The calculation process for this attention coefficient is as follows: The source node's current feature vector is processed through a trainable first attention transformation matrix, and the target node's current feature vector is processed through a trainable second attention transformation matrix. The results of these two processes are concatenated with the edge weights of the current connection edges. The concatenated vector is then multiplied by a trainable attention weight vector and processed through a LeakyReLU non-linear activation function to obtain unnormalized attention coefficients. Next, an exponential function with the natural constant e is calculated for all unnormalized attention coefficients pointing to the target node. The result of the exponential function calculation for each unnormalized attention coefficient is divided by the sum of the exponential function calculation results for all unnormalized attention coefficients pointing to the target node to obtain the normalized attention coefficients. Finally, the updated node feature vector of the target node is obtained as follows: First, all message vectors pointing to the target node are weighted and summed according to their corresponding normalized attention coefficients to obtain an aggregated message vector. Then, this aggregated message vector and the original node feature vector of the target node are input into a gated recurrent unit. The gated recurrent unit controls the degree of forgetting of the original features of the target node and the degree of fusion of the aggregated message vector through its internal gating mechanism. At the same time, the edge weights of the connection edges from each source node to the target node are updated by adding a preset update rate to the current edge weight and multiplying it by the corresponding normalized attention coefficient, and then performing layer normalization on the sum. After completing the L-layer calculation, for each modal node, its importance score is calculated. The calculation process of the importance score is as follows: First, the final node feature vector of the node is input into a multilayer perceptron. The multilayer perceptron contains a hidden layer to extract an importance scalar value representing the state strength of the node from the final node feature vector. Simultaneously, the network influence score of the node is calculated as follows: the normalized attention coefficients of the node pointing to the other two modal nodes in the L-level calculation are added together to obtain the sum of output attention; then the normalized attention coefficients of the other two modal nodes pointing to the node in the L-level calculation are added together to obtain the sum of received attention; finally, the sum of output attention is subtracted from the sum of received attention, and the difference is the network influence score. Next, the importance scalar value output by the multilayer perceptron is multiplied by a preset balance coefficient and the network influence score, and the product is added to obtain the importance score of the node. Finally, an exponential function with the natural constant e is calculated for the importance scores of the three nodes of speech, video and text. The result of the exponential function calculation of the importance score of each node is divided by the sum of the exponential function calculation results of the importance scores of the three nodes, and the three normalized values ​​obtained constitute the target fusion weight set.

[0009] In a preferred embodiment, the specific process of calculating the evaluation quality reward value based on job performance data in the reinforcement learning strategy optimization module is as follows: For each set of initial basic weights output by the meta-learning dynamic weight generation module and its corresponding interview context feature vector stored in the historical database, the target fusion weight set generated by the graph neural network weight correction module and associated with the interview context feature vector is first retrieved from the historical database. Then, the Jensen-Shannon divergence between the initial basic weight set and the target fusion weight set is calculated, and the negative of the divergence value is taken as the alignment reward component. Simultaneously, based on externally acquired candidate job performance data and the final interview evaluation score calculated by weighting the target fusion weight set, the Spearman rank correlation coefficient between the two is calculated. This Spearman rank correlation coefficient is then transformed using a preset exponential scaling function. The exponential scaling function is an exponential function with the natural constant e as its base, where the exponent is a preset scaling factor multiplied by the absolute value of the Spearman rank correlation coefficient. The result is then multiplied by the sign function value of the Spearman rank correlation coefficient to obtain the transformed value, which is used as the predicted reward component. Finally, a preset alignment reward weight is multiplied by the alignment reward component, and a preset predicted reward weight is multiplied by the predicted reward component. The sum is the evaluation quality reward value for this set of historical data.

[0010] In a preferred embodiment, the specific process of constructing a policy gradient optimization objective and iteratively updating it using a proximal policy optimization algorithm is as follows: The meta-learning neural network model in the meta-learning dynamic weight generation module is defined as the policy network to be optimized. Multiple interview context feature vectors stored in the historical database are defined as the state data set. The initial basic weight set output by the meta-learning dynamic weight generation module, corresponding to each interview context feature vector, is defined as the action data set. An evaluation quality reward value, obtained according to the aforementioned evaluation quality reward value calculation process, corresponding to each state-action pair, is defined as the reward data for that data pair. Based on the state data set, action data set, and corresponding multiple reward data, an alternative objective function for proximal policy optimization is constructed and solved. This objective function is defined as calculating the following expression and its expected value for multiple state-action pairs sampled from historical data: First, calculate the policy update ratio, defined as the ratio of the probability that the current policy network outputs the corresponding historical action (i.e., the initial basic weight set) under a given state (i.e., the interview context feature vector), to the probability that the old policy network outputs the same action under the same state before parameter updates. The algorithm iteratively calculates the following: First, it calculates the advantage function estimate using a generalized advantage estimation algorithm. This algorithm iteratively calculates the current reward value, the state value estimates of the current and next states from an independent value network, a preset discount factor, and a preset generalized advantage estimation parameter. Second, it multiplies the policy update ratio by the advantage function estimate to obtain the first product term. Third, it prunes the policy update ratio and multiplies the pruned ratio by the same advantage function estimate to obtain the second product term. Finally, it compares the first and second product terms and takes the smaller value as the contribution of the state action to the objective function. The average of the contributions calculated from all sampled data is used to obtain the value of the substitution objective function. The gradient ascent method is used to iteratively optimize the parameters of the policy network to maximize the substitution objective function. The complete calculation and optimization process, from calculating the policy update ratio to maximizing the substitution objective function to optimize the policy network parameters using the gradient ascent method, is defined as the policy gradient optimization process.

[0011] In a preferred embodiment, the iterative update process specifically includes collaboratively updating an independent value network: In each iteration of optimization, an independent value network is maintained and updated simultaneously. This value network is used to estimate the corresponding state value based on the input interview context feature vector. Before updating the policy network parameters, the value network is first trained with the goal of minimizing a mean squared error loss function. This loss function is calculated as follows: for a batch of sampled state data, the square of the difference between the value network's predicted value for each state and the estimated actual target reward for that state is calculated, and then the average of these squared values ​​is calculated. This loss function is then optimized using the backpropagation algorithm to update the parameters of the value network. Next, the more accurate state value estimate provided by the updated value network is used to recalculate the advantage function estimate. Finally, based on the recalculated advantage function estimate, the policy gradient optimization process is performed to update the parameters of the meta-learning neural network model. After completing one round of parameter iteration updates, the updated parameters of the meta-learning neural network model are synchronized to the meta-learning dynamic weight generation module. After completing parameter synchronization, the reinforcement learning policy optimization module uses a pre-divided and preserved historical validation dataset to validate the optimized policy. The validation method is as follows: using the old policy network before parameter update and the new policy network after parameter update, respectively, the interview context feature vectors in the historical validation dataset are processed to generate corresponding initial basic weight sets, and the average evaluation quality reward value corresponding to these initial basic weight sets is calculated according to the evaluation quality reward value calculation process; by comparing the average evaluation quality reward values ​​obtained by the old and new policies on the validation set, the effectiveness of the policy optimization is confirmed.

[0012] In a preferred embodiment, the model closed-loop iterative update module automatically initiates the model retraining process when a preset trigger condition is met. The specific determination process for the trigger condition includes: During operation, a sliding data window containing records of the most recent interview assessments is dynamically maintained. Two key performance indicators within this sliding data window are calculated periodically. The first key performance indicator is the average decision confidence score, which is calculated as follows: For each interview assessment within the window, the Jensen-Shannon divergence between the initial basic weight set output by the meta-learning dynamic weight generation module and the target fusion weight set output by the graph neural network weight correction module is first calculated. Then, the Jensen-Shannon divergence value is subtracted from the value to obtain the decision confidence score for a single assessment. Finally, the arithmetic mean of the decision confidence scores of all records within the window is taken, and the resulting value is the average decision confidence score. The second key performance indicator is average prediction consistency, which is calculated as follows: For the evaluation records within the sliding data window for which subsequent job performance data has been obtained, calculate the Spearman rank correlation coefficient between the final interview evaluation score obtained by weighting the target fusion weight set and the job performance data. Then, apply an exponential moving average algorithm to these Spearman rank correlation coefficients to calculate their smoothed moving average value. The resulting value is the average prediction consistency. The system continuously monitors changes in the average decision confidence and average prediction consistency relative to their respective historical baselines. When at least one of the average decision confidence or average prediction consistency is detected to have decreased by more than a preset sensitivity threshold compared to its corresponding historical baseline, the system determines that the preset trigger condition is met and automatically initiates the model retraining process. Furthermore, if the total number of newly added interview evaluation records in the historical database reaches a preset quantity threshold, regardless of whether the performance indicators change, the system directly determines that the trigger condition is met and initiates retraining.

[0013] In a preferred embodiment, the specific process of updating the parameters of the meta-learning neural network model using data accumulated in the historical database is as follows: Once the model retraining process is initiated, stratified sampling is first performed from the historical database to construct a balanced training dataset. Then, based on the parameters of the old meta-learning neural network model, incremental meta-learning fine-tuning with an elastic weight consolidation strategy is performed on this training dataset. This process first calculates the importance weight of each parameter in the old model parameters for the existing knowledge. This is done by taking the gradient of the probability of the old model predicting the initial set of base weights with respect to each model parameter on the training dataset and approximating the expected value of the squared gradient. Next, a series of meta-learning tasks are constructed based on the training dataset, each task containing a support set and a query set. During the inner loop adaptation and outer loop meta-parameter update of meta-learning, an additional regularization penalty term is added to the standard meta-learning loss function. This term is proportional to the importance weight and also proportional to the square of the difference between the new and old model parameters. After completing the incremental training, a new set of model parameters is obtained. Then, a progressive update strategy is used to weightedly fuse the new model parameters with the original old model parameters to generate the final updated model candidate parameters. Subsequently, the performance of the updated model candidate parameters is evaluated using an independent validation dataset. The comprehensive performance metric used for evaluation is the weighted sum of the average decision confidence and average prediction consistency on the validation dataset. The updated model candidate parameters are only officially deployed and replace the original model parameters in the meta-learning dynamic weight generation module if the relative improvement of the comprehensive performance metric of the updated model candidate parameters on the validation set compared to the performance of the old model parameters on the same validation set exceeds a preset minimum improvement threshold. Otherwise, the original model parameters are retained, and the retraining process is considered not to have generated a valid update.

[0014] The beneficial effects of this invention are as follows: By dynamically perceiving the interview scenario and generating initial fusion weights through a meta-learning mechanism, and then using graph neural networks to model the complex interaction relationships between modalities to make fine-grained corrections to the weights, the accuracy and scenario adaptability of multimodal assessment are significantly improved. Furthermore, reinforcement learning is introduced to link assessment decisions with long-term talent performance, continuously optimize the weight generation strategy, and ensure that assessment standards are aligned with business objectives. Finally, through a closed-loop iteration mechanism, the entire solution can be automatically updated and continuously evolved based on new data and performance feedback, possessing strong self-optimization capabilities and long-term robustness, and realizing a fundamental transformation from static rules to dynamic intelligent decision-making. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0018] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0019] Example 1 This embodiment provides, for example Figure 1-2 The system illustrates an intelligent interview assessment and feedback system based on multimodal data fusion, specifically comprising: a meta-learning dynamic weight generation module, a graph neural network weight correction module, a reinforcement learning strategy optimization module, and a model closed-loop iterative update module connected sequentially; wherein; Meta-learning dynamic weight generation module: In response to the start of the interview evaluation process for candidates, it obtains the interview context feature vectors that have been collected in real time and have undergone preliminary structuring processing from the upstream interface. The interview context feature vectors are then input into a meta-learning neural network model that has been pre-trained using the meta-learning paradigm. The meta-learning neural network model performs calculations and outputs a set of initial basic weights corresponding to the interview context feature vectors for multimodal fusion. The initial basic weight set contains weight values ​​corresponding to the speech modality, video modality, and text modality, respectively. The graph neural network weight correction module receives the initial basic weight set and the feature vectors corresponding to speech, video, and text from the multimodal feature extraction device. Based on the initial basic weight set, it constructs a heterogeneous graph of modal relationships with each modality as a node. It calls a pre-configured multi-layer graph attention network to perform node feature aggregation and edge weight update calculation on the heterogeneous graph of modal relationships. Based on the representation state of each node in the graph after the update calculation, it recalculates a set of target fusion weights adjusted by the intermodal interaction relationship. The reinforcement learning strategy optimization module, after completing the evaluation of a batch of candidates and obtaining their subsequent job performance data, is triggered to execute a strategy update operation. It uses multiple initial basic weight sets stored in the historical database, generated by multiple rounds of historical evaluations and output by the meta-learning dynamic weight generation module, as the action data set; multiple interview context feature vectors stored in the historical database, corresponding to the multiple initial basic weight sets, as the state data set; and the evaluation quality reward value calculated based on the job performance data as the reward data. It constructs a strategy gradient optimization objective and uses a proximal policy optimization algorithm to iteratively update the parameters of the meta-learning neural network model in the meta-learning dynamic weight generation module. Model closed-loop iterative update module: During operation, the interview context feature vector, modality feature vectors, target fusion weight set, and subsequently acquired job performance data generated from each completed interview evaluation process are continuously added to the historical database. When the preset trigger conditions are met, the model retraining process is automatically started, and the parameters of the meta-learning neural network model are updated using the interview context feature vectors, modality feature vectors, target fusion weight set, and job performance data accumulated in the historical database.

[0020] In this embodiment, the specific process of obtaining the interview context feature vector, which has been collected in real time and has undergone preliminary structuring processing, from the upstream interface in the meta-learning dynamic weight generation module includes: When the interview evaluation process for a single candidate is initiated, the system simultaneously receives speech recognition text and intonation analysis tags from the upstream speech processing unit, candidate facial micro-expression codes and body movement feature sequences from the upstream visual analysis unit, and answer timing and content logic tags from the upstream interaction analysis unit. The aforementioned speech recognition text and intonation analysis labels, candidate facial micro-expression codes and body movement feature sequences, answer timing and content logic tags are vectorized to form corresponding speech feature vectors, visual feature vectors and text interaction feature vectors. Specifically, the vectorization process can use feature embedding techniques, such as extracting text feature vectors through a bag-of-words model or a pre-trained language model, and extracting compact representations of visual feature sequences through a pre-trained behavior encoder, thereby converting the unstructured original analysis labels and sequences into fixed-dimensional numerical vectors, which are convenient for subsequent model processing. Subsequently, the aforementioned speech feature vectors, visual feature vectors, and text interaction feature vectors are aligned and concatenated with the pre-set current interview question type code and interview stage identifier to generate a unified, high-dimensional interview context feature vector, which serves as the input to the meta-learning neural network model. Feature alignment ensures that vectors from different modalities and sources are aligned to the same evaluation segment in the temporal or logical dimension. The concatenation operation links multiple feature vectors end-to-end in the feature dimension to form a comprehensive feature vector. This vector fully represents the context, content, and candidate status at the current interview moment. The interview question type code can be an integer enumeration value, such as 1 representing a technical question, 2 representing a behavioral question, etc. The interview stage identifier can be a standardized label, such as "opening," "core questions," and "ending." The specific process by which the meta-learning neural network model calculates and outputs the initial set of basic weights is as follows: The meta-learning neural network model internally maintains a trainable scene prototype memory matrix, which stores K scene prototype vectors, where K is a positive integer; for example, K can be preset to values ​​such as 128 or 256, and its specific value can be determined through hyperparameter tuning. The scene prototype memory matrix is ​​randomly initialized at the beginning of model training and updated during meta-learning training using gradient descent, so that each row vector (i.e., a scene prototype vector) can represent a specific interview scene pattern (such as "high-pressure technical probing", "relaxed cultural exchange", "logic problem solving", etc.). The meta-learning neural network model first calculates the cosine similarity between the interview context feature vector and each scene prototype vector in the scene prototype memory matrix, obtaining K similarity scores. Next, the meta-learning neural network model performs a normalization exponent operation on the K similarity scores, thereby transforming the K similarity scores into a set of K attention weights, where each attention weight corresponds to a scene prototype vector; the normalization exponent operation is the Softmax function, which maps the K similarity scores to a probability distribution, making the sum of all attention weights equal to 1, thus representing the degree of relevance between the current interview context and each scene prototype; Subsequently, the meta-learning neural network model calculates a weighted sum of K scene prototype vectors, where the weighting coefficient of each scene prototype vector is its corresponding attention weight. The result of this weighted sum is defined as the scene context encoding vector. This step is essentially a retrieval and fusion based on the attention mechanism. The generated scene context encoding vector integrates information from multiple prototypes most relevant to the current scene, forming a distributed representation of the current interview context. Subsequently, the meta-learning neural network model simultaneously inputs the scene context encoding vector into a gating signal generation branch and a basic weight bias generation branch. The gating signal generation branch passes through a fully connected layer containing trainable weight and bias parameters, and applies a non-linear activation function with a value range between zero and one to the output of this fully connected layer, outputting a three-dimensional gating signal vector. The non-linear activation function with a value range between zero and one is typically the Sigmoid function. Each element of the gating signal vector controls the proportion of the corresponding modality's basic weight bias that passes through; a value close to 1 indicates full utilization, and a value close to 0 indicates strong suppression. The basic weight bias generation branch passes through another fully connected layer containing trainable weight and bias parameters, outputting a three-dimensional basic weight bias vector. Then, the meta-learning neural network model performs element-wise multiplication of the gating signal vector and the basic weight bias vector to obtain a set of modulated original weight values ​​containing three values. This element-wise multiplication operation constitutes an adaptive gating mechanism, enabling the model to dynamically adjust the importance benchmarks of each modality according to the scene, rather than using a fixed weight mapping. Finally, the meta-learning neural network model performs a normalized exponential operation on a set of modulated raw weights containing three values, thereby outputting an initial basic weight set containing three basic weight values ​​corresponding to the speech modality, video modality, and text modality, respectively. These three basic weight values ​​are named w_speech, w_video, and w_text, and their sum is equal to one. This normalized exponential operation ensures that the three output weight values ​​form a valid probability distribution, directly applicable to subsequent weighted fusion. w_speech, w_video, and w_text are the three components of the initial basic weight set W_meta. The meta-learning dynamic weight generation module then uses the initial basic weights output by the meta-learning neural network model... The reassembled set is passed to the graph neural network weight correction module; and the interview context feature vector obtained by the meta-learning dynamic weight generation module is passed and stored for use by the reinforcement learning policy optimization module. Specifically, the passing is implemented through a data structure or message interface defined internally by the system. The initial basic weight set W_meta is used as prior knowledge input to the graph neural network weight correction module to initialize the edge weights of its modality graph. The interview context feature vector is associated with the candidate's unique identifier and stored in the historical database. When the candidate's subsequent job performance data is available, it will be retrieved together with the stored interview context feature vector to form the state part of the "state-action" pair in the reinforcement learning policy optimization module.

[0021] In this embodiment, it is specifically necessary to explain the following operation in the graph neural network weight correction module: Constructing a heterogeneous graph of modal relationships with each modality as a node based on the initial basic weight set: The system receives an initial set of basic weights from the meta-learning dynamic weight generation module, which includes three weight values: w_speech, w_video, and w_text. It also receives speech feature vectors, video feature vectors, and text feature vectors from the multimodal feature extraction device. Each of the three modalities (speech, video, and text) is treated as a node in the graph, with the initial node feature vector set to the corresponding modal feature vector. Two directed edges with opposite directions are established between any two nodes of different modalities, forming a fully connected directed graph structure. This directed graph structure can model asymmetric influence relationships between modalities; for example, the influence strength of the speech modality on the text modality may differ from the influence strength of the text modality on the speech modality. The calculation process for the initial edge weight of a directed edge from a first modal node to a second modal node is as follows: First, the natural logarithm of the weight value corresponding to the first modality node (i.e., the corresponding value in w_speech, w_video, and w_text) is summed with a preset positive protection constant to obtain a first intermediate value. The positive protection constant is used to prevent taking the logarithm of zero, and its value is on the order of 10 to the power of -8; for example, this constant can typically be set to 1e-8. Simultaneously, the cosine similarity between the feature vectors of the first modality node and the feature vectors of the second modality node is calculated to obtain a second intermediate value. Next, the first intermediate value is added to the result of multiplying the second intermediate value by a preset mixing coefficient to obtain a third intermediate value. The mixing coefficient is used to balance the contribution ratio of prior basic weights and real-time evidence of feature similarity in the edge weight calculation, and its value is a preset or trainable positive real number in the range of zero to one. As a typical implementation, this mixing coefficient can be preset to 0.5. A balanced combination of prior and real-time evidence is achieved. Finally, taking the second modal node as the normalization target, an exponential function with the natural constant e as the base is calculated for all third intermediate values ​​pointing to the second modal node. The result of the exponential function calculation for each third intermediate value is divided by the sum of the exponential function calculation results for all third intermediate values ​​pointing to the second modal node, ensuring that the sum of the initial edge weights of all directed edges pointing to the second modal node is one. The normalized result is the initial edge weight of the directed edge. This initialization method ensures that the initial weights of the edges not only include the scenario prior provided by the meta-learning module (reflected by the logarithmic value of the basic weights), but also incorporate the real-time correlation of modal features in the current candidate's actual performance data, providing an information-rich and reasonable starting point for subsequent graph learning and avoiding the instability that may be caused by random initialization. The process of calling a pre-configured multi-layer graph attention network to perform node feature aggregation and edge weight update calculation on a heterogeneous graph of modal relationships, and recalculating the target fusion weight set based on the representation state of each node in the graph after the update calculation, is as follows: The multi-layer graph attention network contains L computational layers, where L is a positive integer; for example, L can be set to 2 or 3 to achieve a balance between model complexity and representational power. In each layer, for each target node in the graph, a message vector from each source node to that target node is computed. The message vector is obtained by multiplying the current node feature vector of the source node by a trainable message transformation matrix (denoted as W_msg). Next, the attention coefficient from each source node to that target node is computed. The calculation process for the attention coefficient is as follows: The current feature vector of the source node is processed by a trainable first attention transformation matrix (denoted as W_q), and the current feature vector of the target node is processed by a trainable second attention transformation matrix (denoted as W_k). The results of these two processes are concatenated with the edge weights of the current connection. The concatenated vector is then multiplied by a trainable attention weight vector (denoted as v), and finally processed by a LeakyReLU non-linear activation function to obtain unnormalized attention coefficients. Here, the LeakyReLU function allows small negative gradients to pass through. This helps alleviate the problem of neuron "death." Then, an exponential function with the natural constant e as the base is calculated for all unnormalized attention coefficients pointing to the target node. The result of the exponential function calculation for each unnormalized attention coefficient is divided by the sum of the exponential function calculations for all unnormalized attention coefficients pointing to the target node to obtain the normalized attention coefficient. This normalization operation (i.e., Softmax) ensures that the sum of the attention coefficients of all incoming edges of the target node is 1, forming a competitive attention allocation mechanism. Afterward, the updated node feature vector of the target node is obtained in the following way: First, all message vectors pointing to the target node are weighted and summed according to their corresponding normalized attention coefficients to obtain an aggregated message vector. Then, this aggregated message vector, along with the original node feature vector of the target node, is input into a gated recurrent unit. The gated recurrent unit, through its internal gating mechanism, controls the degree of forgetting of the original features of the target node and the degree of fusion with the aggregated message vector, thereby achieving selective updating of node states based on interaction messages to model the dynamic evolution of modal features during the interaction process. Using a gated recurrent unit instead of simple weighted summation allows nodes to retain important historical state information and filter out irrelevant or conflicting interaction information, thus more accurately modeling the state evolution trajectory of each modality in continuous interaction. Simultaneously, the edge weights of the connections from each source node to the target node are updated by adding a preset update rate to the current edge weight. The sum is multiplied by the corresponding normalized attention coefficient (denoted as η), and then layer normalization is performed on the sum to maintain numerical stability. This iterative accumulation mechanism allows the weights of the connecting edges to be adaptively adjusted as the network computation layers deepen, thereby dynamically recording the changes in the interaction strength between modal nodes. The update rate η is a small positive number, such as 0.1, which controls the magnitude of the adjustment of the edge weights according to the current attention. Layer normalization ensures that the edge weights do not experience gradient explosion or vanishing during the update process, maintaining the stability of the training process. After completing the L-layer computation, for each modal node, its importance score is calculated. The calculation process of this importance score is as follows: First, the final node feature vector of the node is input into a multilayer perceptron. The multilayer perceptron contains at least one hidden layer to extract an importance scalar value representing the state strength of the node itself from the final node feature vector. Simultaneously, the network influence score of the node is calculated. The calculation method is as follows: Add the normalized attention coefficients of the node pointing to the other two modal nodes in the L-level calculation to obtain the sum of output attention; then add the normalized attention coefficients of the other two modal nodes pointing to the node in the L-level calculation to obtain the sum of received attention; finally, subtract the sum of received attention from the sum of output attention, and the difference is the network influence score. This score quantifies whether the modal node is a net output influencer or a net receiver in the interaction relationship. A positive network influence score indicates that the modality is a net outputter of information and has a greater influence on other modalities; a negative score indicates that it is a net receiver and is more easily influenced by other modalities. This indicator supplements global relationship information beyond the node's own characteristics from the perspective of graph structure. Next, the importance scalar value output by the multilayer perceptron is multiplied by a preset balance coefficient (denoted as β) and then by the network influence score. The resulting product is added to obtain the importance score of the node. The balance coefficient β is used to adjust the relative contribution of the node's own state strength and network influence to the final weights, and its value is a preset positive real number; for example, β can be set to 0.2, which means that in the final weight allocation, about 16.7% of the consideration comes from the node's influence in the network, and the rest comes from the node's own state. Finally, an exponential function with the natural constant e is calculated for the importance scores of the three nodes (speech, video, and text). The result of the exponential function calculation of the importance score of each node is divided by the sum of the exponential function calculation results of the three nodes, and the three normalized values ​​obtained constitute the target fusion weight set. The target fusion weight set is output to the system's multimodal evaluation fusion unit as the weight input for calculating the final evaluation score. This final step ensures that the sum of the three weight values ​​is 1, which can be directly used for fusion operations such as weighted averaging. At the same time, the normalized attention coefficients and updated connection edge weights generated during the calculation process are recorded as interpretable log data to trace the decision basis for weight adjustment. For example, when the final weight of the video modality is low, checking the log can reveal that it is because the attention coefficient α_video-speech^(L) from the video node to the speech node is very low in the final layer, indicating that the video information failed to effectively support or enhance the speech information in this interview. Therefore, the system lowers its weight based on data evidence. This interpretability enhances the transparency and credibility of the system.

[0022] In this embodiment, the specific process of calculating the evaluation quality reward value based on job performance data in the reinforcement learning strategy optimization module is as follows: For each set of initial basic weights output by the meta-learning dynamic weight generation module and its corresponding interview context feature vector stored in the historical database, the target fusion weight set generated by the graph neural network weight correction module and associated with the interview context feature vector is first retrieved from the historical database. Then, the Jensen-Shannon divergence between the initial base weight set and the target fusion weight set is calculated, and the negative of this divergence value is used as the alignment reward component. The Jensen-Shannon divergence is a symmetric method for measuring the difference between two probability distributions; its calculation involves the Kullback-Leibler divergence of the two distributions. This alignment reward component is negative; the smaller its absolute value, the closer the distributions of the initial base weight set and the target fusion weight set are, and the higher the reward. This encourages the initial decisions of the meta-learning module to align with data-driven fine-grained corrections. The alignment reward component aims to ensure that the meta-learning neural network model... The generated initial set of basic weights remains largely consistent with the target fusion weight set adjusted by the subsequent graph neural network weight correction module based on real-time data. Simultaneously, based on externally acquired candidate performance data and the final interview evaluation score calculated by weighting the target fusion weight set, a Spearman rank correlation coefficient is calculated between the two. This Spearman rank correlation coefficient is then transformed using a preset exponential scaling function. The exponential scaling function is an exponential function with the natural constant e as its base, where the exponent is a preset, positive-zero scaling factor multiplied by the absolute value of the Spearman rank correlation coefficient. The result is then multiplied by the Spearman rank correlation coefficient. The sign function value of the rank correlation coefficient is used to obtain the transformed value, which serves as the predicted reward component. The exponential scaling function significantly amplifies the signals of successful predictions (strong correlation) or serious misjudgments (strong negative correlation) by providing exponentially increased rewards for high correlation coefficients (whether positive or negative). This allows reinforcement learning algorithms to more accurately capture the correlation patterns between decisions and long-term outcomes. For example, setting the scaling factor to 2 can effectively widen the reward gap between decisions of different quality. The predicted reward component aims to measure the predictive ability of the evaluation decisions guided by the initial set of base weights on long-term performance. Finally, a pre-defined aligned reward weight is applied... The weighted reward is multiplied by the alignment reward component, and then multiplied by a preset predicted reward weight. The sum is the evaluation quality reward value for this set of historical data. The sum of the alignment reward weight and the predicted reward weight is one, and the scaling factor is a configurable positive real number, such as two. The alignment reward weight and the predicted reward weight are configurable positive real numbers, such as 0.3 and 0.7 respectively. This composite reward design combines process consistency (alignment reward) with outcome validity (predicted reward), guiding the policy network to learn and generate weight configurations that are both consistent with immediate data evidence and conducive to long-term talent selection effectiveness. The specific process of constructing a policy gradient optimization objective and iteratively updating it using a proximal policy optimization algorithm is as follows: The meta-learning neural network model in the meta-learning dynamic weight generation module is defined as the policy network to be optimized. Multiple interview context feature vectors stored in the historical database are defined as the state data set. The initial basic weight set output by the meta-learning dynamic weight generation module, corresponding to each interview context feature vector, is defined as the action data set. An evaluation quality reward value, obtained according to the aforementioned evaluation quality reward value calculation process, is defined as the reward data for each state-action pair. Based on the state data set, action data set, and corresponding multiple reward data, an alternative objective function for proximal policy optimization is constructed and solved. The design of this alternative objective function aims to balance the exploratory nature and stability of policy updates. Its core lies in preventing policy performance collapse caused by excessively large single update steps through a policy update ratio pruning mechanism. The objective function is defined as follows: for multiple state-action pairs sampled from historical data, calculate the following expression and obtain its expected value: First, calculate the policy update ratio, defined as the ratio of the probability that the current policy network outputs the corresponding historical action (i.e., the initial base weight set) under a given state (i.e., the interview context feature vector), to the probability that the old policy network output the same action under the same state before parameter updates; simultaneously, calculate the advantage function estimate, which is obtained through generalized advantage estimation. The computational method involves iteratively calculating based on the current reward value, an independent value network estimating the state value of the current and next states, a preset discount factor in the range of zero to one, and a preset generalized advantage estimation parameter in the range of zero to one. The discount factor is typically set to a value close to 1 (e.g., 0.99) to adjust for the importance of future rewards. The generalized advantage estimation parameter (λ) is used to balance the bias and variance of the advantage estimation and is also typically set between zero and one (e.g., 0.95) to quantify the superiority or inferiority of historical actions relative to the average level. Then, the policy update ratio is multiplied by the advantage function estimate to obtain... The first product term is obtained; then, the policy update ratio is pruned, restricting it to the interval between 1 minus the pruning threshold and 1 plus the pruning threshold. The pruning threshold is a small positive number, typically set to 0.1 or 0.2. The pruned policy update ratio is then multiplied by the same advantage function estimate to obtain the second product term. Finally, the first and second product terms are compared, and the smaller value is taken as the contribution of the state-action pair to the objective function. The average of the contribution values ​​calculated from all sampled data is used to obtain the value of the substitution objective function. The parameters of the policy network are iteratively optimized using the gradient ascent method to maximize the substitution objective function.The complete computation and optimization process, from calculating the policy update ratio to maximizing the substitution objective function to optimize the policy network parameters, is defined as the policy gradient optimization process. This process drives the parameters of the policy network (i.e., the meta-learning neural network) to adjust in the direction that can produce higher expected rewards (i.e., evaluation quality reward values) by maximizing the substitution objective function. The iterative update process specifically includes collaboratively updating an independent value network: In each iteration of optimization, an independent value network is maintained and updated simultaneously. This value network estimates the corresponding state value based on the input interview context feature vector. Before updating the policy network parameters, the value network is first trained with the goal of minimizing a mean squared error loss function. This loss function is calculated as follows: for a batch of sampled state data, the square of the difference between the value network's predicted value for each state and the estimated actual target reward for that state is calculated. The average of these squared values ​​is then used to optimize this loss function and update the parameters of the value network. The estimated actual target reward typically uses a discounted cumulative reward estimate, such as the target value from n-step reward or generalized advantage estimation. By minimizing this mean squared error, the value network learns to more accurately predict the expected total reward that can be obtained in a given state, thus providing a more accurate advantage estimation benchmark for policy updates. Then, using the more accurate state value estimate provided by the updated value network, the advantage function estimate is recalculated. Finally, based on the recalculated advantage function estimate, a policy gradient optimization process is performed to update the parameters of the meta-learning neural network model. After completing one round of parameter iteration updates, the updated parameters of the meta-learning neural network model are synchronized to the meta-learning dynamic weight generation module. This synchronization operation directly replaces the parameters of the original model in the meta-learning dynamic weight generation module, enabling it to use the optimized policy to generate weights in subsequent interview evaluations. After parameter synchronization, the reinforcement learning policy optimization module uses a pre-defined and retained historical validation dataset to validate the optimized policy. The validation method involves processing the interview context feature vectors in the historical validation dataset using both the old policy network before and after parameter updates, generating corresponding initial base weight sets, and calculating the average evaluation quality reward value corresponding to these initial base weight sets, obtained from the evaluation quality reward value calculation process. The historical validation dataset is a reserved portion of historical data, not used in the policy network training process, and is used to independently evaluate the policy's generalization performance. The effectiveness of the policy optimization is confirmed by comparing the average evaluation quality reward values ​​obtained by the old and new policies on the validation set. If the average evaluation quality reward value obtained by the new policy on the validation set is significantly higher than that of the old policy (e.g., through statistical testing or setting an improvement threshold), the policy optimization is considered effective and successful, and the optimized model parameters are retained for subsequent online evaluation. Otherwise, the system can revert to the old policy or trigger an alarm. This validation step constitutes a key quality checkpoint for the system's self-evolution, ensuring the correctness of the optimization direction.

[0023] In this embodiment, it is specifically necessary to explain that in the model closed-loop iterative update module, the model retraining process is automatically started when a preset trigger condition is met. The specific determination process of the trigger condition includes: During operation, a sliding data window containing the most recent interview evaluation records is dynamically maintained. Two key performance indicators (KPIs) within this sliding data window are periodically calculated. The first KPI is the average decision confidence score, calculated as follows: For each interview evaluation within the window, the Jensen-Shannon divergence between the initial base weight set output by the meta-learning dynamic weight generation module and the target fusion weight set output by the graph neural network weight correction module is first calculated. This divergence measures the difference in probability distribution between the two weight sets. Then, the Jensen-Shannon divergence value is subtracted from the value to obtain the decision confidence score for a single evaluation. Finally, the arithmetic mean of the decision confidence scores for all records within the window is taken, and the resulting value is the average decision confidence score. The size of the sliding data window can be configured according to the system's processing capacity and sensitivity requirements; for example, it can be set to contain 500 to 1000 recent evaluation records. The calculation of the Jensen-Shannon divergence is based on the discrete probability distribution formed by the modal weight values ​​in the initial base weight set and the target fusion weight set. The smaller the value, the closer the distributions are, and the higher the decision confidence score. The second key performance indicator is average predictive consistency, which is calculated as follows: For evaluation records within the sliding data window for which subsequent job performance data has been obtained, calculate the Spearman rank correlation coefficient between the final interview evaluation score (weighted by the target fusion weight set) and the job performance data. Then, apply an exponential moving average algorithm to these Spearman rank correlation coefficients to calculate their smoothed moving average value. The resulting value is the average predictive consistency. The smoothing factor in the exponential moving average algorithm can be set to 0.1 to give higher weight to recent data, thereby reflecting the latest changing trends in performance correlation more quickly. The system continuously monitors changes in average decision confidence and average prediction consistency relative to their respective historical baselines. The historical baseline is determined by calculating the statistical quantile of each indicator over a longer period; for example, the historical baseline can be defined as the 90th percentile of the indicator in the past 10,000 evaluation records, representing a historically high performance level. When at least one of the indicators, average decision confidence or average prediction consistency, is found to have decreased in value compared to its corresponding historical baseline by more than a preset sensitivity threshold, a preset trigger condition is met, and the model retraining process is automatically initiated. The sensitivity threshold is a configurable small value, such as 0.05, used to define the significance of performance degradation, avoiding frequent retraining triggers due to minor fluctuations. Furthermore, if the total number of newly added interview evaluation records in the historical database reaches a preset quantity threshold, regardless of whether the performance indicators have changed, the trigger condition is directly met, and retraining is initiated. The quantity threshold is a safety mechanism to ensure that even if system performance does not show significant degradation, the system can proactively update its knowledge after accumulating sufficient new data; this threshold can be set, for example, to 5,000 new records. The specific process of updating the parameters of a meta-learning neural network model using data accumulated in a historical database is as follows: Once the model retraining process begins, stratified sampling is first performed from the historical database to construct a balanced training dataset. Stratified sampling is based on the timestamps of the data records and the interview question type codes contained within those records, ensuring the representativeness of the training data in both time and scenario dimensions. Specifically, the dataset can be divided into three time periods: recent, mid-term, and long-term, and sampling is performed uniformly according to question type codes to ensure that data from different periods and different types of interviews can be used for training in a balanced manner. Subsequently, based on the parameters of the existing meta-learning neural network model, incremental meta-learning fine-tuning incorporating an elastic weight consolidation strategy is performed on this training dataset. This process first calculates... The importance weight of each parameter in the old model to existing knowledge is calculated by taking the expected approximate value of the squared gradient of the probability of the old model predicting the initial set of basic weights with respect to each model parameter on the training dataset. Specifically, the time period can be divided into three time periods: recent, medium, and long term, and samples can be uniformly sampled according to question type to ensure that data from different periods and different types of interviews are used for training in a balanced manner. Then, a series of meta-learning tasks are constructed based on the training dataset. Each task contains a support set and a query set. The data samples in the support set tend to be selected more from the data generated in the most recently triggered period to force the model to adapt. The model prioritizes adapting to potential changes in data distribution, while the query set is sampled more evenly from the training dataset to evaluate the model's overall adaptability across a wider range of data. This construction allows the model to quickly adapt to new trends within the inner loop of meta-learning, while maintaining global performance during meta-parameter updates in the outer loop. During both the inner and outer loop meta-parameter updates, an additional regularization penalty term is added to the standard meta-learning loss function. This term is proportional to the importance weights and the square of the difference between the new and old model parameters, thus constraining the update magnitude of important parameters. This allows the model to adapt to potential distribution changes while simultaneously training on new data. This effectively slows down the forgetting of previously learned knowledge; this technique is called elastic weight consolidation, and its regularization strength coefficient is a configurable hyperparameter. After completing the above incremental training, a new set of model parameters is obtained. Then, a progressive update strategy is adopted to weight and fuse the new model parameters with the original old model parameters to generate the final updated model candidate parameters. The weight assigned to the old model parameters is a preset momentum factor close to the value of one, while the weight assigned to the new model parameters is the value of one minus the momentum factor. This momentum factor can be set to, for example, 0.9, so that the parameter update is smooth and avoids the impact on the stability of the online system caused by drastic fluctuations in a single retraining. Next, the performance of the updated model candidate parameters is evaluated using an independent validation dataset. The comprehensive performance metric used for evaluation is the weighted sum of the average decision confidence and average prediction consistency on the validation dataset, where the two weights are configurable positive real numbers and their sum is 1. For example, the weight of average decision confidence can be set to 0.7 and the weight of average prediction consistency to 0.3, to emphasize the intrinsic consistency of the decision-making process. The updated model candidate parameters are only officially deployed and replace the original model parameters in the meta-learning dynamic weight generation module when the relative improvement of the comprehensive performance metric of the updated model candidate parameters on the validation set compared to the performance of the old model parameters on the same validation set exceeds a preset minimum improvement threshold. This minimum improvement threshold can be set, for example, to 2%, to ensure that the performance improvement brought by the update is significant and not random fluctuations. Otherwise, the original model parameters are retained, and the retraining process is considered not to have produced a valid update. After the formal deployment of the new model parameters, data archiving and versioning are performed. The management operation includes: versioning and archiving the old model parameters replaced in this update, the system data state snapshot corresponding to the moment the retraining process was triggered, the performance evaluation report generated on the validation dataset, and the newly deployed model parameters to form a traceable model iteration record; versioning and archiving adopts snapshot technology to package all relevant files and metadata and mark them with a unique version number and timestamp; at the same time, according to the preset data retention strategy, unstructured data such as raw audio, video and interaction logs from front-end acquisition devices stored in the historical database that have exceeded a certain time period are cleaned or archived offline, and only the structured feature vectors and key evaluation results extracted from these raw data are retained, in order to control the size of the online database and retain necessary information for future potential retraining. For example, raw audio and video files older than six months can be transferred to low-cost object storage, while only textual feature vectors and evaluation scores are retained in the online database. This can significantly reduce storage costs and improve query efficiency.

[0024] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0025] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0026] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0027] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0028] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0029] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0030] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An intelligent interview assessment and feedback system based on multimodal data fusion, characterized in that, Specifically, it includes: The module consists of a meta-learning dynamic weight generation module, a graph neural network weight correction module, a reinforcement learning strategy optimization module, and a model closed-loop iterative update module, which are connected in sequence. Meta-learning dynamic weight generation module: In response to the start of the interview evaluation process for candidates, it obtains the interview context feature vectors that have been collected in real time and have undergone preliminary structuring processing from the upstream interface. The interview context feature vectors are then input into a meta-learning neural network model that has been pre-trained using the meta-learning paradigm. The meta-learning neural network model performs calculations and outputs a set of initial basic weights corresponding to the interview context feature vectors for multimodal fusion. The initial basic weight set contains weight values ​​corresponding to the speech modality, video modality, and text modality, respectively. The graph neural network weight correction module receives the initial basic weight set and the feature vectors corresponding to speech, video, and text from the multimodal feature extraction device. Based on the initial basic weight set, it constructs a heterogeneous graph of modal relationships with each modality as a node. It calls a pre-configured multi-layer graph attention network to perform node feature aggregation and edge weight update calculation on the heterogeneous graph of modal relationships. Based on the representation state of each node in the graph after the update calculation, it recalculates a set of target fusion weights adjusted by the intermodal interaction relationship. The reinforcement learning strategy optimization module, after completing the evaluation of a batch of candidates and obtaining their subsequent job performance data, is triggered to execute a strategy update operation. It uses multiple initial basic weight sets stored in the historical database, generated by multiple rounds of historical evaluations and output by the meta-learning dynamic weight generation module, as the action data set; multiple interview context feature vectors stored in the historical database, corresponding to the multiple initial basic weight sets, as the state data set; and the evaluation quality reward value calculated based on the job performance data as the reward data. It constructs a strategy gradient optimization objective and uses a proximal policy optimization algorithm to iteratively update the parameters of the meta-learning neural network model in the meta-learning dynamic weight generation module. Model closed-loop iterative update module: During operation, the interview context feature vector, modality feature vectors, target fusion weight set, and subsequently acquired job performance data generated from each completed interview evaluation process are continuously added to the historical database. When the preset trigger conditions are met, the model retraining process is automatically started, and the parameters of the meta-learning neural network model are updated using the interview context feature vectors, modality feature vectors, target fusion weight set, and job performance data accumulated in the historical database.

2. The intelligent interview assessment and feedback system based on multimodal data fusion according to claim 1, characterized in that: The specific process of obtaining the interview context feature vector, which has been collected in real time and has undergone preliminary structuring processing, from the upstream interface in the meta-learning dynamic weight generation module includes: When the interview evaluation process for a single candidate is initiated, the system simultaneously receives speech recognition text and intonation analysis tags from the upstream speech processing unit, candidate facial micro-expression codes and body movement feature sequences from the upstream visual analysis unit, and answer timing and content logic tags from the upstream interaction analysis unit. The above speech recognition text and intonation analysis labels, candidate facial micro-expression codes and body movement feature sequences, answer timing and content logic tags are vectorized to form corresponding speech feature vectors, visual feature vectors and text interaction feature vectors. Subsequently, the aforementioned speech feature vector, visual feature vector, and text interaction feature vector are aligned and concatenated with the preset current interview question type code and interview stage identifier to generate a unified, high-dimensional interview context feature vector, which serves as the input to the meta-learning neural network model.

3. The intelligent interview assessment and feedback system based on multimodal data fusion according to claim 2, characterized in that: The specific process by which the meta-learning neural network model calculates and outputs the initial set of basic weights is as follows: The meta-learning neural network model maintains a trainable scene prototype memory matrix, which stores K scene prototype vectors. The meta-learning neural network model first calculates the cosine similarity between the interview context feature vector and each scene prototype vector in the scene prototype memory matrix, and obtains K similarity scores. Next, the meta-learning neural network model performs a normalized exponential operation on the K similarity scores, thereby transforming the K similarity scores into a set of K attention weights, where each attention weight corresponds to a scene prototype vector; Subsequently, the meta-learning neural network model calculates a weighted sum of K scene prototype vectors, where the weighting coefficient of each scene prototype vector is its corresponding attention weight, and the result of the weighted sum is defined as the scene context encoding vector. Subsequently, the meta-learning neural network model simultaneously inputs the scene context encoding vector into a gated signal generation branch and a basic weight tendency generation branch; The gated signal generation branch passes through a fully connected layer containing trainable weight and bias parameters, and applies a non-linear activation function to the output of this fully connected layer, outputting a three-dimensional gated signal vector. The basic weight bias generation branch passes through another fully connected layer containing trainable weight and bias parameters, outputting a three-dimensional basic weight bias vector. Then, the meta-learning neural network model multiplies the gated signal vector and the basic weight bias vector element-wise to obtain a set of modulated original weight values ​​containing three values. Finally, the meta-learning neural network model performs a normalized exponential operation on a set of modulated raw weight values ​​containing three values, thereby outputting an initial set of basic weights containing three basic weight values ​​corresponding to the speech modality, video modality, and text modality, respectively, where the three basic weight values ​​are named wspeech, wvideo, and wtext.

4. The intelligent interview assessment and feedback system based on multimodal data fusion according to claim 3, characterized in that: In the graph neural network weight correction module, the specific operation of constructing a heterogeneous graph of modal relationships with each modality as a node based on the initial basic weight set is as follows: The system receives an initial basic weight set from the meta-learning dynamic weight generation module, which includes three weight values: w_speech, w_video, and w_text. It also receives speech feature vectors, video feature vectors, and text feature vectors from the multimodal feature extraction device. Each of the three modalities (speech, video, and text) is treated as a node in the graph, with the initial node feature vector set to the corresponding modal feature vector. Two directed edges with opposite directions are established between any two nodes of different modalities, forming a fully connected directed graph structure. The initial edge weight calculation process for a directed edge from the first modal node to the second modal node is as follows: First, the natural logarithm of the weight value corresponding to the first modal node is calculated and summed with a preset positive protection constant to obtain the first intermediate value. Simultaneously, the cosine similarity between the feature vectors of the first modal node and the feature vectors of the second modal node is calculated to obtain the second intermediate value. Next, the first intermediate value is added to the result of multiplying the second intermediate value by a preset mixing coefficient to obtain the third intermediate value. Finally, using the second modal node as the normalization target, an exponential function with the natural constant e as the base is calculated for all third intermediate values ​​pointing to the second modal node. The result of the exponential function calculation for each third intermediate value is divided by the sum of the exponential function calculation results for all third intermediate values ​​pointing to the second modal node. The resulting normalized result is the initial edge weight of the directed connection edge.

5. The intelligent interview assessment and feedback system based on multimodal data fusion according to claim 4, characterized in that: The process of calling a pre-configured multi-layer graph attention network to perform node feature aggregation and edge weight update calculation on a heterogeneous graph of modal relationships, and recalculating the target fusion weight set based on the representation state of each node in the graph after the update calculation, is as follows: The multi-layer graph attention network contains L computational layers. In each layer, for each target node in the graph, a message vector from each source node to that target node is calculated. This message vector is obtained by multiplying the current feature vector of the source node by a trainable message transformation matrix. Next, the attention coefficient from each source node to that target node is calculated. The calculation process for this attention coefficient is as follows: The current node feature vector of the source node is processed by a trainable first attention transformation matrix, and the current node feature vector of the target node is processed by a trainable second attention transformation matrix. The results of these two processes are concatenated with the edge weight of the current connection edge. The concatenated vector is then multiplied by a trainable attention weight vector and processed by a LeakyReLU nonlinear activation function to obtain the unnormalized attention coefficients. Then, an exponential function with the natural constant e is calculated for all unnormalized attention coefficients pointing to the target node. The result of the exponential function calculation for each unnormalized attention coefficient is divided by the sum of the exponential function calculation results for all unnormalized attention coefficients pointing to the target node to obtain the normalized attention coefficient. Afterward, the updated node feature vector of the target node is obtained as follows: First, all message vectors pointing to the target node are weighted and summed according to their corresponding normalized attention coefficients to obtain an aggregated message vector. Then, this aggregated message vector and the original node feature vector of the target node are input into a gated recurrent unit. The gated recurrent unit controls the degree of forgetting of the original features of the target node and the degree of fusion of the aggregated message vector through its internal gating mechanism. At the same time, the edge weights of the connection edges from each source node to the target node are updated by adding a preset update rate to the current edge weight and multiplying it by the corresponding normalized attention coefficient, and then performing layer normalization on the sum. After completing the L-layer computation, for each modal node, its importance score is calculated. The calculation process for the importance score is as follows: First, the final node feature vector of the node is input into a multilayer perceptron. The multilayer perceptron contains a hidden layer, which is used to extract an importance scalar value representing the state strength of the node itself from the final node feature vector. Simultaneously, the network influence score of the node is calculated. The network influence score is calculated by adding the normalized attention coefficients of the node pointing to the other two modal nodes in the L-level calculation to obtain the sum of the output attention. Next, add the normalized attention coefficients of the other two modal nodes pointing to this node in the Lth layer calculation to obtain the sum of received attention; finally, subtract the sum of received attention from the sum of output attention, and the difference is the network influence score. Next, the importance scalar value output by the multilayer perceptron is multiplied by a preset balance coefficient and the network influence score, and the product is added to obtain the importance score of the node. Finally, an exponential function with the natural constant e is calculated for the importance scores of the three nodes of speech, video and text. The result of the exponential function calculation of the importance score of each node is divided by the sum of the exponential function calculation results of the importance scores of the three nodes, and the three normalized values ​​obtained constitute the target fusion weight set.

6. The intelligent interview assessment and feedback system based on multimodal data fusion according to claim 5, characterized in that: The specific process for calculating the evaluation quality reward value based on job performance data in the reinforcement learning strategy optimization module is as follows: For each set of initial basic weights output by the meta-learning dynamic weight generation module and its corresponding interview context feature vector stored in the historical database, the target fusion weight set generated by the graph neural network weight correction module and associated with the interview context feature vector is first retrieved from the historical database. Then, the Jensen-Shannon divergence between the initial basic weight set and the target fusion weight set is calculated, and the negative of the divergence value is taken as the alignment reward component. Simultaneously, based on externally acquired candidate job performance data and the final interview evaluation score calculated by weighting the target fusion weight set, the Spearman rank correlation coefficient between the two is calculated. This Spearman rank correlation coefficient is then transformed using a preset exponential scaling function. The exponential scaling function is an exponential function with the natural constant e as its base, where the exponent is a preset scaling factor multiplied by the absolute value of the Spearman rank correlation coefficient. The result is then multiplied by the sign function value of the Spearman rank correlation coefficient to obtain the transformed value, which is used as the predicted reward component. Finally, a preset alignment reward weight is multiplied by the alignment reward component, and a preset predicted reward weight is multiplied by the predicted reward component. The sum is the evaluation quality reward value for this set of historical data.

7. The intelligent interview assessment and feedback system based on multimodal data fusion according to claim 6, characterized in that: The specific process of constructing a policy gradient optimization objective and iteratively updating it using a proximal policy optimization algorithm is as follows: The meta-learning neural network model in the meta-learning dynamic weight generation module is defined as the policy network to be optimized. The multiple interview context feature vectors stored in the historical database are defined as the state data set. The initial basic weight set output by the meta-learning dynamic weight generation module corresponding to each interview context feature vector is defined as the action data set. The evaluation quality reward value corresponding to each state and action pair, obtained according to the aforementioned evaluation quality reward value calculation process, is defined as the reward data for that data pair. Based on state data sets, action data sets, and corresponding multiple reward data sets, an alternative objective function for proximal policy optimization is constructed and solved. This objective function is defined as follows for multiple state-action pairs sampled from historical data: First, the policy update ratio is calculated, defined as the ratio of the probability that the current policy network outputs the corresponding historical action (initial base weight set) under a given state (i.e., the interview context feature vector) to the probability that the old policy network output the same action under the same state before parameter updates. Simultaneously, the dominance function estimate is calculated using a generalized dominance estimation algorithm. This algorithm iteratively calculates the current reward value, the state value estimate of the current and next states from an independent value network, a preset discount factor, and a preset generalized dominance estimation parameter. Next, the policy update ratio is multiplied by the dominance function estimate to obtain the first product term. Then, the policy update ratio is pruned, and the pruned policy update ratio is multiplied by the same dominance function estimate to obtain the second product term. Finally, the first and second product terms are compared, and the smaller value is taken as the contribution of the state-action pair to the objective function. The average of the contribution values ​​calculated from all sampled data yields the value of the alternative objective function; The gradient ascent method iteratively optimizes the parameters of the policy network to maximize the substitution objective function. The complete calculation and optimization process, from calculating the policy update ratio to maximizing the substitution objective function to optimize the policy network parameters, is defined as the policy gradient optimization process.

8. The intelligent interview assessment and feedback system based on multimodal data fusion according to claim 7, characterized in that: The iterative update process specifically includes the collaborative update of an independent value network: In each iteration of optimization, an independent value network is maintained and updated simultaneously. This value network is used to estimate the corresponding state value based on the input interview context feature vector. Before updating the policy network parameters, the value network is first trained with the goal of minimizing a mean squared error loss function. This loss function is calculated as follows: for a batch of sampled state data, the square of the difference between the value network's predicted value for each state and the estimated actual target reward for that state is calculated, and then the average of these squared values ​​is calculated. This loss function is then optimized using the backpropagation algorithm to update the parameters of the value network. Then, the more accurate state value estimate provided by the updated value network is used to recalculate the advantage function estimate. Finally, based on the recalculated advantage function estimate, the policy gradient optimization process is performed to update the parameters of the meta-learning neural network model. After completing one round of parameter iteration and update, the updated parameters of the meta-learning neural network model are synchronized to the meta-learning dynamic weight generation module; After completing parameter synchronization, the reinforcement learning policy optimization module uses a pre-divided and preserved historical validation dataset to validate the optimized policy. The validation method is as follows: using the old policy network before parameter update and the new policy network after parameter update, respectively, the interview context feature vectors in the historical validation dataset are processed to generate corresponding initial basic weight sets, and the average evaluation quality reward value corresponding to these initial basic weight sets is calculated according to the evaluation quality reward value calculation process; by comparing the average evaluation quality reward values ​​obtained by the old and new policies on the validation set, the effectiveness of the policy optimization is confirmed.

9. The intelligent interview assessment and feedback system based on multimodal data fusion according to claim 8, characterized in that: In the model closed-loop iterative update module, the model retraining process is automatically started when a preset trigger condition is met. The specific determination process of the trigger condition includes: During operation, a sliding data window containing records of the most recent interview assessments is dynamically maintained. Two key performance indicators within this sliding data window are calculated periodically. The first key performance indicator is the average decision confidence score, which is calculated as follows: For each interview assessment within the window, the Jensen-Shannon divergence between the initial basic weight set output by the meta-learning dynamic weight generation module and the target fusion weight set output by the graph neural network weight correction module is first calculated. Then, the Jensen-Shannon divergence value is subtracted from the value to obtain the decision confidence score for a single assessment. Finally, the arithmetic mean of the decision confidence scores of all records within the window is taken, and the resulting value is the average decision confidence score. The second key performance indicator is average prediction consistency, which is calculated as follows: For the evaluation records within the sliding data window for which subsequent job performance data has been obtained, calculate the Spearman rank correlation coefficient between the final interview evaluation score obtained by weighting the target fusion weight set and the job performance data. Then, apply an exponential moving average algorithm to these Spearman rank correlation coefficients to calculate their smoothed moving average value. The resulting value is the average prediction consistency. The system continuously monitors changes in the average decision confidence and average prediction consistency relative to their respective historical baselines. When at least one of the average decision confidence or average prediction consistency is detected to have decreased by more than a preset sensitivity threshold compared to its corresponding historical baseline, the system determines that the preset trigger condition is met and automatically initiates the model retraining process. Furthermore, if the total number of newly added interview evaluation records in the historical database reaches a preset quantity threshold, regardless of whether the performance indicators change, the system directly determines that the trigger condition is met and initiates retraining.

10. The intelligent interview assessment and feedback system based on multimodal data fusion according to claim 9, characterized in that: The specific process of updating the parameters of the meta-learning neural network model using data accumulated in the historical database is as follows: Once the model retraining process is initiated, stratified sampling is first performed from the historical database to construct a balanced training dataset. Subsequently, based on the parameters of the old meta-learning neural network model, an incremental meta-learning fine-tuning that incorporates an elastic weight consolidation strategy is performed on this training dataset. The process first calculates the importance weight of each parameter in the old model parameters to the old knowledge. The calculation method is to obtain the expected approximate value of the square of the gradient of the probability of the old model predicting the initial basic weight set with respect to each model parameter on the training dataset. Then, a series of meta-learning tasks are constructed based on the training dataset, each task containing a support set and a query set; during the inner loop adaptation and outer loop meta-parameter update of meta-learning, an additional regularization penalty term is added to the standard meta-learning loss function. This term is proportional to the importance weight and proportional to the square of the difference between the new model parameters and the old model parameters; after completing the above incremental training, a new set of model parameters is obtained; then, a progressive update strategy is adopted to weightedly fuse the new model parameters with the original old model parameters to generate the final updated model candidate parameters; Subsequently, the performance of the updated model candidate parameters is evaluated using an independent validation dataset. The comprehensive performance metric used for evaluation is the weighted sum of the average decision confidence and average prediction consistency on the validation dataset. The updated model candidate parameters are only officially deployed and replace the original model parameters in the meta-learning dynamic weight generation module if the relative improvement of the comprehensive performance metric of the updated model candidate parameters on the validation set compared to the performance of the old model parameters on the same validation set exceeds a preset minimum improvement threshold. Otherwise, the original model parameters are retained, and the retraining process is considered not to have generated a valid update.

Images