A behavioral etiquette dimension evaluation method, device and medium for intelligent interviews

Abstract

The application discloses a behavior etiquette dimension evaluation method and device for intelligent interview and a medium, relates to the technical field of artificial intelligence, and comprises the following steps: pre-training a multi-modal Transformer model by using synthetic interview video data, and performing supervised fine-tuning on the pre-trained multi-modal Transformer model by using initial interview video data to output an etiquette fusion model; inputting intelligent interview video data and audio data to be evaluated into the etiquette fusion model for feature extraction and fusion processing to generate an etiquette fusion feature tensor; and inputting the etiquette fusion feature tensor into an evaluation component to generate an explainability evaluation report according to a behavior etiquette rule base of the evaluation component. Through synchronous sampling, feature extraction and cross-modal attention fusion of the video and audio modalities by the etiquette fusion model, the application realizes collaborative modeling of multi-source behavior signals.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, device, and medium for evaluating behavioral etiquette dimensions in intelligent interviews. Background Technology

[0002] With the continuous development of intelligent recruitment and automated interviews, the scope of interview evaluation has expanded from language expression and knowledge-based question answering to multimodal comprehensive performance. The behavioral etiquette displayed by candidates during the interview process, including proper posture, natural facial expressions, polite tone of voice, and interactive coordination, has become an important indicator for measuring professional qualities and interpersonal skills. Behavioral etiquette characteristics are distributed across visual, auditory, and semantic modalities, and exhibit high complexity in time-series logic and contextual dependencies. How to establish an evaluation method for etiquette dimensions that conforms to behavioral logic in the process of multimodal information fusion has become an important issue that cannot be ignored in the research direction of intelligent interviews, and it has practical significance for improving the scientificity and interpretability of evaluation.

[0003] Existing intelligent interview evaluation methods often suffer from incomplete coverage and quantification bias in the behavioral etiquette dimension analysis due to limited training data and insufficient cross-modal feature modeling. Existing methods typically alleviate the problems of insufficient training data and poor generalization ability by expanding manually labeled sample data or optimizing model parameters through transfer learning. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for evaluating behavioral etiquette dimensions in intelligent interviews to address the problems of incomplete coverage of etiquette dimensions and bias in quantitative results.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for evaluating the behavioral etiquette dimension of intelligent interviews, which includes constructing a rule-constrained generative adversarial network and training it using initial interview video data, and outputting synthetic interview video data that conforms to the logic of behavioral etiquette.

[0008] The multimodal Transformer model is pre-trained using synthetic interview video data, and then the pre-trained multimodal Transformer model is supervised and fine-tuned using initial interview video data to optimize the parameters of the multimodal Transformer model and output the final etiquette fusion model.

[0009] The intelligent interview video data and audio data to be evaluated are input into the etiquette fusion model for feature extraction and fusion processing to generate an etiquette fusion feature tensor.

[0010] The etiquette fusion feature tensor is input into the evaluation component. The quantified values ​​in the etiquette fusion feature tensor are weighted according to the preset weights of the evaluation component, and the scores of each dimension of etiquette are output. The scores of each dimension of etiquette are aggregated to generate a comprehensive etiquette score, and an interpretable evaluation report is generated based on the behavioral etiquette rule base of the evaluation component.

[0011] As a preferred embodiment of the behavioral etiquette dimension evaluation method for intelligent interviews described in this invention, the output of synthesized interview video data that conforms to behavioral etiquette logic specifically includes:

[0012] A U-shaped encoding and decoding structure is used to construct a rule-constrained generative adversarial network. Random noise vectors are used as input, and a high-resolution video frame sequence is generated through step-by-step upsampling operations to serve as a simulated interview video.

[0013] The simulated interview video is input into the visual discrimination branch of the dual-branch discriminator. The motion continuity and image realism between video frames in the simulated interview video are analyzed by a 3D convolutional network to generate a visual rationality judgment result.

[0014] A multi-task convolutional network is used to receive simulated interview videos and decompose them into independent frame sequences. Body posture feature recognition, facial expression feature recognition, and clothing recognition are performed through a shared backbone network, and the candidate's body posture features, facial expression features, and clothing features are output.

[0015] The system matches body posture features, facial expression features, and external clothing features with the etiquette logic conditions in the predefined behavioral etiquette rule set item by item, counts the number of violations, and outputs the rule compliance score.

[0016] The visual rationality judgment result and the rule compliance score are used together as the joint feedback signal of the dual-branch discriminator;

[0017] The initial interview video data is loaded as real samples and combined with the simulated interview videos to form a training batch. Based on the joint feedback signal, the neural network weight parameters of the video generator are iteratively updated until the simulated interview videos achieve a realistic visual performance and fully meet the requirements of the behavioral etiquette rule set. The resulting synthetic interview video data conforms to the logic of behavioral etiquette.

[0018] As a preferred embodiment of the behavioral etiquette dimension evaluation method for intelligent interviews described in this invention, the random noise vector is a fixed-length numerical sequence generated by a video generator of a generative adversarial network through random sampling.

[0019] As a preferred embodiment of the behavioral etiquette dimension evaluation method for intelligent interviews described in this invention, the step of pre-training the multimodal Transformer model using synthetic interview video data specifically involves:

[0020] The synthesized interview video data is input into the multimodal Transformer model, and the video frame sequence and corresponding frame-level etiquette dimension annotations in the synthesized interview video data are analyzed to form visual-annotation paired samples.

[0021] Perform masking and cross-modal contrastive learning tasks on visual-labeled paired samples to drive the multimodal Transformer model to learn the association representation between vision and labeling, thus completing the pre-training of the multimodal Transformer model.

[0022] As a preferred embodiment of the behavioral etiquette dimension evaluation method for intelligent interviews described in this invention, the step of supervising and fine-tuning the pre-trained multimodal Transformer model using initial interview video data specifically involves...

[0023] The initial interview video data is input into a pre-trained multimodal Transformer model to extract behavioral sequence features from the initial interview video data;

[0024] A supervised learning task was performed on the behavioral sequence features and the behavioral etiquette rating labels in the initial interview video data to adjust the parameters of the multimodal Transformer model.

[0025] As a preferred embodiment of the behavioral etiquette dimension evaluation method for intelligent interviews described in this invention, the execution of the supervised learning task specifically involves...

[0026] The behavioral sequence features are input into the Transformer encoder, and the video frame sequence and audio spectrum in the behavioral sequence features are processed through the spatiotemporal attention mechanism to generate video modal features and audio modal features respectively.

[0027] By fusing video modal features and audio modal features through a cross-modal attention layer, fused features are generated.

[0028] Dynamic weighting is applied to the fused features to generate a predicted behavioral etiquette score.

[0029] The predicted behavior and etiquette scores are matched with the behavior and etiquette score labels in the initial interview video data to obtain the deviation information between the predicted behavior and etiquette scores and the behavior and etiquette score labels.

[0030] Based on the deviation information between the predicted behavior and etiquette score and the behavior and etiquette score label, the attention parameter matrix of the spatiotemporal attention layer and the cross-modal attention layer is updated layer by layer until the deviation information is minimized, thus completing the parameter adjustment of the multimodal Transformer model.

[0031] As a preferred embodiment of the behavioral etiquette dimension evaluation method for intelligent interviews described in this invention, the step of inputting the intelligent interview video data and audio data to be evaluated into an etiquette fusion model for feature extraction and fusion processing to generate an etiquette fusion feature tensor specifically involves:

[0032] Input the video and audio data of the intelligent interview to be evaluated into the etiquette fusion model;

[0033] The etiquette fusion model samples the intelligent interview video data frame sequence at the same time interval and segments the audio data into time segments to obtain synchronized video frame sequences and audio segment sequences.

[0034] Facial expression and body posture features of candidates are extracted from video frame sequences, and tone and speech rate features of candidates are extracted from audio clip sequences.

[0035] The extracted facial expression features and body posture features are used as video modal features, and the tone and speech rate features are used as audio modal features.

[0036] Video and audio modal features are input into the cross-modal attention mechanism of the etiquette fusion model. The cross-modal attention mechanism performs cross-modal interaction processing on the video and audio modal features to generate weighted fusion features, which serve as the etiquette fusion feature tensor.

[0037] As a preferred embodiment of the behavioral etiquette dimension evaluation method for intelligent interviews described in this invention, the step of weighting the quantized values ​​in the etiquette fusion feature tensor according to the preset weights of the evaluation component and outputting scores for each dimension of etiquette is specifically as follows:

[0038] The etiquette fusion feature tensor is input into the evaluation component. The evaluation component analyzes the facial expression features, body posture features, tone features, and speech rate features in the etiquette fusion feature tensor to obtain the quantitative values ​​of the facial expression features, body posture features, tone features, and speech rate features.

[0039] The quantitative values ​​of the parsed facial expression features, body posture features, tone features, and speech rate features are weighted according to the weight matrix preset by the evaluation component to generate scores for each dimension of etiquette.

[0040] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the behavioral etiquette dimension evaluation method for intelligent interviews as described in the first aspect of the present invention.

[0041] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the behavioral etiquette dimension evaluation method for intelligent interviews as described in the first aspect of the present invention.

[0042] The beneficial effects of this invention are as follows: By synchronously sampling, extracting features, and fusing cross-modal attention in video and audio modalities through an etiquette fusion model, collaborative modeling of multi-source behavioral signals is achieved. Without relying on prior manual annotation, the invention utilizes the spatiotemporal representation capabilities obtained through pre-training to accurately capture key etiquette features such as facial expressions, body posture, tone of voice, and speech rate. Furthermore, by dynamically weighting the correlations between modalities through a cross-modal attention mechanism, a highly discriminative etiquette fusion feature tensor is generated, effectively preserving the consistency of behavioral temporal sequence and the complementary information between modalities, thus improving the accuracy of analyzing subtle nonverbal behaviors. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a method for evaluating behavioral etiquette dimensions in intelligent interviews.

[0045] Figure 2 A flowchart for generating synthetic interview video data.

[0046] Figure 3 A flowchart for training an etiquette integration model.

[0047] Figure 4 A flowchart for evaluating behavioral etiquette dimensions. Detailed Implementation

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0050] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0051] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a method for evaluating behavioral etiquette dimensions in intelligent interviews, including the following steps:

[0052] S1. Construct a rule-constrained generative adversarial network and train it using initial interview video data to output synthetic interview video data that conforms to behavioral etiquette logic.

[0053] A U-shaped encoder-decoder structure is used to construct a rule-constrained generative adversarial network. Random noise vectors are used as inputs, and a high-resolution video frame sequence is generated through stepwise upsampling operations to serve as a simulated interview video. The random noise vectors are fixed-length numerical sequences generated by the video generator of the generative adversarial network through random sampling, and are used as the initial inputs of the video generator.

[0054] Specifically, the random noise vector is input into a rule-constrained generative adversarial network (GAN) video generator. The video generator employs a U-shaped encoder-decoder structure, performing continuous downsampling operations on the random noise vector through the encoding path. The encoding path includes multiple convolutional layers. Each convolutional layer performs convolution processing on the random noise vector or the feature map output from the previous layer to generate the current layer's feature map. The current layer's feature map represents the local spatial features of the random noise vector or the feature map output from the previous layer. After convolution processing, an activation function is applied to introduce a non-linear transformation. At the bottleneck layer, the downsampled feature map undergoes further convolution processing to generate the bottleneck layer feature map. The diagram captures deep semantic information, with the bottleneck layer located between the encoding and decoding paths. The decoding path performs progressive upsampling operations on the bottleneck layer feature maps through multiple transposed convolutional layers, with each transposed convolutional layer restoring spatial dimensions. The decoding path introduces skip connections, connecting the current layer feature maps of the corresponding layers in the encoding path with the upsampled feature maps of the current layers in the decoding path at the channel level, fusing low-level detail information and high-level semantic information to ensure that the high-definition video frame sequence retains fine textures and overall structural consistency. The video generator outputs a high-definition video frame sequence as a simulated interview video.

[0055] The specific steps of convolution processing are as follows: A convolution kernel is applied to perform a point-by-point scanning operation on the random noise vector or the feature map output from the previous layer. The convolution kernel is a set of learnable parameter matrices of a fixed size. It slides along the spatial dimension of the random noise vector or the feature map output from the previous layer, covering a local region of the random noise vector or the feature map output from the previous layer each time. The convolution kernel performs a dot product operation on the local region of the random noise vector or the feature map output from the previous layer, extracts pixel pattern information from the local region, and generates a single convolution output value through activation function processing. The convolution kernel moves row by row and column by column according to a preset stride to cover the random noise vector. The noise vector or the feature map output from the previous layer are used to generate multiple convolutional output values ​​for all regions. These multiple convolutional output values ​​are arranged according to their spatial positions to form a new current layer feature map. The current layer feature map represents the local spatial features of the random noise vector or the feature map output from the previous layer. Through the continuous application of multiple convolutional kernels, each convolutional kernel generates a new current layer feature map to be passed to subsequent processing steps. The preset stride is determined by the distance the convolutional kernel slides on the random noise vector or the feature map output from the previous layer, for example, it is set to 1 or 2 pixels to control the density of the convolutional output values ​​and the spatial resolution of the feature map.

[0056] The simulated interview video is input into the visual discriminant branch of the dual-branch discriminator. The motion continuity and image realism between video frames in the simulated interview video are analyzed by a 3D convolutional network to generate a visual rationality judgment result. The dual-branch discriminator is a component of the rule-constrained generative adversarial network and is used to evaluate the visual rationality and behavioral etiquette compliance of the simulated interview video.

[0057] Specifically, the simulated interview video is input into the visual discriminator branch of a two-branch discriminator within a rule-constrained generative adversarial network (GAN). This visual discriminator branch processes the simulated interview video through a 3D convolutional network, decomposing it into a continuous sequence of video frames. The 3D convolutional network then applies 3D convolutional kernels to perform point-by-point scanning on the video frame sequence. Each 3D convolutional kernel is a set of fixed-size learnable parameter matrices that slides along the spatial and temporal dimensions of the video frame sequence, covering a local spatial-temporal region at each step. The 3D convolutional kernel performs point-by-point dot product operations on the local spatial-temporal regions of the video frame sequence, extracting pixel pattern information and inter-frame variation information from these regions and processing them through an activation function to generate a single 3D convolutional output value. Finally, the 3D convolutional kernel moves row-by-row, column-by-column, and frame-by-frame with a preset stride, covering the entire video frame sequence. For all regions, multiple 3D convolutional output values ​​are generated. These multiple 3D convolutional output values ​​are arranged according to spatial location and temporal order to form a new spatiotemporal feature map. The spatiotemporal feature map represents the motion continuity and image realism features in the video frame sequence. The visual discrimination branch integrates the spatiotemporal feature map through a fully connected layer, mapping the feature values ​​of the spatiotemporal feature map to a single visual reasonableness score. The visual reasonableness score judges the degree to which the motion continuity and image realism of the simulated interview video are close to those of the real interview video through a predefined realism threshold, and outputs the visual reasonableness judgment result, indicating whether the simulated interview video is visually close to the real interview video. The realism threshold is determined based on the visual feature distribution of the initial interview video data, and the value range is, for example, from 0 to 1, where 0 represents completely unrealistic and 1 represents completely close to the real interview video.

[0058] It should be noted that the point-by-point scanning operation is the process by which the three-dimensional convolutional kernel slides along the spatial and temporal dimensions of the video frame sequence, covering a local spatial-temporal region each time.

[0059] The pointwise dot product operation is the process by which a 3D convolution kernel performs dot product processing on pixel values ​​within a covered local spatial-temporal region, extracting pixel pattern information and inter-frame variation information, and generating a single 3D convolution output value.

[0060] A multi-task convolutional network is used to receive simulated interview videos and decompose them into independent frame sequences. Body posture feature recognition, facial expression feature recognition, and clothing recognition are performed simultaneously through a shared backbone network, and the candidate's body posture features, facial expression features, and clothing features are output.

[0061] Specifically, the simulated interview video is input into a multi-task convolutional network. The multi-task convolutional network splits the simulated interview video into individual frame images in chronological order. Each individual frame image is extracted from the simulated interview video one by one and arranged to form an independent frame sequence.

[0062] Multi-task convolutional networks process independent frame sequences through a shared backbone network. The shared backbone network applies convolutional kernels to perform point-by-point scanning operations on each individual frame image in the independent frame sequence. The convolutional kernel is a set of learnable parameter matrices of a fixed size that slides along the spatial dimension of each individual frame image, covering a local region of each individual frame image at a time. The convolutional kernel performs a dot product operation on the local region of each individual frame image, extracts pixel pattern information from the local region, and generates a single shared output value through activation function processing. The convolutional kernel moves row by row and column by column with a preset stride, covering all regions of each individual frame image and generating multiple shared output values. The multiple shared output values ​​are arranged according to their spatial positions to form the shared feature map of each individual frame image. The shared backbone network repeatedly applies multiple layers of convolutional kernels to process the shared feature map and generate a general feature map.

[0063] The pose convolutional layer applies a pose convolutional kernel that slides along the spatial dimension of the general feature map, covering a local region of the general feature map each time. The pose convolutional kernel performs a dot product operation on the local region of the general feature map, extracting information on the candidate's limb joint positions and bending angles to form body pose features. The expression convolutional layer applies an expression convolutional kernel that slides along the spatial dimension of the general feature map, covering a local region of the general feature map each time. The expression convolutional kernel performs a dot product operation on the local region of the general feature map, extracting information on the candidate's eyebrow, eye, and mouth deformations to form facial expression features. The clothing convolutional layer applies a clothing convolutional kernel that slides along the spatial dimension of the general feature map, covering a local region of the general feature map each time. The clothing convolutional kernel performs a dot product operation on the local region of the general feature map, extracting information on the candidate's clothing fabric patterns and color distributions to form external clothing features. The multi-task convolutional network outputs the candidate's body pose features, facial expression features, and external clothing features.

[0064] The system matches body posture features, facial expression features, and external clothing features with the etiquette logic conditions in the predefined behavioral etiquette rule set item by item, counts the number of violations, and outputs the rule compliance score.

[0065] Specifically, this involves loading a predefined set of behavioral etiquette rules, which includes etiquette logic conditions such as standards for upright sitting posture, standards for the degree of smiling, and standards for formal attire; the predefined set of behavioral etiquette rules is determined based on interview etiquette standards in the human resources field.

[0066] Match the limb joint positions in the body posture characteristics with the back verticality standard and shoulder relaxation standard in the sitting upright standard, check whether the back verticality standard and shoulder relaxation standard are met, and record the entries that do not meet the back verticality standard or shoulder relaxation standard as violations.

[0067] Match the bending angle in the body posture characteristics with the standard of avoiding crossed arms in the gesture etiquette standard, check whether the standard of avoiding crossed arms is met, and record the items that do not meet the standard of avoiding crossed arms as violations.

[0068] Match the mouth deformation information in the facial expression features with the corner-of-mouth upturn standard in the smile degree standard, check whether the corner-of-mouth upturn standard is met, and record the entries that do not meet the corner-of-mouth upturn standard as violations.

[0069] Match the eyebrow and eye positions in facial expression features with the eye contact stability standard in the eye contact standard, check whether the eye contact stability standard is met, and record the entries that do not meet the eye contact stability standard as violations.

[0070] Match the fabric pattern in the external clothing features with the no-pattern interference standard in the formal clothing standard, check whether the no-pattern interference standard is met, and record the items that do not meet the no-pattern interference standard as violations.

[0071] Match the color distribution information in the external clothing features with the neutral tone standard in the color distribution standard, check whether the neutral tone standard is met, and record the entries that do not meet the neutral tone standard as violations.

[0072] All violations are summarized and counted one by one. The total number of violations is then compared with the total number of items in the predefined set of behavioral etiquette rules to determine the proportion of violations to the total number of items. Based on the scoring criteria in the predefined set of behavioral etiquette rules, the proportion of violations is mapped to the corresponding scoring range, for example, a lower proportion corresponds to a higher score, and a higher proportion corresponds to a lower score. The rule compliance score is output, indicating the degree of compliance of body posture features, facial expression features, and external clothing features with the predefined set of behavioral etiquette rules. The scoring range is determined by analyzing the distribution of behavioral features in the initial interview video data, based on the interview etiquette standards in the human resources field.

[0073] The visual rationality judgment result and the rule compliance score are used together as the joint feedback signal of the dual-branch discriminator to update the neural network weight parameters of the video generator; the video generator is a component of the rule-constrained generative adversarial network responsible for generating simulated interview videos.

[0074] The initial interview video data is loaded as real samples and combined with the simulated interview videos to form a training batch. Based on the joint feedback signal, the neural network weight parameters of the video generator are iteratively updated until the simulated interview videos achieve a realistic visual performance and fully meet the requirements of the behavioral etiquette rule set. The resulting synthetic interview video data conforms to the logic of behavioral etiquette is then output.

[0075] Specifically, the joint feedback signal is input into the video generator; the parameters of the encoding path convolution kernel are adjusted; the video generator uses the visual rationality judgment result as a guide, which reflects the degree of similarity between the realism of the simulated interview video and the initial interview video data; the deviation information of the visual rationality judgment result is transmitted to each convolution layer of the encoding path through the backpropagation process, and the learnable parameter matrix of the encoding path convolution kernel is updated layer by layer, so that the feature map of the current layer generated by the encoding path convolution kernel is closer to the feature distribution of the initial interview video data;

[0076] The parameters of the transposed convolutional kernel in the decoding path are adjusted. The video generator uses the rule compliance score as a guide. The rule compliance score reflects the degree of conformity between the simulated interview video and the predefined set of behavioral etiquette rules. The deviation information of the rule compliance score is passed to each transposed convolutional layer of the decoding path through the backpropagation process. The learnable parameter matrix of the transposed convolutional kernel in the decoding path is updated layer by layer, so that the upsampled feature map generated by the transposed convolutional kernel in the decoding path better conforms to the requirements of the predefined set of behavioral etiquette rules.

[0077] Among them, the parameters of the encoding path convolution kernel and the parameters of the decoding path transpose convolution kernel are parameters in the video generator;

[0078] The backpropagation process is as follows: A joint feedback signal is received, containing both visual plausibility judgment results and rule compliance scores. For the encoding path convolutional kernel parameters, the visual plausibility judgment results reflect the deviation information between the simulated interview video and the initial interview video data in terms of image realism. Starting from the output of the visual discrimination branch, this deviation information is passed layer by layer from back to front to each convolutional layer of the encoding path, updating the learnable parameter matrix of the convolutional kernel in each convolutional layer, making the current layer feature map generated by the encoding path closer to the feature distribution of the initial interview video data. For the decoding path transposed convolutional kernel parameters, the rule compliance score reflects the visual plausibility judgment results and rule compliance scores. For the decoding path transposed convolutional kernel parameters, the visual plausibility judgment results reflect the deviation information between the simulated interview video and the initial interview video data in terms of image realism. The deviation between the simulated interview video and the predefined set of behavioral etiquette rules is analyzed. Starting from the output of the rule compliance scoring branch, the deviation information is passed layer by layer from back to front to each transposed convolutional layer in the decoding path. The learnable parameter matrix of the transposed convolutional kernel of each transposed convolutional layer is updated, so that the upsampled feature map generated by the decoding path better conforms to the requirements of the predefined set of behavioral etiquette rules. By repeatedly passing the deviation information and updating the learnable parameter matrices of all convolutional kernels and transposed convolutional kernels, a backpropagation adjustment is completed to ensure that the neural network weight parameters of the video generator are optimized to improve the realism of the simulated interview video and the performance of behavioral etiquette.

[0079] The adjusted video generator regenerates the mock interview video using the updated neural network weight parameters; the newly generated mock interview video is input into the two-branch discriminator, which processes the new mock interview video and generates new visual reasonableness judgment results and new rule compliance scores respectively, forming a new joint feedback signal;

[0080] The process of repeatedly adjusting the neural network weight parameters of the video generator using new joint feedback signals and generating new simulated interview videos continues until the visual performance of the new simulated interview videos is close to that of the initial interview video data and fully meets the requirements of the predefined set of behavioral etiquette rules. The output is synthetic interview video data that conforms to the logic of behavioral etiquette. The synthetic interview video data includes, for example, video sequences with a batch size of 10,000 segments, each segment lasting 1 minute, a resolution of 1280×720, and a frame rate of 30fps. Each video sequence has frame-level etiquette dimension annotations.

[0081] It should be noted that this step uses deep learning to construct a rule-constrained generative adversarial network to generate synthetic interview video data that conforms to behavioral etiquette logic, significantly improving the authenticity and etiquette performance of the videos; it employs a U-shaped encoding and decoding structure and a multi-task convolutional network to accurately extract body posture, facial expressions, and external clothing features, meeting the etiquette standards in the human resources field; and it optimizes the continuity of video frame sequences and behavioral norms through a dual-branch discriminator, providing high-quality data support for subsequent steps.

[0082] S2. Use synthetic interview video data to pre-train the multimodal Transformer model, and use the initial interview video data to supervise and fine-tune the pre-trained multimodal Transformer model to optimize the parameters of the multimodal Transformer model and output the final etiquette fusion model.

[0083] Synthetic interview video data that conforms to behavioral etiquette logic is input into a multimodal Transformer model. The video frame sequence in the synthetic interview video data is analyzed and the corresponding frame-level etiquette dimension annotations are matched to form visual-annotation paired samples.

[0084] Perform masking and cross-modal contrastive learning tasks on visual-labeled paired samples to drive the multimodal Transformer model to learn the association representation between vision and labeling, thus completing the pre-training of the multimodal Transformer model;

[0085] Specifically, the multimodal Transformer model sequentially splits the synthetic interview video data into individual video frame images. Each individual video frame image is extracted from the synthetic interview video data and arranged to form a video frame sequence. Simultaneously, it extracts frame-level etiquette dimension annotations corresponding to each individual video frame image from the synthetic interview video data. The frame-level etiquette dimension annotations include dimension labels such as posture, expression, and clothing for each individual video frame image. The multimodal Transformer model combines each individual video frame image in the video frame sequence with its corresponding frame-level etiquette dimension annotation to form visual-annotation pairing samples. Each visual-annotation pairing sample consists of a single video frame image and its frame-level etiquette dimension annotation, which is used for subsequent pre-training tasks.

[0086] The initial interview video data is input into a pre-trained multimodal Transformer model to extract behavioral sequence features from the initial interview video data;

[0087] Specifically, the pre-trained multimodal Transformer model processes the video frame sequence through a video frame sequence encoder, which converts each individual video frame image in the video frame sequence into an embedding vector, which represents the original pixel information of each individual video frame image.

[0088] The video frame sequence encoder uses a spatiotemporal attention mechanism to process video frame sequences and extract features from each video frame image. The spatiotemporal attention mechanism inputs embedded vectors into the attention layer, assigns weights to the spatial pixel distribution, and gives higher weights to key regions based on the importance of pixel positions to extract spatial features. At the same time, it assigns weights to the temporal relationships between frames, and gives higher weights to connections with significant behavioral changes based on the continuity of adjacent frames to extract temporal continuity features. Combining spatial and temporal continuity features, a single-frame feature representation is generated and arranged in chronological order to form a behavioral sequence feature, reflecting the dynamic changes in the candidate's posture, expression, and clothing in the initial interview video.

[0089] A supervised learning task was performed on the behavioral sequence features and the behavioral etiquette rating labels in the initial interview video data to adjust the parameters of the multimodal Transformer model;

[0090] In the supervised learning task, behavioral sequence features are input into a 12-layer Transformer encoder using a pre-trained multimodal Transformer model. The 12-layer Transformer encoder processes the video frame sequence in the behavioral sequence features through a spatiotemporal attention mechanism. The video frame sequence is converted into embedding vectors, which represent the original pixel information of a single video frame in the video frame sequence. Weights are assigned to the spatial pixel distribution in the embedding vectors, giving higher weights to key pixel regions based on the relative importance of pixel positions in the embedding vectors, and spatial features are extracted from the embedding vectors. The spatiotemporal attention mechanism assigns weights to the temporal inter-frame relationships in the embedding vectors, giving higher weights to inter-frame connections with significant behavioral changes based on the continuity of adjacent frames in the embedding vectors, and temporal continuity features are extracted from the embedding vectors. The spatial and temporal continuity features are integrated to generate video modal features.

[0091] The 12-layer Transformer encoder processes video frame sequences while simultaneously handling the audio spectrum in the behavioral sequence features through a spatiotemporal attention mechanism. The audio spectrum is transformed using a Mel-scale filter bank to generate audio embedding vectors, which represent the frequency distribution characteristics of the audio spectrum. Weights are assigned to the Mel-scale filter bank features in the audio embedding vectors, assigning higher weights to key frequency regions based on the significance of the frequency distribution, thus extracting acoustic features from the audio embedding vectors. The spatiotemporal attention mechanism then assigns weights to the temporal segment relationships in the audio embedding vectors, assigning higher weights to segments with significant changes in pitch and speech rate based on the continuity of adjacent temporal segments, thus extracting pitch and speech rate features from the audio embedding vectors. Finally, the spatiotemporal attention mechanism integrates the acoustic features and pitch and speech rate features to generate audio modal features.

[0092] The 12-layer Transformer encoder fuses video modal features and audio modal features through a cross-modal attention layer. The cross-modal attention layer performs association weight allocation on the video modal features and audio modal features, assigning higher weights to complementary information regions based on the semantic association between the video modal features and audio modal features. The fused features are extracted from the video modal features and audio modal features, and the fused features represent the comprehensive etiquette performance in the behavioral sequence features.

[0093] The pre-trained multimodal Transformer model inputs fused features into the output layer. The output layer performs dynamic weighting on the fused features through a gating fusion algorithm. Weights are assigned according to the contribution of each modality in the fused features, and the fused features are integrated to generate a predicted behavior etiquette score. The predicted behavior etiquette score represents the quantified value of the etiquette dimension in the behavior sequence features.

[0094] The pre-trained multimodal Transformer model matches the predicted etiquette score with the etiquette score labels in the initial interview video data to obtain the deviation information between the predicted etiquette score and the etiquette score labels; the etiquette score labels contain the true etiquette score for each behavior sequence in the initial interview video data.

[0095] The pre-trained multimodal Transformer model performs a supervised learning task based on the deviation information between the predicted etiquette score and the etiquette score label. The deviation information represents the difference between the predicted etiquette score and the actual etiquette score.

[0096] The pre-trained multimodal Transformer model uses backpropagation to pass the bias information from the output layer to the cross-modal attention layer and the spatiotemporal attention layer, and updates the attention parameter matrix of the cross-modal attention layer layer by layer, so that the fusion features generated by the cross-modal attention layer are more consistent with the distribution of behavioral etiquette rating labels.

[0097] The pre-trained multimodal Transformer model transmits bias information to the spatiotemporal attention layer through backpropagation, updating the attention parameter matrix of the spatiotemporal attention layer to make the generated video and audio modal features more consistent with the requirements of the behavior etiquette scoring labels. The pre-trained multimodal Transformer model then uses the updated attention parameter matrix to reprocess the behavior sequence features and generate new predicted behavior etiquette scores. The bias information between the new predicted behavior etiquette scores and the behavior etiquette score labels is repeatedly matched, and the attention parameter matrix is ​​updated through backpropagation until the bias information is minimized, thus completing the adjustment of the multimodal Transformer model.

[0098] Repeatedly perform the supervised learning task until the multimodal Transformer model's behavioral etiquette score prediction results on the validation set are stable, and output the final etiquette fusion model.

[0099] S3. Input the intelligent interview video data and audio data to be evaluated into the etiquette fusion model for feature extraction and fusion processing to generate the etiquette fusion feature tensor.

[0100] The intelligent interview video data and audio data to be evaluated are input into the etiquette fusion model; the intelligent interview video data is real interview process data recorded by video acquisition equipment; the audio data is the voice signal corresponding to the intelligent interview video data recorded synchronously by audio acquisition equipment.

[0101] The etiquette fusion model samples the intelligent interview video data frame sequence at the same time interval and segments the audio data into time segments to obtain synchronized video frame sequences and audio segment sequences.

[0102] Specifically, the process involves: extracting all video frame images one by one from the intelligent interview video data, with each video frame image arranged in the recording order; the etiquette fusion model selecting video frame images corresponding to the intervals from all video frame images at the same time intervals, for example, selecting a specific frame from 30 frames per second to form a video frame sequence; simultaneously, extracting all audio signals one by one from the audio data, with each audio signal arranged in the recording order; selecting audio signal segments corresponding to the intervals from all audio signals at the same time intervals, for example, selecting a specific segment from the audio signal per second to form an audio segment sequence; and aligning each video frame image in the video frame sequence with the corresponding audio signal segment in the audio segment sequence according to the recording time point to ensure synchronization between the video frame sequence and the audio segment sequence.

[0103] Facial expression features and body posture features are extracted from video frame sequences, and tone and speech rate features of candidates are extracted from audio clip sequences.

[0104] The extracted facial expression features and body posture features are used as video modal features, and the tone and speech rate features are used as audio modal features.

[0105] Video and audio modal features are input into the cross-modal attention mechanism of the etiquette fusion model. The cross-modal attention mechanism performs cross-modal interaction processing on the video and audio modal features to generate weighted fusion features, which serve as the etiquette fusion feature tensor.

[0106] Specifically, the video modal features are converted into query vectors, which represent the spatial pose and facial expression information in the video modal features; at the same time, the audio modal features are converted into key vectors and value vectors, where the key vectors represent the intonation matching relationship in the audio modal features and the value vectors represent the speech rate details in the audio modal features.

[0107] Perform association weight assignment on the query vector and key vector, assign higher weight to complementary information regions based on the semantic association between the spatial pose information in the query vector and the intonation matching relationship in the key vector, and extract attention-weighted representations from the query vector and key vector;

[0108] The attention-weighted representation is mapped to the value vector element by element. The weights of the attention-weighted representation are applied to the corresponding elements of the value vector one by one through the dot product operation to generate weighted value vector elements. All weighted value vector elements are integrated to generate audio-weighted features of video modality features, which represent the weighted contribution of audio modality features to video modality features.

[0109] Audio modal features are converted into query vectors, which represent the intonation and speech rate information in the audio modal features. Simultaneously, video modal features are converted into key vectors and value vectors, where the key vectors represent facial expression matching relationships and the value vectors represent pose details. A cross-modal attention mechanism repeats the aforementioned association weight allocation and dot product operation process, extracting attention-weighted representations from the query vectors and key vectors. The weighted value vector elements are then integrated to generate video-weighted features of the audio modal features, representing the weighted contribution of the video modal features to the audio modal features.

[0110] The cross-modal attention mechanism integrates the audio-weighted features of video modality features and the video-weighted features of audio modality features to generate a weighted fusion feature. The weighted fusion feature represents the comprehensive etiquette association between video modality features and audio modality features, and is output as the etiquette fusion feature tensor.

[0111] S4. Input the etiquette fusion feature tensor into the evaluation component, perform weighted processing on the quantified values ​​in the etiquette fusion feature tensor according to the preset weights of the evaluation component, and output the scores of each dimension of etiquette; aggregate the scores of each dimension of etiquette to generate a comprehensive etiquette score, and generate an interpretable evaluation report based on the behavioral etiquette rule base of the evaluation component.

[0112] The etiquette fusion feature tensor is input into the evaluation component. The evaluation component analyzes the facial expression features, body posture features, tone features, and speech rate features in the etiquette fusion feature tensor to obtain the quantitative values ​​of the facial expression features, body posture features, tone features, and speech rate features.

[0113] It should be noted that all quantified values ​​represent numerical representations of facial expression features, body posture features, tone features, and speech rate features, respectively, and are used for subsequent weighted processing.

[0114] The quantitative values ​​of the parsed facial expression features, body posture features, tone features, and speech rate features are weighted according to the pre-set weight matrix of the evaluation component to generate scores for each dimension of etiquette; the pre-set weight matrix is ​​determined according to the interview etiquette standards in the human resources field.

[0115] The scores of each dimension of etiquette are aggregated with equal weights to generate a comprehensive etiquette score.

[0116] Specifically, the system obtains scores for facial expression, body posture, tone of voice, and speech rate. Each score represents the etiquette performance of the corresponding feature. Each score is assigned an equal weight value, which is a pre-set equal proportion by the evaluation component based on the interview etiquette standards in the human resources field. For example, facial expression, body posture, tone of voice, and speech rate each account for 25% of the weight.

[0117] The facial expression dimension score is paired with an equal weight value one by one through the dot product operation, and the equal weight value is applied to the corresponding element of the facial expression dimension score to generate the facial expression dimension weighted value.

[0118] Finally, the weighted values ​​of facial expression, body posture, tone of voice, and speech rate are integrated to generate a comprehensive etiquette score, which represents the candidate's overall etiquette performance.

[0119] The comprehensive etiquette score and the scores of each dimension of etiquette are input into the behavioral etiquette rule base of the evaluation component. The behavioral etiquette rule base matches the corresponding evaluation items with the predefined scoring range and description template to generate an interpretable evaluation report.

[0120] Specifically, it loads predefined scoring ranges, which include comprehensive etiquette score ranges such as 0 to 20 points corresponding to poor level, 21 to 40 points corresponding to average level, 41 to 60 points corresponding to medium level, 61 to 80 points corresponding to good level and 81 to 100 points corresponding to excellent level, as well as similar level divisions for facial expression dimension score range, body posture dimension score range, tone dimension score range and speech rate dimension score range.

[0121] The behavioral etiquette rule base also loads predefined description templates. These templates include a comprehensive etiquette score description (e.g., poor level corresponds to significant improvement in overall etiquette, average level corresponds to room for improvement in overall etiquette), a facial expression score description (e.g., poor level corresponds to improved smiling), a body posture score description (e.g., poor level corresponds to upright sitting posture), a tone score description (e.g., poor level corresponds to a steady tone), and a speech rate score description (e.g., poor level corresponds to a moderate speech rate). The predefined scoring ranges are based on human resources interview etiquette standards, and the predefined description templates are based on both human resources interview etiquette standards and psychological behavioral standards.

[0122] The overall etiquette score is matched with a predefined overall etiquette score range. The overall etiquette score falls into which range, and the corresponding overall etiquette score evaluation item is selected from the predefined description template.

[0123] The behavioral etiquette rule base summarizes all selected evaluation items, including comprehensive etiquette score evaluation items, facial expression dimension score evaluation items, body posture dimension score evaluation items, tone dimension score evaluation items, and speech rate dimension score evaluation items.

[0124] The collected evaluation items are organized into a structured JSON format, which includes score details, such as the overall etiquette score and the scores for each dimension, improvement suggestions, such as the evaluation item text for each dimension, and key timestamp evidence, such as the time when the violation occurred. The structured JSON format is then output as an interpretable evaluation report.

[0125] This embodiment also provides a computer device applicable to the behavioral etiquette dimension evaluation method for intelligent interviews, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the behavioral etiquette dimension evaluation method for intelligent interviews as proposed in the above embodiment.

[0126] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0127] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the behavioral etiquette dimension evaluation method for intelligent interviews as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0128] In summary, this invention achieves collaborative modeling of multi-source behavioral signals by simultaneously sampling, extracting features, and fusing cross-modal attention through an etiquette fusion model of video and audio modalities. Without relying on prior manual annotation, it utilizes the spatiotemporal representation capabilities obtained through pre-training to accurately capture key etiquette features such as facial expressions, body posture, tone of voice, and speech rate. Furthermore, by dynamically weighting the correlations between modalities through a cross-modal attention mechanism, it generates a highly discriminative etiquette fusion feature tensor, effectively preserving the consistency of behavioral temporal sequence and complementary information between modalities, thereby improving the accuracy of analyzing subtle nonverbal behaviors.

[0129] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for evaluating behavioral etiquette dimensions for intelligent interviews, characterized in that: Comprising, a generative adversarial network with rule constraints is constructed and trained using initial interview video data to output synthetic interview video data conforming to behavioral etiquette logic; a multi-modal Transformer model is pre-trained using synthetic interview video data, and the pre-trained multi-modal Transformer model is supervised fine-tuned using initial interview video data to optimize multi-modal Transformer model parameters, and output a final etiquette fusion model; the intelligent interview video data and audio data to be evaluated are input into the etiquette fusion model for feature extraction and fusion processing to generate etiquette fusion feature tensors; The etiquette fusion feature tensors are input into the evaluation component, the quantitative values in the etiquette fusion feature tensors are weighted according to the preset weights of the evaluation component, and the etiquette dimension scores are output; aggregate etiquette dimension scores to generate a comprehensive etiquette score, and generate an explainable evaluation report according to the behavioral etiquette rule base of the evaluation component; The output synthetic interview video data conforming to the behavioral etiquette logic is specifically, a U-shaped encoding-decoding structure is adopted to construct a generative adversarial network with rule constraints, random noise vectors are taken as inputs, and high-definition video frame sequences are generated through progressive upsampling operations to simulate interview videos; The simulated interview video is input into the visual discrimination branch of the double-branch discriminator, and the three-dimensional convolutional network is used to analyze the motion continuity and picture authenticity between the video frames in the simulated interview video to generate a visual rationality judgment result; A multi-task convolutional network is used to receive the simulated interview video and decompose it into independent frame sequences, and a shared backbone network is used to perform body posture feature recognition, facial expression feature recognition, and external dressing recognition to output the body posture features, facial expression features, and external dressing features of the candidate; The body posture features, facial expression features, and external dressing features are matched with the etiquette logic conditions in the predefined behavioral etiquette rule set one by one, the number of rule violations is counted, and a rule compliance score is output; The visual rationality judgment result and the rule compliance score are used as joint feedback signals for the double-branch discriminator; Load the initial interview video data as real samples, and form a training batch with the simulated interview video. Based on the joint feedback signal, iteratively update the neural network weight parameters of the video generator until the simulated interview video meets the requirements of the behavioral etiquette rule set in terms of visual performance, and output the synthetic interview video data conforming to the behavioral etiquette logic. The random noise vector is a fixed-length numerical sequence generated by the video generator of the generative adversarial network through random sampling.

2. The behavioral etiquette dimension evaluation method for intelligent interview according to claim 1, characterized in that: The multi-modal Transformer model is pre-trained using synthetic interview video data, specifically, The synthetic interview video data is input into the multi-modal Transformer model, and the video frame sequence and corresponding frame-level etiquette dimension label in the synthetic interview video data are analyzed to form a visual-label paired sample; The mask task and the cross-modal contrast learning task are performed on the visual-label paired sample, a multi-modal Transformer model is driven to learn the correlation representation between the visual and the label, and pre-training of the multi-modal Transformer model is completed.

3. The behavioral etiquette dimension evaluation method for intelligent interview according to claim 1, characterized in that: The pre-trained multi-modal Transformer model is supervised and fine-tuned by using the initial interview video data, specifically, The initial interview video data is input into the pre-trained multi-modal Transformer model, and behavior sequence features in the initial interview video data are extracted. A supervised learning task is performed on the behavior sequence features and the behavior etiquette score label in the initial interview video data, and the multi-modal Transformer model parameters are adjusted.

4. The behavioral etiquette dimension evaluation method for intelligent interview according to claim 3, characterized in that: The supervised learning task is performed, specifically, The behavior sequence features are input into a Transformer encoder, and the video frame sequence and the audio spectrum in the behavior sequence features are processed through a spatio-temporal attention mechanism to generate video modality features and audio modality features, respectively; The video modality features and the audio modality features are fused through a cross-modal attention layer to generate fusion features; A dynamic weighting process is performed on the fusion features to generate a predicted behavior etiquette score; The predicted behavior etiquette score is matched with the behavior etiquette score label in the initial interview video data to obtain deviation information of the predicted behavior etiquette score and the behavior etiquette score label; According to the deviation information of the predicted behavior etiquette score and the behavior etiquette score label, the attention parameter matrix of the spatio-temporal attention layer and the cross-modal attention layer is updated layer by layer until the deviation information is minimized, and the multi-modal Transformer model parameter adjustment is completed.

5. The behavioral etiquette dimension evaluation method for intelligent interview according to claim 1, wherein: The intelligent interview video data and the audio data to be evaluated are input into the etiquette fusion model for feature extraction and fusion processing to generate an etiquette fusion feature tensor, specifically, The intelligent interview video data and the audio data to be evaluated are input into the etiquette fusion model; The etiquette fusion model samples the frame sequence of the intelligent interview video data and segments the audio data into time segments at the same time interval to obtain synchronous video frame sequence and audio segment sequence; The candidate's facial expression features and body posture features are extracted from the video frame sequence, and the candidate's tone features and speech speed features are extracted from the audio segment sequence; The extracted facial expression features and body posture features are taken as video modality features, and the tone features and speech speed features are taken as audio modality features; The video modality features and the audio modality features are input into the cross-modal attention mechanism of the etiquette fusion model, the cross-modal interaction processing is performed on the video modality features and the audio modality features through the cross-modal attention mechanism to generate weighted fusion features as the etiquette fusion feature tensor.

6. The behavioral etiquette dimension evaluation method for intelligent interview according to claim 1, wherein: The quantitative values in the etiquette fusion feature tensor are weighted according to the weights preset by the evaluation component, and the etiquette dimension scores are output, specifically, The etiquette fusion feature tensor is input into the evaluation component, the evaluation component analyzes the facial expression features, the body posture features, the tone features, and the speech speed features in the etiquette fusion feature tensor to obtain the quantitative values of the facial expression features, the body posture features, the tone features, and the speech speed features; The evaluation component pre-sets a weight matrix to weight the quantitative values of the facial expression features, body posture features, tone features and speech speed features, and generate scores of each dimension of etiquette. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The processor implements the steps of the behavior etiquette dimension evaluation method for intelligent interview according to any one of claims 1-6 when executing the computer program.

8. A computer readable storage medium having stored thereon a computer program, characterized in that: The processor implements the steps of the behavior etiquette dimension evaluation method for intelligent interview according to any one of claims 1-6 when executing the computer program.

Images