Virtual job fair interactive experience method and system

Abstract

The invention discloses a virtual recruitment fair interactive experience method and system, and relates to the technical field of virtual interaction. Audio and video data and facial analysis data of a job seeker are comprehensively acquired through a data acquisition module, multi-modal fusion is performed on voice, facial expressions, limb actions and micro-expression features, and a multi-dimensional data set W is constructed; and rich basic information is provided for subsequent analysis. The system can reflect the on-site performance of job seekers more comprehensively, and is helpful for improving the objectivity and comprehensiveness of recruitment decisions. The data processing module realizes a complete process of data cleaning, standardized processing and feature extraction, and ensures the accuracy and consistency of data. By means of an emotion analysis algorithm and a voice emotion recognition technology, the emotion analysis module generates a comprehensive emotion index Ein, and the emotion state of the job seeker can be quantified; and the communication ability analysis module calculates a communication comprehensive score Csc by combining the comprehensive emotion index Ein and facial analysis data, so that multi-dimensional evaluation of the communication ability is realized.

Description

Technical Field

[0001] The present invention relates to the field of virtual interaction technology, and in particular to a virtual job fair interactive experience method and system. Background Art

[0002] The virtual job fair interactive experience system belongs to the field of artificial intelligence and human-computer interaction, especially in the application of intelligent recruitment and emotional computing. In recent years, virtual job fairs have been widely used as an innovative form of remote recruitment. It builds an efficient and convenient communication platform for enterprises and job seekers through virtual reality and intelligent technology. In this context, how to comprehensively analyze the performance of job seekers in virtual interviews has become an important topic to improve the accuracy of recruitment, and emotion recognition and behavior analysis technology is the key tool in this field.

[0003] The current evaluation system of virtual job fairs mainly focuses on resume matching and question and answer content analysis, and pays less attention to the emotional state and communication skills of job seekers during the interview. Traditional interviewers' observations are limited in virtual scenes, and it is difficult to accurately evaluate job seekers' emotional fluctuations, tone stability, and body language in real time. This defect may lead to one-sidedness and misjudgment of recruitment decisions, especially in positions that require high emotional stability or communication skills, and companies may miss out on potential excellent candidates.

[0004] The root cause of the above problems is that the current system lacks deep fusion analysis of multimodal data, especially the lack of processing capabilities for non-verbal information such as voice, facial expressions and body movements. During the virtual job fair, job seekers' performance may be disturbed by external factors such as nervousness and network delays, further exacerbating the difficulty of recruiters' judgment. This deficiency not only reduces the efficiency of enterprises in screening excellent candidates, but may also lead to communication barriers between enterprises and candidates, affecting the recruitment experience and the shaping of corporate image. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method and system for interactive experience of a virtual job fair, which solves the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a virtual job fair interactive experience method and system, including a data collection module, a data processing module, a sentiment analysis module, a communication ability analysis module, a comprehensive evaluation module and a report generation module;

[0007] The data collection module collects audio and video data and facial analysis data of the recruitment process, specifically, the audio and video data includes voice, facial expression and body movement, and the facial analysis data includes facial emotion fluctuation coefficient Emv, voice intonation stability coefficient Tsm, body movement amplitude index Bgm and micro-expression feature occurrence frequency Psf, forming a data set W;

[0008] The data processing module cleans, standardizes and extracts features from the collected data set W to obtain a data standard set Wb;

[0009] The sentiment analysis module analyzes the emotional state of the data standard set Wb by using sentiment analysis algorithms and speech emotion recognition technology to generate a comprehensive sentiment index Ein;

[0010] The communication ability analysis module evaluates the communication ability of job seekers based on the comprehensive emotion index Ein and facial analysis data, and calculates the comprehensive communication score Csc;

[0011] The comprehensive evaluation module integrates the results of the sentiment analysis module and the communication ability analysis module, including the comprehensive sentiment index Ein and the comprehensive communication score Csc, to generate a final interview performance index Msc;

[0012] The report generation module generates an emotional feedback report according to the data standard set Wb, the comprehensive emotional index Ein, the comprehensive communication score Csc and the interview performance index Msc.

[0013] Preferably, the data acquisition module includes an audio and video data acquisition unit and a facial feature extraction unit;

[0014] The audio and video data acquisition unit collects audio and video signals, including voice, facial expressions and body movements, through a camera and a microphone;

[0015] The facial feature extraction unit extracts facial emotion and micro-expression features from the video frame through a computer vision algorithm, including the following parameters: facial emotion fluctuation coefficient Emv, voice intonation stability coefficient Tsm, body movement amplitude index Bgm and micro-expression feature occurrence frequency Psf;

[0016] The facial emotion fluctuation coefficient Emv is obtained by the following formula:

[0017]

[0018] In the formula, N represents the total number of frames collected, E face,i represents the facial emotion value extracted in the i-th frame;

[0019] The speech intonation stability coefficient Tsm is obtained by the difference between 1 and the ratio of the standard deviation of the speech frequency to the average value of the speech frequency;

[0020] The limb movement amplitude index Bgm is obtained by the ratio of the difference between the peak value of the limb movement amplitude and the valley value of the limb movement amplitude to the average value of the movement amplitude;

[0021] The frequency of occurrence of micro-expression features Psf is obtained by the ratio of the number of occurrences of micro-expression features to the total interview duration; the audio and video data and facial analysis data are fitted to form a data set W.

[0022] Preferably, the data processing module includes a data cleaning unit and a data standardization unit;

[0023] The data cleaning unit cleans the data set W to obtain a cleaned data set Wc;

[0024] Cleaning includes missing value filling, outlier removal, and noise removal;

[0025] Fill missing values ​​by using historical averages; identify and remove outliers in the data by using the Z-score method; and eliminate high-frequency noise by using smoothing filters;

[0026] The data standardization unit performs standardization processing on the data cleaning set Wc, eliminates data dimension differences, and forms a data standard set Wb.

[0027] Preferably, the emotion analysis module includes a facial emotion analysis unit and a voice emotion analysis unit;

[0028] The facial emotion analysis unit analyzes the facial emotion fluctuation coefficient Emv and the frequency of occurrence of micro-expression features Psf through an emotion analysis algorithm, analyzes the voice and facial expressions and generates a preliminary facial emotion index Efa;

[0029] The facial emotion index Efa is obtained by the following formula:

[0030] Efa=ω1*Emv+ω2*Psf;

[0031] Where ω1 and ω2 represent the preset weight values ​​of the facial emotion fluctuation coefficient Emv and the frequency of occurrence of micro-expression features Psf, respectively, and ω1+ω2≤1;

[0032] The speech emotion analysis unit analyzes the speech intonation stability coefficient Tsm and the body movement amplitude index Bgm by using speech emotion recognition technology, and generates a speech emotion index Evo in combination with the body movement;

[0033] The speech emotion index Evo is obtained by the following formula:

[0034] Evo=ω3*Tsm+ω4*Bgm;

[0035] Wherein, ω3 and ω4 represent the preset weight values ​​of the voice intonation stability coefficient Tsm and the body movement amplitude index Bgm, respectively, and ω3+ω4≤1;

[0036] Perform weighted fusion of the facial emotion index Efa and the voice emotion index Evo to obtain the comprehensive emotion index Ein;

[0037] The emotion index Ein is obtained by the following formula:

[0038] Ein=α*Efa+β*Evo;

[0039] Wherein, α and β represent the weight coefficients of the facial emotion index Efa and the voice emotion index Evo respectively, and α+β=1.

[0040] Preferably, the communication ability analysis module includes an emotion communication association assessment unit and an expression and action expression analysis unit;

[0041] The emotional communication association evaluation unit analyzes the relationship between emotional fluctuations and communication expressions by using an emotional analysis algorithm and voice emotion recognition technology, filters out interference information caused by excessive fluctuations, and uses a comprehensive emotional index Ein to evaluate the impact of emotional states on communication ability, generating an emotional communication correction coefficient Femo;

[0042] The emotional communication correction coefficient Femo is obtained by the following formula:

[0043] Femo = γ + (1-Ein);

[0044] Where Femo is the emotion communication correction coefficient, which reflects the negative impact of emotion on communication ability, ranging from 0 to γ, where γ is the peak value of the correction coefficient. It is usually set to 1 based on research on the impact of actual emotional fluctuations on communication.

[0045] The expression and action expression analysis unit comprehensively analyzes the facial emotion fluctuation coefficient Emv, the voice intonation stability coefficient Tsm and the body movement amplitude index Bgm to generate a communication index Ncom;

[0046] The communication index Ncom is obtained by the following formula:

[0047] Ncom=ω1*Emv+ω3*Tsm+ω4*Bgm;

[0048] By combining the emotional communication correction coefficient Femo and the communication index Ncom, the comprehensive communication score Csc is obtained and compared with the preset communication threshold score Tcs to judge the communication ability of the job applicant;

[0049] The communication comprehensive score Csc is obtained by the following formula:

[0050] Csc=λ1*Femo+λ2*Ncom;

[0051] In the formula, λ1 and λ2 represent the weight coefficients of the emotional communication correction coefficient Femo and the communication index Ncom respectively;

[0052] The communication skills of the job seekers are matched and obtained in the following ways:

[0053] When the comprehensive communication score Csc ≥ the communication threshold score Tcs, it means that the applicant's communication ability meets the recruitment requirements;

[0054] When the comprehensive communication score Csc is less than the communication threshold score Tcs, it means that the applicant's communication ability does not meet the recruitment requirements.

[0055] Preferably, the comprehensive evaluation module includes a data integration unit and a comprehensive scoring unit;

[0056] The data integration unit integrates the comprehensive emotion index Ein and the comprehensive communication score Csc, and performs normalization processing to unify them under dimensionless, thereby obtaining the normalized emotion index TEin and the normalized comprehensive score TCsc;

[0057] The normalized emotion index TEin is obtained by the following formula:

[0058]

[0059] In the formula, Emin represents the historical valley value of the comprehensive sentiment index Ein, and Emax represents the historical peak value of the comprehensive sentiment index Ein;

[0060]

[0061] Wherein, Cmin represents the historical valley value of the comprehensive communication score Csc, and Cmax represents the historical peak value of the comprehensive communication score Csc.

[0062] Preferably, the comprehensive scoring unit calculates and obtains the interview performance index Msc according to the obtained normalized emotion index TEin and the normalized comprehensive score TCsc;

[0063] The interview performance index Msc is obtained by the following formula:

[0064] Msc=θ1*TEin+θ2*TCsc;

[0065] Where θ1 and θ2 represent the weight coefficients of the normalized emotion index TEin and the normalized comprehensive score TCsc, respectively.

[0066] Setting of weight coefficient:

[0067] External positions: Set θ1 = 0.4, θ2 = 0.6 to highlight the importance of communication skills;

[0068] Technical positions: set θ1=0.6, θ2=0.4, and pay more attention to emotional stability;

[0069] The calculation results of the interview performance index Msc will be used as the basis for the final evaluation and can be divided into multiple levels;

[0070] When the interview performance index Msc ≥ 0.8, it means excellent;

[0071] When 0.8>interview performance index Msc>0.6, it means good;

[0072] When the interview performance index Msc≤0.6, it indicates that improvement is needed.

[0073] Preferably, the report generation module includes a data aggregation unit and a report generation unit;

[0074] The data aggregation unit integrates the data standard set Wb, the comprehensive emotion index Ein, the communication comprehensive score Csc and the interview performance index Msc to provide a data basis for generating a report;

[0075] Specifically, the data standard set Wb includes speech, facial expressions, and body movements;

[0076] The comprehensive emotional index Ein reflects the changes in emotional state at different times during the interview; the comprehensive communication score Csc comprehensively evaluates the job applicant's voice fluency, speaking speed stability, body movement coordination and facial emotional expression; the interview performance index Msc quantifies the overall performance of the job applicant.

[0077] Preferably, the report generating unit generates an emotional feedback report based on the information provided by the data summarizing unit, including statistical information, emotional analysis results, communication ability analysis results, and comprehensive performance and job recommendations;

[0078] Statistics include interview duration, voice analysis data, and facial expression analysis results;

[0079] The results of sentiment analysis include the emotional stability score of job seekers obtained by the comprehensive sentiment index Ein;

[0080] The communication ability analysis results include analysis of voice, body language and facial expression coordination through the comprehensive communication score Csc;

[0081] Overall performance and job recommendations include assessment results of emotional and communication skills obtained through the Interview Performance Index Msc.

[0082] A virtual job fair interactive experience method, comprising the following steps:

[0083] Step 1: The data collection module collects audio and video data and facial analysis data of the recruitment process. Specifically, the audio and video data include voice, facial expressions and body movements, and the facial analysis data includes facial emotion fluctuation coefficient Emv, voice intonation stability coefficient Tsm, body movement amplitude index Bgm and micro-expression feature occurrence frequency Psf, forming a data set W;

[0084] Step 2: The data processing module cleans, standardizes and extracts features from the collected data set W to obtain the data standard set Wb;

[0085] Step 3: The sentiment analysis module uses sentiment analysis algorithms and speech emotion recognition technology to analyze the emotional state of the data standard set Wb and generate a comprehensive sentiment index Ein;

[0086] Step 4: The communication ability analysis module evaluates the applicant's communication ability based on the comprehensive emotion index Ein and facial analysis data, and calculates the comprehensive communication score Csc;

[0087] Step 5: The comprehensive evaluation module integrates the results of the sentiment analysis module and the communication ability analysis module, including the comprehensive sentiment index Ein and the comprehensive communication score Csc, to generate the final interview performance index Msc;

[0088] Step 6: The report generation module generates an emotional feedback report based on the data standard set Wb, the comprehensive emotional index Ein, the comprehensive communication score Csc and the interview performance index Msc.

[0089] The present invention provides a virtual job fair interactive experience method and system, which has the following beneficial effects:

[0090] (1) When the system is running, the data acquisition module comprehensively obtains the audio and video data and facial analysis data of job seekers, and multimodally integrates the voice, facial expression, body movement and micro-expression features to construct a multidimensional data set W, which provides rich basic information for subsequent analysis. Compared with traditional recruitment methods, this system can more comprehensively reflect the on-site performance of job seekers, which helps to improve the objectivity and comprehensiveness of recruitment decisions. The data processing module implements the complete process of data cleaning, standardization processing and feature extraction to ensure the accuracy and consistency of data. By filtering noise and extracting key features, this module optimizes the efficiency of subsequent analysis links, reduces the probability of erroneous evaluation, and improves the intelligence level of the system.

[0091] (2) Through the use of mathematical models, quantitative analysis of multiple characteristic parameters is performed to convert the performance of job seekers into data-based, comparable indicators, further improving the scientificity and objectivity of the evaluation during the recruitment process. This refined processing of characteristic parameters overcomes the problem of one-sided evaluation caused by relying solely on qualitative judgment in the past. The data processing module ensures the quality and consistency of the data through the cooperation of data cleaning and standardization units. Through multi-level data cleaning such as missing value filling, outlier elimination and noise removal, as well as standardization processing to eliminate data dimension differences, the accuracy of data fitting is improved, providing reliable input data for subsequent analysis.

[0092] By constructing a high-quality multimodal dataset W and a data standard set Wb, this system significantly improves the ability to quantify the emotions and communication skills of job seekers, overcoming the one-sidedness of traditional subjective evaluation. In particular, the quantification of micro-expression feature frequency and body movement amplitude index helps to more accurately identify the emotional state and behavioral characteristics of candidates, providing more valuable reference support for recruiters.

[0093] (3) Through the communication ability analysis module, combined with the emotional communication correction coefficient Femo and the communication index Ncom, the complex relationship between emotional fluctuations and communication expression is quantitatively analyzed, avoiding the interference of excessive emotional fluctuations on the communication ability assessment, and generating a more accurate communication comprehensive score Csc, thereby providing a reliable basis for the recruitment party to judge the communication ability. The introduction of quantitative indicators such as the comprehensive emotional index Ein and the comprehensive communication score Csc, and the comparison with the preset communication threshold score Tcs, realizes the automated evaluation of the communication ability of job seekers, avoids the deviation caused by excessive subjectivity in traditional evaluation, and provides support for the scientific and data-driven recruitment process.

[0094] (4) The data collection module collects audio and video data and facial analysis data, including voice, facial expressions, body movements and characteristic coefficients, including the emotion fluctuation coefficient Emv and the tone stability coefficient Tsm, to achieve multi-dimensional data coverage of job seekers' performance, providing rich basic data for subsequent analysis and overcoming the analysis blind spots that may be caused by a single data source. The data processing module cleans, standardizes and extracts features from the collected data to form a data standard set Wb, ensuring the accuracy and consistency of the data, reducing the interference of noise and outliers, and providing a reliable foundation for subsequent sentiment analysis and comprehensive evaluation. The sentiment analysis module quantifies the emotional state of job seekers through sentiment analysis algorithms and voice emotion recognition technology, and generates a comprehensive emotion index Ein. This module effectively identifies the fluctuation patterns of emotions during the interview process, helping companies to more intuitively understand the psychological state and emotional control ability of job seekers. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1This is a flowchart diagram of a virtual job fair interactive experience system of the present invention;

[0096] Figure 2 A schematic diagram of the steps of a virtual job fair interactive experience method of the present invention. DETAILED DESCRIPTION

[0097] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0098] Example 1

[0099] The present invention provides a virtual job fair interactive experience method and system, please refer to Figure 1 , including data collection module, data processing module, sentiment analysis module, communication ability analysis module, comprehensive evaluation module and report generation module;

[0100] The data collection module collects audio and video data and facial analysis data of the recruitment process, specifically, the audio and video data includes voice, facial expression and body movement, and the facial analysis data includes facial emotion fluctuation coefficient Emv, voice intonation stability coefficient Tsm, body movement amplitude index Bgm and micro-expression feature occurrence frequency Psf, forming a data set W;

[0101] The data processing module cleans, standardizes and extracts features from the collected data set W to obtain a data standard set Wb;

[0102] The sentiment analysis module analyzes the emotional state of the data standard set Wb by using sentiment analysis algorithms and speech emotion recognition technology to generate a comprehensive sentiment index Ein;

[0103] The communication ability analysis module evaluates the communication ability of job seekers based on the comprehensive emotion index Ein and facial analysis data, and calculates the comprehensive communication score Csc;

[0104] The comprehensive evaluation module integrates the results of the sentiment analysis module and the communication ability analysis module, including the comprehensive sentiment index Ein and the comprehensive communication score Csc, to generate a final interview performance index Msc;

[0105] The report generation module generates an emotional feedback report according to the data standard set Wb, the comprehensive emotional index Ein, the comprehensive communication score Csc and the interview performance index Msc.

[0106] In this embodiment, the audio and video data and facial analysis data of job seekers are fully obtained through the data acquisition module, and the voice, facial expression, body movement and micro-expression features are multimodally fused to construct a multidimensional data set W, which provides rich basic information for subsequent analysis. Compared with traditional recruitment methods, this system can more comprehensively reflect the on-site performance of job seekers, which helps to improve the objectivity and comprehensiveness of recruitment decisions. The data processing module realizes the complete process of data cleaning, standardization processing and feature extraction to ensure the accuracy and consistency of data. By filtering noise and extracting key features, this module optimizes the efficiency of subsequent analysis links, reduces the probability of erroneous evaluation, and improves the intelligence level of the system.

[0107] With the help of sentiment analysis algorithms and voice emotion recognition technology, the sentiment analysis module generates a comprehensive sentiment index Ein, which can quantify the emotional state of job seekers; the communication ability analysis module further combines Ein and facial analysis data to calculate the comprehensive communication score Csc, realizing a multi-dimensional evaluation of communication ability. This quantitative analysis-based approach effectively makes up for the limitations of traditional subjective judgment and significantly improves the accuracy and reliability of the evaluation.

[0108] The comprehensive evaluation module generates the interview performance index Msc by integrating the results of emotion and communication ability analysis, providing a unified evaluation index for recruiters, helping companies to quickly screen high-potential candidates and simplify the recruitment process. The report generation module presents the analysis results in the form of an emotional feedback report, which is intuitive and comprehensive, providing recruiters with a more scientific basis for decision-making, helping to quickly identify excellent candidates who match the job requirements, and improving the recruitment experience.

[0109] Example 2

[0110] This embodiment is explained in Example 1, please refer to Figure 1 ,Specifically: the data acquisition module includes an audio and video data acquisition unit and a facial feature extraction unit;

[0111] The audio and video data acquisition unit collects audio and video signals, including voice, facial expressions and body movements, through a camera and a microphone;

[0112] The facial feature extraction unit extracts facial emotion and micro-expression features from the video frame through a computer vision algorithm, including the following parameters: facial emotion fluctuation coefficient Emv, voice intonation stability coefficient Tsm, body movement amplitude index Bgm and micro-expression feature occurrence frequency Psf;

[0113] The facial emotion fluctuation coefficient Emv is obtained by the following formula:

[0114]

[0115] In the formula, N represents the total number of frames collected, E face,i represents the facial emotion value extracted in the i-th frame;

[0116] The speech intonation stability coefficient Tsm is obtained by the difference between 1 and the ratio of the standard deviation of the speech frequency to the average value of the speech frequency;

[0117] The limb movement amplitude index Bgm is obtained by the ratio of the difference between the peak value of the limb movement amplitude and the valley value of the limb movement amplitude to the average value of the movement amplitude;

[0118] The frequency of occurrence of micro-expression features Psf is obtained by the ratio of the number of occurrences of micro-expression features to the total interview duration; the audio and video data and facial analysis data are fitted to form a data set W.

[0119] The data processing module includes a data cleaning unit and a data standardization unit;

[0120] The data cleaning unit cleans the data set W to obtain a cleaned data set Wc;

[0121] Cleaning includes missing value filling, outlier removal, and noise removal;

[0122] Fill missing values ​​by using historical averages; identify and remove outliers in the data by using the Z-score method; and eliminate high-frequency noise by using smoothing filters;

[0123] The data standardization unit performs standardization processing on the data cleaning set Wc to eliminate the data dimension difference and form the data standard set Wb

[0124] In this embodiment, the data acquisition module accurately acquires multimodal data such as voice, facial expression, body movements and micro-expression features through the collaborative work of the audio and video data acquisition unit and the facial feature extraction unit. The computer vision algorithm is used to extract facial features, which provides high-precision basic data for emotion analysis and communication ability assessment in the recruitment process, solving the blind spot of emotion and body language analysis in traditional recruitment.

[0125] This embodiment uses a mathematical model to quantitatively analyze multiple characteristic parameters, converts the performance of job seekers into data-based, comparable indicators, and further improves the scientificity and objectivity of the evaluation in the recruitment process. This refined processing of characteristic parameters overcomes the problem of one-sided evaluation caused by relying solely on qualitative judgment in the past. The data processing module ensures the quality and consistency of the data through the cooperation of data cleaning and standardization units. Multi-level data cleaning through missing value filling, outlier elimination and noise removal, as well as standardized processing to eliminate data dimensional differences, improves the accuracy of data fitting, and provides reliable input data for subsequent analysis.

[0126] By constructing a high-quality multimodal dataset W and a data standard set Wb, this system significantly improves the ability to quantify the emotions and communication skills of job seekers, overcoming the one-sidedness of traditional subjective evaluation. In particular, the quantification of micro-expression feature frequency and body movement amplitude index helps to more accurately identify the emotional state and behavioral characteristics of candidates, providing more valuable reference support for recruiters.

[0127] Example 3

[0128] This embodiment is explained in Example 2. Please refer to Figure 1 ,Specifically: the emotion analysis module includes a facial emotion analysis unit and a voice emotion analysis unit;

[0129] The facial emotion analysis unit analyzes the facial emotion fluctuation coefficient Emv and the frequency of occurrence of micro-expression features Psf through an emotion analysis algorithm, analyzes the voice and facial expressions and generates a preliminary facial emotion index Efa;

[0130] The facial emotion index Efa is obtained by the following formula:

[0131] Efa=ω1*Emv+ω2*Psf;

[0132] Where ω1 and ω2 represent the preset weight values ​​of the facial emotion fluctuation coefficient Emv and the frequency of occurrence of micro-expression features Psf, respectively, and ω1+ω2≤1;

[0133] The speech emotion analysis unit analyzes the speech intonation stability coefficient Tsm and the body movement amplitude index Bgm by using speech emotion recognition technology, and generates a speech emotion index Evo in combination with the body movement;

[0134] The speech emotion index Evo is obtained by the following formula:

[0135] Evo=ω3*Tsm+ω4*Bgm;

[0136] Wherein, ω3 and ω4 represent the preset weight values ​​of the voice intonation stability coefficient Tsm and the body movement amplitude index Bgm, respectively, and ω3+ω4≤1;

[0137] Perform weighted fusion of the facial emotion index Efa and the voice emotion index Evo to obtain the comprehensive emotion index Ein;

[0138] The emotion index Ein is obtained by the following formula:

[0139] Ein=α*Efa+β*Evo;

[0140] Wherein, α and β represent the weight coefficients of the facial emotion index Efa and the voice emotion index Evo respectively, and α+β=1.

[0141] The communication ability analysis module includes an emotion communication association assessment unit and an expression and action expression analysis unit;

[0142] The emotional communication association evaluation unit analyzes the relationship between emotional fluctuations and communication expressions by using an emotional analysis algorithm and voice emotion recognition technology, filters out interference information caused by excessive fluctuations, and uses a comprehensive emotional index Ein to evaluate the impact of emotional states on communication ability, generating an emotional communication correction coefficient Femo;

[0143] The emotional communication correction coefficient Femo is obtained by the following formula:

[0144] Femo = γ + (1-Ein);

[0145] In the formula, γ represents the peak value of the correction coefficient;

[0146] The expression and action expression analysis unit comprehensively analyzes the facial emotion fluctuation coefficient Emv, the voice intonation stability coefficient Tsm and the body movement amplitude index Bgm to generate a communication index Ncom;

[0147] The communication index Ncom is obtained by the following formula:

[0148] Ncom=ω1*Emv+ω3*Tsm+ω4*Bgm;

[0149] By combining the emotional communication correction coefficient Femo and the communication index Ncom, the comprehensive communication score Csc is obtained and compared with the preset communication threshold score Tcs to judge the communication ability of the job applicant;

[0150] The communication comprehensive score Csc is obtained by the following formula:

[0151] Csc=λ1*Femo+λ2*Ncom;

[0152] In the formula, λ1 and λ2 represent the weight coefficients of the emotional communication correction coefficient Femo and the communication index Ncom respectively;

[0153] The communication skills of the job seekers are matched and obtained in the following ways:

[0154] When the comprehensive communication score Csc ≥ the communication threshold score Tcs, it means that the applicant's communication ability meets the recruitment requirements;

[0155] When the communication comprehensive score Csc is less than the communication threshold score Tcs, it means that the applicant's communication ability does not meet the recruitment requirements.

[0156] In this embodiment, the multimodal feature parameters are deeply analyzed through the sentiment analysis module to construct the facial emotion index Efa and the voice emotion index Evo, which are then merged into the comprehensive emotion index Ein. This multi-angle sentiment analysis overcomes the misjudgment problem caused by a single emotion dimension, significantly improves the accuracy and comprehensiveness of emotion recognition, and lays the foundation for the comprehensive evaluation of the emotional state of job seekers.

[0157] Through the communication ability analysis module, combined with the emotional communication correction coefficient Femo and the communication index Ncom, the complex relationship between emotional fluctuations and communication expression is quantitatively analyzed, avoiding the interference of excessive emotional fluctuations on the communication ability assessment, and generating a more accurate communication comprehensive score Csc, thereby providing a reliable basis for the recruitment party to judge the communication ability. The introduction of quantitative indicators such as the comprehensive emotional index Ein and the comprehensive communication score Csc, and the comparison with the preset communication threshold score Tcs, realizes the automated evaluation of the communication ability of job seekers, avoids the deviation caused by excessive subjectivity in traditional evaluation, and provides support for the scientific and data-driven recruitment process.

[0158] The joint analysis of the facial emotion fluctuation coefficient Emv, micro-expression feature frequency Psf, voice intonation stability coefficient Tsm, and body movement amplitude index Bgm improves the dimension and accuracy of emotion and communication ability assessment. The weighted fusion of the comprehensive emotion index Ein breaks through the bottleneck of one-sided analysis caused by a single emotion feature. The emotion communication association assessment unit filters out the interference information of excessive emotion fluctuations on communication ability analysis through the emotion communication correction coefficient Femo, ensuring the stability and scientificity of the assessment process.

[0159] Through multimodal feature data fusion analysis, the problem of single dimension in sentiment analysis and communication ability assessment is solved.

[0160] The constructed comprehensive emotional index Ein and comprehensive communication score Csc indicators have significantly improved the scientificity, objectivity and data-driven ability of the evaluation. The relationship between emotional fluctuations and communication skills is converted into a quantitative model, avoiding subjective interference in traditional evaluations and providing a scientific basis for recruitment decisions. The systematic process and algorithm design provide comprehensive support for the intelligentization and efficiency optimization of recruitment evaluation.

[0161] Example 4

[0162] This embodiment is explained in Example 3, please refer to Figure 1 ,Specifically: the comprehensive evaluation module includes a data integration unit and a comprehensive scoring unit;

[0163] The data integration unit integrates the comprehensive emotion index Ein and the comprehensive communication score Csc, and performs normalization processing to unify them under dimensionless, thereby obtaining the normalized emotion index TEin and the normalized comprehensive score TCsc;

[0164] The normalized emotion index TEin is obtained by the following formula:

[0165]

[0166] In the formula, Emin represents the historical valley value of the comprehensive sentiment index Ein, and Emax represents the historical peak value of the comprehensive sentiment index Ein;

[0167]

[0168] Wherein, Cmin represents the historical valley value of the comprehensive communication score Csc, and Cmax represents the historical peak value of the comprehensive communication score Csc.

[0169] The comprehensive scoring unit calculates and obtains the interview performance index Msc according to the obtained normalized emotion index TEin and the normalized comprehensive score TCsc;

[0170] The interview performance index Msc is obtained by the following formula:

[0171] Msc=θ1*TEin+θ2*TCsc;

[0172] Where θ1 and θ2 represent the weight coefficients of the normalized emotion index TEin and the normalized comprehensive score TCsc, respectively.

[0173] The report generation module includes a data aggregation unit and a report generation unit;

[0174] The data aggregation unit integrates the data standard set Wb, the comprehensive emotion index Ein, the communication comprehensive score Csc and the interview performance index Msc to provide a data basis for generating a report;

[0175] Specifically, the data standard set Wb includes speech, facial expressions, and body movements;

[0176] The comprehensive emotional index Ein reflects the changes in emotional state at different times during the interview; the comprehensive communication score Csc comprehensively evaluates the job applicant's voice fluency, speaking speed stability, body movement coordination and facial emotional expression; the interview performance index Msc quantifies the overall performance of the job applicant.

[0177] The report generating unit generates an emotional feedback report based on the information provided by the data aggregation unit, including statistical information, emotional analysis results, communication ability analysis results, and comprehensive performance and job recommendations;

[0178] Statistics include interview duration, voice analysis data, and facial expression analysis results;

[0179] The results of sentiment analysis include the emotional stability score of job seekers obtained by the comprehensive sentiment index Ein;

[0180] The communication ability analysis results include analysis of voice, body language and facial expression coordination through the comprehensive communication score Csc;

[0181] Overall performance and job recommendations including emotional and communication skills assessments obtained through the Interview Performance Index Msc

[0182] In this embodiment, the data integration unit normalizes the comprehensive emotion index Ein and the comprehensive communication score Csc, eliminating the interference problem caused by different dimensions of the data, and providing a scientific basis for subsequent comprehensive evaluation and comparison. By calculating the interview performance index Msc, the emotion and communication ability indicators are effectively integrated to reflect the comprehensive interview performance of job seekers in a quantitative manner, avoiding the limitations of single-dimensional evaluation and providing a comprehensive reference for recruitment decisions. The data aggregation unit integrates the emotion index Ein, the comprehensive communication score Csc and the standardized data Wb into a unified data set, and ensures that the evaluation results are comprehensive and accurate through multi-dimensional data analysis, taking into account both the emotional stability during the interview and the coordination of voice, expression and body movements.

[0183] The report generation unit converts the analysis results into a visual emotional feedback report, covering statistical information, sentiment analysis, communication ability evaluation and job recommendations, which significantly improves the interpretability and operability of the interview results and provides intelligent support for the company's recruitment decisions. The normalized emotional index TEin and the normalized comprehensive score TCsc are calculated through the normalization formula, eliminating the dimensional difference, ensuring the reasonable weight distribution of different indicators, and improving the scientific nature of the comprehensive analysis. This improvement overcomes the calculation bias caused by unit differences in traditional analysis and enhances the ability to compare and screen job applicants.

[0184] The introduction of the interview performance index Msc organically combines the two key indicators of emotion and communication ability to form quantitative data that can be directly used for comparison and decision-making. This improvement solves the problem of one-sided information caused by the fragmentation of separate analysis indicators and provides a comprehensive reference for interview evaluation.

[0185] The report generation module generates detailed emotional feedback reports through standardized data integration and multi-dimensional analysis, including interview statistics, emotional analysis, communication ability assessment, and job recommendations. This improvement optimizes the processing efficiency of interview data, reduces human involvement and subjective bias, and brings an efficient and professional automated experience to the interview process.

[0186] Example 5

[0187] A virtual job fair interactive experience method, please refer to Figure 2 , specifically: including the following steps:

[0188] Step 1: The data collection module collects audio and video data and facial analysis data of the recruitment process. Specifically, the audio and video data include voice, facial expressions and body movements, and the facial analysis data includes facial emotion fluctuation coefficient Emv, voice intonation stability coefficient Tsm, body movement amplitude index Bgm and micro-expression feature occurrence frequency Psf, forming a data set W;

[0189] Step 2: The data processing module cleans, standardizes and extracts features from the collected data set W to obtain the data standard set Wb;

[0190] Step 3: The sentiment analysis module uses sentiment analysis algorithms and speech emotion recognition technology to analyze the emotional state of the data standard set Wb and generate a comprehensive sentiment index Ein;

[0191] Step 4: The communication ability analysis module evaluates the applicant's communication ability based on the comprehensive emotion index Ein and facial analysis data, and calculates the comprehensive communication score Csc;

[0192] Step 5: The comprehensive evaluation module integrates the results of the sentiment analysis module and the communication ability analysis module, including the comprehensive sentiment index Ein and the comprehensive communication score Csc, to generate the final interview performance index Msc;

[0193] Step 6: The report generation module generates an emotional feedback report based on the data standard set Wb, the comprehensive emotional index Ein, the comprehensive communication score Csc and the interview performance index Msc.

[0194] In this embodiment, through steps one to six: the data acquisition module collects audio and video data and facial analysis data, including voice, facial expressions, body movements and characteristic coefficients including emotion fluctuation coefficient Emv and tone stability coefficient Tsm, thereby achieving multi-dimensional data coverage of job applicants' performance, providing rich basic data for subsequent analysis, and overcoming the analysis blind spots that may be caused by a single data source.

[0195] The data processing module cleans, standardizes and extracts features from the collected data to form a data standard set Wb, ensuring the accuracy and consistency of the data, reducing the interference of noise and outliers, and providing a reliable basis for subsequent sentiment analysis and comprehensive evaluation. The sentiment analysis module quantifies the emotional state of job seekers through sentiment analysis algorithms and voice emotion recognition technology, and generates a comprehensive emotion index Ein. This module effectively identifies the fluctuation patterns of emotions during the interview process, helping companies to more intuitively understand the psychological state and emotional control ability of job seekers.

[0196] The communication ability analysis module evaluates the job applicant's voice fluency, intonation stability and body coordination based on the comprehensive emotion index Ein and facial analysis data, and quantifies the comprehensive communication score Csc. This method directly reflects the job applicant's communication ability and provides a scientific evaluation basis for companies in the recruitment process. The comprehensive evaluation module integrates the comprehensive emotion index Ein and the comprehensive communication score Csc to generate the final interview performance index Msc. Through this quantitative indicator, companies can fully understand the performance of job applicants and avoid the problem of one-sided information that may be caused by single-dimensional evaluation.

[0197] The report generation module generates detailed sentiment feedback reports based on multi-dimensional data, including comprehensive sentiment analysis, communication ability assessment and comprehensive evaluation of interview performance, which significantly improves the efficiency of the recruitment process and the interpretability of feedback results, and provides comprehensive support for decision-making.

[0198] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A virtual job fair interactive experience system, characterized by: It includes data collection module, data processing module, sentiment analysis module, communication ability analysis module, comprehensive evaluation module and report generation module; The data collection module collects audio and video data and facial analysis data of the recruitment process, the audio and video data includes voice, facial expression and body movement, the facial analysis data includes facial emotion fluctuation coefficient Emv, voice intonation stability coefficient Tsm, body movement amplitude index Bgm and micro-expression feature occurrence frequency Psf, forming a data set W; The data processing module cleans, standardizes and extracts features from the collected data set W to obtain a data standard set Wb; The sentiment analysis module analyzes the emotional state of the data standard set Wb by using sentiment analysis algorithms and speech emotion recognition technology to generate a comprehensive sentiment index Ein; The communication ability analysis module evaluates the communication ability of job seekers based on the comprehensive emotion index Ein and facial analysis data, and calculates the comprehensive communication score Csc; The comprehensive evaluation module integrates the results of the sentiment analysis module and the communication ability analysis module, including the comprehensive sentiment index Ein and the comprehensive communication score Csc, to generate a final interview performance index Msc; The report generation module generates an emotional feedback report according to the data standard set Wb, the comprehensive emotional index Ein, the comprehensive communication score Csc and the interview performance index Msc.

2. A virtual job fair interactive experience system according to claim 1, characterized in that: The data acquisition module includes an audio and video data acquisition unit and a facial feature extraction unit; The audio and video data acquisition unit collects audio and video signals, including voice, facial expressions and body movements, through a camera and a microphone; The facial feature extraction unit extracts facial emotion and micro-expression features from the video frame through a computer vision algorithm, including the following parameters: facial emotion fluctuation coefficient Emv, voice intonation stability coefficient Tsm, body movement amplitude index Bgm and micro-expression feature occurrence frequency Psf; The facial emotion fluctuation coefficient Emv is obtained by the following formula: In the formula, N represents the total number of frames collected, E face,i represents the facial emotion value extracted in the i-th frame; The speech intonation stability coefficient Tsm is obtained by the difference between 1 and the ratio of the standard deviation of the speech frequency to the average value of the speech frequency; The limb movement amplitude index Bgm is obtained by the ratio of the difference between the peak value of the limb movement amplitude and the valley value of the limb movement amplitude to the average value of the movement amplitude; The frequency of occurrence of micro-expression features Psf is obtained by the ratio of the number of occurrences of micro-expression features to the total interview duration; the audio and video data and facial analysis data are fitted to form a data set W.

3. A virtual job fair interactive experience system according to claim 1, characterized in that: The data processing module includes a data cleaning unit and a data standardization unit; The data cleaning unit cleans the data set W to obtain a cleaned data set Wc; Cleaning includes missing value filling, outlier removal, and noise removal; Fill missing values ​​by using historical averages; identify and remove outliers in the data by using the Z-score method; and eliminate high-frequency noise by using smoothing filters; The data standardization unit performs standardization processing on the data cleaning set Wc, eliminates data dimension differences, and forms a data standard set Wb.

4. A virtual job fair interactive experience system according to claim 1, characterized in that: The emotion analysis module includes a facial emotion analysis unit and a voice emotion analysis unit; The facial emotion analysis unit analyzes the facial emotion fluctuation coefficient Emv and the frequency of occurrence of micro-expression features Psf through an emotion analysis algorithm, analyzes the voice and facial expressions and generates a preliminary facial emotion index Efa; The facial emotion index Efa is obtained by the following formula: Efa=ω1*Emv+ω2*Psf; Where ω1 and ω2 represent the preset weight values ​​of the facial emotion fluctuation coefficient Emv and the frequency of occurrence of micro-expression features Psf, respectively, and ω1+ω2≤1; The speech emotion analysis unit analyzes the speech intonation stability coefficient Tsm and the body movement amplitude index Bgm by using speech emotion recognition technology, and generates a speech emotion index Evo in combination with the body movement; The speech emotion index Evo is obtained by the following formula: Evo=ω3*Tsm+ω4*Bgm; Wherein, ω3 and ω4 represent the preset weight values ​​of the voice intonation stability coefficient Tsm and the body movement amplitude index Bgm, respectively, and ω3+ω4≤1; Perform weighted fusion of the facial emotion index Efa and the voice emotion index Evo to obtain the comprehensive emotion index Ein; The emotion index Ein is obtained by the following formula: Ein=α*Efa+β*Evo; Wherein, α and β represent the weight coefficients of the facial emotion index Efa and the voice emotion index Evo respectively, and α+β=1.

5. The virtual job fair interactive experience system according to claim 1, characterized in that: The communication ability analysis module includes an emotion communication association assessment unit and an expression and action expression analysis unit; The emotional communication association evaluation unit analyzes the relationship between emotional fluctuations and communication expressions by using an emotional analysis algorithm and voice emotion recognition technology, filters out interference information caused by excessive fluctuations, and uses a comprehensive emotional index Ein to evaluate the impact of emotional states on communication ability, generating an emotional communication correction coefficient Femo; The emotional communication correction coefficient Femo is obtained by the following formula: Femo = γ + (1-Ein); In the formula, γ represents the peak value of the correction coefficient; The expression and action expression analysis unit comprehensively analyzes the facial emotion fluctuation coefficient Emv, the voice intonation stability coefficient Tsm and the body movement amplitude index Bgm to generate a communication index Ncom; The communication index Ncom is obtained by the following formula: Ncom=ω1*Emv+ω3*Tsm+ω4*Bgm; By combining the emotional communication correction coefficient Femo and the communication index Ncom, the comprehensive communication score Csc is obtained and compared with the preset communication threshold score Tcs to judge the communication ability of the job applicant; The communication comprehensive score Csc is obtained by the following formula: Csc=λ1*Femo+λ2*Ncom; In the formula, λ1 and λ2 represent the weight coefficients of the emotional communication correction coefficient Femo and the communication index Ncom respectively; The communication skills of the job seekers are matched and obtained in the following ways: When the comprehensive communication score Csc ≥ the communication threshold score Tcs, it means that the applicant's communication ability meets the recruitment requirements; When the comprehensive communication score Csc is less than the communication threshold score Tcs, it means that the applicant's communication ability does not meet the recruitment requirements.

6. A virtual job fair interactive experience system according to claim 1, characterized in that: The comprehensive evaluation module includes a data integration unit and a comprehensive scoring unit; The data integration unit integrates the comprehensive emotion index Ein and the comprehensive communication score Csc, and performs normalization processing to obtain the normalized emotion index TEin and the normalized comprehensive score TCsc; The normalized emotion index TEin is obtained by the following formula: In the formula, Emin represents the historical valley value of the comprehensive sentiment index Ein, and Emax represents the historical peak value of the comprehensive sentiment index Ein; Wherein, Cmin represents the historical valley value of the comprehensive communication score Csc, and Cmax represents the historical peak value of the comprehensive communication score Csc.

7. A virtual job fair interactive experience system according to claim 6, characterized in that: The comprehensive scoring unit calculates and obtains the interview performance index Msc according to the obtained normalized emotion index TEin and the normalized comprehensive score TCsc; The interview performance index Msc is obtained by the following formula: Msc=θ1*TEin+θ2*TCsc; Where θ1 and θ2 represent the weight coefficients of the normalized emotion index TEin and the normalized comprehensive score TCsc, respectively.

8. The virtual job fair interactive experience system according to claim 1, characterized in that: The report generation module includes a data aggregation unit and a report generation unit; The data aggregation unit integrates the data standard set Wb, the comprehensive emotion index Ein, the communication comprehensive score Csc and the interview performance index Msc to provide a data basis for generating a report; Specifically, the data standard set Wb includes speech, facial expressions, and body movements; The comprehensive emotional index Ein reflects the changes in emotional state at different times during the interview; the comprehensive communication score Csc comprehensively evaluates the job applicant's voice fluency, speaking speed stability, body movement coordination and facial emotional expression; the interview performance index Msc quantifies the overall performance of the job applicant.

9. A virtual job fair interactive experience system according to claim 8, characterized in that: The report generating unit generates an emotional feedback report based on the information provided by the data aggregation unit, including statistical information, emotional analysis results, communication ability analysis results, and comprehensive performance and job recommendations; Statistics include interview duration, voice analysis data, and facial expression analysis results; The sentiment analysis results include the emotional stability score of job seekers obtained by the comprehensive sentiment index Ein; The communication ability analysis results include analysis of voice, body language and facial expression coordination through the comprehensive communication score Csc; Overall performance and job recommendations include assessment results of emotional and communication skills obtained through the Interview Performance Index Msc.

10. A virtual job fair interactive experience method, applied to a virtual job fair interactive experience system as claimed in any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: The data collection module collects audio and video data and facial analysis data of the recruitment process. Specifically, the audio and video data include voice, facial expressions and body movements, and the facial analysis data includes facial emotion fluctuation coefficient Emv, voice intonation stability coefficient Tsm, body movement amplitude index Bgm and micro-expression feature occurrence frequency Psf, forming a data set W; Step 2: The data processing module cleans, standardizes and extracts features from the collected data set W to obtain the data standard set Wb; Step 3: The sentiment analysis module uses sentiment analysis algorithms and speech emotion recognition technology to analyze the emotional state of the data standard set Wb and generate a comprehensive sentiment index Ein; Step 4: The communication ability analysis module evaluates the applicant's communication ability based on the comprehensive emotion index Ein and facial analysis data, and calculates the comprehensive communication score Csc; Step 5: The comprehensive evaluation module integrates the results of the sentiment analysis module and the communication ability analysis module, including the comprehensive sentiment index Ein and the comprehensive communication score Csc, to generate the final interview performance index Msc; Step 6: The report generation module generates an emotional feedback report based on the data standard set Wb, the comprehensive emotional index Ein, the comprehensive communication score Csc and the interview performance index Msc.

Images