A multi-modal emotion recognition index construction system and method

Abstract

This specification provides a multimodal emotion recognition index construction system and method, wherein the method includes: collecting first questionnaire data and first voice data of a user to be identified; preprocessing the first questionnaire data and the first voice data to obtain observed variables in the first questionnaire data and the first voice data; inputting the observed variables of the user to be identified into a trained emotion recognition model, wherein the emotion recognition model outputs values ​​corresponding to different emotion types for the user to be identified based on the correlation between the observed variables and different emotion types; and determining the true emotion type of the user to be identified based on the values ​​of the different emotion types. The method of this specification can improve the accuracy of prediction.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence, and in particular, to a multimodal emotion recognition index construction system and method. Background Technology

[0002] Emotion recognition technology has important application value in fields such as mental health assessment, workplace psychological intervention, human-computer interaction, and intelligent customer service. Accurate emotion type prediction is the core support for optimizing service experience and reducing psychological risks. Current emotion prediction technology still has significant limitations. The core problems are concentrated in the single data dimension, incomplete variable coverage, and lack of targeted analysis methods, which makes it difficult to meet the needs of emotion prediction in complex scenarios. Related technologies can be mainly divided into two categories: (1) Emotion prediction technology based on subjective data, which mostly relies on single-dimensional emotion scales (such as focusing only on depression and anxiety) to obtain feedback. The variable coverage is limited to core emotions and does not include derivative variables closely related to emotions such as sleep quality, work burnout, and somatization symptoms, resulting in an incomplete emotional characterization. Moreover, it is easily affected by the subjective bias of the subjects, and the static filling mode is difficult to capture real-time emotional fluctuations. (2) Emotion prediction technology based on objective data, which takes voice, facial expression and other data as the core. However, the ability of single objective data to distinguish complex emotions is limited, and it lacks correlation calibration with multi-dimensional subjective variables, so the prediction results are prone to deviation.

[0003] Therefore, there is an urgent need for an emotion type prediction technology that integrates multi-dimensional subjective questionnaire variables and objective voice data and adopts scientific modeling methods to overcome the problems of limited variable coverage and insufficient data integration, and improve prediction accuracy. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a multimodal emotion recognition index construction system and method to improve prediction accuracy.

[0005] To achieve the above objectives, on the one hand, embodiments of this specification provide a method for constructing multimodal emotion recognition metrics, including: Collect the first questionnaire data and the first voice data of the user to be identified; Preprocess the first questionnaire data and the first voice data to obtain the observed variables in the first questionnaire data and the first voice data; The observed variables of the user to be identified are input into the trained emotion recognition model. The emotion recognition model outputs the values ​​of the user to be identified corresponding to different emotion types based on the correlation between the observed variables and different emotion types. Based on the values ​​of the different emotion types, the true emotion type of the user to be identified is determined.

[0006] Preferably, the first questionnaire data is preprocessed to obtain the observed variables in the first questionnaire data, including: Fill in the missing values ​​in the first questionnaire data; The completed first questionnaire data was standardized to obtain first questionnaire data with uniform dimensions. The observed variables in the first questionnaire data with unified dimensions are extracted.

[0007] Preferably, the observation variables in the first speech data are obtained by preprocessing the first speech data, including: The first voice data is then subjected to noise reduction processing; The silent segments in the first speech data after noise reduction are removed to obtain the effective first speech data; The effective first speech data is extracted to obtain the observed variables in the first speech data.

[0008] Preferably, the training method for the emotion recognition model includes: Obtain a batch of user samples, as well as the second questionnaire data and second voice data of the batch of user samples; The batch of user samples were labeled using an expert double-blind annotation method, and at least a portion of the selected user samples and the true values ​​of the selected user samples corresponding to different emotion types were selected. The second questionnaire data and the second voice data of the selected user sample are preprocessed to obtain the observed variables in the second questionnaire data and the second voice data; The observed variables of the selected user samples, as well as the true values ​​of the selected user samples corresponding to different emotion types, are input into the emotion recognition model for training. The emotion recognition model obtains the corresponding emotion influencing factors based on the observed variables of the selected user samples; The emotion recognition model obtains predicted values ​​for different emotion types based on the path relationship between emotion influencing factors and different emotion types, and verifies whether the fitting index meets the standard based on the difference between the actual value and the predicted value. If so, the emotion recognition model has been trained. If not, the emotion recognition model is then optimized by first optimizing the observed variables and then optimizing the path relationships.

[0009] Preferably, the path relationship between the emotion influencing factors and different emotion types is as follows: The first weight of the emotion influencing factors for different emotion types.

[0010] Preferably, obtaining predicted values ​​for different emotion types based on the path relationship between emotion influencing factors and different emotion types includes: Obtain the second weights of the observed variables obtained from training the emotion recognition model on the emotion influencing factors; Based on the second weight and the data values ​​corresponding to the observed variables, the different emotional impact values ​​are calculated. Based on the first weight and the different emotion influence values, the predicted values ​​for different emotion types are calculated.

[0011] Preferably, the method of first optimizing the observed variables and then optimizing the path relationship includes: First, based on the loading coefficients of the observed variables obtained from training the emotion recognition model, the observed variables are added or removed. Then, based on the path relationship between the emotion influencing factors and different emotion types, the emotion influencing factors are merged or split.

[0012] On the other hand, embodiments of this specification provide a multimodal emotion recognition index construction system, the system comprising: The data acquisition module is used to collect the first questionnaire data and the first voice data of the user to be identified.

[0013] The data preprocessing module is used to preprocess the first questionnaire data and the first voice data to obtain the observed variables in the first questionnaire data and the first voice data.

[0014] The model validation module is used to input the observed variables of the user to be identified into the trained emotion recognition model. The emotion recognition model outputs the values ​​of the user to be identified corresponding to different emotion types based on the correlation between the observed variables and different emotion types.

[0015] A multimodal emotion recognition module is used to determine the true emotion type of the user to be identified based on the values ​​of the different emotion types.

[0016] In another aspect, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements the steps of any of the methods described above.

[0017] In another aspect, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of a computer device, implements the steps of any of the methods described above.

[0018] As can be seen from the technical solutions provided in the embodiments of this specification above, the embodiments of this specification, through dual-modal acquisition of questionnaires and voice, take into account both subject self-report and objective behavioral signals, ensuring the integrity and diversity of information sources. Through preprocessing, the raw data is cleaned, denoised, standardized, and the observed variables are extracted, unifying the units of measurement, filtering out redundancy, and reducing the bias caused by noise interference and individual differences. Using a trained emotion recognition model, the observed variables are mapped to scores or probabilities of different emotion types, capturing the nonlinear and interactive relationship between multiple variables and emotions, improving the ability to distinguish complex and mixed emotions and the generalization across scenarios. Decision fusion is performed based on multiple emotion scores, which can still robustly give the most credible true emotion type in conflict or ambiguous situations.

[0019] Overall, it achieves multi-source information fusion, robust noise resistance, and mutual verification of subjective and objective factors, significantly improving recognition accuracy, stability, and real-time performance. It reduces single-modal bias and human subjectivity, supports large-scale automated assessment, continuous monitoring, and the construction of individualized emotion profiles, and overcomes the problems of limited variable coverage and insufficient data fusion, thereby improving prediction accuracy and scenario applicability.

[0020] To make the above and other objects, features and advantages of this specification more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a method for constructing multimodal emotion recognition metrics provided in an embodiment of this specification is shown. Figure 2 This document illustrates a flowchart of preprocessing first questionnaire data to obtain observed variables from the first questionnaire data, as provided in an embodiment of this specification. Figure 3 This document illustrates a flowchart of preprocessing first speech data to obtain observed variables in the first speech data, as provided in an embodiment of this specification. Figure 4 A flowchart illustrating the training method of the emotion recognition model provided in the embodiments of this specification is shown. Figure 5 This document illustrates a flowchart of an embodiment of the process for obtaining predicted values ​​for different emotion types based on the path relationship between emotion influencing factors and different emotion types. Figure 6 A flowchart illustrating the method provided in the embodiments of this specification, which optimizes the observed variables first and then the path relationship, is shown. Figure 7 This diagram illustrates the modular structure of a multimodal emotion recognition index construction system provided in an embodiment of this specification. Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this specification is shown.

[0023] Explanation of symbols in the attached drawings: 100. Data acquisition module; 200. Data preprocessing module; 300. Model Validation Module; 400. Multimodal emotion recognition module; 802. Computer equipment; 804, Processor; 806. Memory; 808. Drive mechanism; 810. Input / Output Module; 812. Input devices; 814. Output devices; 816. Presentation equipment; 818. Graphical User Interface; 820. Network interface; 822. Communication link; 824. Communication bus. Detailed Implementation

[0024] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the embodiments of this specification.

[0025] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0026] To address the aforementioned issues, this specification provides a method for constructing multimodal emotion recognition metrics. Figure 1 This is a flowchart illustrating a method for constructing multimodal emotion recognition indicators provided in the embodiments of this specification. This specification provides the operational steps of the method described in the embodiments or flowcharts, but based on conventional or non-creative labor, it may include more or fewer operational steps. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0027] It should be noted that the terms "first," "second," etc., in the description, claims, and accompanying drawings of the embodiments in this specification are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0028] Reference Figure 1 A method for constructing a multimodal emotion recognition metric may include the following steps: S101: Collect the first questionnaire data and the first voice data of the user to be identified; S102: Preprocess the first questionnaire data and the first voice data to obtain the observed variables in the first questionnaire data and the first voice data; S103: Input the observed variables of the user to be identified into the trained emotion recognition model. The emotion recognition model outputs the values ​​of the user to be identified corresponding to different emotion types based on the correlation between the observed variables and different emotion types. S104: Based on the values ​​of the different emotion types, determine the true emotion type of the user to be identified.

[0029] In general, the first questionnaire data of the user to be identified is the completed questionnaire form, and the first voice data is the interview recording file of the user to be identified.

[0030] The questionnaire may consist of at least one scale and may contain multiple scales, with data from multiple dimensions. For example, the questionnaire may be the PHQ-9 scale (used to assess the severity of depression), which includes dimensions such as appetite change and attention loss. For instance, the attention loss dimension may use a range of 0-5 to represent different degrees of attention loss. Based on any data value from 0-5 actually selected by the user to be identified, the corresponding data value for the attention loss dimension in the PHQ-9 scale is obtained.

[0031] Among them, reference Figure 2 The first questionnaire data is preprocessed to obtain the observed variables in the first questionnaire data, including: S201: Fill in the missing values ​​in the first questionnaire data; Specifically, the missing values ​​in the first questionnaire data are filled with the mean of the same dimension, that is, within the same dimension, the missing parts are filled with the arithmetic mean of the non-missing data in that dimension.

[0032] S202: Standardize the completed first questionnaire data to obtain first questionnaire data with unified dimensions; The first questionnaire data, after being filled in, is standardized to obtain first questionnaire data with uniform dimensions. In this first questionnaire data with uniform dimensions, all data values ​​across all dimensions are of the same dimension, thus achieving standardization of all data values. Specifically, the Z-score standardization formula (X'=(X-μ) / σ) is used to eliminate dimensional differences, where μ is the mean of the data values ​​for each dimension, σ is the standard deviation of the data values ​​for each dimension, X is the data value before standardization, and X' is the data value after standardization, unifying data values ​​of different units or orders of magnitude into unitless values.

[0033] In addition to the data values ​​for each dimension, it is also necessary to sum the data values ​​for each dimension in the scale to obtain the total value of the scale. The total value of the scale should be consistent with the dimensions of the data values ​​for each dimension.

[0034] S203: Extract the first questionnaire data with unified dimensions to obtain the observed variables in the first questionnaire data.

[0035] For any given scale, the data values ​​of each dimension and the total value of the scale are extracted to obtain the observed variables of that scale. The observed variables of all scales are then summarized to obtain the observed variables in the first questionnaire data. The specific dimensions to extract can be set according to the actual situation, and this manual will not elaborate on this further.

[0036] In addition, refer to Figure 3 The first speech data is preprocessed to obtain the observed variables in the first speech data, including: S301: Perform noise reduction processing on the first voice data; The first voice data is an interview recording of the user to be identified. The effective duration of the recording is not less than 8 seconds and there is no obvious noise. The Librosa spectral subtraction method is used to reduce the environmental noise in the recording.

[0037] S302: Remove silent segments from the first speech data after noise reduction processing to obtain valid first speech data; Among them, the energy threshold method is used to remove the silent segments at the beginning and end of the speech, and retain the speech data with a valid speech duration greater than or equal to the set duration. The set duration can be set according to actual needs, for example, 8 seconds.

[0038] S303: Extract the valid first speech data to obtain the observed variables in the first speech data.

[0039] For example, the fundamental frequency (F0), volume (RMS), speech rate (syllables / second), spectral bandwidth, 13-dimensional Mel-frequency cepstral coefficients (MFCC), formant frequency, and fundamental frequency perturbation of the speech data are extracted as observation variables of the first speech data.

[0040] Reference Figure 4 The training method for the emotion recognition model includes: S401: Obtain a batch of user samples, as well as the second questionnaire data and second voice data of the batch of user samples; S402: The batch of user samples are labeled using an expert double-blind labeling method, and at least a portion of the selected user samples and the true values ​​of different emotion types corresponding to the selected user samples are selected. In this study, two or more psychology experts were selected. The experts were unaware of the observed variables of the sample and could only obtain the second questionnaire data and the second voice data. User identity information was not labeled to avoid the subjective bias of the experts, i.e., "double-blind labeling".

[0041] Experts are provided with the original second questionnaire data and second audio data (such as interview transcripts, questionnaires, etc.) of the user samples. Experts independently assign values ​​to each emotion type for each sample, labeling those matching the emotion type as 1 and those not matching as 0. For any user sample, only samples with a label consistency rate greater than or equal to a set percentage are accepted and selected as user samples. The label consistency rate can be set according to actual needs, for example, 80%. When multiple experts assign the same label to the same emotion type, the labeling is considered consistent; otherwise, it is considered inconsistent. The proportion of consistent labels to the total number of consistent and inconsistent labels is the label consistency rate.

[0042] 70% of the selected user samples are used as the training set and 30% as the test set. The model is first trained using the selected user samples in the training set, and then tested using the selected user samples in the test set.

[0043] S403: Preprocess the second questionnaire data and second voice data of the selected user sample to obtain the observed variables in the second questionnaire data and second voice data; For specific implementation methods, please refer to steps S201-S203 and S301-S303 above, which will not be repeated here.

[0044] S404: Input the observed variables of the selected user samples and the true values ​​of the selected user samples corresponding to different emotion types into the emotion recognition model for training; The emotion training model is a partial least squares structure equation model (PLS-SEM model), with 300 iterations and a convergence criterion of 1e. -7 .

[0045] S405: The emotion recognition model obtains the corresponding emotion influencing factor based on the observed variables of the selected user sample; The model pre-defined the correspondence between observed variables and emotion-influencing factors. These factors included: PHQ-9 depression, anxiety, sleep, burnout, somatization, susceptible personality, work-family conflict, stress, tone characteristics, volume characteristics, rhythm characteristics, and spectral characteristics. The first eight emotion-influencing factors corresponded to the observed variables in the second questionnaire data, while the last four corresponded to the observed variables in the second speech data.

[0046] Taking the correspondence between emotion influencing factors and observed variables in the second speech data as an example, pitch features correspond to fundamental frequency (F0), fundamental frequency perturbation, and formant frequency; volume features correspond to volume; rhythm features correspond to speech rate (syllables / second); and spectral features correspond to spectral bandwidth and 13-Vimel frequency cepstral coefficients (MFCC).

[0047] S406: The emotion recognition model obtains predicted values ​​for different emotion types based on the path relationship between emotion influencing factors and different emotion types, and verifies whether the fitting index meets the standard based on the difference between the actual value and the predicted value. If so, S4061: Then the emotion recognition model has been trained. If not, S4062: Then optimize the emotion recognition model by first optimizing the observed variables and then optimizing the path relationship.

[0048] In the initial state, each emotion type has a path relationship with all emotion influencing factors. The path relationship between the emotion influencing factors and different emotion types is as follows: the first weight of the emotion influencing factor on different emotion types. The first weight is used to reflect the importance of the emotion influencing factor to the emotion type. Different emotion influencing factors have different first weights on different emotion types.

[0049] The 12 emotion influencing factors (depression, anxiety, tone features, etc.) have different effects on different emotion types. By normalizing the path coefficients to obtain the first weight, the relative importance of each emotion influencing factor can be quantified (for example, it was found that "stress" and "anxiety" have the highest first weights, indicating that these two emotion influencing factors are the core dimensions affecting users' emotions).

[0050] Among them, reference Figure 5 The step of obtaining predicted values ​​for different emotion types based on the path relationship between emotion influencing factors and different emotion types includes: S501: Obtain the second weights of the observed variables obtained from the training of the emotion recognition model on the emotion influencing factors; S502: Based on the second weight and the data values ​​corresponding to the observed variables, calculate the different emotional impact values; S503: Based on the first weight and the different emotion influence values, the predicted values ​​for different emotion types are calculated.

[0051] The second weight reflects the contribution of the observed variables. Each emotion influencing factor corresponds to multiple observed variables (e.g., the latent variable "anxiety" corresponds to the total score and dimensional scores of the anxiety scale), and the contribution of these observed variables varies. By combining the first and second weights, the comprehensive weight of the observed variables can be calculated, thereby selecting the core observed variables that are most effective in predicting emotion type.

[0052] The formula for the overall weight is: Wfinal,k=Wlatent,m×Wobs,k,m; Wherein, Wlatent,m is the first weight of the m-th emotion influencing factor, Wobs,k,m is the second weight of the k-th observed variable under the m-th emotion influencing factor, and Wfinal,k is the comprehensive weight of the k-th observed variable under the m-th emotion influencing factor.

[0053] In the initial state, the emotion recognition model sets the second weight of each observed variable under the same emotion influencing factor to an average weight. The product of the second weight of any observed variable and its corresponding data value is calculated, and the sum of these products across all observed variables under the same influencing factor yields the emotion influence value of that emotion influencing factor.

[0054] In the initial state, for the same emotion type, the emotion recognition model derives the first weight of the emotion influencing factor based on the second weight of each observed variable under the same emotion influencing factor. Specifically, when the second weight is taken as an average weight in the initial state, the first weight can be directly derived using a least-squares closed-form solution.

[0055] In the initial state, the first weight of any emotion influencing factor is calculated and the product of the corresponding emotion influence value is obtained. The products of each emotion influencing factor under the same emotion type are summed to obtain the predicted value of that emotion type.

[0056] The fitting index is verified based on the difference between the true value and the predicted value. The fitting index includes the coefficient of determination R², the predictive correlation Q², and the standardized residual root mean square SRMR.

[0057] If the fit indices satisfy R²≥0.6, Q²≥0.5, and SRMR≤0.08, the model fit is considered satisfactory; otherwise, if any one of these indices is not met, the model does not meet the criteria.

[0058] If the target is not met, the model will be optimized. See details below. Figure 6 This involves optimizing the observed variables first, and then optimizing the path relationships, including: S601: First, based on the loading coefficients of the observed variables obtained from the training of the emotion recognition model, add or remove the observed variables; S602: Then, based on the path relationship between the emotion influencing factors and different emotion types, the emotion influencing factors are merged or split.

[0059] Among these, observed variables with low loading coefficients are removed, generally requiring a loading coefficient ≥ 0.7 (if the sample size is small, this can be relaxed to ≥ 0.6). Observed variables with loading coefficients < 0.6 are directly removed (for example, the loading of "a certain dimension of the anxiety scale" is only 0.4, indicating that this dimension cannot effectively reflect the latent variable of "anxiety").

[0060] To increase the number of effective observed variables, if the number of observed variables for a certain emotion influencing factor is less than the set number, the set number can be set to 3 according to actual needs (e.g., only 1 observed variable, "speech rate," for "rhythm characteristics"). Observed variables of the same type as the emotion influencing factor can be added (e.g., "number of pauses" or "average sentence length") to enhance the measurement reliability of the emotion influencing factor. The loading coefficients of the added observed variables of the same type should generally be ≥0.7 (if the sample size is small, this can be relaxed to ≥0.6). Whether the observed variables are of the same type can be determined according to a pre-set type table.

[0061] If the correlation coefficient of the path coefficients of two emotion-influencing factors is greater than the set coefficient, where the set coefficient can be set to 0.8 according to actual needs (e.g., the correlation coefficient between "stress" and "work-family conflict" is >0.8), it indicates that the two factors are collinear and can be merged into one emotion-influencing factor (e.g., "work-related stressors") to reduce path redundancy. The correlation coefficient of the path coefficient is usually obtained using Pearson correlation.

[0062] If the observed variables of a certain emotion-influencing factor can be clearly divided into two categories (such as "fundamental frequency" and "fundamental frequency perturbation" under "pitch characteristics," with large differences in loading coefficients), this latent variable can be split into two new emotion-influencing factors (such as "basic pitch" and "pitch stability") to improve measurement accuracy. Specifically, the loading coefficients of the observed variables in the emotion-influencing factor can be used to determine whether the observed variables are of the same type. For example, if the difference in the loading coefficients of two observed variables is greater than a set difference, they are considered to be of different types. The set difference can be set to 0.3 depending on the actual situation.

[0063] After optimization, the model is run again to check if the fit index has improved. The above model optimization steps are repeated until the fit index is met, and a well-trained emotion recognition model is obtained.

[0064] After the model is trained, the selected test samples in the test set are substituted into the trained emotion recognition model to calculate the emotion type prediction accuracy, precision, recall and F1 score. The accuracy is required to be ≥88%. The stability of the model is verified by confusion matrix, ROC curve and AUC value (≥0.88).

[0065] This specification's embodiments utilize a dual-modal data acquisition approach, combining questionnaires and voice recordings to consider both subjective self-reports and objective behavioral signals. This ensures the integrity and diversity of information sources. Preprocessing cleans, denoises, and standardizes the raw data, extracting observed variables, unifying dimensions, and eliminating redundancy to reduce noise interference and biases caused by individual differences. A trained emotion recognition model maps observed variables to scores or probabilities of different emotion types, capturing the nonlinear and interactive relationships between multiple variables and emotions, improving the ability to distinguish complex and mixed emotions and generalize across scenarios. Decision fusion based on multiple emotion scores ensures robust identification of the most reliable true emotion type even in conflict or ambiguous situations.

[0066] Overall, it achieves multi-source information fusion, robust noise resistance, and mutual verification of subjective and objective factors, significantly improving recognition accuracy, stability, and real-time performance. It reduces single-modal bias and human subjectivity, supports large-scale automated assessment, continuous monitoring, and the construction of individualized emotion profiles, and overcomes the problems of limited variable coverage and insufficient data fusion, thereby improving prediction accuracy and scenario applicability.

[0067] This application provides users with access to relevant big data analysis (such as personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), allowing users to choose to agree to or reject automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0068] Based on the multimodal emotion recognition index construction method described above, this specification also provides a multimodal emotion recognition index construction system. The system may include a system (including a distributed system), software (application), modules, components, servers, clients, etc., using the method described in this specification, combined with necessary hardware implementation devices. Based on the same innovative concept, the devices in one or more embodiments provided in this specification are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the devices are similar, the implementation of specific devices in this specification can refer to the implementation of the aforementioned method, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0069] Specifically, Figure 7 This is a schematic diagram of the module structure of one embodiment of a multimodal emotion recognition index construction system provided in this specification. (Refer to...) Figure 7 As shown in the embodiments of this specification, a multimodal fusion emotion recognition device includes: a data acquisition module 100, a data preprocessing module 200, a model verification module 300, and a multimodal emotion recognition module 400.

[0070] The data acquisition module 100 is used to collect the first questionnaire data and the first voice data of the user to be identified.

[0071] The data preprocessing module 200 is used to preprocess the first questionnaire data and the first voice data to obtain the observed variables in the first questionnaire data and the first voice data.

[0072] The model validation module 300 is used to input the observed variables of the user to be identified into the trained emotion recognition model. The emotion recognition model outputs the values ​​of the user to be identified corresponding to different emotion types based on the correlation between the observed variables and different emotion types.

[0073] The multimodal emotion recognition module 400 is used to determine the true emotion type of the user to be identified based on the values ​​of the different emotion types.

[0074] Reference Figure 8As shown, based on the multimodal emotion recognition index construction method described above, one embodiment of this specification also provides a computer device 802, wherein the above method runs on the computer device 802. The computer device 802 may include one or more processors 804, such as one or more central processing units (CPUs) or graphics processing units (GPUs), each processing unit may implement one or more hardware threads. The computer device 802 may also include any memory 806 for storing any kind of information such as code, settings, data, etc. In one specific embodiment, a computer program is stored on the memory 806 and can run on the processor 804. When the computer program is run by the processor 804, it can execute instructions according to the above method. Non-limitingly, for example, the memory 806 may include any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 802. In one scenario, when processor 804 executes associated instructions stored in any memory or combination of memories, computer device 802 can perform any operation of the associated instructions. Computer device 802 also includes one or more drive mechanisms 808 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0075] Computer device 802 may also include an input / output module 810 (I / O) for receiving various inputs (via input device 812) and providing various outputs (via output device 814). A specific output mechanism may include a presentation device 816 and an associated graphical user interface 818 (GUI). In other embodiments, the input / output module 810 (I / O), input device 812, and output device 814 may be omitted, and the device may function solely as a computer device within a network. Computer device 802 may also include one or more network interfaces 820 for exchanging data with other devices via one or more communication links 822. One or more communication buses 824 couple the components described above together.

[0076] Communication link 822 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0077] Corresponding to Figures 1-6In addition to the methods described above, embodiments of this specification also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the methods described above.

[0078] This specification also provides computer-readable instructions, wherein when a processor executes the instructions, the program therein causes the processor to perform the following... Figures 1 to 6 The method shown.

[0079] This specification also provides a computer program product, which, when executed by the processor of a computer device, performs the following... Figures 1 to 6 The method shown.

[0080] The computer program product described in this specification is a software product that mainly implements the methods described in this specification through a computer program.

[0081] It should be understood that in the various embodiments of this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0082] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the embodiments of this specification, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments in this specification.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0085] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described in this specification, depending on actual needs.

[0087] Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this specification, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] This specification uses specific embodiments to illustrate the principles and implementation methods of the embodiments. The above description of the embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments in this specification. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments in this specification. Therefore, the content of this specification should not be construed as a limitation on the embodiments in this specification.

Claims

1. A multi-modal emotion recognition index construction method, characterized in that, The method comprises the following steps: Collecting first questionnaire data and first voice data of a user to be identified; Preprocessing the first questionnaire data and the first voice data to obtain observation variables in the first questionnaire data and the first voice data; Inputting the observation variables of the user to be identified into a trained emotion recognition model, the emotion recognition model outputting values of different emotion types corresponding to the user to be identified based on an association between the observation variables and the different emotion types; Determining a real emotion type of the user to be identified based on the values of the different emotion types. 2.The method of claim 1, wherein, The preprocessing of the first questionnaire data to obtain observation variables in the first questionnaire data comprises the following steps: Performing missing value filling on the first questionnaire data; Performing standardization processing on the filled first questionnaire data to obtain first questionnaire data with unified dimensions; Extracting the first questionnaire data with unified dimensions to obtain observation variables in the first questionnaire data. 3.The method of claim 1, wherein, The preprocessing of the first voice data to obtain observation variables in the first voice data comprises the following steps: Performing noise reduction processing on the first voice data; Removing silent segments in the first voice data after noise reduction processing to obtain effective first voice data; Extracting the effective first voice data to obtain observation variables in the first voice data. 4.The method of claim 1, wherein, The training method of the emotion recognition model comprises the following steps: Obtaining a batch of user samples, second questionnaire data and second voice data of the batch of user samples; Labeling the batch of user samples by using an expert double-blind labeling method, selecting at least part of selected user samples and real values of different emotion types corresponding to the selected user samples; Preprocessing the second questionnaire data and the second voice data of the selected user samples to obtain observation variables in the second questionnaire data and the second voice data; Inputting the observation variables of the selected user samples and the real values of different emotion types corresponding to the selected user samples into an emotion recognition model for training; The emotion recognition model obtains emotion influence factors corresponding to the observation variables of the selected user samples; The emotion recognition model obtains prediction values of different emotion types according to a path relationship between the emotion influence factors and the different emotion types, and verifies whether a fitting index meets a standard based on a difference between the real values and the prediction values; If yes, the training of the emotion recognition model is completed; If no, the emotion recognition model is optimized based on a mode of optimizing the observation variables first and then optimizing the path relationship. 5.The method of claim 4, wherein, The path relationship between the emotion influence factors and the different emotion types is: A first weight of the emotion influence factors on the different emotion types. 6.The method of claim 5, wherein, The obtaining of the prediction values of the different emotion types according to the path relationship between the emotion influence factors and the different emotion types comprises the following steps: Obtaining a second weight of the observation variables on the emotion influence factors obtained by the emotion recognition model; Calculating different emotion influence values according to the second weight and data values corresponding to the observation variables; Calculating prediction values of different emotion types according to the first weight and the different emotion influence values. 7.The method of claim 4, wherein, The mode of optimizing the observation variables first and then optimizing the path relationship comprises the following steps: First, based on the loading coefficients of the observed variables obtained from training the emotion recognition model, the observed variables are added or removed. Then, based on the path relationship between the emotion influencing factors and different emotion types, the emotion influencing factors are merged or split. 8.A multi-modal emotion recognition index construction system, characterized in that, The system includes: The data acquisition module is used to collect the first questionnaire data and the first voice data of the user to be identified; The data preprocessing module is used to preprocess the first questionnaire data and the first voice data to obtain the observed variables in the first questionnaire data and the first voice data. The model validation module is used to input the observed variables of the user to be identified into the trained emotion recognition model. The emotion recognition model outputs the values ​​of the user to be identified corresponding to different emotion types based on the correlation between the observed variables and different emotion types. A multimodal emotion recognition module is used to determine the true emotion type of the user to be identified based on the values ​​of the different emotion types.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein, When the computer program is run by the processor, it implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is run by the processor of the computer device, it implements the steps of the method according to any one of claims 1-7.

Images