A human emotion recognition method and system based on multi-modal sensor fusion

Abstract

The application relates to the fields of artificial intelligence, biometric identification and human-computer interaction, and provides a human emotion recognition method and system based on multi-modal sensor fusion. The method comprises the following steps: collecting audio, face and physiological three modal human data and recording time stamps respectively; calculating the offset of the time stamp of the human data corresponding to each mode from the current time, updating the clock drift compensation value of each mode through an exponential moving average method, and correcting the time stamp of the human data corresponding to each mode; calculating emotion category weights according to the human data, generating a few-class emotion synthesis sample, detecting available human data and corresponding modes in the synthesis sample; if only one mode has available human data, single-modal emotion prediction is performed; if multiple modes have available human data, the human data are input into a preset emotion prediction model for multi-modal fusion prediction, the reliability of each mode is evaluated, and a final emotion prediction result is obtained.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, biometric recognition and human-computer interaction, and in particular to a method and system for human emotion recognition based on multimodal sensor fusion. Background Technology

[0002] With the rapid development of artificial intelligence and human-computer interaction technologies, human emotion recognition technology has been widely applied in fields such as intelligent customer service, medical diagnosis, and education and training. However, single-modal emotion recognition technology is limited by a single data source, is easily affected by environmental interference, and its recognition accuracy and robustness are insufficient to meet the needs of practical applications. Therefore, multimodal fusion emotion recognition technology has become a current research hotspot.

[0003] However, existing multimodal emotion recognition technologies generally suffer from the following three technical shortcomings: 1. The clocks of multimodal data acquisition devices have inherent drift, and the timestamps of data acquired by different devices cannot be accurately aligned, resulting in timing misalignment during fusion, which seriously affects the recognition accuracy; 2. Emotional data naturally suffers from an imbalance in category distribution. Samples of minority emotions such as fear and disgust are scarce, and existing models generally favor the majority of emotions, resulting in extremely low accuracy in identifying minority emotions. 3. In real-world applications, some modalities may temporarily fail due to equipment malfunctions, environmental occlusion, or other reasons. Existing multimodal fusion models cannot adaptively handle the situation of missing modalities. Once data for a particular modality is lost, the performance of the entire system will drop sharply or even become completely unusable.

[0004] Therefore, a method is urgently needed to solve at least one of the above problems. Summary of the Invention

[0005] This application provides a method and system for human emotion recognition based on multimodal sensor fusion, aiming to solve the problem that existing technologies can usually only solve one of the problems individually, and there is no multimodal emotion recognition method that can simultaneously and systematically solve the three core technical challenges. This greatly limits the practical application of multimodal emotion recognition technology.

[0006] In a first aspect, embodiments of this application provide a method for human emotion recognition based on multimodal sensor fusion, the method comprising: Collect human data in three modalities: audio, facial, and physiological, and record timestamps for each. Calculate the offset between the timestamp of the human data corresponding to each modality and the current time, update the clock drift compensation value for each modality using an exponential moving average method, and correct the timestamp of the human data corresponding to each modality. Based on the human body data, calculate the weight of the emotion category, generate synthetic samples of minority emotions, and detect available human body data and corresponding modalities in the synthetic samples. If only one modality has available human data, single-modal emotion prediction is performed. If multiple modalities have available human data, the data is input into a preset emotion prediction model for multimodal fusion prediction. The reliability of each modality is evaluated and combined to obtain the final emotion prediction result.

[0007] In some embodiments, after evaluating the reliability of each modality and combining them to obtain the final emotion prediction result, the method further includes: constructing a retrieval-enhanced generative knowledge base containing knowledge about the laws of human physiological changes and the correlation between medical emotions; extracting human data features corresponding to the final emotion prediction result; converting the human data features into retrieval query statements; retrieving medical knowledge entries that match the human data features from the retrieval-enhanced generative knowledge base; verifying and correcting the final emotion prediction result based on the retrieved medical knowledge entries; and outputting the corrected final emotion prediction result.

[0008] This embodiment, based on the basic embodiment, adds a step for verifying and correcting emotion prediction results based on medical knowledge. The technical problem addressed by this embodiment is that existing technologies rely solely on collected actual data for training and prediction, lacking the support of authoritative medical knowledge, and are prone to misjudgment due to individual differences or data noise, thus requiring further improvement in recognition accuracy. This embodiment enhances the knowledge base by introducing a retrieval system that links human physiological changes with medical emotions, providing a scientific external verification basis for emotion prediction and significantly improving the reliability of the recognition results.

[0009] This involved collecting knowledge about the relationship between human emotions and physiological changes from authoritative medical literature, clinical research reports, physiology textbooks, and psychiatric guidelines. Specifically, the collected knowledge included: Knowledge about the relationship between physiological signals and emotions: the relationship between increased heart rate, decreased heart rate variability, increased skin conductivity and anger and anxiety; the relationship between increased respiratory rate and tension; the relationship between slightly elevated body temperature and shyness, etc.

[0010] Medical correspondence between facial expressions and emotions: AU4 (contraction of the corrugator supercilii muscle) corresponds to anger; AU12 (upturned corners of the mouth) corresponds to happiness; AU1+AU4 (upturned inner eyebrows + frowning) corresponds to sadness, etc.

[0011] Medical research findings on the relationship between speech characteristics and emotions include: a correlation between increased speech rate and higher pitch and anger; a correlation between decreased speech rate and lower pitch and sadness; and a correlation between increased volume and excitement.

[0012] The collected knowledge content is structured and converted into knowledge entries in a unified format. Each knowledge entry contains six fields: emotion category, corresponding human physiological characteristics, facial features, voice features, medical explanation, and level of evidence. The level of evidence is divided into three levels: high, medium, and low, corresponding to randomized controlled trials, cohort studies, and case reports, respectively. All structured knowledge entries are converted into vector representations and stored in a vector database to form a retrieval-enhanced generative knowledge base.

[0013] From the final emotion prediction result output in step S103 of the basic embodiment, corresponding audio features, facial features, and physiological features are extracted. Specifically, the extracted features include: Audio features: speech rate, average pitch, pitch variation range, average volume, fundamental frequency, formants, and spectral entropy. Facial features: activation intensity of each facial action unit, eye opening and closing, mouth corner curvature, frowning degree, and blinking frequency. Physiological features: average heart rate, heart rate variability SDNN, average skin conductivity, number of skin conductance responses, respiratory rate, and body temperature change rate.

[0014] Using preset feature and text mapping templates, numerical human body data features are converted into natural language query statements. For example, when the extracted features are heart rate of 110 beats per minute, skin conductivity of 0.8 microSiemens, frowning degree of 0.8, and pitch increase of 20%, the generated query statement is "emotion category and medical basis corresponding to heart rate of 110 beats per minute, skin conductivity of 0.8 microSiemens, frowning degree of 0.8, and pitch increase of 20%".

[0015] The generated natural language query is converted into a vector representation, and a cosine similarity search is performed in the vector database. The top 3 knowledge entries with the highest similarity are returned, with a similarity threshold of 0.7; knowledge entries with similarity below this threshold are not returned.

[0016] First, the sentiment category and evidence level of each returned knowledge item are extracted. Then, the sentiment category of the medical knowledge item is compared with the final sentiment prediction result output by the basic implementation: If the sentiment category of all returned knowledge items is consistent with the final sentiment prediction result, the final sentiment prediction result remains unchanged, and the overall confidence level is the original confidence level multiplied by 0.9 plus the confidence level of the knowledge item with the highest evidence level multiplied by 0.1. If the sentiment category of any returned knowledge item is inconsistent with the final sentiment prediction result, the confidence level of the original prediction result is compared with the confidence level of the knowledge item with the inconsistent highest evidence level: If the confidence level of the medical knowledge item is higher, the final sentiment prediction result is corrected to the corresponding sentiment category in the medical knowledge item, and the overall confidence level is the confidence level of the medical knowledge item multiplied by 0.6 plus the confidence level of the original prediction result multiplied by 0.4. If the confidence level of the original prediction result is higher, the original prediction result remains unchanged, and the overall confidence level is the confidence level of the original prediction result multiplied by 0.7 plus the confidence level of the knowledge item with the highest evidence level multiplied by 0.3. The final sentiment prediction results, after verification and correction, along with the corresponding comprehensive confidence scores, will be output.

[0017] This embodiment, by introducing authoritative medical knowledge as external verification, can effectively correct misjudgments caused by individual differences, data noise, or model bias, significantly improving the accuracy and reliability of emotion recognition. Especially for emotional states with atypical physiological characteristics, it can provide a more scientific basis for judgment, reducing errors caused by subjective judgment.

[0018] In some embodiments, before collecting human data in three modalities (audio, facial, and physiological) and recording timestamps respectively, the method further includes: collecting scene image data through an image sensor; collecting the position and motion data of human bodies in the scene through millimeter-wave radar; collecting speech data in the scene through an audio sensor and extracting voiceprint features; identifying the number of human bodies in the scene based on image data, radar data, and voiceprint features; assigning a unique identifier to each human body when the number of identified human bodies does not exceed two; tracking the position and motion trajectory of each human body; associating the collected audio data, facial data, and physiological data with the human body corresponding to the identifier; and performing subsequent timestamp correction, emotion category weight calculation, synthetic sample generation, and emotion prediction steps for each human body corresponding to the identifier.

[0019] This embodiment, based on the basic embodiment, adds steps for identity differentiation and individual emotion recognition in multi-person scenarios. The technical problem this embodiment solves is that most existing technologies are designed for single-person scenarios and cannot accurately distinguish the identities of different people in multi-person scenarios, leading to multimodal data confusion and the inability to achieve individual emotion recognition, thus limiting the system's application scenarios. This embodiment, by fusing image processing, 60G millimeter-wave radar, and voiceprint recognition technologies, can accurately distinguish the identities of different people in scenarios with up to two people and achieve precise association of multimodal data.

[0020] An RGB camera with a resolution of 1920×1080 and a frame rate of 30 frames per second is used to acquire color image data of the scene in real time. After the image data is acquired, preprocessing operations are performed, including Gaussian filtering for noise reduction, automatic white balance adjustment, and contrast enhancement, to improve the accuracy of subsequent target detection.

[0021] This system employs a 60GHz frequency-modulated continuous wave radar with a 4GHz bandwidth, transmitting linear frequency-modulated continuous wave signals and receiving echo signals reflected from human bodies. The range dimension spectrum is obtained by performing a fast Fourier transform on the echo signals, angular dimension information is obtained through digital beamforming, and the velocity and micro-motion information of the human body is obtained through phase difference analysis. Finally, it outputs data on the distance, azimuth, elevation, radial velocity, respiratory rate, and heart rate for each detected human body. The 60GHz millimeter-wave radar is unaffected by environmental factors such as light, smoke, and dust, and can operate stably in dark and harsh environments without compromising user facial privacy.

[0022] A four-microphone array was used to collect speech data from the scene, with a sampling rate of 16kHz and a sampling precision of 16 bits. Beamforming algorithms were employed to enhance the speech signal in the target direction while suppressing environmental noise and interference. A pre-trained voiceprint recognition model was used to extract a 128-dimensional voiceprint feature vector from the enhanced speech data. Voiceprint features are unique and stable, enabling accurate differentiation between different speakers.

[0023] Human targets are detected in the preprocessed image data using a target detection algorithm, resulting in the number of human targets detected (N1). The number of human targets detected by radar data is then used to obtain N2. All extracted voiceprint features are clustered to obtain the number of speakers (N3). A majority voting method is used to determine the final number of human targets: if the number detected by at least two sensors is consistent, that number is taken as the final result; if the number detected by all three sensors is inconsistent, the maximum value is taken as the final result and marked as an uncertain state.

[0024] When only one person is identified, an identity identifier (ID001) is assigned to that person. When two people are identified, identity identifiers (ID001 and ID002) are assigned to the two people respectively. When more than two people are identified, a voice prompt message "Too many people in the current scene, emotion recognition cannot be performed" is output, and the emotion recognition process is paused until the number of people in the scene is reduced to two or less.

[0025] A multi-target tracking algorithm is employed, combining image and radar data to continuously track each human body. For human targets in images, their appearance features are extracted; for human targets in radar data, their position and motion features are extracted. Through feature fusion and data association, stable tracking of human targets is achieved. Tracking remains intact even when the human body moves, turns, or is partially occluded in the scene.

[0026] Based on the position of the human body in the image and the tracking results, the detected facial regions are associated with the human bodies corresponding to the identification tags.

[0027] The extracted voiceprint features are matched with the voiceprint template corresponding to each identity. The identity with the highest matching degree is the person corresponding to the voice data. Initially, the voiceprint template is empty. When voice data of a certain identity is detected for the first time, its voiceprint features are used as the initial template for that identity. Subsequently, each time voice data of that identity is detected, the voiceprint template is updated using an exponential moving average method.

[0028] If the physiological sensor is a wearable device, it is associated with the identity information pre-bound to the device. If the physiological sensor is a non-contact physiological radar, the detected physiological signals are associated with the corresponding location's identity identifier based on the human body's location information.

[0029] An independent data processing thread and data storage space are created for each human body with a unique identifier. Each thread independently executes all operations from steps S101 to S103 in the basic embodiment, obtaining the emotion prediction result for each human body. The data processing processes for human bodies with different identifiers are independent of each other and do not interfere with each other.

[0030] This embodiment achieves accurate identity differentiation and data association in multi-person scenarios by fusing image processing, 60G millimeter-wave radar, and voiceprint recognition technologies. It solves the problem of data ambiguity in multi-person scenarios, expanding the application scenarios of this invention to include common scenarios with two or fewer people, such as homes and offices. Simultaneously, the use of 60G millimeter-wave radar improves the system's adaptability to harsh environments and its privacy protection.

[0031] In some embodiments, after outputting the final emotion prediction result, the method further includes: storing each emotion prediction result, the corresponding human data features, and time information to form a historical memory bank; extracting multiple emotion prediction results and corresponding human data features of the same human from the historical memory bank; learning the personalized emotion expression features of the same human; generating an interactive query statement when a new emotion prediction result differs from the historical personalized emotion expression features of the human; outputting the interactive query statement to the corresponding human through voice interaction; receiving voice feedback information from the corresponding human; updating the personalized emotion expression features and historical memory bank of the human based on the voice feedback information; and optimizing the subsequent emotion prediction process using the updated personalized emotion expression features.

[0032] This embodiment, based on the basic embodiment, adds personalized feature learning and interactive memory steps. The technical problem this embodiment addresses is that significant individual differences exist in emotional expression among different people, making it difficult for general emotion recognition models to adapt to the expressive characteristics of all users, resulting in low recognition accuracy for specific users. Existing technologies lack personalized learning and memory functions, failing to continuously optimize recognition performance based on user feedback. This embodiment establishes a memory bank by storing users' historical emotion data, learns users' personalized emotional expression characteristics, and obtains user feedback through voice interaction to continuously optimize model performance.

[0033] A historical memory database was constructed using a MySQL relational database. The historical memory database contains three tables: a user table, an emotion record table, and a characteristic table. User table: Stores user identification, voiceprint template, personalized model parameters and other information.

[0034] Emotion Record Table: Stores information such as ID, user identity, emotion category, confidence level, prediction time, and interaction feedback results for each emotion prediction record.

[0035] Feature table: Stores the audio feature vector, facial feature vector, and physiological feature vector corresponding to each emotion prediction record, and is associated with the emotion record table through record ID.

[0036] After each sentiment prediction is completed, the predicted sentiment category, confidence level, time information, corresponding multimodal feature vector, and user identifier are written to the history memory bank. The storage period for the history memory bank is set to 1 year, and records older than 1 year are automatically deleted to save storage space.

[0037] Based on the user's identity, retrieve all correctly confirmed or reported sentiment prediction records from the historical memory database for that user within the past 30 days. If the number of valid records for that user within the past 30 days is less than 50, then retrieve all valid historical records.

[0038] The gradient boosting tree algorithm is used to train the extracted historical data to learn the user's personalized emotional expression features. During training, multimodal feature vectors are used as input and emotion categories are used as outputs to adjust the model's parameters, enabling the model to accurately identify the user's unique emotional expression patterns. For example, the model learns that the user does not frown when angry, but exhibits a significant increase in heart rate and faster speech; and that the user displays specific facial expressions when happy.

[0039] Simultaneously, the user's voice characteristics are analyzed to identify frequently used verbal tics. When a word appears more than 5 times per 100 sentences in the user's voice data, the word is marked as the user's verbal tics, and the corresponding emotion category when the verbal tics appears is recorded.

[0040] The multimodal feature vector corresponding to the new sentiment prediction result is input into the trained personalized model to obtain the personalized prediction result. The prediction results of the general model and the personalized model are compared: If the two are consistent, the prediction result of the general model will be output directly.

[0041] If the two are inconsistent, and the confidence level of the general model is less than 0.7, then the corresponding interactive query statement will be generated.

[0042] Interactive query statements are divided into the following three categories based on their specific circumstances: Emotion confirmation category: When there is a significant difference between the prediction result and the personalized characteristics, generate statements such as "Master, were you angry just now?" or "Master, are you happy now?".

[0043] Catchphrase Association Class: When a user is detected using a catchphrase and that catchphrase is associated with a specific emotion in the history, statements such as "Master, why do you keep saying XXX?" or "Master, were you happy when you said XXX last time? Are you happy now?" are generated.

[0044] Historical similarity class: When the similarity between the current multimodal feature and a feature from a previous sentiment prediction is detected to be greater than 0.8, statements such as "Master, you were angry when you said XXX last time, are you still angry now?" are generated.

[0045] The generated interactive queries are input into a text-to-speech engine, which converts them into natural and fluent speech signals. These speech signals are then played through a speaker to the corresponding identified person. During playback, the speech rate, pitch, and volume are adjusted based on the user's past preferences.

[0046] User voice feedback is collected via a microphone array. A speech recognition algorithm converts the speech signal into text. A sentiment analysis algorithm analyzes the emotional tone and semantic content of the feedback to extract confirmation or denial information from the user. For example, "yes" or "right" indicates confirmation; "no" or "I'm not angry" indicates denial.

[0047] If the user confirms the sentiment prediction is correct, the prediction is marked as correct and added to the historical memory. This data is then used to incrementally train the personalized model, updating the personalized sentiment expression features.

[0048] If the user rejects the emotion prediction result, they are prompted to enter the correct emotion category. The correct emotion category provided by the user and its corresponding multimodal features are added to the historical memory. The personalized model is retrained using the updated historical data, and the personalized emotion expression features are updated.

[0049] In the subsequent sentiment prediction process, the prediction results of the general model and the personalized model are weighted and combined. The weight of the personalized model gradually increases with the number of user history records. When the number of valid history records exceeds 200, the weight of the personalized model is set to 0.7, and the weight of the general model is set to 0.3.

[0050] By storing users' historical emotional data to form a memory bank, the system learns users' personalized emotional expression characteristics and continuously optimizes model parameters by obtaining user feedback through voice interaction. This effectively solves the problem of low emotion recognition accuracy caused by large individual differences, enabling the system to gradually adapt to users' unique expression characteristics and significantly improve user experience and recognition accuracy.

[0051] In some embodiments, the step of calculating the emotion category weights based on the human body data and generating synthetic samples of minority emotions includes: extracting emotion labels corresponding to all human body data and counting the number of samples corresponding to each emotion label; calculating the weight of the corresponding emotion category based on the number of samples for each emotion label; identifying emotion categories with fewer than a preset threshold of samples as minority emotions; and using a generative adversarial network to generate a specified number of synthetic samples of minority emotions.

[0052] In some embodiments, calculating the emotion category weights based on the human body data includes: extracting emotion tags corresponding to all human body data, obtaining all unique emotion tags to form an emotion category set; counting the number of samples corresponding to each emotion category in the emotion category set; using a balanced category weight calculation method, calculating the weight of the corresponding emotion category based on the number of samples in each emotion category; establishing a mapping relationship between each emotion category and its corresponding weight and storing it.

[0053] In some embodiments, the step of performing single-modal emotion prediction if only one modality has available human data includes: if only the audio modality has available human data, inputting the available data of the audio modality into a preset audio emotion recognition model for emotion prediction; if only the facial modality has available human data, inputting the available data of the facial modality into a preset facial emotion recognition model for emotion prediction; if only the physiological modality has available human data, inputting the available data of the physiological modality into a preset physiological emotion recognition model for emotion prediction; and outputting the single-modal emotion prediction result and the corresponding confidence level.

[0054] In some embodiments, if available human data exists for the multiple modalities, it is input into a preset emotion prediction model for multimodal fusion prediction, evaluating the reliability of each modality and combining them to obtain the final emotion prediction result. This includes: inputting human data from all available modalities into a preset multimodal fusion emotion prediction model for fusion prediction to obtain a fusion prediction result; calculating the reliability score of each modality based on the completeness, continuity, and signal-to-noise ratio of the data; weighting and combining the fusion prediction result and the emotion prediction results of each single modality based on the reliability scores of each modality; and outputting the final emotion prediction result and its corresponding confidence level after weighted combination.

[0055] In some embodiments, the method further includes: performing dynamic range quantization on all preset emotion prediction models to obtain quantized emotion prediction models; performing low-amplitude pruning on the quantized emotion prediction models to remove parameters with small absolute weight values ​​in the emotion prediction models; forming data batches of human data to be inferred according to preset batch sizes; sequentially inputting the data batches into the optimized emotion prediction models for batch inference and outputting batch inference results.

[0056] Secondly, this application provides a human emotion recognition system based on multimodal sensor fusion, the system comprising: The data acquisition unit is used to acquire human data in three modalities: audio, facial, and physiological, and record timestamps for each modality. It calculates the offset between the timestamp of the human data corresponding to each modality and the current time, updates the clock drift compensation value of each modality using an exponential moving average method, and corrects the timestamp of the human data corresponding to each modality. The sample generation unit is used to calculate the emotion category weights based on the human data, generate synthetic samples of minority emotions, and detect available human data and corresponding modalities in the synthetic samples. The result generation unit is used to perform single-modal emotion prediction if only one modality has available human data, and input the data into a preset emotion prediction model for multimodal fusion prediction if multiple modalities have available human data. The unit evaluates the reliability of each modality and combines them to obtain the final emotion prediction result.

[0057] This application employs a dynamic clock drift compensation mechanism to control the synchronization error of multimodal data, ensuring the temporal consistency of fused data and significantly improving the recognition accuracy of multimodal fusion. By combining category weight adjustment and minority class sample synthesis techniques, it effectively alleviates the model's bias towards majority class emotions, greatly improving the recognition accuracy of minority class emotions such as fear and disgust. An adaptive modality detection and degradation prediction mechanism is designed, which allows the system to automatically switch to the prediction mode of available modalities when some modalities fail, ensuring stable operation of the system in complex environments. The above three core technologies are organically integrated to form a complete multimodal emotion recognition solution that can be directly applied to various complex real-world scenarios.

[0058] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic flowchart illustrating the steps of a human emotion recognition method based on multimodal sensor fusion provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the principle of a human emotion recognition method based on multimodal sensor fusion, provided in one embodiment of this application. Figure 3 This is a schematic block diagram of a human emotion recognition system based on multimodal sensor fusion provided in one embodiment of this application; Figure 4 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0061] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0064] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0065] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0066] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0067] With the rapid development of artificial intelligence and human-computer interaction technologies, human emotion recognition technology has been widely applied in fields such as intelligent customer service, medical diagnosis, and education and training. However, single-modal emotion recognition technology is limited by a single data source, is easily affected by environmental interference, and its recognition accuracy and robustness are insufficient to meet the needs of practical applications. Therefore, multimodal fusion emotion recognition technology has become a current research hotspot.

[0068] However, existing multimodal emotion recognition technologies generally suffer from the following three technical shortcomings: 1. The clocks of multimodal data acquisition devices have inherent drift, and the timestamps of data acquired by different devices cannot be accurately aligned, resulting in timing misalignment during fusion, which seriously affects the recognition accuracy; 2. Emotional data naturally suffers from an imbalance in category distribution. Samples of minority emotions such as fear and disgust are scarce, and existing models generally favor the majority of emotions, resulting in extremely low accuracy in identifying minority emotions. 3. In real-world applications, some modalities may temporarily fail due to equipment malfunctions, environmental occlusion, or other reasons. Existing multimodal fusion models cannot adaptively handle the situation of missing modalities. Once data for a particular modality is lost, the performance of the entire system will drop sharply or even become completely unusable.

[0069] The three technical problems mentioned above are interconnected and mutually influential. Existing technologies can usually only solve one of the problems individually. There is no multimodal emotion recognition method that can systematically solve all three core technical challenges at the same time. This greatly limits the practical application of multimodal emotion recognition technology.

[0070] To solve the above problem, please refer to Figure 1 This application provides a method for human emotion recognition based on multimodal sensor fusion, applied to computer devices. The computer devices can be deployed on a single server or server cluster. They can also be deployed on handheld terminals, laptops, wearable devices, or robots, etc. It should be noted that all information involved in the method provided in this application is extracted with the authorization of the relevant user and in accordance with relevant regulations, and will not infringe on user privacy.

[0071] The provided method for human emotion recognition based on multimodal sensor fusion includes steps S101 to S103. Details are as follows: Step S101. Collect human body data in three modalities: audio, facial, and physiological, and record the timestamps for each modality; calculate the offset between the timestamp of the human body data corresponding to each modality and the current time, update the clock drift compensation value of each modality using an exponential moving average method, and correct the timestamp of the human body data corresponding to each modality.

[0072] Specifically, the human emotion recognition method based on multimodal sensor fusion provided by this invention is implemented through three core steps executed sequentially: multimodal data acquisition and time synchronization correction, data imbalance processing and synthetic sample validity detection, and adaptive modal emotion prediction and result fusion. These three steps are interconnected and work together organically to systematically solve the three core technical problems existing in current multimodal emotion recognition technologies: temporal misalignment, data imbalance, and poor robustness due to modality loss.

[0073] The core objective of this step is to resolve the timing misalignment issue in multimodal data caused by inherent clock drift from different sensor devices, providing a foundation of time-consistent data for subsequent multimodal fusion. This step consists of four sub-steps executed sequentially: multimodal data acquisition and timestamp recording, time offset calculation, clock drift compensation value update, and timestamp correction.

[0074] Multimodal data acquisition and timestamp recording involves simultaneously activating audio acquisition devices, facial image acquisition devices, and physiological signal acquisition devices to collect human voice data, facial image data, and physiological signal data, respectively. At the moment each set of data acquisition is completed, the current system's global time is immediately recorded as the corresponding acquisition timestamp. All acquisition devices are calibrated with the same system clock to ensure a consistent timestamp reference.

[0075] Time offset calculation is performed by immediately acquiring the system's current global time after obtaining any set of multimodal data. The time offset is calculated separately for the audio modality data acquisition timestamp, the facial modality data acquisition timestamp, and the physiological modality data acquisition timestamp. The time offset reflects the time difference between the data acquisition moment and the data processing moment, and also includes errors caused by device clock drift.

[0076] Clock drift compensation value updates are achieved by pre-establishing and storing a clock drift compensation value mapping table to store the clock drift compensation value corresponding to each mode. If a mode does not have a stored clock drift compensation value in the mapping table, the time offset calculated for that mode is used as its initial clock drift compensation value and stored in the mapping table. If a mode already has a clock drift compensation value in the mapping table, the stored clock drift compensation value is updated using the time offset calculated for the current time using an exponential moving average method. The exponential moving average method can effectively suppress random noise in a single time offset measurement, while quickly tracking the slow changes in device clock drift, ensuring the accuracy and stability of time synchronization.

[0077] Timestamp correction is achieved by adding the acquisition timestamp of the data corresponding to each modality to the latest clock drift compensation value corresponding to that modality in the clock drift compensation value mapping table, thus obtaining the corrected timestamp of the modality data. All modal data are then processed using the corrected timestamps for subsequent multimodal data alignment and fusion, ensuring that the data from different modalities correspond precisely in the time dimension.

[0078] Step S102. Calculate the emotion category weights based on the human body data, generate synthetic samples of minority emotions, and detect available human body data and corresponding modalities in the synthetic samples.

[0079] Specifically, the core objective of this step is to address the problem of low accuracy in minority class emotion recognition caused by the inherent imbalance in the distribution of emotion categories in the data. By adjusting category weights and using minority class sample synthesis techniques, the distribution of training data for all emotion categories is balanced, thereby improving the model's generalization ability across all emotion categories. This step consists of three sub-steps executed sequentially: emotion category weight calculation, minority class emotion recognition and synthetic sample generation, and synthetic sample validity detection and modality recognition.

[0080] Emotion category weight calculation involves extracting emotion labels from all training data to form an emotion category set, consisting of all unique emotion labels. The number of samples for each emotion category is then counted, and a balanced class weighting method is used to calculate the weight of each emotion category based on its sample count. Emotion categories with fewer samples have higher weights, and vice versa. A mapping relationship is established between each emotion category and its calculated weight, and this mapping is stored for use in the loss function weighting during subsequent model training.

[0081] Minority emotion recognition and synthetic sample generation: By pre-setting a sample size threshold, emotion categories with fewer than this threshold in the emotion category set are identified as minority emotions. A generative adversarial network (GAN) is used to learn from existing samples for each minority emotion, generating a specified number of synthetic minority emotion samples. The generated synthetic samples maintain consistency with real samples in data distribution and feature representation, effectively expanding the training dataset for minority emotions.

[0082] Synthetic sample validity detection and modality recognition involve individually testing each generated minority class emotion synthetic sample. The presence of audio modality, facial modality, and physiological modality data in each synthetic sample is checked, and the data is verified against preset validity criteria. Data that simultaneously meets both existence and validity requirements is deemed usable, and the modality corresponding to all usable data in each synthetic sample is recorded. Synthetic samples with invalid data across all modalities are discarded to prevent invalid data from negatively impacting model training.

[0083] Step S103. If only one modality has available human data, perform single-modal emotion prediction. If multiple modalities have available human data, input them into the preset emotion prediction model for multimodal fusion prediction, evaluate the reliability of each modality, and combine them to obtain the final emotion prediction result.

[0084] Specifically, the core objective of this step is to address the problem of system performance degradation or even complete malfunction caused by the temporary failure of some modalities in real-world application scenarios. Through adaptive modality detection and hierarchical prediction mechanisms, it ensures that the system can stably output emotion recognition results under any combination of available modalities. This step consists of three sub-steps executed sequentially: available modality detection, single-modal emotion prediction, and multimodal fusion prediction and reliability assessment.

[0085] Available modality detection involves examining each of the multimodal data to be identified, determining whether audio, facial, and physiological modal data exist and are valid. The set of available modalities is then obtained by statistically analyzing all existing and valid data.

[0086] Unimodal sentiment prediction works by calling the pre-trained unimodal sentiment recognition model corresponding to only one modality if the available modality set contains only that modality. The available data for that modality is input into the corresponding unimodal sentiment recognition model, and the model outputs the sentiment prediction result and the corresponding confidence score, which serve as the final sentiment recognition result.

[0087] Multimodal fusion prediction and reliability assessment: If the available modality set contains two or three modalities, the human data of all available modalities are first input into a pre-trained multimodal fusion emotion prediction model for fusion prediction, obtaining the fusion prediction result and the corresponding confidence score. Simultaneously, the single-modal emotion recognition model corresponding to each available modality is called for prediction, obtaining the prediction result and the corresponding confidence score for each single modality.

[0088] Based on three metrics—completeness, continuity, and signal-to-noise ratio—a reliability score is calculated for each available modality. Higher data completeness, continuity, and signal-to-noise ratio result in a higher reliability score. The multimodal fusion prediction result and the individual single-modal prediction results are then weighted and combined according to their reliability scores. Prediction results with higher reliability scores have a greater weight in the final result. The final sentiment prediction result after weighted combination and its corresponding overall confidence score are output.

[0089] In some embodiments, after evaluating the reliability of each modality and combining them to obtain the final emotion prediction result, the method further includes: constructing a retrieval-enhanced generative knowledge base containing knowledge about the laws of human physiological changes and the correlation between medical emotions; extracting human data features corresponding to the final emotion prediction result; converting the human data features into retrieval query statements; retrieving medical knowledge entries that match the human data features from the retrieval-enhanced generative knowledge base; verifying and correcting the final emotion prediction result based on the retrieved medical knowledge entries; and outputting the corrected final emotion prediction result.

[0090] In some embodiments, before collecting human data in three modalities (audio, facial, and physiological) and recording timestamps respectively, the method further includes: collecting scene image data through an image sensor; collecting the position and motion data of human bodies in the scene through millimeter-wave radar; collecting speech data in the scene through an audio sensor and extracting voiceprint features; identifying the number of human bodies in the scene based on image data, radar data, and voiceprint features; assigning a unique identifier to each human body when the number of identified human bodies does not exceed two; tracking the position and motion trajectory of each human body; associating the collected audio data, facial data, and physiological data with the human body corresponding to the identifier; and performing subsequent timestamp correction, emotion category weight calculation, synthetic sample generation, and emotion prediction steps for each human body corresponding to the identifier.

[0091] In some embodiments, after outputting the final emotion prediction result, the method further includes: storing each emotion prediction result, the corresponding human data features, and time information to form a historical memory bank; extracting multiple emotion prediction results and corresponding human data features of the same human from the historical memory bank; learning the personalized emotion expression features of the same human; generating an interactive query statement when a new emotion prediction result differs from the historical personalized emotion expression features of the human; outputting the interactive query statement to the corresponding human through voice interaction; receiving voice feedback information from the corresponding human; updating the personalized emotion expression features and historical memory bank of the human based on the voice feedback information; and optimizing the subsequent emotion prediction process using the updated personalized emotion expression features.

[0092] In some embodiments, the acquisition of human body data in three modalities—audio, facial, and physiological—and the recording of timestamps for each modality includes: acquiring human voice data through an audio sensor, acquiring human facial image data through an image sensor, and acquiring human physiological signal data through a physiological sensor; and recording the acquisition timestamp of each data set while acquiring each set of data.

[0093] A microphone with a sampling rate of 16 kHz is used as an audio sensor to collect human voice data, acquiring one frame of voice data every 100 milliseconds, with each frame containing 1600 sampling points. A camera with a resolution of 1920×1080 is used as an image sensor to collect human facial image data, acquiring 30 frames of facial images per second. A wearable device integrating a heart rate sensor, a skin conductance sensor, and a body temperature sensor is used as a physiological sensor to collect human physiological signal data, acquiring 10 sets of physiological signal data per second.

[0094] Upon system startup, the internal clocks of the microphone, camera, and wearable devices are calibrated with the computer's global system clock to ensure all devices use a unified time reference. Immediately upon completion of the acquisition of each frame of audio data, facial image, or set of physiological signal data, the computer system's current global time is read, accurate to the millisecond level. This time is used as the acquisition timestamp for the corresponding data frame and stored in conjunction with the data frame.

[0095] In some embodiments, the step of calculating the offset between the timestamp of the human data corresponding to each modality and the current time, updating the clock drift compensation value of each modality using an exponential moving average method, and correcting the timestamp of the human data corresponding to each modality includes: obtaining the current system time, calculating the difference between the acquisition timestamp of the data corresponding to the audio modality, facial modality, and physiological modality and the current system time to obtain the time offset of each modality; if no stored clock drift compensation value exists for any modality, storing the time offset of that modality as the clock drift compensation value for that modality; if a stored clock drift compensation value exists for any modality, updating the stored clock drift compensation value using the time offset of that modality using an exponential moving average method; and adding the acquisition timestamp of the human data corresponding to each modality to the clock drift compensation value of the corresponding modality to obtain the corrected timestamp of the human data corresponding to each modality.

[0096] An empty clock drift compensation value dictionary is pre-built in computer memory. The key of the dictionary is the modality name, and the value is the clock drift compensation value for the corresponding modality. When the system receives a frame of audio data for the first time, the current global time of the system is read as 100050 milliseconds, and the acquisition timestamp of the audio data is 100000 milliseconds. The time offset of the audio modality is calculated to be 50 milliseconds. Since there is no compensation value corresponding to the audio modality in the clock drift compensation value dictionary, 50 milliseconds is stored in the dictionary as the initial clock drift compensation value for the audio modality.

[0097] When the system receives a second frame of audio data, it reads the current global time as 200052 milliseconds, and the audio data acquisition timestamp as 200000 milliseconds, calculating the audio modality time offset to be 52 milliseconds. At this point, the clock drift compensation value dictionary already contains a compensation value of 50 milliseconds for the audio modality. Therefore, it updates the value using an exponential moving average method. The updated audio modality clock drift compensation value is 45 milliseconds plus 5.2 milliseconds, equal to 50.2 milliseconds. The updated compensation value is then stored in the dictionary, overwriting the original compensation value.

[0098] Adding the acquisition timestamp of the audio data (200,000 milliseconds) to the updated clock drift compensation value (50.2 milliseconds) yields the corrected timestamp of the audio data as 200050.2 milliseconds. This corrected timestamp is then used to align with the corrected timestamps of other modal data, achieving precise time synchronization of multimodal data.

[0099] The process of updating clock drift compensation values ​​and correcting timestamps for facial and physiological modalities is completely consistent with the process for audio modalities.

[0100] In some embodiments, the step of calculating the emotion category weights based on the human body data and generating synthetic samples of minority emotions includes: extracting emotion labels corresponding to all human body data and counting the number of samples corresponding to each emotion label; calculating the weight of the corresponding emotion category based on the number of samples for each emotion label; identifying emotion categories with fewer than a preset threshold of samples as minority emotions; and using a generative adversarial network to generate a specified number of synthetic samples of minority emotions.

[0101] The training dataset contains samples representing six emotion categories: happiness, sadness, anger, fear, disgust, and surprise. Emotion labels were extracted from all training samples, resulting in 10,000 samples for happiness, 8,000 for sadness, 5,000 for anger, 500 for fear, 300 for disgust, and 2,000 for surprise.

[0102] Based on the number of samples for each emotion category, a balanced class weighting method is used to calculate the weight of each emotion category. A pre-set threshold of 1000 samples is used to identify fear and disgust categories, which have fewer than 1000 samples, as minority emotions.

[0103] A generative adversarial network (GAN) was constructed and trained using 500 real samples from the fear category and 300 real samples from the aversion category. After training, the GAN generated 2000 synthetic samples from the fear category and 2000 synthetic samples from the aversion category. The generated synthetic samples were then merged with the original real samples to form a new training dataset for subsequent training of the emotion recognition model.

[0104] In some embodiments, calculating the emotion category weights based on the human body data includes: extracting emotion tags corresponding to all human body data, obtaining all unique emotion tags to form an emotion category set; counting the number of samples corresponding to each emotion category in the emotion category set; using a balanced category weight calculation method, calculating the weight of the corresponding emotion category based on the number of samples in each emotion category; establishing a mapping relationship between each emotion category and its corresponding weight and storing it.

[0105] The training dataset contains 25,800 samples across six emotion categories. The weight for each emotion category is calculated as follows: total number of samples divided by the total number of emotion categories, then divided by the number of samples in that specific emotion category.

[0106] According to the above calculation method, the weight of the happy category is 25800 divided by 6 and then divided by 10000, which equals 0.43; the weight of the sad category is 25800 divided by 6 and then divided by 8000, which equals 0.5375; the weight of the angry category is 25800 divided by 6 and then divided by 5000, which equals 0.86; the weight of the fear category is 25800 divided by 6 and then divided by 500, which equals 8.6; the weight of the disgust category is 25800 divided by 6 and then divided by 300, which equals approximately 14.33; and the weight of the surprised category is 25800 divided by 6 and then divided by 2000, which equals 2.15.

[0107] Simultaneously, a mapping relationship between emotion categories and weights is established: for example, happiness corresponds to 0.43, sadness to 0.5375, anger to 0.86, fear to 8.6, disgust to 14.33, and surprise to 2.15. This mapping relationship is stored in computer memory, and during model training, these weights are used to weight the loss function, making the model pay more attention to minority emotion samples during training. In some embodiments, generating synthetic samples of minority emotions, and detecting available human data and corresponding modalities in the synthetic samples, includes: identifying emotion categories with fewer than a preset threshold as minority emotions; generating a specified number of synthetic samples of minority emotions using a generative adversarial network; detecting the existence and validity of audio modality, facial modality, and physiological modality data in each synthetic sample; and recording the modality corresponding to the existing and valid data in each synthetic sample.

[0108] In some embodiments, generating synthetic samples of minority emotions, and detecting available human data and corresponding modalities in the synthetic samples, includes: identifying emotion categories with fewer than a preset threshold as minority emotions; generating a specified number of synthetic samples of minority emotions using a generative adversarial network; detecting the existence and validity of audio modal, facial modal, and physiological modal data in each synthetic sample; and recording the modality corresponding to the valid data in each synthetic sample.

[0109] Each synthetic sample generated by the generative adversarial network contains data from three modalities: audio, facial, and physiological. Each synthetic sample undergoes a validity check, with the following validity criteria: Audio data: Length between 0.5 and 5 seconds, no obvious silences, and sampling rate meets preset requirements; Facial image data: resolution not less than 224×224 pixels, the image contains a complete face area, without obvious occlusion or blurring; Physiological signal data: The length is between 1 and 10 seconds, with no obvious signal disconnection or abnormal jumps.

[0110] For the first generated synthetic sample, the audio data length was found to be 0.3 seconds, which did not meet the validity standard and was therefore deemed invalid; the facial image data resolution was 480×480 pixels, containing a complete face, and was therefore deemed valid; the physiological signal data length was 3 seconds, with no breaks or jumps, and was therefore deemed valid. Therefore, the usable modalities of this synthetic sample are the facial modality and the physiological modality.

[0111] For the second synthesized sample, the detection found that the data of all three modalities met the validity criteria. Therefore, the usable modalities of this synthesized sample are audio modality, facial modality, and physiological modality.

[0112] For the third synthetic sample generated, the detection found that the data of all three modalities did not meet the validity criteria, so the synthetic sample was removed from the training dataset.

[0113] In some embodiments, the step of performing single-modal emotion prediction if only one modality has available human data includes: if only the audio modality has available human data, inputting the available data of the audio modality into a preset audio emotion recognition model for emotion prediction; if only the facial modality has available human data, inputting the available data of the facial modality into a preset facial emotion recognition model for emotion prediction; if only the physiological modality has available human data, inputting the available data of the physiological modality into a preset physiological emotion recognition model for emotion prediction; and outputting the single-modal emotion prediction result and the corresponding confidence level.

[0114] Three unimodal models—audio emotion recognition, facial emotion recognition, and physiological emotion recognition—were pre-trained. The inputs to each model were pre-processed data for their respective modality, and the outputs were the probability distributions of six emotion categories, the emotion category with the highest probability, and its confidence score.

[0115] When only facial modality data is available in the data to be identified, the facial image data is preprocessed, adjusted to a size of 224×224 pixels, and normalized. The preprocessed facial image data is then input into the facial emotion recognition model. The model outputs the probability distributions for six emotion categories: happy 0.92, sad 0.03, angry 0.02, fear 0.01, disgust 0.01, and surprise 0.01. The model outputs the final single-modal emotion prediction result as happy, with a corresponding confidence score of 0.92. This result is then output as the final emotion recognition result.

[0116] When only audio modality or only physiological modality data is available in the data to be identified, the prediction process is completely consistent with the prediction process of facial modality, and the corresponding single-modality model is called to make predictions and output results.

[0117] In some embodiments, if available human data exists for the multiple modalities, it is input into a preset emotion prediction model for multimodal fusion prediction, evaluating the reliability of each modality and combining them to obtain the final emotion prediction result. This includes: inputting human data from all available modalities into a preset multimodal fusion emotion prediction model for fusion prediction to obtain a fusion prediction result; calculating the reliability score of each modality based on the completeness, continuity, and signal-to-noise ratio of the data; weighting and combining the fusion prediction result and the emotion prediction results of each single modality based on the reliability scores of each modality; and outputting the final emotion prediction result and its corresponding confidence level after weighted combination.

[0118] When usable data exists for the audio, facial, and physiological modalities in the data to be identified, the preprocessed data for each modality is first input into a multimodal fusion emotion prediction model. The model outputs a fusion prediction result of "happy," with a confidence level of 0.85. Simultaneously, three single-modal models are called for prediction, yielding the following results: audio modality prediction of "happy," with a confidence level of 0.8; facial modality prediction of "happy," with a confidence level of 0.9; and physiological modality prediction of "sad," with a confidence level of 0.6.

[0119] A reliability score was calculated for each modality. The reliability score was obtained by weighting three indicators: data integrity, data continuity, and data signal-to-noise ratio (SNR). Data integrity accounted for 40%, data continuity for 30%, and data SNR for 30%. The calculated reliability scores were 0.8 for the audio modality, 0.9 for the facial modality, and 0.7 for the physiological modality.

[0120] Based on the reliability scores of each modality, the fusion prediction result and the three individual modality prediction results are weighted and combined. The final emotion prediction result obtained after weighted calculation is "happy," with a corresponding comprehensive confidence level of 0.87. This result is output as the final emotion recognition result.

[0121] In some embodiments, the method further includes: performing dynamic range quantization on all preset emotion prediction models to obtain quantized emotion prediction models; performing low-amplitude pruning on the quantized emotion prediction models to remove parameters with small absolute weight values ​​in the emotion prediction models; forming data batches of human data to be inferred according to preset batch sizes; sequentially inputting the data batches into the optimized emotion prediction models for batch inference and outputting batch inference results.

[0122] Dynamic range quantization was performed on all pre-trained emotion recognition models to convert the 32-bit floating-point weights to 8-bit integers, resulting in a quantized model. The quantized model was then subjected to low-amplitude pruning, removing parameters whose absolute weight values ​​were less than a preset threshold. The pruning ratio was set to 50%, resulting in a pruned and optimized model.

[0123] During the inference phase, a data queue is established to cache the human data to be inferred. The human data to be inferred is grouped into batches of eight according to a preset batch size. When the number of data in the data queue reaches the batch size, the batch is input into the optimized sentiment prediction model for batch inference. When the data queue is empty and there is remaining unprocessed data, the remaining data is grouped into a small batch for inference. The inference result and corresponding timestamp for each data point are output and stored in a result queue for subsequent modules to use.

[0124] In some embodiments, by providing a distributed multi-node collaborative emotion recognition implementation method suitable for large-scale deployment scenarios, the problems of insufficient computing power and limited coverage of a single device are solved.

[0125] A distributed system consisting of a data acquisition node cluster, data preprocessing nodes, sentiment analysis nodes, and result processing nodes is deployed, with all nodes communicating via a message bus. The data acquisition node cluster includes multiple edge acquisition devices, each integrating audio sensors, image sensors, and physiological sensors, responsible for collecting multimodal human data within its corresponding area.

[0126] Each data acquisition node executes the multimodal data acquisition and timestamp recording sub-step in step S101, recording a local timestamp while acquiring each set of data, and encapsulating the data and timestamp into a standardized message and sending it to the message bus. The data preprocessing node subscribes to the raw data messages in the message bus and executes the time offset calculation, clock drift compensation value update, and timestamp correction sub-steps in step S101 to perform unified time synchronization correction on the data uploaded by all acquisition nodes.

[0127] After preprocessing, the data preprocessing node sends the corrected data to the sentiment analysis node. The sentiment analysis node executes the data imbalance handling and synthetic sample validity detection sub-steps in step S102, and the adaptive modality sentiment prediction and result fusion sub-steps in step S103, generating sentiment recognition results. The result processing node subscribes to the result messages output by the sentiment analysis node and is responsible for storing, displaying, and pushing the results.

[0128] The system employs a load balancing mechanism. When the load on a single sentiment analysis node exceeds a preset threshold, new tasks are automatically distributed to other idle sentiment analysis nodes. All nodes periodically send heartbeat messages to ensure system connectivity and stability.

[0129] In some embodiments, by providing a real-time monitoring method for system operating status and an automatic fault recovery mechanism, the problem of service interruption caused by system failure during long-term continuous operation is solved.

[0130] A monitoring agent is deployed on each node of the distributed system. The monitoring agent collects node status data every 5 seconds, including node status, data processing rate, internal queue size, error count, CPU utilization, memory utilization, disk utilization, and network traffic. All monitoring agents send the collected status data to the system monitoring center.

[0131] The system monitoring center executes the following monitoring logic: If the error count of a node exceeds a preset threshold for three consecutive monitoring periods, the node is determined to have experienced a software failure. If the internal queue size of a node continuously increases and exceeds a preset threshold for five consecutive monitoring periods, the node is determined to have insufficient processing capacity. If a node fails to send a heartbeat message for three consecutive monitoring periods, the node is determined to have experienced a hardware or network failure. When the system monitoring center detects a node failure, it immediately triggers an automatic recovery mechanism: it starts a pre-deployed backup node and synchronizes the configuration information of the failed node to the backup node; it updates the routing rules of the message bus, forwarding messages originally sent to the failed node to the backup node; it reassigns unfinished tasks from the failed node to other normally operating nodes; it generates a fault alarm and records it in the system log, notifying maintenance personnel for further processing. This embodiment ensures that the system can automatically recover services without manual intervention in the event of a single point of failure, achieving stable operation 24 / 7.

[0132] In some embodiments, by providing online incremental learning methods and user-personalized emotion recognition mechanisms, the problem of insufficient recognition accuracy of general models in specific users or specific scenarios is solved.

[0133] After outputting the final sentiment prediction result in step S103, the system provides a user feedback interface, allowing users to confirm or correct the prediction result. The system binds the correct sentiment labels provided by the user with the corresponding multimodal data, generating incremental training samples and storing them in the incremental training dataset.

[0134] The system automatically performs an incremental learning process once daily at midnight: extracting all newly added samples from the incremental training dataset for the day; executing the emotion category weight calculation sub-step in step S102 to calculate the weights of each emotion category in the incremental training dataset; fine-tuning the existing emotion recognition model using incremental training samples, freezing the model's lower-level feature extraction layer during the fine-tuning process and updating only the top-level classification layer to avoid catastrophic forgetting; saving the updated model parameters and replacing the original model parameters for subsequent inference processes. Simultaneously, the system establishes a personalized emotion feature database for each registered user, recording the user's historical emotion data and corresponding prediction results. When recognizing a registered user's emotion, in the result fusion sub-step of step S103, the system dynamically adjusts the weights of the general model's prediction results based on the user's historical prediction accuracy. If the historical prediction accuracy for a user's emotion category is lower than a preset threshold, the weight of the corresponding single-modal model's prediction result is increased to further improve personalized recognition accuracy.

[0135] Meanwhile, during the prediction process, predictions can also be made by combining a rag knowledge base that integrates human body changes with medicine, thereby improving the accuracy of personalized identification.

[0136] In some embodiments, by providing a batch emotion recognition implementation method suitable for multi-user concurrent scenarios such as intelligent customer service centers and large conference rooms, the problem of excessive system response latency under high concurrency is solved.

[0137] The system assigns a unique user identifier to each connected user and binds and stores the multimodal data of different users with the corresponding user identifier. During the inference phase, the system establishes multiple data queues, each corresponding to a sentiment analysis node. The length of the queue is dynamically adjusted according to the processing capacity of the corresponding analysis node.

[0138] The system allocates the data to be identified to different data queues according to user identifiers, and the data in each queue is sorted by arrival time. When the amount of data in the queue reaches the preset batch size, the system inputs the batch of data into the corresponding sentiment analysis node and executes the complete process from steps S101 to S103 for batch inference.

[0139] After batch inference is completed, the system distributes the inference results to the corresponding user sessions based on the user identifier associated with each data point. If the amount of data in the data queue does not reach the batch size within a preset time, the system groups all existing data in the queue into a small batch for inference to avoid excessively long waiting times for users.

[0140] The system employs a dynamic batch size adjustment mechanism, automatically adjusting the batch size based on the current number of concurrent users and CPU utilization. When the number of concurrent users is high, the batch size is increased to improve system throughput; when the number of concurrent users is low, the batch size is decreased to reduce system response latency.

[0141] In some embodiments, by providing a dynamic evaluation method for modal reliability under extreme environments and a multi-level degradation prediction mechanism, the problem of reduced recognition accuracy caused by a sharp decline in the quality of some modal data under extreme environments is solved.

[0142] The system additionally deploys environmental sensors to collect environmental parameters in real time, such as noise intensity, light intensity, temperature, and user motion acceleration. In the modal reliability assessment sub-step of step S103, the system uses environmental parameters as important evaluation indicators to dynamically adjust the reliability score of each mode.

[0143] The specific adjustment rules include: when the ambient noise intensity exceeds 60 dB, the reliability score of the audio modality decreases by 10% for every 5 dB increase; when the ambient noise intensity exceeds 90 dB, the reliability score of the audio modality is set to 0; when the ambient light intensity is below 50 lux, the reliability score of the facial modality decreases by 15% for every 10 lux decrease; when the ambient light intensity is below 10 lux, the reliability score of the facial modality is set to 0; when the user's motion acceleration exceeds 2 m / s², the reliability score of the physiological modality decreases by 10% for every 0.5 m / s² increase; when the user's motion acceleration exceeds 5 m / s², the reliability score of the physiological modality is set to 0.

[0144] The system employs a multi-level degradation prediction mechanism: if the reliability scores of all modalities are greater than 0.5, a complete multimodal fusion prediction is performed; if the reliability score of one modality is less than or equal to 0.5, that modality is excluded, and the remaining two modalities are used for fusion prediction; if the reliability scores of two modalities are less than or equal to 0.5, the system automatically degrades to single-modal prediction, using the modality with the highest reliability score for prediction; if the reliability scores of all modalities are less than or equal to 0.5, the system outputs a result indicating unknown sentiment and prompts the user to improve environmental conditions.

[0145] In some embodiments, by providing a multi-source data fusion emotion recognition implementation across multiple smart devices, the problem of incomplete data collection from a single device is solved, further improving the recognition accuracy.

[0146] Users can simultaneously use three devices: a smartphone, a smartwatch, and a smart speaker. Each device integrates different sensors: the smartphone integrates an image sensor and an audio sensor, the smartwatch integrates a physiological sensor and an accelerometer, and the smart speaker integrates a high-fidelity audio sensor.

[0147] All devices are synchronized with the cloud system clock. Each device records a local timestamp while collecting data and sends the data and timestamp to the cloud server. The cloud server executes the time offset calculation, clock drift compensation value update, and timestamp correction sub-steps in step S101 to perform unified time synchronization on data from different devices.

[0148] For multi-source data of the same modality, the system uses a weighted fusion approach to enhance the data: For audio data, the fusion weights are calculated based on the signal-to-noise ratio (SNR) of the audio data collected by the smart speaker and the smartphone. The higher the SNR, the greater the weight. For physiological data, the fusion weights are calculated based on the continuity of heart rate, skin conductance, and body temperature data collected by the smartwatch; the better the continuity, the greater the weight.

[0149] After data fusion is complete, the system executes the full process of steps S102 and S103, using the fused high-quality data for emotion recognition. When a device goes offline or malfunctions, the system automatically uses data collected from other devices to continue recognition, ensuring service continuity.

[0150] Please see Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a human emotion recognition system 200 based on multimodal sensor fusion provided in this application embodiment. The human emotion recognition system 200 based on multimodal sensor fusion is used to execute the steps of the human emotion recognition method based on multimodal sensor fusion shown in the above embodiments. The human emotion recognition system 200 based on multimodal sensor fusion can be a single server or a server cluster, or it can be a terminal, such as a handheld terminal, a laptop computer, a wearable device, or a robot.

[0151] like Figure 3 As shown, the human emotion recognition system 200 based on multimodal sensor fusion includes: The data acquisition unit 201 is used to acquire human body data in three modalities: audio, facial, and physiological, and record timestamps for each modality; calculate the offset between the timestamp of the human body data corresponding to each modality and the current time; update the clock drift compensation value of each modality using an exponential moving average method; and correct the timestamp of the human body data corresponding to each modality. The sample generation unit 202 is used to calculate the emotion category weight based on the human body data, generate synthetic samples of minority emotions, and detect available human body data and corresponding modalities in the synthetic samples. The result generation unit 203 is used to perform single-modal emotion prediction if only one modality has available human data, and input the data into a preset emotion prediction model for multimodal fusion prediction if multiple modalities have available human data, evaluate the reliability of each modality, and combine them to obtain the final emotion prediction result.

[0152] In some embodiments, after evaluating the reliability of each modality and combining them to obtain the final emotion prediction result, the method further includes: constructing a retrieval-enhanced generative knowledge base containing knowledge about the laws of human physiological changes and the correlation between medical emotions; extracting human data features corresponding to the final emotion prediction result; converting the human data features into retrieval query statements; retrieving medical knowledge entries that match the human data features from the retrieval-enhanced generative knowledge base; verifying and correcting the final emotion prediction result based on the retrieved medical knowledge entries; and outputting the corrected final emotion prediction result.

[0153] In some embodiments, before collecting human data in three modalities (audio, facial, and physiological) and recording timestamps respectively, the method further includes: collecting scene image data through an image sensor; collecting the position and motion data of human bodies in the scene through millimeter-wave radar; collecting speech data in the scene through an audio sensor and extracting voiceprint features; identifying the number of human bodies in the scene based on image data, radar data, and voiceprint features; assigning a unique identifier to each human body when the number of identified human bodies does not exceed two; tracking the position and motion trajectory of each human body; associating the collected audio data, facial data, and physiological data with the human body corresponding to the identifier; and performing subsequent timestamp correction, emotion category weight calculation, synthetic sample generation, and emotion prediction steps for each human body corresponding to the identifier.

[0154] In some embodiments, after outputting the final emotion prediction result, the method further includes: storing each emotion prediction result, the corresponding human data features, and time information to form a historical memory bank; extracting multiple emotion prediction results and corresponding human data features of the same human from the historical memory bank; learning the personalized emotion expression features of the same human; generating an interactive query statement when a new emotion prediction result differs from the historical personalized emotion expression features of the human; outputting the interactive query statement to the corresponding human through voice interaction; receiving voice feedback information from the corresponding human; updating the personalized emotion expression features and historical memory bank of the human based on the voice feedback information; and optimizing the subsequent emotion prediction process using the updated personalized emotion expression features.

[0155] In some embodiments, the step of calculating the emotion category weights based on the human body data and generating synthetic samples of minority emotions includes: extracting emotion labels corresponding to all human body data and counting the number of samples corresponding to each emotion label; calculating the weight of the corresponding emotion category based on the number of samples for each emotion label; identifying emotion categories with fewer than a preset threshold of samples as minority emotions; and using a generative adversarial network to generate a specified number of synthetic samples of minority emotions.

[0156] In some embodiments, calculating the emotion category weights based on the human body data includes: extracting emotion tags corresponding to all human body data, obtaining all unique emotion tags to form an emotion category set; counting the number of samples corresponding to each emotion category in the emotion category set; using a balanced category weight calculation method, calculating the weight of the corresponding emotion category based on the number of samples in each emotion category; establishing a mapping relationship between each emotion category and its corresponding weight and storing it.

[0157] In some embodiments, the step of performing single-modal emotion prediction if only one modality has available human data includes: if only the audio modality has available human data, inputting the available data of the audio modality into a preset audio emotion recognition model for emotion prediction; if only the facial modality has available human data, inputting the available data of the facial modality into a preset facial emotion recognition model for emotion prediction; if only the physiological modality has available human data, inputting the available data of the physiological modality into a preset physiological emotion recognition model for emotion prediction; and outputting the single-modal emotion prediction result and the corresponding confidence level.

[0158] In some embodiments, if available human data exists for the multiple modalities, it is input into a preset emotion prediction model for multimodal fusion prediction, evaluating the reliability of each modality and combining them to obtain the final emotion prediction result. This includes: inputting human data from all available modalities into a preset multimodal fusion emotion prediction model for fusion prediction to obtain a fusion prediction result; calculating the reliability score of each modality based on the completeness, continuity, and signal-to-noise ratio of the data; weighting and combining the fusion prediction result and the emotion prediction results of each single modality based on the reliability scores of each modality; and outputting the final emotion prediction result and its corresponding confidence level after weighted combination.

[0159] In some embodiments, the method further includes: performing dynamic range quantization on all preset emotion prediction models to obtain quantized emotion prediction models; performing low-amplitude pruning on the quantized emotion prediction models to remove parameters with small absolute weight values ​​in the emotion prediction models; forming data batches of human data to be inferred according to preset batch sizes; sequentially inputting the data batches into the optimized emotion prediction models for batch inference and outputting batch inference results.

[0160] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the human emotion recognition system based on multimodal sensor fusion and each module described above can be referred to the corresponding content in the various embodiments of the human emotion recognition method based on multimodal sensor fusion, and will not be repeated here.

[0161] The aforementioned human emotion recognition method based on multimodal sensor fusion can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the device shown.

[0162] Please see Figure 4 , Figure 4 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application. The computer device includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and internal memory.

[0163] The storage medium may store operating devices and computer programs. The computer program includes program instructions that, when executed, cause the processor to perform any human emotion recognition method based on multimodal sensor fusion.

[0164] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0165] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program can enable the processor to perform any human emotion recognition method based on multimodal sensor fusion.

[0166] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0167] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0168] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: Collect human data in three modalities: audio, facial, and physiological, and record timestamps for each. Calculate the offset between the timestamp of the human data corresponding to each modality and the current time, update the clock drift compensation value for each modality using an exponential moving average method, and correct the timestamp of the human data corresponding to each modality. Based on the human body data, calculate the weight of the emotion category, generate synthetic samples of minority emotions, and detect available human body data and corresponding modalities in the synthetic samples. If only one modality has available human data, single-modal emotion prediction is performed. If multiple modalities have available human data, the data is input into a preset emotion prediction model for multimodal fusion prediction. The reliability of each modality is evaluated and combined to obtain the final emotion prediction result.

[0169] In some embodiments, after evaluating the reliability of each modality and combining them to obtain the final emotion prediction result, the method further includes: constructing a retrieval-enhanced generative knowledge base containing knowledge about the laws of human physiological changes and the correlation between medical emotions; extracting human data features corresponding to the final emotion prediction result; converting the human data features into retrieval query statements; retrieving medical knowledge entries that match the human data features from the retrieval-enhanced generative knowledge base; verifying and correcting the final emotion prediction result based on the retrieved medical knowledge entries; and outputting the corrected final emotion prediction result.

[0170] In some embodiments, before collecting human data in three modalities (audio, facial, and physiological) and recording timestamps respectively, the method further includes: collecting scene image data through an image sensor; collecting the position and motion data of human bodies in the scene through millimeter-wave radar; collecting speech data in the scene through an audio sensor and extracting voiceprint features; identifying the number of human bodies in the scene based on image data, radar data, and voiceprint features; assigning a unique identifier to each human body when the number of identified human bodies does not exceed two; tracking the position and motion trajectory of each human body; associating the collected audio data, facial data, and physiological data with the human body corresponding to the identifier; and performing subsequent timestamp correction, emotion category weight calculation, synthetic sample generation, and emotion prediction steps for each human body corresponding to the identifier.

[0171] In some embodiments, after outputting the final emotion prediction result, the method further includes: storing each emotion prediction result, the corresponding human data features, and time information to form a historical memory bank; extracting multiple emotion prediction results and corresponding human data features of the same human from the historical memory bank; learning the personalized emotion expression features of the same human; generating an interactive query statement when a new emotion prediction result differs from the historical personalized emotion expression features of the human; outputting the interactive query statement to the corresponding human through voice interaction; receiving voice feedback information from the corresponding human; updating the personalized emotion expression features and historical memory bank of the human based on the voice feedback information; and optimizing the subsequent emotion prediction process using the updated personalized emotion expression features.

[0172] In some embodiments, the step of calculating the emotion category weights based on the human body data and generating synthetic samples of minority emotions includes: extracting emotion labels corresponding to all human body data and counting the number of samples corresponding to each emotion label; calculating the weight of the corresponding emotion category based on the number of samples for each emotion label; identifying emotion categories with fewer than a preset threshold of samples as minority emotions; and using a generative adversarial network to generate a specified number of synthetic samples of minority emotions.

[0173] In some embodiments, calculating the emotion category weights based on the human body data includes: extracting emotion tags corresponding to all human body data, obtaining all unique emotion tags to form an emotion category set; counting the number of samples corresponding to each emotion category in the emotion category set; using a balanced category weight calculation method, calculating the weight of the corresponding emotion category based on the number of samples in each emotion category; establishing a mapping relationship between each emotion category and its corresponding weight and storing it.

[0174] In some embodiments, the step of performing single-modal emotion prediction if only one modality has available human data includes: if only the audio modality has available human data, inputting the available data of the audio modality into a preset audio emotion recognition model for emotion prediction; if only the facial modality has available human data, inputting the available data of the facial modality into a preset facial emotion recognition model for emotion prediction; if only the physiological modality has available human data, inputting the available data of the physiological modality into a preset physiological emotion recognition model for emotion prediction; and outputting the single-modal emotion prediction result and the corresponding confidence level.

[0175] In some embodiments, if available human data exists for the multiple modalities, it is input into a preset emotion prediction model for multimodal fusion prediction, evaluating the reliability of each modality and combining them to obtain the final emotion prediction result. This includes: inputting human data from all available modalities into a preset multimodal fusion emotion prediction model for fusion prediction to obtain a fusion prediction result; calculating the reliability score of each modality based on the completeness, continuity, and signal-to-noise ratio of the data; weighting and combining the fusion prediction result and the emotion prediction results of each single modality based on the reliability scores of each modality; and outputting the final emotion prediction result and its corresponding confidence level after weighted combination.

[0176] In some embodiments, the method further includes: performing dynamic range quantization on all preset emotion prediction models to obtain quantized emotion prediction models; performing low-amplitude pruning on the quantized emotion prediction models to remove parameters with small absolute weight values ​​in the emotion prediction models; forming data batches of human data to be inferred according to preset batch sizes; sequentially inputting the data batches into the optimized emotion prediction models for batch inference and outputting batch inference results.

[0177] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the steps of the human emotion recognition method based on multimodal sensor fusion as provided in any embodiment of this application.

[0178] The computer-readable storage medium can be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the computer device.

[0179] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for human emotion recognition based on multimodal sensor fusion, characterized in that, include: Collect human data in three modalities: audio, facial, and physiological, and record timestamps for each. Calculate the offset between the timestamp of the human body data corresponding to each modality and the current time, update the clock drift compensation value of each modality using an exponential moving average method, and correct the timestamp of the human body data corresponding to each modality. Based on the human body data, calculate the weight of the emotion category, generate synthetic samples of minority emotions, and detect available human body data and corresponding modalities in the synthetic samples. If only one modality has available human data, single-modal emotion prediction is performed. If multiple modalities have available human data, the data is input into a preset emotion prediction model for multimodal fusion prediction. The reliability of each modality is evaluated and combined to obtain the final emotion prediction result.

2. The method according to claim 1, characterized in that, After evaluating the reliability of each modality and combining them to obtain the final sentiment prediction result, the process also includes: Construct a retrieval-enhanced knowledge base that includes knowledge about the relationship between human physiological changes and medical emotions; Extract the human data features corresponding to the final emotion prediction results; Convert human body data features into search query statements; Retrieve medical knowledge entries that match human data features from the enhanced knowledge base; The final emotion prediction results were verified and corrected based on the retrieved medical knowledge entries. Output the corrected final sentiment prediction result.

3. The method according to claim 1, characterized in that, Before collecting human data in three modalities—audio, facial, and physiological—and recording timestamps for each, the process also includes: Acquire scene image data using an image sensor; The location and movement data of the human body in the scene are collected using millimeter-wave radar; Collect speech data from the scene using an audio sensor and extract voiceprint features; Identify the number of human bodies in a scene based on image data, radar data, and voiceprint features; When the number of human bodies identified does not exceed two, a unique identifier is assigned to each human body. Track the position and movement trajectory of each individual human body; The collected audio data, facial data, and physiological data are associated with the corresponding human body identifiers; For each individual identity, the subsequent steps of timestamp correction, emotion category weight calculation, synthetic sample generation, and emotion prediction are performed.

4. The method according to claim 1, characterized in that, After outputting the final sentiment prediction result, it also includes: Store each emotion prediction result, corresponding human data characteristics, and time information to form a historical memory bank; Extract multiple emotion prediction results and corresponding human data features of the same human being from the historical memory bank; Learn the personalized emotional expression characteristics of the same person; When the new emotion prediction result differs from the individual's historical personalized emotion expression characteristics, an interactive query is generated. The system outputs interactive query statements to the corresponding human body via voice interaction. Receive voice feedback information from the corresponding human body; The personalized emotional expression characteristics and historical memory database of the human body are updated based on voice feedback information; The subsequent sentiment prediction process is optimized using updated personalized sentiment expression features.

5. The method according to claim 1, characterized in that, The step of calculating emotion category weights based on the human body data and generating synthetic samples of minority emotions includes: Extract the emotion labels corresponding to all human data, and count the number of samples corresponding to each emotion label; The weight of the corresponding emotion category is calculated based on the number of samples for each emotion tag; Emotion categories with fewer than a preset threshold of samples are identified as minority emotions. Generative adversarial networks are used to generate a specified number of synthetic samples of minority class sentiment.

6. The method according to claim 1, characterized in that, The calculation of emotion category weights based on the human body data includes: Extract the emotion tags corresponding to all human data, and obtain a set of emotion categories composed of all unique emotion tags; Count the number of samples corresponding to each emotion category in the set of emotion categories; A balanced category weighting method is used to calculate the weight of each emotion category based on the number of samples in each emotion category. Establish and store a mapping relationship between each emotion category and its corresponding weight.

7. The method according to claim 1, characterized in that, If only one modality has available human data, then single-modal emotion prediction is performed, including: If only the audio modality has usable human data, then the usable data of the audio modality is input into the preset audio emotion recognition model for emotion prediction. If only facial modality has usable human data, then the usable data of the facial modality is input into the preset facial emotion recognition model for emotion prediction. If only physiological modalities have usable human data, then the usable data of the physiological modalities will be input into the preset physiological emotion recognition model for emotion prediction. Output the unimodal sentiment prediction results and their corresponding confidence levels.

8. The method according to claim 1, characterized in that, If usable human data exists for each of the multiple modalities, it is input into a preset emotion prediction model for multimodal fusion prediction. The reliability of each modality is evaluated and combined to obtain the final emotion prediction result, including: All available modal human data are input into a pre-set multimodal fusion emotion prediction model for fusion prediction to obtain the fusion prediction result; The reliability score for each mode is calculated based on the completeness, continuity, and signal-to-noise ratio of the data. Based on the reliability scores of each modality, the fusion prediction results and the sentiment prediction results of each individual modality are weighted and combined. Output the final sentiment prediction result after weighted combination and the corresponding confidence level.

9. The method according to claim 1, characterized in that, The method further includes: Dynamic range quantization is performed on all preset emotion prediction models to obtain quantized emotion prediction models. The quantified sentiment prediction model is pruned with a low amplitude to remove parameters with small absolute weights. The human data to be inferred is grouped into data batches according to a preset batch size; the data batches are then sequentially input into the optimized emotion prediction model for batch inference, and the batch inference results are output.

10. A human emotion recognition system based on multimodal sensor fusion, used to implement the human emotion recognition method based on multimodal sensor fusion as described in any one of claims 1-9; characterized in that, include: The data acquisition unit is used to collect human data in three modalities: audio, facial, and physiological, and record timestamps for each. Calculate the offset between the timestamp of the human body data corresponding to each modality and the current time, update the clock drift compensation value of each modality using an exponential moving average method, and correct the timestamp of the human body data corresponding to each modality. The sample generation unit is used to calculate the emotion category weights based on the human data, generate synthetic samples of minority emotions, and detect available human data and corresponding modalities in the synthetic samples. The result generation unit is used to perform single-modal emotion prediction if only one modality has available human data, and input the data into a preset emotion prediction model for multimodal fusion prediction if multiple modalities have available human data. The unit evaluates the reliability of each modality and combines them to obtain the final emotion prediction result.

Images