Intelligent mood adjusting system for preschool children based on multi-modal biological recognition

Abstract

The invention provides a preschool child emotion intelligent adjusting system based on multi-modal biological recognition, and the system comprises a multi-modal data obtaining module which is used for obtaining the multi-modal biological data of a target child, and carrying out the data preprocessing of the multi-modal biological data, and obtaining a multi-modal data set; the hierarchical feature extraction module is used for performing hierarchical feature extraction on the multi-modal data set to generate a hierarchical vector set; the cross-modal fusion module is used for performing time sequence alignment and dynamic weight fusion on the hierarchical vector set to generate a joint emotion representation vector; the emotion recognition module is used for performing emotion classification processing based on the joint emotion representation vector to generate an emotion recognition result; and the emotion adjustment execution module is used for generating a feedback instruction according to the emotion recognition result, and the feedback instruction is used for indicating the intelligent adjustment device to execute an adjustment strategy matched with the emotion recognition result. The system can effectively capture the hierarchical association of the emotion of the child, and improves the emotion recognition accuracy and the emotion intelligent adjustment pertinence.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to an intelligent emotion regulation system for preschool children based on multimodal biometrics. Background Art

[0002] Research on children's emotion regulation is an important area of ​​research at the intersection of psychology and artificial intelligence, crucial for promoting children's mental health and optimizing educational interventions. Multimodal biometric recognition technology can capture children's emotional states in different contexts, providing support for personalized emotion management. Existing technologies often use single-modal data (such as facial expressions or voice signals) for emotion recognition and integrate multimodal data through pre-set rules or simple weighted fusion. However, current single-modal emotion recognition approaches are sensitive to environmental interference and poorly adaptable to individual differences. For example, children's expressions or voices are easily distorted in bright light or noisy environments, resulting in a significant decrease in recognition accuracy. Furthermore, traditional methods often employ fixed weighting strategies when fusing cross-modal data, ignoring the temporal correlation and hierarchical semantic relationships between physiological signals and behavioral data. This makes it difficult to capture the dynamic mapping from low-level physiological responses (such as skin conductance changes) to high-level emotional states (such as anger and anxiety). Furthermore, children's emotional expressions are transient, ambiguous, and highly context-dependent. Existing systems lack the capabilities for hierarchical feature modeling and personalized dynamic regulation, making them unable to meet the practical needs of emotional intelligence intervention for preschool children. Summary of the Invention

[0003] Based on this, it is necessary to provide a preschool children's emotional intelligence regulation system based on multimodal biometrics to address the above technical problems, which can effectively capture the hierarchical correlation of children's emotions, improve the accuracy of emotion recognition and the targeted emotional intelligence regulation.

[0004] In the first aspect, the present application provides a preschool children's emotional intelligence regulation system based on multimodal biometrics, including a multimodal data acquisition module, a hierarchical feature extraction module, a cross-modal fusion module, an emotion recognition module, and an emotion regulation execution module:

[0005] A multimodal data acquisition module is used to obtain multimodal biological data of the target child and perform data preprocessing on the multimodal biological data to obtain a multimodal dataset. The multimodal dataset includes facial image data, speech signals, and skin conductance signals.

[0006] Hierarchical feature extraction module, used to extract hierarchical features from multimodal datasets and generate hierarchical vector sets, which include facial spatial feature vectors, speech temporal feature vectors, and physiological frequency domain feature vectors;

[0007] A cross-modal fusion module is used to perform temporal alignment and dynamic weight fusion on the layered vector set to generate a joint emotion representation vector;

[0008] Emotion recognition module, which is used to perform emotion classification based on the joint emotion representation vector and generate emotion recognition results;

[0009] The emotion regulation execution module is used to generate a feedback instruction based on the emotion recognition result, and the feedback instruction is used to instruct the intelligent regulation device to execute the regulation strategy matching the emotion recognition result.

[0010] In a possible embodiment, the multimodal data acquisition module includes:

[0011] A facial data processing unit is used to perform key point normalization processing on facial image data to generate standardized facial data;

[0012] A voice data processing unit is used to extract Mel-frequency cepstral coefficients of the voice signal and filter out silent segments to generate voice feature data;

[0013] Physiological data processing unit, used to perform wavelet denoising and skin conductance response event detection on skin conductance signals to generate physiological characteristic data;

[0014] The data synchronization unit is used to synchronize and align the timestamps of standardized facial data, voice feature data, and physiological feature data based on a hardware-level precise time protocol to generate a multimodal dataset.

[0015] In a possible embodiment, the hierarchical feature extraction module includes:

[0016] Facial feature extraction unit, which performs spatial convolution processing on standardized facial data to extract local muscle movement features, and extracts dynamic change features between frames through a three-dimensional convolutional network to generate facial spatial feature vectors;

[0017] The speech feature extraction unit is used to perform one-dimensional convolution processing on the speech feature data to extract short-term acoustic features, and extract context-related features through a bidirectional long short-term memory network to generate a speech time series feature vector;

[0018] The physiological feature extraction unit is used to perform wavelet transform processing on the physiological feature data, decompose the frequency domain energy distribution characteristics, and extract the physiological response intensity through the skin conductance response event detection algorithm to generate the physiological frequency domain feature vector.

[0019] In a possible embodiment, the emotion recognition module includes:

[0020] A classification processing unit is used to input the joint emotion representation vector into a multi-layer perceptron for classification processing and output the probability distribution and confidence score of seven basic emotions;

[0021] Uncertainty fusion unit, used to resolve conflicts in the probability distribution of seven basic emotions based on evidence theory and generate initial emotion recognition results;

[0022] A personalized adaptation unit, which uses a meta-learning adapter to fine-tune the classification parameters of the multilayer perceptron online based on the target child's historical emotion data to generate a personalized emotion classification model;

[0023] The personalized emotion recognition unit is used to input the initial emotion recognition results into the personalized emotion classification model for calibration to generate emotion recognition results.

[0024] In a possible embodiment, the emotion regulation execution module includes:

[0025] A context matching subunit, which dynamically adjusts the weight distribution strategy of emotion recognition results based on the current context label, generating a context adaptation adjustment strategy. The current context label is generated based on the sound and light intensity data collected by the environmental sensor.

[0026] The individual mapping subunit is used to match the physiological baseline and behavioral preferences of the target child from a preset child individual feature database to generate individualized adjustment parameters;

[0027] The instruction issuing subunit is used to generate feedback instructions based on the situational adaptation adjustment strategy and individual adjustment parameters.

[0028] In a second aspect, the present application also provides a method for regulating emotional intelligence in preschool children based on multimodal biometrics, comprising:

[0029] Acquire multimodal biological data of the target child and perform data preprocessing on the multimodal biological data to obtain a multimodal dataset, which includes facial image data, speech signals, and skin conductance signals;

[0030] Perform hierarchical feature extraction on the multimodal dataset to generate a hierarchical vector set, which includes facial spatial feature vectors, speech temporal feature vectors, and physiological frequency domain feature vectors;

[0031] Perform temporal alignment and dynamic weight fusion on the layered vector set to generate a joint emotion representation vector;

[0032] Perform emotion classification processing based on the joint emotion representation vector to generate emotion recognition results;

[0033] A feedback instruction is generated according to the emotion recognition result, and the feedback instruction is used to instruct the intelligent adjustment device to execute an adjustment strategy that matches the emotion recognition result.

[0034] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method for regulating emotional intelligence of preschool children based on multimodal biometrics when executing the computer program.

[0035] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for regulating emotional intelligence of preschool children based on multimodal biometrics.

[0036] The aforementioned intelligent emotion regulation system for preschool children based on multimodal biometrics acquires multimodal biometric data, including facial image data, voice signals, and skin conductance signals, from the target child and performs data preprocessing to form a multimodal dataset. It then performs hierarchical feature extraction on the multimodal dataset to generate a hierarchical vector set consisting of facial spatial feature vectors, voice temporal feature vectors, and physiological frequency domain feature vectors. The hierarchical vector set is then subjected to temporal alignment and dynamic weight fusion to generate a joint emotion representation vector. The emotion recognition module then performs emotion classification processing based on the joint emotion representation vector to generate an emotion recognition result. The emotion regulation execution module generates feedback instructions based on the emotion recognition result, instructing the intelligent regulation device to execute a regulation strategy that matches the emotion recognition result. The aforementioned technical solution can effectively capture the hierarchical associations and dynamic changes of children's emotions, thereby improving the accuracy of emotion recognition and the targeted nature of intelligent emotion regulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of the structure of a preschool children's emotional intelligence regulation system based on multimodal biometrics provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the structure of a multimodal data acquisition module provided in an embodiment of the present invention;

[0040] Figure 3 A flowchart of a method for regulating emotional intelligence in preschool children based on multimodal biometrics provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0042] First, a brief introduction is given to the terms involved in the embodiments of this application.

[0043] Skin conductance (EDA) is a bioelectrical signal that reflects the activity of the sympathetic nervous system by measuring changes in the electrical conductivity of the skin surface. Its changes are highly correlated with emotional arousal (such as tension, excitement, or anxiety). When children experience emotional fluctuations, sympathetic nerve activation triggers increased sweat gland secretion, leading to a transient increase in skin conductivity. By detecting these conductance response events (SCRs) and their frequency domain energy distribution, the intensity and dynamic changes of the physiological response to emotion can be quantified, providing objective physiological indicators for multimodal emotion recognition. Compared to single behavioral data, skin conductance signals can effectively capture unconscious physiological reactions induced by emotions, addressing the blind spots in the recognition of facial expressions or voice signals in scenarios of emotional suppression or disguise, thereby enhancing the system's anti-interference ability and the reliability of emotional state discrimination.

[0044] Spatial convolution is a local feature extraction technology based on convolutional neural networks. It extracts the texture, edge and structural information of local areas layer by layer in the two-dimensional spatial dimension by sliding the convolution kernel. Compared with the global feature extraction method, spatial convolution processing can effectively focus on local areas related to emotions, avoid background noise interference, and provide high-resolution spatial semantic representation for subsequent temporal dynamic feature extraction.

[0045] The Bidirectional Long Short-Term Memory (Bi-LSTM) is a specialized recurrent neural network (RNN) architecture specifically designed for processing sequential data. It consists of two LSTM networks operating in opposite directions: a forward LSTM processes information from the beginning to the end of the sequence, while a reverse LSTM processes information from the end to the beginning. This bidirectional structure enables the Bi-LSTM to capture both forward and backward information in the sequence data, thereby more comprehensively understanding the contextual relationships within the sequence. For example, in natural language processing, the Bi-LSTM can better understand the semantics of each word in a sentence by considering not only the preceding words but also the influence of subsequent words on the current word. This bidirectional information fusion capability enables the Bi-LSTM to perform well in tasks such as speech recognition, text classification, and machine translation, effectively improving the model's modeling capabilities and prediction accuracy for sequential data.

[0046] Based on the above explanations, the multimodal biometric-based intelligent emotional regulation system for preschool children provided in the embodiments of this application is described. Schematically, the implementation environment includes: a sensor group, a feedback regulation device, a terminal, and a processor. The terminal is connected to the sensor group, intelligent regulation device, and processor via a network; the sensor group includes, but is not limited to, a high-frame-rate near-infrared camera, an optical camera, an acoustic sensor, a bioelectric sensor, and an environmental sensor; the intelligent regulation device can be a temperature and humidity controller, a tactile feedback device, a visual guidance device, an auditory interaction device, and the like; and the processor can be a central processing unit, a neural network processor, or a multi-core processor, without limitation here.

[0047] In combination with the above explanations of terms and implementation environments, the application scenarios of the embodiments of this application are explained. The preschool children's emotional intelligence regulation system based on multimodal biometrics provided in the embodiments of this application can be applied to, but not limited to, the following scenarios:

[0048] In preschool settings, this system uses high-sensitivity cameras and microphone arrays deployed in classrooms to capture children's facial expressions, voice intonation, and body movements in real time. Combined with wristband biosensors, it monitors skin conductance and heart rate variability. If it detects individual children experiencing anxiety (such as frowning, trembling voice, or increased heart rate) during group activities due to conflict, the system automatically triggers AR projection to guide interactive games, distracting children and promoting calm. Simultaneously, it sends notifications to the teacher's terminal, enabling them to quickly intervene and maintain classroom order.

[0049] In child psychology clinics, this system can assist doctors in quantitatively evaluating the effectiveness of emotional interventions. For example, during exposure therapy, by real-time monitoring of skin conductance signals and facial micro-expressions, it can analyze children's physiological and behavioral responses to specific stimuli, dynamically adjust the stimulation intensity of the virtual reality scene, and record emotional change curves, providing doctors with objective evidence for efficacy evaluation and optimizing treatment paths.

[0050] In home settings, children can wear smart bracelets with integrated biosensors, which, in conjunction with smart cameras and voice capture equipment in the living room, provide real-time analysis of their emotional state. For example, if a child experiences persistent crying due to separation anxiety (characterized by a high-pitched voice and a sudden increase in skin conductivity), the system will activate a smart speaker to play the mother's pre-recorded soothing voice, adjust the ambient lighting to warm tones, and provide rhythmic tactile feedback through the vibrating bracelet, simulating the feeling of a hug, to help the child gradually calm down.

[0051] Illustratively, the multimodal biometric-based intelligent emotional regulation system for preschool children provided in the embodiments of the present application can also be applied to other application scenarios. This is only used as an example and does not limit the specific application scenarios.

[0052] In an exemplary embodiment, Figure 1 As shown, a preschool children's emotional intelligence regulation system 10 based on multimodal biometrics is provided. This embodiment uses the system as an example of a terminal in the aforementioned implementation environment. It is understandable that the system can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the system includes a multimodal data acquisition module 11, a hierarchical feature extraction module 12, a cross-modal fusion module 13, an emotion recognition module 14, and an emotion regulation execution module 15:

[0053] The multimodal data acquisition module 11 is used to acquire the multimodal biological data of the target child and perform data preprocessing on the multimodal biological data to obtain a multimodal data set. The multimodal data set includes facial image data, voice signals and skin conductance signals.

[0054] Specifically, multimodal biometric data from the target child can be acquired through a variety of sensor devices, including facial image data, voice signals, and skin conductance signals. For example, facial image data can be acquired using a high-definition camera, voice signals using a high-sensitivity microphone, and skin conductance signals using a skin conductance sensor. The collected data undergoes data preprocessing, such as denoising and normalization, to form a multimodal dataset, providing high-quality data input for subsequent modules.

[0055] The hierarchical feature extraction module 12 is used to perform hierarchical feature extraction on the multimodal data set to generate a hierarchical vector set. The hierarchical vector set includes facial spatial feature vectors, speech temporal feature vectors and physiological frequency domain feature vectors.

[0056] Specifically, hierarchical feature extraction is performed on multimodal datasets. For example, for facial image data, convolutional neural networks are used to extract facial spatial feature vectors to capture the key features of facial expressions; for speech signals, Mel-frequency cepstral coefficients are used to extract speech temporal feature vectors to analyze the emotional characteristics of speech; for skin conductance signals, physiological frequency domain feature vectors are extracted through Fourier transform to reflect changes in physiological state. Through hierarchical feature extraction technology, the multi-level characteristics of emotional expression are analyzed from the spatial, temporal and frequency domain dimensions to improve the discriminability of feature representation.

[0057] The cross-modal fusion module 13 is used to perform temporal alignment and dynamic weight fusion on the layered vector set to generate a joint emotion representation vector.

[0058] Specifically, the layered vector sets are time-aligned to ensure temporal consistency across modal data. A dynamic weighted fusion algorithm assigns weights to each modal data based on its importance in emotion recognition, fusing the feature vectors from each modality into a joint emotion representation vector. For example, when speech signals are more critical in emotion recognition, a higher weight is assigned to the speech temporal feature vector.

[0059] The emotion recognition module 14 is used to perform emotion classification processing based on the joint emotion representation vector and generate an emotion recognition result.

[0060] Specifically, emotion classification processing is performed based on the joint emotion representation vector. For example, the joint emotion representation vector can be mapped to specific emotion categories, such as happiness, sadness, anger, etc., through a trained deep learning model, such as a support vector machine (SVM) or a deep neural network (DNN), to generate emotion recognition results.

[0061] The emotion regulation execution module 15 is used to generate a feedback instruction according to the emotion recognition result, and the feedback instruction is used to instruct the intelligent regulation device to execute the regulation strategy matching the emotion recognition result.

[0062] Specifically, feedback instructions are generated based on the emotion recognition results. For example, if a child is identified as anxious, the feedback instructions can instruct the intelligent control device to play soothing music or display a soothing image, implementing a control strategy that matches the emotion recognition results to help the child relieve their emotions.

[0063] The aforementioned intelligent emotion regulation system for preschool children based on multimodal biometrics acquires multimodal biometric data, including facial image data, voice signals, and skin conductance signals, from the target child and performs data preprocessing to form a multimodal dataset. It then performs hierarchical feature extraction on the multimodal dataset to generate a hierarchical vector set consisting of facial spatial feature vectors, voice temporal feature vectors, and physiological frequency domain feature vectors. The hierarchical vector set is then subjected to temporal alignment and dynamic weight fusion to generate a joint emotion representation vector. The emotion recognition module then performs emotion classification processing based on the joint emotion representation vector to generate an emotion recognition result. The emotion regulation execution module generates feedback instructions based on the emotion recognition result, instructing the intelligent regulation device to execute a regulation strategy that matches the emotion recognition result. The aforementioned technical solution can effectively capture the hierarchical associations and dynamic changes of children's emotions, thereby improving the accuracy of emotion recognition and the targeted nature of intelligent emotion regulation.

[0064] like Figure 2 As shown, in a possible embodiment, the multimodal data acquisition module 11 includes:

[0065] The facial data processing unit 111 is configured to perform key point normalization processing on the facial image data to generate standardized facial data.

[0066] Specifically, facial image data captured by a camera can be received, and a keypoint detection algorithm can be used to locate facial features (such as the corners of the eyes and mouth). The image can then be rotated and scaled based on an affine transformation, and the facial area can be normalized to a standard size and normal viewing angle to generate standardized facial data. This normalization process eliminates lighting differences and head posture deviations, ensuring consistency of facial expression features across different scenarios. Furthermore, the facial image data can be continuous frame images or video stream data.

[0067] The speech data processing unit 112 is used to extract Mel-frequency cepstral coefficients and perform silent segment filtering on the speech signal to generate speech feature data.

[0068] Specifically, the original speech signal collected by the microphone is framed and windowed, and the Mel-frequency cepstral coefficients are extracted as acoustic features. The silent segments and non-speech noise are filtered out through energy threshold detection and dynamic time warping algorithm, and the valid speech segments are retained to generate speech feature data to eliminate the interference of environmental noise and invalid speech segments and improve the signal-to-noise ratio of the speech timing features.

[0069] The physiological data processing unit 113 is used to perform wavelet denoising and skin conductance response event detection processing on the skin conductance signal to generate physiological characteristic data.

[0070] Specifically, the original signal collected by the skin conductance sensor is decomposed by wavelet transform to separate high-frequency noise and low-frequency effective components, retaining the frequency band reflecting sympathetic nerve activity; skin conductance response events are identified through a sliding window peak detection algorithm, and physiological characteristic data containing event intensity and timing information are generated to suppress baseline drift and transient interference, and accurately capture emotion-related physiological response events.

[0071] The data synchronization unit 114 is used to synchronize and align the timestamps of the standardized facial data, voice feature data, and physiological feature data based on a hardware-level precise time protocol to generate a multimodal data set.

[0072] Specifically, based on a hardware-level precise time protocol, a unified clock source is allocated to facial image data, voice signals, and skin conductance signals. The acquisition delay of each sensor is compensated through a timestamp alignment algorithm, and the multimodal data stream is synchronized to millisecond-level accuracy to generate a time-consistent multimodal data set. This solves the problem of timing mismatch in multi-source data and ensures the reliability of cross-modal feature fusion.

[0073] In a possible embodiment, the hierarchical feature extraction module 12 includes:

[0074] The facial feature extraction unit is used to perform spatial convolution processing on the standardized facial data, extract local muscle movement features, and extract inter-frame dynamic change features through a three-dimensional convolutional network to generate a facial spatial feature vector.

[0075] Specifically, by using a spatial convolutional network to slide the convolution kernel on the spatial dimension of the facial image, the muscle movement texture features of local areas such as the eye area and mouth corners are extracted layer by layer; further, a three-dimensional convolutional network is used to continuously analyze the dynamic changes of multiple frames of images in the time dimension, capture the evolution of expressions from calm to peak state, and generate a facial spatial feature vector that integrates spatial structure and temporal dynamics. The above method focuses on local micro-expression details through spatial convolution, and models the dynamic continuity of expressions through three-dimensional convolution, comprehensively capturing the static and dynamic information of facial expressions, so as to enhance the ability to capture short-term emotional changes.

[0076] The speech feature extraction unit is used to perform one-dimensional convolution processing on the speech feature data to extract short-term acoustic features, and extract context-related features through a bidirectional long short-term memory network to generate a speech time series feature vector.

[0077] Specifically, a one-dimensional convolutional layer is used to extract short-term acoustic features of speech signals (such as pitch and resonance peaks) in the time domain to capture emotional fluctuations at the syllable level. A bidirectional long short-term memory network is further used to scan the speech sequence bidirectionally along the time axis to model the contextual relevance between speech segments (such as the rising ending of the interrogative tone), and to generate a speech time series feature vector containing short-term details and long-term dependencies. The above method strengthens local acoustic features through one-dimensional convolution, and the bidirectional network integrates contextual semantics to improve the accuracy of emotional discrimination of complex intonations.

[0078] The physiological feature extraction unit is used to perform wavelet transform processing on the physiological feature data, decompose the frequency domain energy distribution characteristics, and extract the physiological response intensity through the skin conductance response event detection algorithm to generate the physiological frequency domain feature vector.

[0079] Specifically, the skin conductance signal can be decomposed into energy distribution characteristics of different frequency bands through wavelet transform, separating the low-frequency components related to sympathetic nerve activity and the high-frequency components of noise; further, a sliding window can be used to detect the peak intensity and duration of skin conductance response events, quantify the intensity of physiological responses induced by emotions, and generate physiological frequency domain feature vectors that characterize frequency domain characteristics and event intensity. Wavelet transform suppresses noise interference, and event detection focuses on significant physiological reactions, enhancing the distinguishability of emotion-related physiological characteristics.

[0080] In a possible embodiment, the emotion recognition module 14 includes:

[0081] The classification processing unit is used to input the joint emotion representation vector into the multi-layer perceptron for classification processing and output the probability distribution and confidence score of seven basic emotions.

[0082] Specifically, the joint emotion representation vector can be input into a multilayer perceptron for fully connected computation and nonlinear activation, outputting probability distributions for seven basic emotions and corresponding confidence scores. These seven basic emotions are derived from Paul Ekman's classic theory, describing universally shared core human emotions: happiness, sadness, anger, fear, disgust, surprise, and neutrality. The probability distribution reflects the likelihood of each emotion category, while the confidence score measures the reliability of the classification result. The nonlinear mapping capabilities of the multilayer perceptron convert the high-dimensional emotion representation vector into interpretable emotion category probabilities, providing highly discriminative features for subsequent decision-making.

[0083] The uncertainty fusion unit is used to resolve conflicts in the probability distribution of seven basic emotions based on evidence theory and generate initial emotion recognition results.

[0084] Specifically, based on the Dempster combination rule in evidence theory, we can calculate the conflict measure between different emotion categories, eliminate the interference of low-confidence categories through normalization, and fuse the multimodal decision results to generate the initial emotion recognition result. This solution reduces the impact of single-modal misjudgments through conflict resolution and improves the consistency and reliability of multimodal emotion recognition results.

[0085] A personalized adaptation unit is used to fine-tune the classification parameters of the multi-layer perceptron online based on the historical emotion data of the target child through a meta-learning adapter to generate a personalized emotion classification model.

[0086] Specifically, the historical emotional data of the target children are loaded from a preset database, and the fully connected layer parameters of the multi-layer perceptron are incrementally updated with gradients through a meta-learning adapter, so that the classification model can adapt to the differences in individual emotional expression and generate an individualized emotion classification model. Through online fine-tuning based on historical data, the individual differences in children's emotional patterns can be resolved, and the adaptability of the classification model to specific users can be enhanced.

[0087] The personalized emotion recognition unit is used to input the initial emotion recognition results into the personalized emotion classification model for calibration to generate emotion recognition results.

[0088] Specifically, the initial emotion recognition results are fed into the personalized emotion classification model. A forward propagation algorithm calculates the calibrated emotion category probabilities, and the emotion category with the highest probability is selected as the emotion recognition result. Combining the universal model with personalized parameters allows for precise calibration of emotion recognition results, improving the accuracy of discerning the target child's emotional state. For example, if a child deliberately conceals their expression due to social anxiety, the universal model may misclassify them as "calm." However, the personalized model, based on historical data on the child's physiological patterns of anxiety, calibrates them to "nervous."

[0089] In a possible embodiment, the emotion regulation execution module 15 includes:

[0090] The context matching unit is used to dynamically adjust the weight distribution strategy of the emotion recognition results according to the current environment label and generate a context adaptation adjustment strategy, where the current environment label is generated based on the sound and light intensity data collected by the environmental sensor.

[0091] Specifically, the system receives sound and light intensity data collected by environmental sensors, obtains the ambient brightness level through a light sensor, measures the ambient noise intensity through a decibel meter, and generates a current environmental label (such as "classroom activity," "nap," or "outdoor game") based on a preset brightness-noise mapping table. Based on the environmental label, the corresponding weight allocation strategy is invoked. For example, in the "nap" scenario, the weight of voice features is reduced to avoid misjudgment, while in the "outdoor game" scenario, the weight of body movement features is increased, thus generating a contextual adaptation adjustment strategy that adapts to the environmental state.

[0092] The individual mapping subunit is used to match the physiological baseline and behavioral preferences of the target child from the preset child individual feature database to generate individualized adjustment parameters.

[0093] Specifically, the target child's physiological baseline data (such as resting heart rate and skin conductance baseline) and behavioral preference records (such as sensitivity to tactile feedback and preferred soothing methods) are retrieved from a preset database of individual child characteristics. A rule engine is then used to match the current emotion recognition results with individual characteristics to generate personalized adjustment parameters, including adjustment intensity and feedback method priority. For example, vibration feedback is prioritized for children who are sensitive to touch, while animation interaction is triggered for children who respond well to visual guidance. By combining individual physiological and behavioral characteristics, precise customization of adjustment strategies is achieved, improving children's acceptance and intervention effectiveness.

[0094] The instruction issuing subunit is used to generate feedback instructions based on the situational adaptation adjustment strategy and individual adjustment parameters.

[0095] Specifically, by integrating contextually adaptive adjustment strategies with individualized adjustment parameters, multimodal feedback instructions are generated. For example, in a "high-anxiety classroom" scenario, control instructions are generated that include progressive vibration intensity, AR animation guidance paths, and teacher-side prompts. These instructions are distributed to smart bracelets, projection devices, and teacher terminals via the Internet of Things protocol, ensuring that the adjustment strategy is executed collaboratively across multiple devices.

[0096] In summary, the multimodal biometric-based intelligent emotion regulation system 10 for preschool children provided in the embodiment of the present application synchronously collects facial images, voice signals and skin conductance signals through a multimodal data acquisition module, and generates a time-aligned multimodal data set through preprocessing such as key point normalization, acoustic feature extraction and physiological signal denoising; then, the hierarchical feature extraction module extracts hierarchical feature vectors with emotion representation from the spatial dimension (facial muscle movement), temporal dimension (voice context association) and frequency domain dimension (skin conductance response event), respectively, to capture the hierarchical association of children's emotions at the physiological, behavioral and semantic levels; compensates for the multimodal data acquisition delay through temporal alignment, and adaptively adjusts the contribution ratio of facial, voice and physiological features in combination with a dynamic weight fusion strategy to generate a joint emotion representation vector that can represent the dynamic evolution law of emotions; fuses multimodal decision results based on evidence theory, and fine-tunes parameters in combination with individual historical data through a meta-learning adapter to achieve dual optimization of general models and personalized calibration; and dynamically generates multimodal adjustment instructions based on environmental conditions and individual characteristics, and realizes the precise execution of emotional intervention strategies through the coordinated response of tactile, visual and auditory devices.

[0097] It should be understood that although the steps in the flowcharts of the various embodiments described above are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Furthermore, at least a portion of the steps in the flowcharts of the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The order of execution of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or at least a portion of steps or stages in other steps.

[0098] Based on the same inventive concept, the embodiments of the present application also provide a method for regulating emotional intelligence for preschool children based on multimodal biometrics, which is used to implement the aforementioned system 10 for regulating emotional intelligence for preschool children based on multimodal biometrics. The implementation solution provided by this method is similar to the implementation solution described in the aforementioned system. Therefore, the specific limitations of one or more embodiments of the method for regulating emotional intelligence for preschool children based on multimodal biometrics provided below can be found in the above-mentioned limitations of the system 10 for regulating emotional intelligence for preschool children based on multimodal biometrics, and will not be repeated here.

[0099] In an exemplary embodiment, Figure 3 As shown, a method for regulating emotional intelligence of preschool children based on multimodal biometrics is provided, comprising:

[0100] Step 101: Acquire multimodal biological data of a target child and perform data preprocessing on the multimodal biological data to obtain a multimodal dataset. The multimodal dataset includes facial image data, speech signals, and skin conductance signals.

[0101] Step 102: Perform hierarchical feature extraction on the multimodal dataset to generate a hierarchical vector set, which includes facial spatial feature vectors, speech temporal feature vectors, and physiological frequency domain feature vectors.

[0102] Step 103: perform temporal alignment and dynamic weight fusion on the layered vector set to generate a joint emotion representation vector.

[0103] Step 104 : performing emotion classification processing based on the joint emotion representation vector to generate an emotion recognition result.

[0104] Step 105 : generating a feedback instruction according to the emotion recognition result, wherein the feedback instruction is used to instruct the intelligent adjustment device to execute an adjustment strategy that matches the emotion recognition result.

[0105] In a possible embodiment, multimodal biological data of a target child is obtained, and data preprocessing is performed on the multimodal biological data to obtain a multimodal dataset, including:

[0106] Perform key point normalization on facial image data to generate standardized facial data.

[0107] The speech signal is processed by extracting Mel-frequency cepstral coefficients and filtering silent segments to generate speech feature data.

[0108] The skin conductance signal is subjected to wavelet denoising and skin conductance response event detection processing to generate physiological characteristic data.

[0109] Based on the hardware-level precise time protocol, the timestamps of standardized facial data, voice feature data and physiological feature data are synchronized and aligned to generate a multimodal dataset.

[0110] In a possible embodiment, performing hierarchical feature extraction on a multimodal dataset to generate a hierarchical vector set includes:

[0111] The standardized facial data is spatially convolved to extract local muscle movement features, and the dynamic change features between frames are extracted through a three-dimensional convolutional network to generate facial spatial feature vectors.

[0112] One-dimensional convolution processing is performed on the speech feature data to extract short-term acoustic features, and context-related features are extracted through a bidirectional long short-term memory network to generate a speech time series feature vector.

[0113] The physiological characteristic data are processed by wavelet transform to decompose the frequency domain energy distribution characteristics, and the physiological response intensity is extracted through the skin conductance response event detection algorithm to generate the physiological frequency domain feature vector.

[0114] In a possible embodiment, performing emotion classification processing based on the joint emotion representation vector to generate an emotion recognition result includes:

[0115] The joint emotion representation vector is input into a multi-layer perceptron for classification processing, and the probability distribution and confidence score of seven basic emotions are output.

[0116] Based on evidence theory, the probability distribution of seven basic emotions is processed for conflict resolution to generate initial emotion recognition results.

[0117] The classification parameters of the multi-layer perceptron are fine-tuned online based on the historical emotion data of the target children through a meta-learning adapter to generate an individualized emotion classification model.

[0118] The initial emotion recognition results are input into the individualized emotion classification model for calibration to generate emotion recognition results.

[0119] In a possible embodiment, generating a feedback instruction according to the emotion recognition result includes:

[0120] The weight distribution strategy of emotion recognition results is dynamically adjusted according to the current environmental label to generate a situational adaptation adjustment strategy, where the current environmental label is generated based on the sound and light intensity data collected by the environmental sensor.

[0121] The physiological baseline and behavioral preferences of the target child are matched from the preset child individual characteristic database to generate individualized adjustment parameters.

[0122] Generate feedback instructions based on situational adaptation regulation strategies and individualized regulation parameters.

[0123] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for regulating emotional intelligence of preschool children based on multimodal biometrics as described above are implemented.

[0124] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0125] Since the device embodiments generally correspond to the method embodiments, reference will be made to the description of the method embodiments for relevant details. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in a single location or distributed across multiple network elements. Some or all of these modules may be selected based on actual needs to achieve the objectives of the disclosed solutions. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0126] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A preschool children's emotional intelligence regulation system based on multimodal biometrics, characterized by: The system includes a multimodal data acquisition module, a hierarchical feature extraction module, a cross-modal fusion module, an emotion recognition module, and an emotion regulation execution module: The multimodal data acquisition module is used to acquire multimodal biological data of the target child and perform data preprocessing on the multimodal biological data to obtain a multimodal dataset, wherein the multimodal dataset includes facial image data, voice signals, and skin conductance signals; The hierarchical feature extraction module is used to perform hierarchical feature extraction on the multimodal data set to generate a hierarchical vector set, wherein the hierarchical vector set includes a facial spatial feature vector, a speech temporal feature vector, and a physiological frequency domain feature vector; The cross-modal fusion module is used to perform temporal alignment and dynamic weight fusion on the layered vector set to generate a joint emotion representation vector; The emotion recognition module is used to perform emotion classification processing based on the joint emotion representation vector to generate an emotion recognition result; The emotion regulation execution module is used to generate a feedback instruction according to the emotion recognition result, and the feedback instruction is used to instruct the intelligent regulation device to execute a regulation strategy that matches the emotion recognition result.

2. The system according to claim 1, wherein: The multimodal data acquisition module includes: A facial data processing unit, configured to perform key point normalization processing on the facial image data to generate standardized facial data; A voice data processing unit, configured to extract Mel-frequency cepstral coefficients and perform silent segment filtering on the voice signal to generate voice feature data; a physiological data processing unit, configured to perform wavelet denoising and skin conductance response event detection processing on the skin conductance signal to generate physiological characteristic data; A data synchronization unit is used to synchronize and align the timestamps of the standardized facial data, the voice feature data, and the physiological feature data based on a hardware-level precise time protocol to generate the multimodal data set.

3. The system according to claim 2, characterized in that The hierarchical feature extraction module includes: A facial feature extraction unit is used to perform spatial convolution processing on the standardized facial data to extract local muscle movement features, and extract inter-frame dynamic change features through a three-dimensional convolutional network to generate the facial spatial feature vector; a speech feature extraction unit, configured to perform one-dimensional convolution processing on the speech feature data to extract short-term acoustic features, and extract context-related features through a bidirectional long short-term memory network to generate the speech time series feature vector; The physiological feature extraction unit is used to perform wavelet transform processing on the physiological feature data, decompose the frequency domain energy distribution characteristics, and extract the physiological response intensity through the skin conductance response event detection algorithm to generate the physiological frequency domain feature vector.

4. The system according to claim 1, wherein: The emotion recognition module includes: A classification processing unit, configured to input the joint emotion representation vector into a multi-layer perceptron for classification processing, and output a probability distribution and confidence score of seven basic emotions; An uncertainty fusion unit, configured to perform conflict resolution processing on the probability distribution of the seven basic emotions based on evidence theory to generate an initial emotion recognition result; a personalized adaptation unit, configured to perform online fine-tuning of the classification parameters of the multilayer perceptron according to the historical emotion data of the target child through a meta-learning adapter to generate a personalized emotion classification model; The personalized emotion recognition unit is used to input the initial emotion recognition result into the personalized emotion classification model for calibration to generate the emotion recognition result.

5. The system according to claim 1, wherein: The emotion regulation execution module includes: A context matching subunit, configured to dynamically adjust the weight distribution strategy of the emotion recognition result according to the current context label to generate a context adaptation adjustment strategy, wherein the current context label is generated based on the sound and light intensity data collected by the environmental sensor; An individual mapping subunit, configured to match the physiological baseline and behavioral preferences of the target child from a preset child individual feature database to generate individualized adjustment parameters; The instruction issuing subunit is used to generate the feedback instruction according to the situation adaptation adjustment strategy and the individual adjustment parameters.

6. A method for regulating emotional intelligence in preschool children based on multimodal biometrics, characterized in that: The method comprises: Acquiring multimodal biological data of a target child and performing data preprocessing on the multimodal biological data to obtain a multimodal dataset, wherein the multimodal dataset includes facial image data, speech signals, and skin conductance signals; Performing hierarchical feature extraction on the multimodal dataset to generate a hierarchical vector set, wherein the hierarchical vector set includes a facial spatial feature vector, a speech temporal feature vector, and a physiological frequency domain feature vector; Performing temporal alignment and dynamic weight fusion on the layered vector set to generate a joint emotion representation vector; Performing emotion classification processing based on the joint emotion representation vector to generate an emotion recognition result; A feedback instruction is generated according to the emotion recognition result, and the feedback instruction is used to instruct the intelligent adjustment device to execute an adjustment strategy that matches the emotion recognition result.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to claim 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to claim 6 is implemented.

Images