Multi-mode fusion AI accompanying intelligent system applied to single-chip microcomputer

Abstract

The invention relates to the technical field of artificial intelligence and embedding, and discloses a multi-modal fusion AI accompanying intelligent system applied to a single-chip microcomputer, and the system comprises a data collection unit which is used for carrying out the synchronous collection of an asynchronous original signal according to a DMA channel and a hardware timer; the parallel quantization unit is used for carrying out parallel processing on the multi-modal data flow by adopting a three-stage pipeline architecture; the self-adaptive fusion unit is used for adopting a self-adaptive fusion method for the real-time quantitative feature flow; the emotional state recognition unit inputs the low-delay fusion features into a lightweight neural network to recognize the current emotional state of the user; the interaction strategy generation unit is used for performing multi-thread parallel processing on the emotional state recognition result to generate a personalized interaction strategy; the output control unit adopts a multi-channel DMA controller and a priority scheduling algorithm to realize concurrent operation of output equipment for the parallelization interaction instruction; according to the invention, by adopting a hardware-level data synchronization technology, accurate acquisition and processing of multi-modal data are realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and embedded technology, and more specifically, to a multimodal fusion AI companion intelligent system applied to a microcontroller. Background Technology

[0002] With the rapid development of artificial intelligence technology and the widespread application of embedded systems, intelligent companion systems are playing an increasingly important role in fields such as medical rehabilitation, education and training, and elderly care. Traditional intelligent companion systems mainly rely on high-performance processors and cloud computing resources. Although they can provide rich functionality, they face problems such as high cost, high power consumption, and high privacy risks, making them difficult to deploy on a large scale in resource-constrained embedded environments.

[0003] In the field of rehabilitation medicine, cognitive rehabilitation training for stroke patients requires long-term, continuous, personalized guidance and emotional support. Existing rehabilitation training equipment mostly uses PCs or tablets as hardware platforms, resulting in large, expensive devices that are difficult to deploy in home rehabilitation environments. Simultaneously, changes in the patient's emotional state and rehabilitation progress require real-time monitoring and response, placing higher demands on the system's real-time performance and personalization capabilities. While microcontrollers, as core components of embedded systems, offer advantages such as low cost, low power consumption, and miniaturization, their limited computing resources and storage space restrict the deployment of complex AI algorithms.

[0004] Existing technologies have the following technical problems: excessive computational resource requirements; existing multimodal fusion technologies require a large amount of computing resources and storage space, making it impossible to operate normally under typical microcontroller hardware configurations; difficulty in balancing real-time performance and accuracy; rehabilitation training scenarios require the system to complete the entire process from multimodal data acquisition to emotional state recognition within 100 milliseconds, but existing technologies cannot meet such stringent real-time requirements while ensuring accuracy; and a lack of personalized emotional modeling capabilities; the emotional expression and cognitive responses of stroke patients differ from those of healthy individuals, and existing general-purpose emotional computing models cannot accurately identify subtle emotional changes in patients. Summary of the Invention

[0005] This invention provides a multimodal fusion AI companion intelligent system applied to a microcontroller, solving the technical challenges of real-time multimodal data processing, emotional state recognition, and personalized interaction on a resource-constrained microcontroller platform, and addressing technical issues in intelligent rehabilitation training, smart homes, and educational companionship.

[0006] This invention provides a multimodal fusion AI companion intelligent system applied to a microcontroller, comprising: The data acquisition unit is used to synchronously acquire asynchronous raw signals according to the DMA channel and hardware timer to obtain a multimodal data stream; The parallel quantization unit is used to process the multimodal data stream in parallel using a three-stage pipeline architecture to obtain a real-time quantized feature stream; An adaptive fusion unit is used to apply an adaptive fusion method to the real-time quantized feature stream to obtain low-latency fused features. The emotion state recognition unit is used to input low-latency fused features into a lightweight neural network to identify the user's current emotion state and obtain the emotion state recognition result. The interaction strategy generation unit is used to generate personalized interaction strategies by using multi-threaded parallel processing of the emotion state recognition results, and obtain parallelized interaction instructions. The output control unit is used to implement concurrent operation of the output device by using a multi-channel DMA controller and priority scheduling algorithm to obtain multimodal synchronous response results for parallel interactive instructions.

[0007] In a preferred embodiment, the step of acquiring the multimodal data stream includes: The system master clock generates sensor synchronization clocks through a frequency divider, and the sensor synchronization clocks generate corresponding sampling frequencies for different sensors. Each sensor is allocated an independent circular buffer. The buffer size is calculated based on the data rate and processing latency requirements. The circular buffer includes a voice data buffer, an image data buffer, a tactile data buffer, and a heart rate data buffer. The voice data buffer is 512 bytes, the image data buffer is 8KB, and the tactile data buffer and heart rate data buffer are each 256 bytes. When any sensor data reaches the first preset threshold, a hardware interrupt is triggered, and the data synchronization processing flow is started. The nearest neighbor interpolation method is used to map data with different sampling rates to a unified time base.

[0008] In a preferred embodiment, the three-stage pipeline architecture includes: The first-stage pipeline acquires multimodal data streams. For the speech data in the multimodal data stream, an improved MFCC feature extraction method is used. The 16kHz audio signal is pre-emphasized, framed, and windowed before undergoing FFT transformation to extract 13-dimensional MFCC coefficients. For the image data in the multimodal data stream, a depthwise separable convolutional network is used to extract visual features, outputting a 64-dimensional feature vector. For the tactile data in the multimodal data stream, time-domain and frequency-domain statistical features are calculated, extracting 8-dimensional features. For the heart rate data in the multimodal data stream, heart rate variability-related features are extracted, obtaining 6-dimensional features. The second-stage pipeline uses a symmetric quantization method to convert floating-point features into 8-bit fixed-point numbers, and achieves feature quantization by calculating quantization scale parameters and rounding functions; The third-stage pipeline stores the quantized features in a dedicated feature buffer, maintaining the feature timestamps and modal identifiers.

[0009] In a preferred embodiment, the step of obtaining the low-latency fusion feature includes: To obtain the real-time quantized feature stream, the query matrix, key matrix, and value matrix are designed with dimensions of 32, 32, and 64, respectively. The input features in the real-time quantized feature stream are mapped to the non-negative space through a feature mapping function, reducing the computational complexity of attention from quadratic to linear, thus obtaining linear attention features. Obtain linear attention features, set the window size to 16 time steps, automatically remove the oldest features when new linear attention features enter, perform time-series management on the linear attention features within the window, and obtain a windowed feature sequence. Obtain a windowed feature sequence, evaluate the data quality based on three dimensions: signal-to-noise ratio, completeness, and consistency of different modal data in the windowed feature sequence, calculate the modality fusion weights using the softmax function, and generate a unified multimodal feature representation by weighted summation of different modal features to obtain low-latency fusion features.

[0010] In a preferred embodiment, the step of obtaining the emotion state recognition result includes: Low-latency fusion features are obtained to construct a lightweight emotion classification network. The lightweight emotion classification network contains two fully connected layers. The first layer has 32 neurons, and the second layer has 8 neurons corresponding to 8 basic emotion states. The ReLU activation function and softmax output function are used to process the low-latency fusion features, calculate the probability distribution of each emotion state, and obtain the current emotion recognition result. To obtain low-latency fusion features, a simplified LSTM network with 16 hidden units is used for sentiment state prediction. The information flow is controlled by forget gate, input gate, and output gate to predict the sentiment state at the next moment. The current emotion recognition result is weighted and fused with the predicted emotion state. The fusion weight is dynamically adjusted according to the historical prediction accuracy to obtain the emotion state recognition result.

[0011] In a preferred embodiment, the step of obtaining the parallelized interaction instructions includes: The system obtains the emotional state recognition results and makes decisions based on the multi-dimensional information in the emotional state recognition results. The multi-dimensional information includes the user's emotional state, rehabilitation progress and historical preferences. A decision tree is built with a three-layer structure, which corresponds to the emotional state classification, rehabilitation task type classification and user personalized feature classification, to obtain the strategy selection results. Obtain the strategy selection result, and allocate speech synthesis, display content generation, and interaction logic processing to different execution threads for parallel processing based on the strategy selection result, to obtain multi-threaded processing instructions; The system obtains multi-threaded program instructions, defines five event types, schedules these instructions using a priority queue, establishes a preloading mechanism to predict and prepare response content in advance based on emotional states, and obtains parallelized interaction instructions.

[0012] In a preferred embodiment, the step of obtaining the multimodal synchronization response result includes: Obtain parallelized interaction instructions, allocate dedicated DMA channels for different types of output devices, configure each channel in cyclic mode to support continuous data transmission, process the parallelized interaction instructions for device allocation, and obtain device allocation instructions. Obtain device allocation instructions, define four priority levels: urgent, high, medium, and low, determine the processing order based on the urgency of the task, the impact on user experience, and resource consumption in the device allocation instructions, and obtain priority scheduling instructions; The priority scheduling instruction is obtained, and a synchronization signal is generated using a hardware timer. All output channels are time-aligned with the priority scheduling instruction based on the synchronization signal, and the synchronization accuracy is controlled within 5 milliseconds to obtain the multimodal synchronization response result.

[0013] In a preferred embodiment, it further includes; The resource allocation and power management unit is used to monitor the execution process of multimodal synchronous response results to obtain system load status, and to perform intelligent pre-allocation and predictive power optimization management of computing resources to obtain optimized system resource configuration and power status.

[0014] In a preferred embodiment, the step of obtaining the low-latency fusion feature further includes: Windowed feature sequences are obtained, and specialized emotion modeling is performed on windowed feature sequences for stroke patients. The impact of impaired cognitive function on emotional expression is considered, and a patient emotion expression mapping table is established to map subtle facial expression changes in the windowed feature sequences to the corresponding emotional states, thereby obtaining specialized emotion features. Specialized emotional features are obtained, and the speech features within these specialized emotional features are analyzed by focusing on intonation changes to obtain optimized speech features. Optimized speech features are obtained, and an incremental learning strategy is used to process the optimized speech features. Only the most recent 100 training samples are retained for model updates, adapting to the storage limitations of the microcontroller, and low-latency fusion features are obtained.

[0015] In a preferred embodiment, a computer-readable storage medium is provided for storing computer-readable instructions that, when read by a computer, enable the operation of a multimodal fusion AI companion intelligent system applied to a microcontroller as described above.

[0016] The beneficial effects of this invention are as follows: by adopting hardware-level data synchronization technology, accurate acquisition and processing of multimodal data are achieved; by applying pipelined feature extraction technology, feature extraction efficiency is improved; by introducing a fast linear attention mechanism, computational complexity is reduced; by designing a three-stage pipeline architecture and parallel quantization processing technology, the problem of limited microcontroller storage capacity is effectively solved; through the comprehensive application of the above technologies, ultra-low latency multimodal fusion processing on a microcontroller platform is successfully realized, enabling the microcontroller system to have real-time multimodal AI processing capabilities for the first time, laying a solid technical foundation for the widespread application of intelligent companion systems in resource-constrained environments.

[0017] By employing intelligent resource pre-allocation technology, efficient utilization of system resources is achieved; by applying predictive power management technology, overall system power consumption is reduced; by introducing a Markov chain prediction model, accurate prediction of user behavior patterns is achieved; by designing dynamic voltage and frequency adjustment technology, system operating efficiency is optimized; and by developing a personalized learning mechanism, the system can continuously optimize the user experience locally. Through the organic combination of the above technologies, the system's energy efficiency ratio and practicality are improved, while effectively protecting user privacy and reducing operating costs, providing a feasible technical path for the industrial application of intelligent companion systems. Attached Figure Description

[0018] Figure 1 This is a block diagram of a multimodal fusion AI companion intelligent system applied to a single-chip microcomputer in this invention; Figure 2 This is a bar chart comparing the multimodal processing delays of traditional schemes, compression schemes, and the method of this invention; Figure 3 It is a line graph comparing power consumption changes between fixed power consumption mode and intelligent power management; Figure 4 It is a radar chart for multi-dimensional evaluation of user satisfaction. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0020] At least one embodiment of the present invention discloses a multimodal fusion AI companion intelligent system applied to a microcontroller, such as... Figure 1 As shown, it includes: The data acquisition unit is used to synchronously acquire asynchronous raw signals according to the DMA channel and hardware timer to obtain a multimodal data stream; Based on multiple DMA (Direct Memory Access) channels and hardware timers of a microcontroller, asynchronous raw signals from a camera, microphone, tactile sensor, and heart rate sensor are synchronously acquired and processed.

[0021] A hardware synchronization clock reference is configured, dividing the system master clock to generate a sampling clock signal. This signal is then output via GPIO (General Purpose Input Output) to provide a unified sampling trigger signal for all sensors. Specifically, the 480MHz master clock is divided by 16 to obtain a 30MHz sensor synchronization clock. A programmable frequency divider is then used to generate corresponding sampling frequencies for different sensors: 16kHz for the voice sensor, 30fps for the image sensor, 1kHz for the haptic sensor, and 100Hz for the heart rate sensor. The system master clock generates the sensor synchronization clock via the frequency divider, and the sensor synchronization clock generates corresponding sampling frequencies for different sensors. The clock division factor is calculated by dividing the master clock frequency by the target sampling frequency, ensuring accurate sampling timing for each sensor.

[0022] A dedicated DMA buffer management strategy is designed, allocating an independent circular buffer to each sensor. The buffer size is calculated based on the product of the data transfer rate, target latency, and security factor. The voice data buffer is set to 512 bytes, capable of storing 32 milliseconds of audio data; the image data buffer is set to 8KB, capable of storing two frames of 64×64 pixel grayscale images; and tactile and heart rate data are each allocated a 256-byte buffer.

[0023] A hardware interrupt-based data triggering mechanism is implemented. An interrupt is triggered when any sensor data reaches a first preset threshold, initiating the data synchronization processing flow. The first preset threshold is set based on the data buffer size of each sensor, specifically: an interrupt is triggered when the voice data buffer reaches 256 bytes (approximately 16ms of audio data), the image data buffer reaches 4KB (approximately one frame of image data), the haptic data buffer reaches 128 bytes, and the heart rate data buffer reaches 64 bytes. The interrupt service routine checks the status of all buffers and combines data with similar timestamps into a synchronization data packet. The timestamp alignment algorithm uses nearest neighbor interpolation to map data with different sampling rates to a unified time base.

[0024] Data quality is assessed and anomaly handling is performed by calculating metrics such as signal-to-noise ratio (SNR) and dynamic range. SNR is calculated by taking the logarithm of the ratio of signal power to noise power ten times; a higher SNR indicates better signal quality. For outlier data, historical data smoothing filtering is used for repair, ensuring the stability of subsequent processing.

[0025] This step outputs a hardware-synchronized multimodal data stream, including aligned timestamps and quality identifiers, providing a reliable data foundation for subsequent feature extraction.

[0026] Furthermore, an adaptive sampling rate adjustment mechanism can be employed to dynamically adjust the sensor sampling frequency based on the activity level of the current scene. This mechanism calculates a user activity intensity index, quantifying the activity level of the current scene by analyzing the variance and energy changes of multimodal signals within a continuous time window. When the activity index falls below a second preset threshold (set to 60% of the historical average activity), the system automatically reduces the sensor sampling frequency, lowering the voice sampling rate from 16kHz to 8kHz and the image sampling rate from 30fps to 15fps to save system power. When increased user activity is detected, and the activity index exceeds the second preset threshold, the system immediately resumes a high sampling rate to ensure data integrity. The purpose of this adaptive mechanism is to maximize the system's energy efficiency while ensuring data quality, continuously optimizing the sampling strategy parameters through online learning algorithms.

[0027] The parallel quantization unit is used to process the multimodal data stream in parallel using a three-stage pipeline architecture to obtain a real-time quantized feature stream; A three-stage pipeline architecture is adopted to perform parallel execution of feature extraction, quantization, and feature caching on hardware-synchronized multimodal data streams.

[0028] The first-stage pipeline performs feature extraction, employing lightweight feature extraction algorithms designed specifically for different modalities. For speech data, an improved MFCC (Mel-scale Frequency Cephalic Coefficients) feature extraction method is used. The 16kHz audio signal is pre-emphasized to enhance high-frequency components. Frame segmentation is used to divide the continuous signal into 20-millisecond short frames. A Hamming window is applied to each frame to reduce spectral leakage. A Fast Fourier Transform (FFT) is performed to obtain the spectrum. A Mel filter bank is used to simulate human hearing characteristics, ultimately extracting 13-dimensional MFCC coefficients. The computational complexity is reduced by optimizing the FFT algorithm and using a lookup table method instead of trigonometric function calculations.

[0029] For image data, a depthwise separable convolutional network is used to extract visual features. This network processes the input feature map by first performing depthwise convolution operations followed by pointwise convolution operations to generate the output feature map. The network structure consists of three depthwise convolutional layers and two pointwise convolutional layers, each followed by a ReLU activation function, ultimately outputting a 64-dimensional feature vector.

[0030] For tactile data, calculate time-domain and frequency-domain statistical characteristics, including eight-dimensional features such as mean, variance, peak value, and dominant frequency.

[0031] For heart rate data, extract heart rate variability-related features, including six-dimensional features such as mean heart rate, RMSSD, and pNN50.

[0032] The second-stage pipeline performs quantization, converting floating-point features into 8-bit fixed-point numbers to reduce storage requirements and computational complexity. The quantization process uses a symmetric quantization method, statistically analyzing the dynamic range of feature values. The quantization scale parameter is calculated by dividing the larger of the absolute values ​​of the feature's minimum and maximum values ​​by 127.

[0033] Before calculating the quantization parameters, the calibration dataset needs to be preprocessed to remove outliers (data points exceeding 3 times the standard deviation), and the data is normalized to ensure numerical stability. The maximum and minimum values ​​of the statistically processed data are used for quantization parameter calculation. The floating-point feature values ​​are converted to 8-bit integers, and the quantized 8-bit integer value is obtained by dividing the original floating-point feature value by the quantization scale parameter and rounding it.

[0034] The third-stage pipeline performs feature caching operations, storing the quantized features in a dedicated feature buffer and maintaining the feature timestamps and modality identifiers. A circular buffer structure is used, automatically overwriting the oldest data when the buffer is full, ensuring continuous system operation.

[0035] The entire pipeline employs a double-buffering mechanism, where one buffer processes the current data while the other receives new input data, preventing data loss and processing blockage. Each stage of the pipeline is connected via a hardware FIFO (First In First Out) queue to ensure real-time and reliable data transmission.

[0036] This step outputs a real-time quantized feature stream, compressing the feature dimension from thousands of dimensions to 91 dimensions, reducing the amount of data and laying the foundation for subsequent fusion processing.

[0037] Furthermore, a dynamic feature selection mechanism can be employed to adaptively select the most relevant subset of features based on current task requirements and system load. This mechanism establishes a feature importance scoring model, quantifying the contribution of each feature by calculating the Pearson correlation coefficient between each feature and the target task, as well as the mutual information between features. The linear correlation is measured by calculating the Pearson correlation coefficient between features and the target task, while non-linear dependencies are measured by calculating mutual information.

[0038] The comprehensive feature importance score is derived by weighted fusion of correlation coefficients and mutual information, with weights set to 0.6 and 0.4 respectively to balance the influence of linear correlation and non-linear dependence. The comprehensive score is calculated by multiplying the absolute value of the correlation coefficient by a weight of 0.6 and the mutual information value by a weight of 0.4. Before calculating the comprehensive score, the correlation coefficients and mutual information values ​​need to be standardized: the absolute values ​​of the correlation coefficients and mutual information values ​​are mapped to the zero-to-one interval to ensure that the two indicators are weighted and fused under the same units, avoiding the influence of differences in numerical ranges on the scoring results. When the system load exceeds 80%, the algorithm automatically activates the feature selection mode, prioritizing the retention of the top 50% of features in terms of importance score, compressing the 91-dimensional features to 45 dimensions to reduce computational overhead. When the system load drops below 60%, the full 91-dimensional feature set is restored to ensure recognition accuracy. The purpose of this dynamic feature selection mechanism is to maximize feature expressive power under real-time constraints, balancing system performance and accuracy requirements through adaptive feature dimension adjustment.

[0039] An adaptive fusion unit is used to apply an adaptive fusion method to the real-time quantized feature stream to obtain low-latency fused features. Based on real-time quantized feature streams, a fast linear attention mechanism and a sliding window strategy are used to adaptively assign weights and perform deep fusion processing on multimodal features.

[0040] A fast linear attention mechanism is implemented, reducing the quadratic complexity of traditional attention to linear complexity. Lightweight query, key, and value matrices are designed with dimensions of 32, 32, and 64 respectively, ensuring normal operation under microcontroller memory constraints. A feature mapping function maps the input features to a non-negative space, reducing the attention computational complexity from quadratic to linear, resulting in linear attention features and avoiding the impact of negative values ​​on attention calculation. Linear attention achieves linear calculation of attention weights by applying the feature mapping function to the query and key matrices respectively and calculating the matrix multiplication of the transpose of the query matrix and the key matrix, plus the product of the value matrix. This method reduces the attention computational complexity from the square of the sequence length to the square of the feature dimension, improving computational efficiency. The feature mapping function uses an exponential linear unit activation function plus 1, outputting directly for positive values ​​and exponentially decaying for negative values, mitigating the vanishing gradient problem and maintaining the smoothness of the output for negative values.

[0041] A sliding window mechanism is employed to reduce computational load and memory consumption. The window size is set to 16 time steps, corresponding to approximately 500 milliseconds of historical data. Features within the window are organized chronologically; when a new feature enters, the oldest feature is automatically removed. The linear attention features within the window are managed temporally to obtain a windowed feature sequence, maintaining a constant window size. The sliding window attention mechanism performs linear attention calculations on the query, key, and value matrices of the previous 16 time steps.

[0042] An adaptive weight allocation algorithm is designed to evaluate data quality based on three dimensions: signal-to-noise ratio (SNR), data integrity, and consistency of different modal data in a windowed feature sequence. The quality score is calculated by assigning weight coefficients to the SNR, data integrity, and consistency with historical data for each modality, followed by a weighted summation. Data integrity is determined by calculating the ratio of the current sampled data to the expected data volume. Consistency is represented by the inverse of the KL divergence between the current and historical data distributions; a smaller divergence indicates higher consistency. The scores for each dimension need to be normalized, mapping indicators with different dimensions to a unified interval of 0 to 1. The normalization method uses maximum and minimum value standardization to ensure the comparability of quality scores. The modality fusion weights are calculated using the softmax function: multiplying each modality quality score by a temperature parameter, taking the exponent, and then dividing by the sum of all modality exponent values ​​to achieve normalized weight allocation.

[0043] A weighted feature fusion operation is performed, summing the features from different modalities according to their respective weights to generate a unified multimodal feature representation. The fusion process involves multiplying each modal feature vector by its corresponding fusion weight and summing all the weighted feature vectors to obtain the final fused feature vector.

[0044] To improve the fusion effect, a residual connection mechanism is introduced, which adds the fused features to the simple concatenation of all modal features according to the residual weights, thus preserving important modality-specific information.

[0045] This step outputs low-latency fusion features with a 64-dimensional feature dimension, which is a deep fusion representation containing multimodal information, and the computation latency is controlled within 15 milliseconds.

[0046] Furthermore, a hierarchical attention mechanism can be employed, applying attention computation at both the feature level and the decision level to achieve more refined multimodal fusion. This mechanism performs intramodal attention computation at the feature level, weighting the importance of features within each modality to highlight the feature components most discriminative to the current emotional state. Subsequently, intermodal attention computation is performed at the decision level, adaptively selecting the most advantageous modality combination strategy based on historical fusion results and the characteristics of the current task. The purpose of the hierarchical attention mechanism is to better capture the complex relationships and complementary information between multimodal data through two-layer attention modeling, thereby improving the discriminative power and robustness of the fused features.

[0047] The emotion state recognition unit is used to input low-latency fused features into a lightweight neural network to identify the user's current emotion state and obtain the emotion state recognition result. Based on low-latency fusion features and combined with historical sentiment patterns, a lightweight neural network and prediction algorithm are used to achieve accurate identification and prediction of users' sentiment states.

[0048] A lightweight emotion classification network is constructed, consisting of two fully connected layers. The first layer contains 32 neurons, and the second layer contains 8 neurons corresponding to 8 basic emotion states (happiness, sadness, anger, fear, surprise, disgust, calmness, and confusion). The ReLU activation function is chosen for its computational efficiency. The output layer uses a softmax function to calculate the probability distribution of each emotion state, yielding the current emotion recognition result and the policy selection result. The probability calculation process is as follows: the fused features are input into the first fully connected layer with a bias, processed by the ReLU activation function, and then input into the second layer. The output of the second layer, after adding the bias, is used to calculate the exponent for each emotion category. Finally, the result is divided by the sum of the exponent values ​​of all emotion categories to obtain the normalized probability distribution.

[0049] A mood state prediction mechanism is implemented to predict the mood state of the next moment based on the user's historical mood change patterns. A simplified LSTM (Long Short-Term Memory) network with 16 hidden units is used. The forget gate controls the retention of historical information, the input gate determines the storage of new information, and the output gate controls the transmission of cell state to hidden state. The forget gate calculates the retention ratio using the sigmoid activation function, the input gate also uses the sigmoid function to determine the update weights, new candidate values ​​are generated using the tanh function, the cell state is obtained by multiplying the forget gate output by the cell state at the previous moment, plus the input gate output by the new candidate value, and finally the hidden state is calculated by multiplying the output of the output gate by the current cell state after tanh activation.

[0050] A probabilistic fusion strategy is designed to weight and fuse the current sentiment recognition result with the prediction result, thereby improving the accuracy and stability of the recognition. The fusion weights are dynamically adjusted based on historical prediction accuracy. The current recognition weight and the prediction weight are calculated separately using normalized accuracy ratios, ensuring that the sum of the two weights is 1. The final sentiment state probability distribution is calculated using a weighted average method, which involves multiplying the currently recognized sentiment probability by the current recognition weight, and adding the predicted sentiment probability multiplied by the prediction weight to obtain the final fused sentiment state probability distribution.

[0051] This study aims to achieve specialized emotion modeling for windowed feature sequences in stroke patients, considering the impact of cognitive impairment on emotional expression. An emotion expression mapping table is established to map subtle facial expression changes in the windowed feature sequences to corresponding emotional states, resulting in specialized emotion features. For the speech features within the specialized emotion features, the focus is on intonation variations rather than linguistic content, as some patients experience language expression difficulties.

[0052] An online learning algorithm is used to continuously optimize model performance, and gradient descent is used to update network parameters. The parameter update process is as follows: the model parameters at the current time step are subtracted from the product of the learning rate and the gradient of the loss function to obtain the updated parameters for the next time step. To adapt to the storage limitations of the microcontroller, an incremental learning strategy is used to process the optimized speech features, retaining only the most recent 100 training samples for model updates.

[0053] This step outputs the emotion state recognition results, including the probability distribution and confidence score of 8 emotion states. The recognition accuracy is high and the response latency is controlled within 20 milliseconds.

[0054] Furthermore, a multi-scale temporal modeling mechanism can be adopted to consider both short-term emotional changes and long-term emotional trends, providing a more comprehensive assessment of emotional state. Specifically, multiple LSTM networks with different time windows are used to model immediate emotional responses (5-second time window), short-term emotional states (30-second time window), and long-term emotional trends (300-second time window), respectively, and emotional information from different time scales is adaptively fused through an attention mechanism.

[0055] Attention weights are calculated using the softmax function to ensure that the sum of the weights across the three time scales is 1. Attention energy is calculated using the hyperbolic tangent activation function to determine the correlation between the hidden state and the context vector. The final multi-scale sentiment representation is obtained by weighted summation, where the attention weights at each of the three time scales are multiplied by their respective LSTM hidden states and then summed to obtain the multi-scale fused sentiment representation.

[0056] This mechanism can better understand the dynamic changes in users' emotions, capturing immediate emotional fluctuations through a short-term window, identifying emotional state transitions through a medium-term window, and analyzing emotional trend patterns through a long-term window, thus providing a richer information foundation for personalized interaction strategies.

[0057] The interaction strategy generation unit is used to generate personalized interaction strategies by using multi-threaded parallel processing of the emotion state recognition results, and obtain parallelized interaction instructions. Based on the enhanced emotion state recognition results, combined with the user's personalized profile and the current task context, a multi-threaded parallel processing method is used to generate personalized interaction strategies and response content.

[0058] A strategy selection framework based on a decision tree is established to make decisions based on multi-dimensional information such as user emotional state, rehabilitation progress, and historical preferences. The decision tree has a three-layer structure: the first layer classifies based on emotional state (positive, negative, neutral); the second layer classifies based on rehabilitation task type (cognitive training, motor rehabilitation, emotional regulation); and the third layer classifies based on user personalized characteristics (extroverted, introverted, mixed), thus obtaining the strategy selection result. Each leaf node corresponds to a set of predefined interaction strategy templates.

[0059] A multi-threaded strategy generation mechanism is designed to obtain strategy selection results. Based on these results, speech synthesis, display content generation, and interaction logic processing are distributed to different execution threads for parallel processing, resulting in multi-threaded processing instructions. The speech synthesis thread uses template concatenation technology to select appropriate words and phrases from a pre-stored audio segment library for combination. The display content generation thread selects corresponding icons, colors, and animation effects based on emotional states. The interaction logic processing thread calculates the parameters for the next training task and the difficulty adjustment strategy.

[0060] Implement an event-driven architecture to reduce CPU polling overhead. Obtain multi-threaded program instructions and define five event types: emotion state change events, user input events, task completion events, error handling events, and timed trigger events. Each event type corresponds to a dedicated handler, and multi-threaded program instructions are scheduled using a priority queue. Event priority is calculated using a weighted scoring mechanism, comprehensively considering three dimensions: urgency, scope of impact, and resource consumption, with weighting coefficients set to 0.5, 0.3, and 0.2, respectively. The weighted scoring mechanism calculates priority by multiplying urgency by a weight of 0.5, scope of impact by a weight of 0.3, and the reciprocal of resource consumption by a weight of 0.2, then summing the results. Before calculation, the raw values ​​for the three dimensions need to be normalized, mapping indicators of different dimensions to a uniform range of zero to one.

[0061] Establish a preloading mechanism to prepare response content in advance based on emotional state predictions. Maintain a content cache pool containing commonly used audio clips, image materials, text templates, etc. When a change in the user's emotional state is predicted, preload the corresponding response content into memory to reduce processing latency during actual response.

[0062] A personalized parameter adjustment algorithm is designed to dynamically adjust strategy parameters based on the user's historical interaction data. A simplified reinforcement learning method is adopted, using a tabular Q-learning algorithm: the Q-value is updated by adding the product of the current Q-value and the learning rate and the temporal difference error, where the temporal difference error is equal to the immediate reward plus a discount factor multiplied by the maximum Q-value of the next state minus the current Q-value. In the algorithm, the current state includes emotional state, task type, and user characteristics; the action is the interaction strategy adopted; the reward value comes from user feedback; the learning rate controls the learning speed; and the discount factor determines the importance attached to future rewards.

[0063] The state space is discretized, quantifying the continuous probability distribution of emotional states into 8 discrete intervals. Task types are categorized into 3 types, and user characteristics into 3 types, totaling 72 states. The action space includes 12 strategy combinations, such as voice response intensity, displayed content type, and interaction frequency. The reward value is calculated by comprehensively considering user participation, satisfaction, and task completion, weighted by 0.4, 0.4, and 0.2 respectively for weighted averaging. Participation is quantified by interaction duration and frequency, satisfaction is assessed by facial expression recognition scores, and task completion is measured by rehabilitation training progress. Before calculating the reward value, the three indicators need to be standardized: participation is normalized by the ratio to the historical average, satisfaction is weighted and normalized by emotion recognition confidence, and task completion is normalized by the percentage of target progress, ensuring that each indicator is weighted and fused under the same dimensions.

[0064] To achieve dynamic optimization of the interaction strategy, the effectiveness of the strategy is continuously improved by analyzing user behavioral feedback. Three evaluation metrics are defined: user engagement (calculated through interaction frequency and duration), satisfaction (derived through facial expressions and voice emotion analysis), and rehabilitation effect (assessed through task completion). A weighted average method is used to calculate the comprehensive score: user engagement, satisfaction, and rehabilitation progress are each multiplied by their corresponding weight coefficients, and the three weighted results are summed to obtain the final comprehensive score.

[0065] The overall score is calculated using a weighted average, with user engagement, satisfaction, and recovery effectiveness weighted at 0.4, 0.4, and 0.2, respectively. Before calculating the overall score, data preprocessing is required for the three indicators: user engagement is normalized over time (divided by the total session duration), satisfaction is weighted by confidence level, and recovery effectiveness is normalized using baseline comparison to ensure consistent numerical ranges and comparability for each indicator.

[0066] This step outputs parallel interactive commands, including voice output commands, display update commands, LED control commands, etc., with a generation latency controlled within 25 milliseconds, supporting the simultaneous processing of up to 8 concurrent interactive tasks.

[0067] Furthermore, a context-aware strategy adaptation mechanism can be employed to dynamically adjust the expression and content focus of the interaction strategy based on environmental conditions and user status. Specifically, an ambient light sensor detects current lighting conditions, enhancing voice feedback and reducing visual displays in low light; a noise detection sensor assesses ambient noise levels, increasing voice volume and using more concise expressions in noisy environments. This mechanism ensures that the interaction strategy adapts to different usage environments, providing a consistent user experience quality.

[0068] The output control unit is used to implement concurrent operation of the output device by using a multi-channel DMA controller and priority scheduling algorithm to obtain multimodal synchronous response results for parallel interactive instructions; Based on parallelized interactive instructions, a multi-channel DMA controller and priority scheduling algorithm are used to realize concurrent operation and synchronous control of multiple output devices such as voice, display, LED, and haptic feedback.

[0069] A multi-channel DMA controller is configured to allocate dedicated DMA channels for different types of output devices, processing parallel interactive commands to obtain device allocation instructions. Voice output uses DMA1 channel 1, connected to the DAC module for audio playback; display output uses DMA2 channel 2, connected to the SPI interface to drive the TFT display; LED control uses DMA1 channel 3, connected to the TIM module to generate PWM signals; haptic feedback uses DMA2 channel 4, connected to the vibration motor drive circuit. Each DMA channel is configured in loop mode, supporting continuous data transmission without CPU intervention.

[0070] Design an output priority scheduling algorithm to determine the processing order based on task urgency, user experience impact, and resource consumption. Obtain device allocation instructions and define four priority levels: Urgent (safety warnings, error messages), High (emotional feedback, task guidance), Medium (status display, progress updates), and Low (decorative animations, background sound effects). Based on the task urgency, user experience impact, and resource consumption in the device allocation instructions, determine the processing order to obtain priority scheduling instructions. Priority calculation uses a weighted scoring mechanism, multiplying urgency by 0.6, user experience impact by 0.3, and the reciprocal of resource consumption by 0.1, then summing the results to obtain the final score. Before calculation, data preprocessing is required for each parameter: urgency is normalized to a zero-to-one range using time urgency; user experience impact is mapped to a zero-to-one range using importance level; and resource consumption is directly normalized using occupancy ratio, ensuring that each dimension is weighted under the same dimensions. High-priority tasks can preempt resources from low-priority tasks, ensuring that critical information is delivered to users in a timely manner.

[0071] Adaptive buffer management is implemented, dynamically adjusting the buffer size of each output channel based on the current system load and output requirements. The base buffer size is calculated based on the data rate: 2KB for voice output (125 milliseconds of audio data), 4KB for display output (two frames of image data), 512 bytes for LED control (256 PWM cycles), and 256 bytes for haptic feedback (64 vibration modes). When the system load is high, the buffer size is appropriately increased to improve system stability. The dynamic buffer size is calculated by multiplying the base buffer size by the product of the load factor and the expansion scaling factor, and then adding 1.

[0072] An output synchronization mechanism is established to ensure time consistency across multiple outputs, particularly for synchronized playback of audio and display content. Priority scheduling instructions are acquired, and a hardware timer generates a synchronization signal. All output channels use this signal as a reference for time alignment with the priority scheduling instructions. Synchronization accuracy is controlled within 5 milliseconds to obtain multimodal synchronization response results that meet human perception requirements. For outputs requiring precise synchronization (such as audio and lip-sync animation), a pre-calculated delay compensation method is employed. The latency compensation value is calculated by summing three parts: data processing time, data transmission time, and device response latency. Data processing time is calculated using algorithm complexity and CPU frequency; data transmission time is calculated using data volume and transmission rate; and device response latency is obtained by looking up device characteristic parameters in a table. Before calculation, the time parameters need to be standardized by converting all time values ​​to milliseconds to ensure calculation accuracy.

[0073] The design incorporates adaptive output quality control, dynamically adjusting output accuracy based on system load to ensure real-time performance. Three quality levels are defined: High quality (16kHz sampling rate, 128×128 display resolution, 1kHz LED PWM frequency), Medium quality (8kHz sampling rate, 64×64 display resolution, 500Hz LED PWM frequency), and Low quality (4kHz sampling rate, 32×32 display resolution, 250Hz LED PWM frequency). The quality level automatically switches according to system load. The quality level automatically switches according to the system load: high quality mode is used when the load is below 70%, medium quality mode is used between 70% and 90%, and low quality mode is used when the load exceeds 90%.

[0074] A closed-loop feedback control mechanism is implemented, using sensors to monitor user responses to the output, evaluate the output effect, and make dynamic adjustments. For example, the microphone detects changes in the volume and tone of the user's voice response to determine whether the voice output is appropriate; the camera detects the user's gaze direction to determine whether the displayed content is attention-grabbing. Output parameters are automatically adjusted based on feedback information. The adjusted output parameters are calculated by adding the feedback gain coefficient to the current output parameters and multiplying it by the difference between the expected and actual responses.

[0075] This step outputs a multimodal synchronous response result, including coordinated voice, visual, and tactile feedback signals, with output latency controlled within 30 milliseconds, and supports up to 4 concurrent output channels working simultaneously.

[0076] Furthermore, an intelligent output content optimization mechanism can be employed to automatically select the optimal combination of output modes based on the user's sensory preferences and current state. Specifically, by analyzing the user's response intensity and preference patterns to different output modes, a personalized output strategy model can be established. For example, for visually sensitive users, the clarity and color contrast of the displayed content can be enhanced; for auditory sensitive users, the tone and rhythm of the speech can be optimized. This mechanism continuously learns the user's feedback patterns and gradually optimizes the output strategy to provide the most suitable interactive experience for the user's characteristics.

[0077] In one embodiment of the present invention, in order to perform intelligent pre-allocation and predictive power consumption optimization management of system computing resources, a multimodal fusion AI companion intelligent system applied to a microcontroller further includes: The resource allocation and power management unit is used to monitor the execution process of multimodal synchronous response results to obtain system load status, and to perform intelligent pre-allocation and predictive power optimization management of computing resources to obtain optimized system resource configuration and power status. Based on system load and user behavior patterns, a Markov chain prediction model and dynamic voltage and frequency regulation technology are used to achieve intelligent pre-allocation of computing resources and predictive optimization management of system power consumption.

[0078] A user behavior prediction model is established, analyzing user activity patterns and state transition probabilities based on Markov chain theory. Six user states are defined: active interaction, passive observation, pause for thought, rest, task switching, and system waiting. By statistically analyzing historical user behavior data, a state transition matrix is ​​constructed: the state transition probability equals the number of times a user transitions from a given state to the target state divided by the total number of times that state occurs.

[0079] Based on the current state, the probability distribution of the next state is predicted to obtain the user behavior prediction result: multiply the current state probability vector with the state transition matrix to obtain the probability distribution of each state at the next time step.

[0080] Implement a prediction-based resource pre-allocation strategy, adjusting system resource configuration in advance based on user behavior predictions to obtain the resource allocation strategy. Define four resource allocation modes: High-performance mode (CPU frequency 480MHz, memory allocation 80%, all peripherals activated), Balanced mode (CPU frequency 240MHz, memory allocation 60%, some peripherals activated), Energy-saving mode (CPU frequency 120MHz, memory allocation 40%, few peripherals activated), and Standby mode (CPU frequency 60MHz, memory allocation 20%, only critical peripherals activated). The pre-allocation algorithm selects the appropriate configuration mode based on the predicted user activity level. Resource allocation mode is automatically selected based on predicted user activity level: high performance mode is used when activity level exceeds 0.8, balanced mode is used between 0.5 and 0.8, energy saving mode is used between 0.2 and 0.5, and standby mode is used when activity level is below 0.2.

[0081] A Dynamic Voltage and Frequency Scaling (DVFS) strategy is designed to dynamically adjust resource allocation based on real-time computing demands and battery level. Processor dynamic power consumption equals the product of the switching activity factor, load capacitance, the square of the operating voltage, and the operating frequency. Reducing voltage and frequency can decrease power consumption but impacts computing performance. The DVFS control algorithm employs a PID controller to achieve precise voltage regulation. By calculating the error between target and actual performance and combining proportional, integral, and derivative control elements, it dynamically adjusts the processor voltage to achieve the optimal performance-power consumption balance.

[0082] Design a task priority scheduling algorithm to dynamically allocate processor time based on task importance, time urgency, and resource consumption. Priority is calculated by adding the following three factors: task importance score multiplied by its importance weight, the reciprocal of the deadline multiplied by its time urgency weight, and the reciprocal of resource availability multiplied by its resource consumption weight. A preemptive scheduling strategy is employed, allowing higher-priority tasks to interrupt the execution of lower-priority tasks.

[0083] Before calculation, data preprocessing is required for each parameter: importance scores are normalized to the zero-to-one range using a classification mapping; deadlines are calculated using the current time difference and logarithmically transformed to avoid excessively large values; and resource quantities are normalized using the system's total resource ratio to ensure that each parameter is within a reasonable range for priority calculation. A preemptive scheduling strategy is adopted, allowing high-priority tasks to interrupt the execution of low-priority tasks.

[0084] Establish an intelligent sleep mechanism to automatically reduce system power consumption when the user is inactive, while maintaining fast wake-up capability. The sleep depth is determined based on the predicted next activity time. The sleep depth is divided into three levels: shallow sleep is used when the predicted next activity time is less than 10 seconds, deep sleep is used between 10 and 60 seconds, and sleep mode is used when it exceeds 60 seconds.

[0085] The predicted time is calculated by analyzing the user's historical activity patterns and current environmental conditions, taking into account a weighted combination of three factors: historical average interval, circadian rhythm factor, and task urgency. Before calculation, each factor undergoes data preprocessing: historical intervals are smoothed using a time window moving average, the circadian rhythm factor is normalized using a day-night cycle, and task urgency is mapped using priority levels to ensure the accuracy and stability of the prediction results. Shallow hibernation maintains power to memory data and critical peripherals, with the CPU frequency reduced to its minimum operating frequency, resulting in power consumption of approximately 30% of normal operation. Deep hibernation shuts down most peripherals, retaining only the RTC and critical sensors, and puts the memory into self-refresh mode, reducing power consumption to 5% of normal operation. Hibernation mode saves the system state to non-volatile memory, shuts down almost all power supplies, retaining only the wake-up circuitry, reducing power consumption to the microamplitude level.

[0086] This system enables battery management and lifespan prediction by monitoring parameters such as battery voltage, current, and temperature to assess battery status and predict remaining usage time. Data preprocessing is required during parameter monitoring: voltage values ​​are sampled by an ADC and then filtered to eliminate noise; current values ​​are acquired by a Hall sensor and then temperature-compensated; and temperature values ​​are measured by a thermistor and then linearized to ensure the accuracy and reliability of the monitoring data. The battery capacity decay model uses an exponential decay function, where the remaining battery capacity is calculated by multiplying the initial battery capacity by the negative decay coefficient (base of the natural logarithm) and then by the usage time raised to the power of the result. A dynamic performance adjustment strategy based on battery status is implemented, automatically activating an energy-saving mode to extend usage time when the battery level is low.

[0087] This step outputs optimized system resource configuration and power consumption status, achieving reduced power consumption and improved performance, extending battery life from 8 hours to 12 hours.

[0088] Furthermore, a machine learning-enhanced power consumption prediction model can be employed. By analyzing long-term user usage patterns and environmental factors, a more accurate power consumption prediction algorithm can be established. This prediction model uses a multilayer perceptron network structure. The input layer receives a 12-dimensional feature vector, including user behavior characteristics, ambient temperature, humidity, light intensity, task type, and historical power consumption data. The hidden layer contains 8 neurons using the ReLU activation function. The output layer predicts the average power consumption value per hour for the next 6 hours. The forward propagation calculation process of the network is as follows: the 12-dimensional input feature vector is multiplied by the weight matrix from the input layer to the hidden layer and the hidden layer bias is added. After processing by the ReLU activation function, the hidden layer output vector is obtained. The hidden layer output is multiplied by the weight matrix from the hidden layer to the output layer and the output layer bias is added to obtain the final 6-hour power consumption prediction value. The model training uses the backpropagation algorithm, and the loss function uses mean squared error, which is calculated by dividing the sum of the squares of the differences between the actual power consumption value and the predicted power consumption value of all training samples by the number of samples.

[0089] Weight updates are performed using gradient descent with a learning rate of 0.001. The purpose of machine learning-enhanced power consumption prediction is to achieve accurate power consumption prediction several hours in advance. Based on long-term prediction results, optimized resource allocation strategies are developed, such as adjusting CPU frequency and activating peripherals before predicting high-load periods, and postponing the execution of non-urgent tasks when predicting low-load periods, thereby further optimizing overall energy management and extending device battery life.

[0090] A computer-readable storage medium for storing computer-readable instructions that, when read by a computer, enable the execution of a multimodal fusion AI companion intelligent system applied to a microcontroller, as described above.

[0091] In one embodiment of the present invention, a specific example is provided: A six-month field test was conducted in the rehabilitation department of a hospital in City A. During the test, 15 sets of multimodal fusion AI companion intelligent systems based on STM32H7 microcontrollers were deployed, covering an area of ​​approximately 200 square meters in the rehabilitation training room, serving 45 stroke rehabilitation patients. The test environment included a cognitive function training area, a motor rehabilitation area, and an emotional regulation area, with five sets of devices configured in each area. Data synchronization and centralized management were achieved through a wireless network.

[0092] Examples of voice and image data acquisition are shown in Table 1: Table 1: Examples of Voice and Image Data Acquisition Table 2 shows an example of system operation status data collection: Table 2: Example of System Operation Status Data Collection As shown in Tables 1 and 2, the system can stably acquire multimodal sensor data, the data acquisition frequency meets the design requirements, and all operating parameters are within the normal range. The MFCC coefficients extracted from speech features can effectively reflect the emotional characteristics of speech, the image feature values ​​reflect the changing trends of facial expressions, and tactile and heart rate data provide physiological support information for emotional state recognition.

[0093] like Figure 2 As shown, this invention demonstrates its advantages in handling latency. Compared to the traditional solution's 420 millisecond latency, this invention achieves an ultra-low latency of 95 milliseconds, a reduction of 77.4%. This improvement is mainly attributed to the application of hardware-level data synchronization, a pipelined processing architecture, and a fast linear attention mechanism, ensuring smooth real-time interaction during rehabilitation training.

[0094] like Figure 3 As shown, the effect of the intelligent power management strategy is demonstrated. In the traditional fixed power consumption mode, the system power consumption remains at a high level throughout the entire working cycle. However, the intelligent power management of this invention achieves a gradient decrease in power consumption through predictive resource allocation and dynamic voltage and frequency adjustment. The average power consumption is reduced by 33% over a 12-hour working cycle, effectively extending the device's battery life.

[0095] like Figure 4The results show user satisfaction assessments of the system's various functions. Among the six dimensions, usability scored the highest (9.1 points), indicating a reasonable design of the system's interface and interaction flow. Emotional understanding scored 8.2 points, validating the effectiveness of multimodal fusion technology in emotion recognition. The interaction naturalness score was relatively low (7.9 points), mainly due to room for improvement in the naturalness of speech synthesis. Overall, user satisfaction scores were all above 7.5 points, indicating that the system effectively meets the actual needs of rehabilitation training scenarios.

[0096] Through the above performance tests and user evaluations, the feasibility and effectiveness of the technical solution of this invention in realizing a multimodal fusion AI companion intelligent system on a single-chip microcomputer platform have been verified, providing important technical support for the industrial application of intelligent rehabilitation training equipment.

[0097] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A multimodal fusion AI companion intelligent system applied to a microcontroller, characterized in that, include: The data acquisition unit is used to synchronously acquire asynchronous raw signals according to the DMA channel and hardware timer to obtain a multimodal data stream; The parallel quantization unit is used to process the multimodal data stream in parallel using a three-stage pipeline architecture to obtain a real-time quantized feature stream; An adaptive fusion unit is used to apply an adaptive fusion method to the real-time quantized feature stream to obtain low-latency fused features. The emotion state recognition unit is used to input low-latency fused features into a lightweight neural network to identify the user's current emotion state and obtain the emotion state recognition result. The interaction strategy generation unit is used to generate personalized interaction strategies by using multi-threaded parallel processing of the emotion state recognition results, and obtain parallelized interaction instructions. The output control unit is used to implement concurrent operation of the output device by using a multi-channel DMA controller and priority scheduling algorithm to obtain multimodal synchronous response results for parallel interactive instructions.

2. The multimodal fusion AI companion intelligent system applied to a single-chip microcomputer as described in claim 1, characterized in that, The steps for acquiring the multimodal data stream include: The system master clock generates sensor synchronization clocks through a frequency divider, and the sensor synchronization clocks generate corresponding sampling frequencies for different sensors. Each sensor is allocated an independent circular buffer. The buffer size is calculated based on the data rate and processing latency requirements. The circular buffer includes a voice data buffer, an image data buffer, a tactile data buffer, and a heart rate data buffer. The voice data buffer is 512 bytes, the image data buffer is 8KB, and the tactile data buffer and heart rate data buffer are each 256 bytes. When any sensor data reaches the first preset threshold, a hardware interrupt is triggered, and the data synchronization processing flow is started. The nearest neighbor interpolation method is used to map data with different sampling rates to a unified time base.

3. The multimodal fusion AI companion intelligent system applied to a microcontroller according to claim 1, characterized in that, The three-stage pipeline architecture includes: The first-stage pipeline acquires multimodal data streams. For the speech data in the multimodal data stream, an improved MFCC feature extraction method is used. The 16kHz audio signal is pre-emphasized, framed, and windowed before undergoing FFT transformation to extract 13-dimensional MFCC coefficients. For the image data in the multimodal data stream, a depthwise separable convolutional network is used to extract visual features, outputting a 64-dimensional feature vector. For the tactile data in the multimodal data stream, time-domain and frequency-domain statistical features are calculated, extracting 8-dimensional features. For the heart rate data in the multimodal data stream, heart rate variability-related features are extracted, obtaining 6-dimensional features. The second-stage pipeline uses a symmetric quantization method to convert floating-point features into 8-bit fixed-point numbers, and achieves feature quantization by calculating quantization scale parameters and rounding functions; The third-stage pipeline stores the quantized features in a dedicated feature buffer, maintaining the feature timestamps and modal identifiers.

4. The multimodal fusion AI companion intelligent system applied to a microcontroller according to claim 1, characterized in that, The steps for obtaining the low-latency fusion features include: To obtain the real-time quantized feature stream, the query matrix, key matrix, and value matrix are designed with dimensions of 32, 32, and 64, respectively. The input features in the real-time quantized feature stream are mapped to the non-negative space through a feature mapping function, reducing the computational complexity of attention from quadratic to linear, thus obtaining linear attention features. Obtain linear attention features, set the window size to 16 time steps, automatically remove the oldest features when new linear attention features enter, perform time-series management on the linear attention features within the window, and obtain a windowed feature sequence. Obtain a windowed feature sequence, evaluate the data quality based on three dimensions: signal-to-noise ratio, completeness, and consistency of different modal data in the windowed feature sequence, calculate the modality fusion weights using the softmax function, and generate a unified multimodal feature representation by weighted summation of different modal features to obtain low-latency fusion features.

5. A multimodal fusion AI companion intelligent system applied to a single-chip microcomputer according to claim 1, characterized in that, The steps for obtaining the emotion state recognition result include: Low-latency fusion features are obtained to construct a lightweight emotion classification network. The lightweight emotion classification network contains two fully connected layers. The first layer has 32 neurons, and the second layer has 8 neurons corresponding to 8 basic emotion states. The ReLU activation function and softmax output function are used to process the low-latency fusion features, calculate the probability distribution of each emotion state, and obtain the current emotion recognition result. To obtain low-latency fusion features, a simplified LSTM network with 16 hidden units is used for sentiment state prediction. The information flow is controlled by forget gate, input gate, and output gate to predict the sentiment state at the next moment. The current emotion recognition result is weighted and fused with the predicted emotion state. The fusion weight is dynamically adjusted according to the historical prediction accuracy to obtain the emotion state recognition result.

6. A multimodal fusion AI companion intelligent system applied to a microcontroller according to claim 1, characterized in that, The steps for obtaining the parallelized interactive instructions include: The system obtains the emotional state recognition results and makes decisions based on the multi-dimensional information in the emotional state recognition results. The multi-dimensional information includes the user's emotional state, rehabilitation progress and historical preferences. A decision tree is built with a three-layer structure, which corresponds to the emotional state classification, rehabilitation task type classification and user personalized feature classification, to obtain the strategy selection results. Obtain the strategy selection result, and allocate speech synthesis, display content generation, and interaction logic processing to different execution threads for parallel processing based on the strategy selection result, to obtain multi-threaded processing instructions; The system obtains multi-threaded program instructions, defines five event types, schedules these instructions using a priority queue, establishes a preloading mechanism to predict and prepare response content in advance based on emotional states, and obtains parallelized interaction instructions.

7. A multimodal fusion AI companion intelligent system applied to a microcontroller according to claim 1, characterized in that, The steps for obtaining the multimodal synchronization response results include: Obtain parallelized interaction instructions, allocate dedicated DMA channels for different types of output devices, configure each channel in cyclic mode to support continuous data transmission, process the parallelized interaction instructions for device allocation, and obtain device allocation instructions. Obtain device allocation instructions, define four priority levels: urgent, high, medium, and low, determine the processing order based on the urgency of the task, the impact on user experience, and resource consumption in the device allocation instructions, and obtain priority scheduling instructions; The priority scheduling instruction is obtained, and a synchronization signal is generated using a hardware timer. All output channels are time-aligned with the priority scheduling instruction based on the synchronization signal, and the synchronization accuracy is controlled within 5 milliseconds to obtain the multimodal synchronization response result.

8. A multimodal fusion AI companion intelligent system applied to a microcontroller according to claim 1, characterized in that, Also includes; The resource allocation and power management unit is used to monitor the execution process of multimodal synchronous response results to obtain system load status, and to perform intelligent pre-allocation and predictive power optimization management of computing resources to obtain optimized system resource configuration and power status.

9. A multimodal fusion AI companion intelligent system applied to a single-chip microcomputer according to claim 4, characterized in that, The step of obtaining the low-latency fusion feature further includes: Windowed feature sequences are obtained, and specialized emotion modeling is performed on windowed feature sequences for stroke patients. The impact of impaired cognitive function on emotional expression is considered, and a patient emotion expression mapping table is established to map subtle facial expression changes in the windowed feature sequences to the corresponding emotional states, thereby obtaining specialized emotion features. Specialized emotional features are obtained, and the speech features within these specialized emotional features are analyzed by focusing on intonation changes to obtain optimized speech features. Optimized speech features are obtained, and an incremental learning strategy is used to process the optimized speech features. Only the most recent 100 training samples are retained for model updates, adapting to the storage limitations of the microcontroller, and low-latency fusion features are obtained.

10. A computer-readable storage medium, characterized in that, It is used to store computer-readable instructions, which, when read by a computer, enable the execution of a multimodal fusion AI companion intelligent system applied to a microcontroller as described in any one of claims 1-9.

Images