A multimodal emotion self-adaptive sensory compensation interaction method and system

Abstract

The present application relates to the technical field of human-computer interaction, and particularly relates to a multi-modal emotion self-adaptive sensory compensation interaction method and system, comprising the following steps: step S1: in response to the start of a trigger signal, multi-modal perception data of a user is collected; the multi-modal perception data at least comprises voice audio data and touch pressure time sequence signals; step S2: the multi-modal perception data is input into a pre-trained emotion calculation model, each modal feature is extracted and feature-level fusion is performed, and a PAD three-dimensional emotion vector is output by reasoning; by constructing a visual-tactile mapping sub-rule and an auditory-visual mapping sub-rule, the present application converts visual texture features into a vibration waveform, converts audio main frequency features into light effect tones and a breathing cycle, so that visually impaired users can perceive flower texture through tactile perception, and hearing impaired users can perceive sound rhythm through light effect perception, the barriers of traditional sensory interaction are effectively broken, and cross-modal sensory compensation is achieved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a multimodal emotion-adaptive sensory compensation interaction method and system. Background Technology

[0002] Horticultural therapy, as an adjunctive rehabilitation method, relieves stress and improves mood by allowing participants to interact with plants and the natural environment, and has been widely used in the fields of psychological rehabilitation and special education. However, traditional horticultural therapy relies heavily on visual observation of plant shapes and colors, and auditory listening to natural sounds such as wind, water, and birdsong. Visually impaired individuals cannot perceive the visual beauty of flowers, and hearing impaired individuals cannot hear natural sounds. These sensory barriers naturally exclude visually and hearing impaired individuals from the horticultural therapy experience, making it difficult for them to obtain the corresponding rehabilitation effects.

[0003] Currently, assistive devices for people with disabilities mainly include screen reader software, voice prompt devices, and single-frequency vibration motors. These devices can only provide single-modal sensory compensation and lack a collaborative compensatory mechanism involving multiple sensory channels such as vision, touch, hearing, and light effects. Meanwhile, existing emotional state regulation methods mostly rely on pre-set fixed scripts or human rehabilitation instructors, employing uniform intervention strategies that cannot dynamically adjust to the user's real-time emotional fluctuations. This results in low personalization, a one-size-fits-all approach, and high service costs. Furthermore, existing algorithms are heavily tied to specific hardware platforms, requiring redevelopment for each new terminal, leading to poor scalability and portability. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies, such as the inability to provide sensory compensation in a single modality and the lack of a collaborative compensation mechanism for multiple sensory channels, including vision, touch, hearing, and light effects. Therefore, this invention proposes a multimodal emotion-adaptive sensory compensation interaction method and system.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of this invention proposes an adaptive sensory compensation interaction method based on multimodal emotion computing, which specifically includes the following steps: Step S1: In response to the trigger signal, collect the user's multimodal perception data; the multimodal perception data includes at least voice audio data and touch pressure timing signals; Step S2: Input the multimodal perception data into the pre-trained emotion computing model, extract features of each modality and perform feature-level fusion, and infer and output the PAD three-dimensional emotion vector; the PAD three-dimensional emotion vector includes pleasure value, activation value and dominance value; Step S3: Obtain the device capability description file of the target interactive terminal. The device capability description file includes at least the frequency range, intensity level and layout information of the haptic feedback unit, and the color and brightness capability parameters of the visual feedback unit. Step S4: Based on the PAD three-dimensional emotion vector, call the cross-modal sensory compensation mapping rule library and combine it with the device capability description file of the target interactive terminal to generate a multimodal driving instruction set adapted to the capability range of the target interactive terminal; the multimodal driving instruction set includes at least vibration waveform driving parameters and light effect driving parameters. Step S5: Send the multimodal driving instruction set to the target interactive terminal to drive the target interactive terminal to execute tactile feedback and / or visual feedback, forming an adaptive closed-loop control of sensory compensation and interactive state.

[0006] Furthermore, the cross-modal sensory compensation mapping rule base includes: The visual-tactile mapping sub-rule is used to map the visual texture roughness features of a target object to the frequency and waveform type of a vibration waveform. Its mapping function is: frequency f = 60 + R × 240 Hz, where R is the roughness value; the waveform type is divided into sine wave, square wave or random noise according to the value range of R. The auditory-visual mapping sub-rule is used to map the dominant frequency characteristics of an audio signal to the hue and breathing cycle of an LED light effect. Its mapping function is: hue H = (f_aud - 100) / 800 × 240°, breathing cycle T_br = 5000 / (f_aud / 100) ms, where f_aud is the dominant frequency of the audio signal.

[0007] Furthermore, the sentiment computing model described in step S2 is a lightweight neural network model, which is deployed on edge computing units through quantization compression and efficient parameter fine-tuning; the quantization compression includes INT8 quantization, and the efficient parameter fine-tuning adopts at least one method from low-rank decomposition or adapter module.

[0008] Furthermore, step S4 also includes an emotion adaptive regulation sub-step: Monitor the activation value A and the pleasure value P in the three-dimensional emotion vector of the PAD; If the activation value A is greater than a preset first threshold, the vibration intensity parameter in the multimodal driving instruction set is adaptively reduced and / or the light effect breathing cycle is extended to reduce the user's wake-up level. If the pleasure value P is less than a preset second threshold, the color saturation parameter and vibration rhythm parameter in the multimodal driving instruction set are adaptively increased to enhance the user's positive emotions.

[0009] Furthermore, the multimodal perception data also includes at least one of physiological signal data or facial expression data; in step S2, the feature-level fusion uses a masking mechanism to process missing modalities and dynamically adjusts the fusion weights of each modality feature in the emotion computing model according to the currently available modality types.

[0010] Furthermore, this also includes crisis intervention and data recording steps: Monitor whether the activation value A in the three-dimensional emotion vector of the PAD is greater than the third threshold and whether the pleasure value P is less than the fourth threshold, and whether this state lasts for more than a preset time; if so, trigger the crisis intervention process, which includes transferring to a human hotline or notifying a preset emergency contact. Record complete log data for each interaction and generate a long-term rehabilitation profile based on the accumulated log data. The long-term rehabilitation profile includes patterns of emotional fluctuations, topic preferences, and trends in intervention effects.

[0011] The second aspect of this invention also proposes an adaptive sensory compensation interaction system based on multimodal emotion computing, specifically including: Triggering medium, used to store a unique scene identifier and generate trigger signals in response to external interactive actions; A multimodal input device includes at least a microphone array and a pressure sensor array, used to collect the user's voice audio signal and touch force timing signal, respectively; The computing processing unit includes an edge computing device and / or a cloud server. The computing processing unit integrates an emotion reasoning engine and a cross-modal mapping engine to execute the above-described methods and output a multimodal driving instruction set. A multimodal interactive terminal includes at least a linear motor array and a controllable LED light source array, used to receive the multimodal driving instruction set and generate vibration feedback and / or light effect feedback; The computing processing unit is connected to the multimodal interactive terminal via wireless communication, and the multimodal interactive terminal reports its device capability description file to the computing processing unit. The computing processing unit dynamically adapts driver instructions based on the device capability description file.

[0012] Furthermore, the multimodal interactive terminal is selected from any one or more of the following forms: Smartphones, tablets, smart bracelets, modular landscape interaction terminals, haptic gloves.

[0013] Furthermore, the modular landscape interaction terminal includes at least: Embedded microcontrollers; A near-field communication read head is used to read the identification information of the trigger medium; At least eight independently controllable linear vibration actuators are arranged in an array. Programmable LED light source array, using a single-bus controlled full-color RGB light strip; And a touch pressure sensor array, used to collect data on the distribution of user pressure on the terminal surface.

[0014] Furthermore, the triggering medium is selected from any of the following: NFC cards, RFID tags, QR code cards, or Bluetooth beacons.

[0015] The present invention proposes a multimodal emotion-adaptive sensory compensation interaction method and system, the beneficial effects of which are as follows: by constructing visual-tactile mapping sub-rules and auditory-visual mapping sub-rules, the present invention transforms visual texture features into vibration waveforms and audio main frequency features into light effect hues and breathing cycles, enabling visually impaired users to perceive flower textures through touch and hearing-impaired users to perceive sound rhythms through light effects, effectively breaking down the barriers of traditional sensory interaction and realizing cross-modal sensory compensation.

[0016] Secondly, by deploying a lightweight emotion computing model on the edge computing unit, the PAD three-dimensional emotion vector is output in real time. Combined with the emotion adaptive adjustment mechanism, the vibration intensity and light effect parameters are dynamically adjusted according to the real-time changes in user activation and pleasure. This allows the interactive feedback to adapt to the fluctuations in user emotions, avoiding the rigid mode of traditional preset scripts and realizing truly personalized emotional healing intervention. Attached Figure Description

[0017] Figure 1 This is a flowchart of Embodiment 1 of the present invention; Figure 2 This is a flowchart of the multimodal emotion computing process according to Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the cross-modal mapping rule in Embodiment 1 of the present invention; Figure 4 This is a structural block diagram of Embodiment 2 of the present invention; Figure 5 This is a diagram showing the internal module connections of the interactive terminal in Embodiment 2 of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0019] Example 1: Refer to Figures 1-3 As one embodiment of the present invention, an adaptive sensory compensation interaction method based on multimodal emotion computing is disclosed, the method specifically including the following steps: Step S1: In response to the trigger signal, collect the user's multimodal perception data; the multimodal perception data includes at least voice audio data and touch pressure timing signals; Specifically, in this embodiment, the trigger signal is emitted by the Near Field Communication (NFC) trigger medium. The user places an NFC card containing a unique identifier for a flower or natural scene close to the NFC sensing area of ​​the smartphone. This NFC card can be an NTAG213 tag with a storage capacity of 144 bytes, compatible with the ISO 14443A protocol standard. The phone's built-in NFC controller can be a PN548 or SN100T chip, supporting a reading distance of 0 to 5 cm and a communication rate of 106 kbps. After reading the card identifier, the NFC controller triggers system startup, simultaneously recording the reading timestamp and the Received Signal Strength Indicator (RSSI) value, providing a data basis for subsequent user behavior analysis.

[0020] After triggering the interrupt response, the system opens a data acquisition window, which is set to last from 1 second before the trigger to 3 seconds after the trigger, to ensure complete capture of the user's reaction information before and after the trigger. The system acquires multimodal data through the following sub-modules: Voice data acquisition is accomplished through the phone's built-in MEMS microphone array, which communicates with the main processor via a PDM or I²S digital interface. Acquisition parameters are set as follows: a sampling rate of 16kHz, covering the full frequency range of human voice (80Hz to 8kHz); a quantization bit depth of 16bit, providing a dynamic range of 96dB; and the use of mono recording in PCM or WAV lossless encoding format to reduce computational load. The acquired raw voice signal is first processed by a bandpass filter with a passband set to 80Hz to 8kHz to effectively remove ambient noise. Then, a Voice Endpoint Detection (VAD) algorithm is used to extract valid voice segments, ensuring the accuracy of subsequent feature extraction.

[0021] Touch pressure data acquisition is accomplished through a pressure sensor array integrated into the touch surface of the interactive terminal. This pressure sensor array uses capacitive or resistive thin-film sensors, such as the FSR-406 model, with a range of 0 to 10 Newtons, covering the typical range of human finger pressure, and a resolution of 0.01 Newtons, enabling the detection of subtle changes in force. The system captures the dynamic process of pressing at a sampling rate of 100Hz, recording characteristic parameters including peak pressure F_max, pressure rise time t_rise, pressure duration t_hold, release speed v_release, and pressure fluctuation variance σ²_pressure. These characteristics reflect the user's emotional state from different dimensions; for example, anxious individuals typically apply more force and have a higher pressure variance, while emotionally stable individuals exhibit stronger pressing patterns.

[0022] Optionally, if the phone has a front-facing camera and the user has explicitly authorized it, the system can also simultaneously collect the user's facial micro-expression data. The camera captures data at a resolution of 640×480 pixels and a frame rate of 30fps, which is sufficient to identify facial feature points and capture micro-expression changes. The system extracts facial action unit (AU) intensity using MediaPipe or OpenFace tools, focusing on key action units such as AU4 (corresponding to negative emotions by the corrugator supercilii muscle), the combination of AU6 and AU12 (corresponding to genuine smiles by the orbicularis oculi and zygomaticus major muscles), and the combination of AU1 and AU2 (corresponding to surprise or fear by raising eyebrows), extracting a total of 17 AU intensity values ​​as auxiliary emotion discrimination criteria. When data for a certain modality is unavailable, the system fills the corresponding dimension of that modality with zeros through a masking mechanism and attaches a modality availability mask vector to the model input layer, informing the inference model of the currently effective modality type, thereby ensuring that the system can still operate stably when some sensors are missing.

[0023] Step S2: Input the multimodal perception data into the pre-trained emotion computing model, extract features of each modality and perform feature-level fusion, and infer and output the PAD three-dimensional emotion vector; the PAD three-dimensional emotion vector includes pleasure value, activation value and dominance value; The multimodal perception data collected in step S1 is input into a pre-trained emotion computing model. Features of each modality are extracted and fused at the feature level to infer and output a three-dimensional emotion vector of PAD. The three-dimensional emotion vector of PAD includes a pleasure value, an arousal value, and a dominance value, with each dimension ranging from -1 to +1.

[0024] This step specifically includes the following sub-procedures: First, speech feature extraction is performed. The standard Mel-frequency cepstral coefficient (MFCC) process is applied to the preprocessed effective speech segments. This process includes pre-emphasis processing with a pre-emphasis coefficient of 0.97, frame segmentation with a frame length of 25 milliseconds, Hamming windowing, Fast Fourier Transform (FFT), filtering through 40 Mel filter banks, logarithmic operations, and Discrete Cosine Transform (DCT), ultimately extracting 13-dimensional MFCC coefficients. Further calculations of the first-order difference Delta MFCC and second-order difference Delta-Delta MFCC are performed to capture the dynamic changes in speech. These, along with the fundamental frequency F0 representing pitch, the root mean square energy (RMS) representing volume, and the zero-crossing rate (ZCR) representing voiced / unvoiced speech, constitute a total of 42-dimensional speech feature vectors. The mean and variance of the statistics for each frame are used to compress the temporal dimension information.

[0025] Next, pressure features are extracted. The mean pressure μ, pressure variance σ², peak pressure F_max, rise slope k_rise, duration t_hold, and pressure entropy H are calculated for the pressure time series obtained in step S1. The pressure entropy represents the regularity of pressure, and together they form a 12-dimensional pressure feature vector.

[0026] Optionally, when a wearable device such as a smart bracelet is connected, the system acquires photoplethysmography (PPG) signals and calculates heart rate variability (HRV) using the MAX30102 sensor, and acquires electrical skin response (EDA) signals using the AD8233 sensor, extracting a total of 6-dimensional physiological features. When a camera is connected, the system extracts 17-dimensional motion unit intensity and 7-dimensional basic emotion probabilities using the MobileNet-FER model, forming a maximum 26-dimensional facial expression feature vector.

[0027] The system then employs an early fusion strategy, namely feature-level fusion, to concatenate the feature vectors of the aforementioned modalities into a fused feature vector of 54 to 80 dimensions. Specifically, the fused vector consists of 42 dimensions of speech features, 12 dimensions of stress features, 6 dimensions of physiological features, and 17 dimensions of facial expression features. The latter two dimensions are selectively concatenated based on actual availability. When a modality becomes unavailable, the system fills the corresponding dimension with a zero vector and dynamically adjusts the contribution weights of each modality through a masking mechanism.

[0028] Finally, the fused feature vector is input into a lightweight large language model (LLM) deployed on the mobile edge computing unit for inference. This model is based on a Transformer architecture with 1.5B to 3B parameters, and can use models such as LLaMA-3B or ChatGLM3-1.5B as its foundation. The model undergoes LoRA low-rank adaptive fine-tuning, where the rank r of the low-rank decomposition is set to 8, the scaling factor α is set to 16, and the number of fine-tuned parameters is approximately 4.2M, accounting for only 0.3% of the total model size. The model also undergoes INT8 quantization compression, resulting in a memory footprint of less than 200MB. During inference, the fused vector is first mapped to 512 dimensions through a linear projection layer, then processed by a Transformer encoder, and finally output as a three-dimensional continuous vector, i.e., the PAD emotion vector, through the emotion head. In the mobile operating environment of this embodiment, the latency of a single inference operation does not exceed 50 milliseconds, fully meeting the requirements of real-time interaction. As an example, the output in the anxious state is -0.3 pleasure, +0.7 activation, and -0.4 dominance, while the output in the calm state is +0.5 pleasure, -0.6 activation, and +0.2 dominance.

[0029] Step S3: Obtain the device capability description file of the target interactive terminal. The device capability description file includes at least the frequency range, intensity level and layout information of the haptic feedback unit, and the color and brightness capability parameters of the visual feedback unit. The system obtains the device capability description file of the target interactive terminal. This device capability description file includes at least the frequency range, intensity level, and layout information of the haptic feedback unit, as well as the color and brightness capability parameters of the visual feedback unit. The device capability description file is defined in JSON format and is the core mechanism for hardware decoupling in this invention.

[0030] like Figure 4 As shown, during system operation, the edge computing unit, i.e., the smartphone in this embodiment, scans for nearby available interactive terminals via Bluetooth Low Energy (BLE) broadcast or mDNS protocol. The target terminal, i.e., the smartphone itself currently executing the feedback or an external wristband and landscape terminal, responds and reports a JSON-formatted device capability description file. Taking the smartphone in this embodiment as an example, its capability description file includes haptic capability definitions such as linear resonant motor (LRA) type, frequency range of 50 to 300 Hz, intensity level of 256, and support for waveform types such as sine wave, square wave, and pulse sequences; display capability definitions such as screen resolution of 1080×2400 pixels, 24-bit color depth, brightness level of 255, and support for HDR display; and LED capability definitions such as screen area simulating LEDs, support for red, green, blue, and white colors, and brightness level of 255.

[0031] After parsing the JSON file, the system dynamically builds a mapping rule adapter, tailoring all subsequently generated general driving parameters, such as abstract vibration frequency and intensity, to the actual physical range supported by the terminal. If the terminal's capabilities change, for example, if the user connects an external vibration glove, the system automatically retrieves the description file again and adjusts the driving instructions online, without requiring a restart or manual configuration. This mechanism ensures that the same core algorithm can be deployed on multiple hardware platforms such as mobile phones, tablets, smart bracelets, and landscape interaction terminals without modification, significantly reducing development costs.

[0032] Step S4: Based on the PAD three-dimensional emotion vector, call the cross-modal sensory compensation mapping rule library and combine it with the device capability description file of the target interactive terminal to generate a multimodal driving instruction set adapted to the capability range of the target interactive terminal; the multimodal driving instruction set includes at least vibration waveform driving parameters and light effect driving parameters. The system first selects a healing theme based on emotion. It uses a rule-based decision-making strategy based on the activation value A and the pleasure value P in the PAD emotion vector. If the activation value A is greater than 0.5, the user is considered to be in a high-arousal state, such as anxiety or tension, and the Calm Forest theme is selected to lower arousal. If the activation value A is less than -0.3 and the pleasure value P is less than 0, the user is considered to be in a low-arousal and negative emotional state, such as depression, and the Vibrant Garden theme is selected to increase both arousal and pleasure. If the pleasure value P is less than -0.3 and the activation value A is less than 0.3, the user is considered to be in a calm but depressed state, and the Sunny Grassland theme is selected to increase pleasure. If the pleasure value P is greater than 0.5 and the activation value A is less than 0, the user is considered to be in a calm and pleasant state, and the Starry Sky Meditation theme is selected to maintain this positive state. Other themes, such as Rose Garden, Lavender Field, and Lotus Pond, are randomly selected to provide an exploratory experience. The above thresholds are the system's preset baseline values. In actual use, the system records the emotional vectors of each user's past 10 interactions and calculates the mean and standard deviation of each dimension. The thresholds are dynamically set to the mean plus or minus 0.5 times the standard deviation, thereby achieving personalized emotional zoning.

[0033] Secondly, cross-modal sensory compensation mapping is performed, which is a key technical means to achieve sensory substitution.

[0034] For visual-to-touch mapping, also known as visual-touch compensation, the system generates vibration waveforms based on the texture features of flowers or natural scenes, including roughness R and temperature sensation T. The mapping function is specifically defined as: vibration frequency f = 60 + roughness R multiplied by 240, in Hertz (Hz), where roughness R ranges from 0 to 1, covering the tactile sensitive frequency band from 60Hz to 300Hz. The waveform type is selected based on roughness R: if roughness R is less than 0.3, a sine wave is used to simulate a smooth surface like a lotus leaf; if roughness R is greater than or equal to 0.3 and less than 0.7, a square wave is used to simulate medium roughness like a rose; if roughness R is greater than or equal to 0.7, random noise is used to simulate complex rough textures like pine needles. Vibration intensity I = 128 + temperature sensation T multiplied by 127, where temperature sensation T ranges from -1 to +1, quantifying temperature perception into intensity levels from 0 to 255. For example, warmth corresponds to high-intensity vibration, and coolness corresponds to low-intensity vibration. Taking a rose scene as an example, with a roughness R of 0.7, a random noise waveform with a frequency of 60 + 0.7 × 240 = 228Hz is generated. A temperature sensing value T of 0.3 is used to calculate the intensity as 128 + 0.3 × 127 = 166, with a default duration of 3000 milliseconds. The system also supports spatial vibration rendering of multi-motor arrays. For example, in a 4×4 mesh layout composed of 16 linear motors, the system samples the spatial distribution map of the target texture based on the position coordinates of each actuator, independently calculating the vibration intensity and start-up delay for each actuator. This achieves a sense of growth, with vibrations gradually moving from the petiole to the leaf tip, allowing the user to perceive the shape and direction of the leaves.

[0035] For auditory-to-visual mapping, also known as auditory-optical compensation, the system maps the dominant frequency characteristics of natural sounds to light effects, colors, and breathing cycles. The mapping function is defined as: Hue H = Audio dominant frequency f_aud - 100 divided by 800 multiplied by 240, where the audio dominant frequency f_aud ranges from 100 to 900Hz. This mapping corresponds approximately 200Hz of wind sound to a pale green hue, approximately 500Hz of birdsong to a pale yellow hue, and approximately 300Hz of flowing water to a blue-green hue. The breathing cycle T_br is equal to 5000 divided by the audio dominant frequency (within parentheses) divided by 100, in milliseconds, and the calculation result is limited to the range of 500 to 5000 milliseconds. This ensures that low-frequency sounds correspond to long cycles (soothing breathing) and high-frequency sounds correspond to short cycles (bright flashing). The brightness amplitude dynamically changes with the audio amplitude envelope, achieving a visual representation of the audio signal envelope.

[0036] Simultaneously, the system also performs emotional adaptive adjustment, dynamically changing mapping parameters based on the PAD vector to proactively intervene in the user's emotions. If the activation value A is greater than 0.6, indicating that the user is in a state of anxiety or tension, the system automatically reduces the vibration intensity by 20% and extends the light effect breathing cycle to 1.5 times to reduce the overall sensory stimulation intensity and guide the user to relax. If the pleasure value P is less than -0.3, indicating that the user is in a state of displeasure or depression, the system increases the color saturation by 20% and accelerates the vibration rhythm (i.e., the frequency is increased by 10%) to enhance positive emotional stimulation.

[0037] Based on the above mapping results, the system generates a structured multimodal driving instruction set. This instruction set includes tactile instructions such as waveform types (sine wave, square wave, random noise), vibration frequency (hertz), intensity level, duration (milliseconds), and actuator identifiers to support independent control of multi-motor arrays; light effect instructions such as target color (RGB values), brightness level, breathing cycle (milliseconds), and lighting modes (static breathing rainbow custom sequences); and optional voice instructions, which are natural language descriptions generated based on LLM, such as "You are now experiencing the delicate texture of rose petals, accompanied by gentle wind sounds and light effects," which are broadcast through a mobile phone speaker or Bluetooth headset.

[0038] Step S5: Send the multimodal driving instruction set to the target interactive terminal to drive the target interactive terminal to execute tactile feedback and / or visual feedback, forming an adaptive closed-loop control of sensory compensation and interactive state.

[0039] In this embodiment, the smartphone executes feedback by calling the underlying interfaces of the Android or iOS system according to the driving instruction set. Vibration execution drives the linear motor through the system's Vibrator application programming interface, outputting tactile feedback according to a specified frequency, intensity, and waveform pattern. For motor arrays supporting multi-point independent control, such as the landscape terminal in Embodiment 2, the system controls each actuator separately through independent pulse width modulation (PWM) channels to achieve spatial vibration rendering. Light effect execution displays light effects of specified colors and breathing rhythms in specific areas of the screen, such as around the NFC sensing area or in full-screen ambient light, through the screen display application programming interface or LED driving interface such as SurfaceFlinger or LEDService, or through the fill light on the back of the phone to achieve ambient lighting effects. Voice broadcasting plays natural language descriptions generated by a large language model through a text-to-speech engine, providing an auxiliary information channel.

[0040] The system employs a time synchronization protocol to ensure consistent perception across multimodal feedback. All executed commands are accompanied by a unified timestamp, and the time difference between tactile output and light effect output is strictly controlled within 20 milliseconds, while the human body's perceptible synchronization threshold is approximately 50 milliseconds. This ensures that users perceive a synchronized and coordinated overall experience between tactile and visual signals.

[0041] After the feedback process is completed, the system records a complete interaction log, including structured information such as trigger medium identifier, multimodal input feature vector, PAD emotion vector output results, generated driving instruction parameters, and user self-assessment feedback. This forms a complete rehabilitation log entry stored in a local database or uploaded to a cloud-based rehabilitation record system. Based on the accumulated log data, the system generates rehabilitation trend analyses across different time dimensions, including daily, weekly, monthly, and annual reports. The daily report presents the user's mood curve, usage frequency, and average pleasure level for self-awareness reference; the weekly report presents the pattern of mood fluctuations, theme preferences, and intervention effect trends for rehabilitation therapists' evaluation; the monthly report presents baseline comparisons, days of significant improvement, and crisis warning information for doctors' decision-making reference; and the annual report presents long-term trends and rehabilitation milestones for research data analysis.

[0042] The following describes the working process of this embodiment using a complete scenario. In a special education school, a student in an anxious state has a PAD emotion vector of -0.4 for pleasure, +0.8 for activation, and -0.5 for dominance. The student holds an NFC card with a rose icon close to a smartphone.

[0043] The phone's NFC reader reads the card identifier and triggers system startup within 100 milliseconds. The microphone captures the student's unintentional utterance of "So annoying!" with a sampling rate of 16kHz and a quantization depth of 16bit. The pressure sensor records the peak force of the pressure applied to the device as 3.2 Newtons. Within 1.2 seconds, the system extracts and fuses the 42-dimensional Mel-frequency cepstral coefficient features of the speech and the 12-dimensional pressure features, outputting a three-dimensional emotion vector for the PAD through a lightweight large language model. The cross-modal mapping engine selects the "Calm Forest" theme based on the activation value of 0.8 (greater than 0.5), mapping the rose's roughness of 0.7 to a 228Hz random noise vibration waveform, and mapping the wind's main frequency of 200Hz to a pale green breathing light effect with a breathing cycle of 2 seconds.

[0044] The phone's linear motor and screen lighting effects initiate feedback within 1.35 seconds of triggering. Users perceive the rough texture of a rose through vibration and feel the rhythm of the wind through the lighting effects. The interaction ends after 4.35 seconds, the system records a complete log, and generates a rehabilitation profile. The total latency from triggering to the first feedback is approximately 1.35 seconds, including a 45-millisecond calculation latency, a 30-millisecond mapping latency, and a communication latency of less than 10 milliseconds, fully meeting the response requirements for real-time interaction.

[0045] In some embodiments, the cross-modal sensory compensation mapping rule base includes: The visual-tactile mapping sub-rule is used to map the visual texture roughness features of a target object to the frequency and waveform type of a vibration waveform. Its mapping function is: frequency f = 60 + R × 240 Hz, where R is the roughness value; the waveform type is divided into sine wave, square wave, or random noise based on the R value range. The visual-tactile mapping sub-rule is designed based on the frequency selectivity and waveform recognition ability of the human tactile perception system. The roughness value R is the core parameter, and its value is derived from the surface texture calibration of various flowers or natural scenes in the sensory feature parameter library. When a user touches the terminal, the system calculates the vibration frequency based on the R value corresponding to the current interaction theme. The larger the R value, the rougher the texture, and the higher the vibration frequency generated by the mapping. At the same time, the waveform type transitions from a sine wave to random noise, creating a sensory correspondence between the tactile sensation and the roughness of the visual texture. The auditory-visual mapping sub-rule is used to map the dominant frequency characteristics of an audio signal to the hue and breathing cycle of an LED light effect. Its mapping function is: Hue H = (f_aud - 100) / 800 × 240°, Breathing cycle T_br = 5000 / (f_aud / 100) ms, where f_aud is the dominant audio frequency. The auditory-visual mapping sub-rule is designed based on the high sensitivity of the human visual system to changes in color and brightness. The dominant audio frequency f_aud determines the dominant hue and breathing rhythm of the light effect. Low-frequency sounds correspond to cooler hue regions and longer breathing cycles in the color wheel, while high-frequency sounds correspond to warmer hue regions and shorter breathing cycles. The brightness amplitude changes in real time with the audio amplitude envelope, creating a sensory correspondence between the dynamics of the light effect and the intensity fluctuations of the sound. Through this mapping rule, the system transfers texture information, which originally relied on visual perception, and sound information, which relied on auditory perception, to the tactile and visual channels respectively, thereby achieving cross-modal sensory compensation.

[0046] Furthermore, the sentiment computing model described in step S2 is a lightweight neural network model, which is deployed on edge computing units through quantization compression and efficient parameter fine-tuning; the quantization compression includes INT8 quantization, and the efficient parameter fine-tuning adopts at least one method from low-rank decomposition or adapter module.

[0047] The emotion computing model employs a lightweight neural network architecture, with its parameter count controlled within the range of 1.5B to 3B to balance emotion recognition accuracy with edge deployment capabilities. To achieve real-time inference on edge computing units, the model is optimized through two techniques: quantization compression and efficient parameter fine-tuning. Quantization compression uses the INT8 integer quantization scheme, converting model weights and activation values ​​from 32-bit floating-point numbers to 8-bit integer representations, reducing model storage space by approximately 75% and keeping memory usage below 200MB. Simultaneously, it leverages the INT8 computing instruction set provided by edge devices to accelerate the inference process, significantly reducing computational latency. Efficient parameter fine-tuning employs at least one method from low-rank decomposition (LoRA) or adapter modules. Low-rank decomposition involves adding a trainable low-rank decomposition matrix next to the weight matrix of the pre-trained model for fine-tuning, updating only a small number of parameters without changing the overall structure of the original pre-trained weights. The number of fine-tuned parameters is approximately 4.2M, accounting for only 0.3% of the total model size. Through the above optimizations, the affective computing model can independently complete inference on edge devices such as mobile phones, with a single inference latency of no more than 50 milliseconds. It eliminates the need to transmit user data to the cloud, effectively protecting user privacy while ensuring real-time interactive performance. This deployment scheme eliminates the need for dedicated computing hardware or a continuous network connection, significantly reducing deployment barriers and operating costs.

[0048] In an optional embodiment, step S4 further includes an emotion adaptive adjustment sub-step: Monitor the activation value A and the pleasure value P in the three-dimensional emotion vector of the PAD; If the activation value A is greater than a preset first threshold, the vibration intensity parameter in the multimodal driving instruction set is adaptively reduced and / or the light effect breathing cycle is extended to reduce the user's wake-up level. If the pleasure value P is less than a preset second threshold, the color saturation parameter and vibration rhythm parameter in the multimodal driving instruction set are adaptively increased to enhance the user's positive emotions.

[0049] Specifically, the system receives the PAD three-dimensional emotion vector output in step S2 in real time and continuously monitors its activation value A and pleasure value P. When the activation value A is greater than a preset first threshold, it indicates that the user is currently in a high arousal state, exhibiting emotional characteristics such as anxiety, tension, or excitement. At this time, the system adaptively reduces the vibration intensity parameter in the multimodal driving instruction set, that is, reduces the overall vibration intensity output by the visual-tactile mapping sub-rule by 20%, while extending the light effect breathing cycle, that is, extending the breathing cycle output by the auditory-visual mapping sub-rule to 1.5 times the original value. By weakening the intensity of tactile stimulation and slowing down the rhythm of visual light effect changes, the system reduces the arousal level of external stimuli on the user through multi-sensory channels, guiding the user from a high arousal state to a calm state.

[0050] When the pleasure value P is less than the preset second threshold, it indicates that the user is currently in a negative emotional state, manifested as displeasure, depression, or sadness. At this time, the system adaptively increases the color saturation parameter in the multimodal driving instruction set, that is, increases the color saturation of the light effect by 20% to enhance the vividness and appeal of the visual stimulus. At the same time, it increases the vibration rhythm parameter, that is, increases the vibration frequency by 10% to enhance the activity and rhythm of the tactile feedback. By enhancing the positive attributes of multisensory stimulation, the system improves the user's pleasure level.

[0051] The first and second thresholds mentioned above are set using a combination of system-preset baseline values ​​and user-personalized adaptive methods. The baseline values ​​are set as A>0.6 and P<-0.3, respectively. Based on this, the system records the statistical distribution of each user's emotional vectors from multiple past interactions and dynamically adjusts the thresholds to offset the positive or negative values ​​of the user's individual mean, thereby achieving personalized emotional regulation boundaries. Through this adaptive adjustment mechanism, the system's output feedback is no longer a fixed mapping result, but a personalized interactive experience that can dynamically adjust with the user's emotional fluctuations, effectively improving the accuracy and effectiveness of emotional intervention.

[0052] In an optional embodiment, the multimodal perception data further includes at least one of physiological signal data or facial expression data; the feature-level fusion in step S2 uses a masking mechanism to process missing modalities and dynamically adjusts the fusion weights of each modality feature in the emotion computing model according to the currently available modality types.

[0053] The acquisition of multimodal perception data is not limited to voice and touch pressure modes. The system also supports the access of at least one of physiological signal data and facial expression data as supplementary perception channels. Physiological signal data is collected through wearable devices such as smart bracelets, specifically including heart rate variability (HRV) obtained by photoplethysmography (PPG) sensors and skin conductance level (EDA) obtained by skin conductance sensors. These signals reflect the user's autonomic nervous activity state and are important physiological indicators of emotional arousal. Facial expression data is acquired through camera-captured facial image sequences. The intensity of facial action units (AUs) is extracted using MediaPipe or OpenFace tools, focusing on key emotion-related action units such as the corrugator supercilii (AU4), the orbicularis oculi and zygomaticus major muscle combination (AU6+12), and the eyebrow raising (AU1+2). The probability distribution of each basic emotion category is also output.

[0054] Because users may not be wearing wearable devices or the terminal may not have a camera in different usage scenarios, the data for each modality is not always complete. To address this modality incompleteness, the feature-level fusion in step S2 employs a masking mechanism to handle missing modalities. Specifically, when constructing the fusion feature vector, the system fills all feature values ​​of the corresponding dimension of the currently unavailable modality with zeros, and generates a mask vector corresponding to the modality type. In this mask vector, the flag bits corresponding to available modalities are marked as 1, and the flag bits corresponding to missing modalities are marked as 0. The fusion feature vector, along with the mask vector, is input into the sentiment computing model. During inference, the model uses the mask vector to determine the type and number of currently effective modalities, and dynamically adjusts the fusion weights of each modality feature based on the mask value before each modality feature enters the projection layer. For example, when only voice and stress modalities are available, the model assigns higher weight coefficients to these two modalities; when physiological signals or facial expression data are available simultaneously, the model maintains the weights of the basic modalities while reasonably allocating the contribution ratio of the newly added modalities, so that multimodal information is fully utilized and the model output is not significantly biased due to the absence of a certain modality.

[0055] Furthermore, this invention also includes crisis intervention and data recording steps: Monitor whether the activation value A in the three-dimensional emotion vector of the PAD is greater than the third threshold and whether the pleasure value P is less than the fourth threshold, and whether this state lasts for more than a preset time; if so, trigger the crisis intervention process, which includes transferring to a human hotline or notifying a preset emergency contact. Record complete log data for each interaction and generate a long-term rehabilitation profile based on the accumulated log data. The long-term rehabilitation profile includes patterns of emotional fluctuations, topic preferences, and trends in intervention effects.

[0056] The crisis intervention and data recording steps constitute an important functional extension of this invention in terms of emotion monitoring and rehabilitation management. During continuous system operation, the emotion computing engine outputs a PAD three-dimensional emotion vector in real time at a fixed frequency. The crisis intervention module simultaneously monitors the activation value A and the pleasure value P in this emotion vector. When the activation value A is greater than a third threshold and the pleasure value P is less than a fourth threshold, and this dual condition persists for more than a preset duration, the system determines that the user is currently in an extreme negative emotional state, such as high anxiety accompanied by deep displeasure. This state carries a potential psychological crisis risk, and the system automatically triggers the crisis intervention process. The third and fourth thresholds are set higher than the first and second thresholds used in the aforementioned emotion adaptive adjustment steps. Specifically, they can be set to an activation value A greater than 0.7 and a pleasure value P less than -0.5 to exclude general emotional fluctuations and target only extreme states that truly require external intervention. The preset duration can be set to 3 to 5 seconds to prevent false triggering due to instantaneous changes in the emotion vector, ensuring the reliability and robustness of the crisis assessment.

[0057] Once the crisis intervention process is initiated, the system will perform at least one of the following operations: First, it will issue a clear vibration alert and visual cue via the interactive terminal, informing the user that the system has detected that their emotional state requires support, and providing a one-click call interface for the user to actively seek help. If the user does not actively cancel or respond within the set time, the system will automatically transfer the user to a pre-set human psychological assistance hotline or online psychological counseling service platform, establishing a real-time communication channel between the user and a professional therapist. Simultaneously, the system will send a notification message containing the user's real-time location information and a summary of their current emotional state to pre-set emergency contacts via cellular or wireless networks, ensuring timely external support intervention. After the crisis intervention process is completed, the system will mark the complete data of this event, including the trigger time, emotional vector trajectory, intervention measures, and user response results, as a crisis event and store it separately in a cloud database.

[0058] In terms of data recording, the system generates a complete rehabilitation log entry for each interaction. The log entries include the identifier and timestamp of the triggering medium, multimodal sensory data features before user interaction (i.e., voice MFCC feature vector, stress feature vector, and optional physiological and facial feature vectors), the PAD three-dimensional emotion vector and probability distributions of each basic emotion output by the emotion computing model, generated cross-modal driving command parameters (i.e., vibration waveform specifications and light effect parameters), the user's self-assessment scale score after interaction, and an indicator of whether the complete interaction process was completed. All log data is anonymized and stored locally, and periodically synchronized to the cloud-based rehabilitation record system. Based on the accumulated log data, the system automatically generates rehabilitation trend reports according to different time dimensions. Daily reports present the user's emotional curves at different times of the day, the frequency of use of each theme, and the average level of pleasure, providing users with a reference for daily self-awareness and emotion management. Weekly reports summarize the temporal distribution patterns of emotional fluctuations within a week, the user's preference ranking for different healing themes, and the trend of emotional improvement after each interaction, allowing rehabilitation therapists to assess the effectiveness of phased interventions and adjust subsequent rehabilitation plans.

[0059] Example 2: Refer to Figure 4 , Figure 5 In another embodiment of the present invention, an adaptive sensory compensation interaction system based on multimodal emotion computing is disclosed, the system specifically comprising: Triggering medium, used to store a unique scene identifier and generate trigger signals in response to external interactive actions; A multimodal input device includes at least a microphone array and a pressure sensor array, used to collect the user's voice audio signal and touch force timing signal, respectively; The computing processing unit includes an edge computing device and / or a cloud server. The computing processing unit integrates an emotion reasoning engine and a cross-modal mapping engine to execute the above-described methods and output a multimodal driving instruction set. A multimodal interactive terminal includes at least a linear motor array and a controllable LED light source array, used to receive the multimodal driving instruction set and generate vibration feedback and / or light effect feedback; The computing processing unit is connected to the multimodal interactive terminal via wireless communication, and the multimodal interactive terminal reports its device capability description file to the computing processing unit. The computing processing unit dynamically adapts driver instructions based on the device capability description file.

[0060] Based on the above embodiments, the multimodal interactive terminal is selected from any one or more of the following forms: smartphone, tablet computer, smart bracelet, modular landscape interactive terminal, tactile gloves.

[0061] The modular landscape interaction terminal includes at least the following: Embedded microcontrollers; A near-field communication read head is used to read the identification information of the trigger medium; At least eight independently controllable linear vibration actuators are arranged in an array. Programmable LED light source array, using a single-bus controlled full-color RGB light strip; And a touch pressure sensor array, used to collect data on the distribution of user pressure on the terminal surface.

[0062] The triggering medium is selected from any of the following: NFC card, RFID tag, QR code card or Bluetooth beacon.

[0063] The complete workflow of the system is as follows: When the trigger medium approaches the landscape terminal, the near-field communication reader reads the medium identifier and wakes up the system. The embedded microcontroller simultaneously starts the microphone array to collect voice and the pressure sensor array to collect the tactile force distribution. All data is collected to the embedded microcontroller via the bus. The lightweight emotion computing model running on it extracts the features of each modality, performs feature-level fusion, and infers and outputs a three-dimensional emotion vector for the PAD. The cross-modal mapping engine selects a healing theme based on this emotion vector, reads the texture features and audio features of the corresponding flowers or natural scenes from the sensory feature parameter library, generates vibration waveform parameters and light effect parameters through visual-touch and auditory-light mapping rules, and then adapts the parameters to the physical range supported by the hardware by combining the current terminal's own capability description file. The finally generated drive instruction set is simultaneously distributed to each motor driver chip and LED light strip via the I²C bus. The 16 linear motors are started sequentially according to the spatialized tactile rendering algorithm to generate texture vibration, and the 60 LED light strips synchronously present breathing light effects associated with the main audio frequency, thus forming a collaborative multi-sensory feedback experience in the user's touch area. The total delay from triggering to the first feedback in the entire process does not exceed 1.2 seconds, which fully meets the requirements for real-time interaction.

[0064] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An adaptive sensory compensation interaction method based on multimodal emotion computing, characterized in that, Includes the following steps: Step S1: In response to the trigger signal, collect the user's multimodal perception data; The multimodal sensing data includes at least voice audio data and touch pressure timing signals; Step S2: Input the multimodal perception data into the pre-trained sentiment computing model, extract the features of each modality and perform feature-level fusion, and infer and output the PAD three-dimensional sentiment vector; The PAD three-dimensional emotion vector includes a pleasure value, an activation value, and a dominance value. Step S3: Obtain the device capability description file of the target interactive terminal. The device capability description file includes at least the frequency range, intensity level and layout information of the haptic feedback unit, and the color and brightness capability parameters of the visual feedback unit. Step S4: Based on the PAD three-dimensional emotion vector, call the cross-modal sensory compensation mapping rule library and combine it with the device capability description file of the target interactive terminal to generate a multimodal driving instruction set adapted to the capability range of the target interactive terminal; the multimodal driving instruction set includes at least vibration waveform driving parameters and light effect driving parameters. Step S5: Send the multimodal driving instruction set to the target interactive terminal to drive the target interactive terminal to execute tactile feedback and / or visual feedback, forming an adaptive closed-loop control of sensory compensation and interactive state.

2. The method according to claim 1, characterized in that, The cross-modal sensory compensation mapping rule base includes: The visual-tactile mapping sub-rule is used to map the visual texture roughness features of a target object to the frequency and waveform type of a vibration waveform. Its mapping function is: frequency f = 60 + R × 240 Hz, where R is the roughness value; the waveform type is divided into sine wave, square wave or random noise according to the value range of R. The auditory-visual mapping sub-rule is used to map the dominant frequency characteristics of an audio signal to the hue and breathing cycle of an LED light effect. Its mapping function is: hue H = (f_aud - 100) / 800 × 240°, breathing cycle T_br = 5000 / (f_aud / 100) ms, where f_aud is the dominant frequency of the audio signal.

3. The method according to claim 1, characterized in that, The sentiment computing model described in step S2 is a lightweight neural network model, which is deployed on edge computing units through quantization compression and efficient parameter fine-tuning; the quantization compression includes INT8 quantization, and the efficient parameter fine-tuning adopts at least one method from low-rank decomposition or adapter module.

4. The method according to claim 1, characterized in that, Step S4 also includes an emotion adaptive regulation sub-step: Monitor the activation value A and the pleasure value P in the three-dimensional emotion vector of the PAD; If the activation value A is greater than a preset first threshold, the vibration intensity parameter in the multimodal driving instruction set is adaptively reduced and / or the light effect breathing cycle is extended to reduce the user's wake-up level. If the pleasure value P is less than a preset second threshold, the color saturation parameter and vibration rhythm parameter in the multimodal driving instruction set are adaptively increased to enhance the user's positive emotions.

5. The method according to claim 1, characterized in that, The multimodal perception data also includes at least one of physiological signal data or facial expression data; the feature-level fusion in step S2 uses a masking mechanism to process missing modalities and dynamically adjusts the fusion weights of each modality feature in the emotion computing model according to the currently available modality types.

6. The method according to claim 1, characterized in that, It also includes crisis intervention and data recording steps: Monitor whether the activation value A in the three-dimensional emotion vector of the PAD is greater than the third threshold and whether the pleasure value P is less than the fourth threshold, and whether this state lasts for more than a preset time. If so, the crisis intervention process is triggered, which includes transferring to a live hotline or notifying a pre-set emergency contact. Record complete log data for each interaction and generate a long-term rehabilitation profile based on the accumulated log data. The long-term rehabilitation profile includes patterns of emotional fluctuations, topic preferences, and trends in intervention effects.

7. An adaptive sensory compensation interaction system based on multimodal emotion computing, characterized in that, include: Triggering medium, used to store a unique scene identifier and generate trigger signals in response to external interactive actions; A multimodal input device includes at least a microphone array and a pressure sensor array, used to collect the user's voice audio signal and touch force timing signal, respectively; The computing processing unit includes an edge computing device and / or a cloud server, the computing processing unit integrating an emotion reasoning engine and a cross-modal mapping engine, for executing the method as described in any one of claims 1 to 6, and outputting a multimodal driving instruction set; A multimodal interactive terminal includes at least a linear motor array and a controllable LED light source array, used to receive the multimodal driving instruction set and generate vibration feedback and / or light effect feedback; The computing processing unit is connected to the multimodal interactive terminal via wireless communication, and the multimodal interactive terminal reports its device capability description file to the computing processing unit. The computing processing unit dynamically adapts driver instructions based on the device capability description file.

8. The system according to claim 7, characterized in that, The multimodal interactive terminal is selected from any one or more of the following forms: Smartphones, tablets, smart bracelets, modular landscape interaction terminals, haptic gloves.

9. The system according to claim 8, characterized in that, The modular landscape interaction terminal includes at least: Embedded microcontrollers; A near-field communication read head is used to read the identification information of the trigger medium; At least eight independently controllable linear vibration actuators are arranged in an array. Programmable LED light source array, using a single-bus controlled full-color RGB light strip; And a touch pressure sensor array, used to collect data on the distribution of user pressure on the terminal surface.

10. The system according to claim 7, characterized in that, The triggering medium is selected from any of the following: NFC cards, RFID tags, QR code cards, or Bluetooth beacons.

Images