Embodied interaction intelligent accessory system based on affective computing and tactile feedback

Abstract

This invention proposes an embodied interactive smart accessory system based on affective computing and haptic feedback. It comprises a wearable biofeedback and haptic interaction device, a mobile computing device, and a cloud device. The wearable biofeedback and haptic interaction device is responsible for collecting data from electrical activity data (EDA), photoplethysmography (PPG), and inertial measurement unit (IMU), and executing haptic rendering commands. The mobile computing device handles Bluetooth communication, signal preprocessing, affective feature extraction, haptic rendering algorithm calculation, and communication with the cloud device. The cloud device runs a large language model, receives affective vectors, and generates personalized content. This invention addresses the lack of human touch in AI interaction by introducing an affective computing closed loop; and it resolves safety concerns in outdoor AR experiences through spatialized haptic navigation. This design aligns more closely with the scientific principles of embodied cognition, providing a more natural interface for human-machine symbiosis.

Description

Technical Field

[0001] This invention relates to the field of wearable devices, specifically to an embodied interactive smart accessory system based on emotion computing and haptic feedback. Background Technology

[0002] With the rise of spatial computing technology, the consumption scenarios for digital content are expanding from two-dimensional screens to three-dimensional physical space. In museum tours, urban cultural explorations, and immersive games, users are no longer static observers but mobile participants. This "embodied" transformation requires interactive devices to possess "invisibility" and "environmental awareness." According to embodied cognition theory, the depth of human understanding of space depends on the body's sensorimotor system. However, current interactive device design remains at the "instrumental" stage, meaning the device is an operating tool in the user's hand, rather than a "digital prosthetic" that extends the user's perception. Traditional mobile device haptic feedback is limited to simple vibration alerts. With the widespread adoption of linear resonant actuators and piezoelectric actuators, high-precision haptic feedback has become possible. Psychophysical research shows that by controlling the time and intensity differences of multiple stimulation points on the skin, "phantom perception" or "tactile motion illusion" can be induced, making the user feel the vibration points moving continuously on the skin surface. This technology provides a theoretical basis for achieving high-resolution spatial navigation within a limited wearable area.

[0003] While existing smart bracelets and similar technologies achieve accurate input of "intentions," they neglect the perception of "state." In a virtual-real fusion experience, the system cannot determine whether the user is bored, anxious, or immersed. Although existing surface electromyography (sEMG) technology attempts to be used for emotion recognition, in a non-resting state, the electrical signals of muscle activity completely drown out the weak emotional EMG responses, and sEMG's ability to distinguish arousal and valence is far lower than that of electrical skin activity (EDA) and heart rate variability (HRV).

[0004] Furthermore, both mobile phone screen navigation and virtual arrows on AR glasses require users to focus their attention on the visual channel. This visual monopoly not only causes users to ignore hazards in the physical environment (such as steps or vehicles) but also undermines the sense of immersion in the environment. Existing haptic devices cannot provide accurate directional guidance in a miniaturized form (such as the size of a ring), forcing users to frequently check the screen to confirm their location. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an embodied interactive smart accessory system based on affective computing and haptic feedback. By integrating EDA (electrodermal oscillation) and PPG (photoplethysmography) sensor arrays, it captures signals reflecting the state of the autonomic nervous system in real time, quantifying the user's arousal and emotional valence. Utilizing a surround-type micro LRA array, combined with a "haptic funnel illusion" algorithm, it generates continuously movable, directional virtual haptic points within a limited annular area of ​​contact between the wearable device and the skin. This allows for navigation guided by human tactile intuition, completely liberating vision. This solves the problems mentioned in the background section. The technical solution provided by this invention is as follows:

[0006] An embodied interactive smart accessory system based on affective computing and haptic feedback consists of a wearable biofeedback and haptic interaction device, a mobile computing device, and a cloud device. The wearable biofeedback and haptic interaction device is responsible for collecting data from electrodermal activity (EDA), photoplethysmography (PPG), and inertial measurement unit (IMU), and executing haptic rendering commands. The mobile computing device is responsible for Bluetooth communication, signal preprocessing, affective feature extraction, haptic rendering algorithm calculation, and communication with the cloud device. The cloud device is responsible for running a large language model, receiving affective vectors, and generating personalized content.

[0007] The wearable biofeedback and tactile interaction device includes a shell, several rigid motherboards, EDA sensors, PPG sensors, and actuators. The motherboards integrate a microcontroller unit, an inertial measurement unit, EDA sensor front-end circuits, PPG sensor front-end circuits, and a battery. The actuators execute tactile rendering instructions through a tactile rendering algorithm.

[0008] Preferably, the EDA sensor is a dry electrode made of Ag / AgCl coating or conductive ceramic, and is attached to the pad of the finger or the inside of the wrist; the PPG sensor integrates multi-wavelength LEDs and is attached to the arteries of the finger or wrist.

[0009] Preferably, the actuator is a spatialized haptic actuation array consisting of at least four miniature linear resonant actuators (LRAs).

[0010] Preferably, the haptic rendering algorithm specifically involves: causing two adjacent actuators to vibrate simultaneously and merging them into a virtual stimulation point located in the middle of the line connecting the two actuators. The position of the virtual stimulation point depends on the ratio of the vibration intensity of the two actuators and is allocated according to the trigonometric function law.

[0011] Preferably, the actuator vibration adopts an asymmetric waveform with rapid rise and slow decay characteristics.

[0012] Preferably, the wearable biofeedback and tactile interaction device is in the form of a wristband, consisting of multiple functional islands and flexible bridges. The functional islands contain several rigid mainboards that avoid the styloid processes of the ulna and radius, and only conform to the soft tissue area of ​​the muscles. The flexible bridges are serpentine copper foil wires that connect the islands. The entire structure is encapsulated in skin-friendly silicone or elastic fabric.

[0013] Preferably, the mobile computing device and the cloud device perform emotion computing and generate personalized content through the following methods:

[0014] Step 1: Using IMU data as a reference, the acceleration signal measured by the IMU is used as a reference noise source. The motion-induced component is subtracted from the original PPG and EDA signals. For the EDA signal, the skin conductance response (SCR) is separated by high-pass filtering, and the skin conductance level (SCL) is separated by low-pass filtering.

[0015] Step 2: Extract the number and amplitude of SCR peaks per minute from the EDA signal, and extract the successive heartbeat interval and heart rate variability (HRV) from the PPG signal. The heart rate variability (HRV) includes the root mean square difference (RMSSD) and standard deviation normal to normal interval (SDNN) of the time domain indicators, and the low frequency to high frequency ratio (LF / HF) of the frequency domain indicators.

[0016] Step 3: Construct an emotion coordinate system based on the Russell ring model. The X-axis represents valence (V), determined by the HRV index; the Y-axis represents arousal (A), mainly determined by the SCR frequency and amplitude of EDA. The output is a normalized emotion vector. ;

[0017] Step 4: Input the emotion vector into the cloud-based large language model to generate personalized content, and trigger feedback through a haptic rendering algorithm.

[0018] Compared with existing technologies, the beneficial effects achieved by this invention are: by introducing an emotional computing closed loop, it solves the problem of AI interaction lacking human touch; and by using spatialized haptic navigation, it addresses the safety hazards of outdoor AR experiences. This design is more in line with the scientific principles of "embodied cognition," providing a more natural interface for human-machine symbiosis. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0020] Figure 1 This is a schematic diagram of the miniaturized smart ring architecture of Embodiment 1 of the present invention;

[0021] Figure 2 This is a schematic diagram of the flexible smart bracelet architecture of Embodiment 2 of the present invention;

[0022] Figure 3 This is a flowchart of the closed-loop process of emotion computing and AI processing in this invention;

[0023] Figure 4 This is a schematic diagram illustrating the tactile illusion navigation principle of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To make the above-mentioned objectives, features and effects of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Example 1: An embodied interactive smart accessory system based on affective computing and haptic feedback, comprising a wearable biofeedback and haptic interaction device, a mobile computing device, and a cloud device. The wearable biofeedback and haptic interaction device is responsible for collecting data on electrical activity of the skin (EDA), photoplethysmography (PPG), and inertial measurement unit (IMU), and executing haptic rendering instructions. The mobile computing device is responsible for Bluetooth communication, signal preprocessing, affective feature extraction, haptic rendering algorithm calculation, and communication with the cloud device. The cloud device is responsible for running a large language model, receiving affective vectors, and generating personalized content.

[0027] In this embodiment, the wearable biofeedback and tactile interaction device adopts a miniaturized smart ring architecture, such as... Figure 1 As shown, considering the small size of fingers and their sensitivity to thickness, the following technical solution is adopted:

[0028] The ring features a dual-layer structure. The inner layer is made of medical-grade resin or ceramic and integrates sensor contacts; the outer layer is a replaceable decorative shell (such as imitation jade, enamel, or fabric) to suit different cultural themes.

[0029] The internal circuitry of the ring employs a "sandwich" rigid-flexible structure. The rigid mainboard on the back side is located at the top of the ring, integrating the main control microcontroller unit (MCU, such as nRF5340), the 6-axis inertial measurement unit (IMU), and the PPG optical front end. The rigid board is chosen to ensure the focal length stability of the optical components and the reliable soldering of the BGA chip. The rigid sensing board on the belly side is located at the bottom of the ring, integrating the EDA analog front end and the wireless charging coil. A lateral flexible connecting strip connects the back and belly boards. A custom-designed curved lithium polymer battery is placed on the side wall of the ring.

[0030] The PPG sensor's optical window is located at the top of the inner ring of the finger ring (i.e., the back of the finger). The dorsal digital artery is relatively shallow, and when the hand grips an object (such as a mobile phone), the mechanical pressure on the back of the finger is much less than that on the fingertip, thus ensuring signal continuity during operation. The sensor's light source integrates multi-wavelength LEDs: green light is used for heart rate monitoring during exercise (with strong resistance to motion interference), and infrared light is used for deep blood flow detection in static or slightly moving states. It boasts a high signal-to-noise ratio and is invisible, so it does not interfere with the user experience. The sampling rate is set to ≥64Hz (preferably 100Hz) to ensure accurate capture of the peak time of each pulse wave, thereby calculating the millisecond-level RR interval.

[0031] The EDA sensor electrodes are two gold-plated dry electrodes made of Ag / AgCl plating or conductive ceramic, exposed at the bottom of the inner ring of the finger (i.e., the pad of the finger). The fingertip has an extremely high density of sweat glands (>600 / cm²). 2 This device can capture subtle skin conductance responses that reflect emotional arousal. The acquisition circuit employs the exosomatic measurement method, applying a weak, constant DC voltage (<0.5V) to measure changes in skin resistance. The sampling rate is set to 4Hz-10Hz to capture skin conductance responses induced by emotional arousal.

[0032] The inertial measurement unit (IMU) integrates a three-axis accelerometer and a three-axis gyroscope for motion artifact elimination and attitude interaction. When rapid motion is detected, the system automatically reduces the weights of EDA and HRV to prevent false alarms.

[0033] Miniature linear resonant actuators (LRAs) are selected as the spatial haptic actuation array. The LRAs have a start-stop time of <10ms, producing a clear "click" sensation rather than a "buzzing" sound. Furthermore, the vibration frequency and amplitude can be decoupled and controlled, making them suitable for detailed rendering. The actuation array has at least four LRAs distributed around the circumference of the ring (e.g., at 0°, 90°, 180°, and 270° positions). Each actuator is encased in a liquid silicone rubber (LSR, Shore A 30-40 hardness) sheath, avoiding direct contact with the circuit board or housing. Each LRA is controlled by an independent drive channel (e.g., a TI DRV2605L series driver chip), supporting real-time waveform library calls.

[0034] This invention employs a weighted coefficient-based actuator array control method to achieve emotion computing and haptic rendering, the flowchart of which is shown below. Figure 3 As shown, it includes the following steps:

[0035] Step 1, Signal Preprocessing and Artifact Removal: Using IMU data as a reference, an adaptive LMS filter is employed to subtract motion-induced components from the original PPG and EDA signals, using the acceleration signal measured by the IMU as a reference noise source. For the EDA signal, the skin response (SCR) is separated through a high-pass filter (cutoff frequency 0.05Hz), and the skin conductance level (SCL) is separated through a low-pass filter.

[0036] Step 2, Physiological Feature Extraction: Extract the number and amplitude of SCR peaks per minute from the EDA signal, and extract successive heartbeat intervals and heart rate variability (HRV) from the PPG signal. HRV includes time-domain indices RMSSD (root mean square difference between adjacent heartbeat intervals, reflecting parasympathetic activity and associated with relaxation) and SDNN (normal to normal standard deviation interval), as well as frequency-domain indices LF / HF (low-frequency to high-frequency ratio, reflecting sympathetic / parasympathetic balance and associated with stress).

[0037] Step 3, Multimodal Sentiment Mapping: Construct a sentiment coordinate system based on the Russell ring model. The X-axis represents valence (V), primarily determined by the HRV (High RMSSD) index (high LF / HF indicates positive / relaxed, high LF / HF indicates negative / stressful); the Y-axis represents arousal (A), primarily determined by the frequency and magnitude of SCR (Selective Criteria) in EDA (Emotional Arousal). An example is shown below:

[0038] Case 1: High EDA (high arousal) + low HRV (sympathetic dominance / high stress) = anxiety / fear;

[0039] Case 2: High EDA (high arousal) + High HRV (parasympathetic dominance / balance) = Excitation / flow;

[0040] Case 3: Low EDA (low arousal) + High HRV (parasympathetic dominance / balance) = relaxation / boredom.

[0041] The output is a normalized sentiment vector. .

[0042] Step 4: Input the emotion vector into the cloud-based large language model to generate personalized content, and trigger feedback through a haptic rendering algorithm.

[0043] The haptic rendering algorithm described is based on the haptic funnel illusion. When two adjacent actuators (let's call them...) When two stimuli vibrate simultaneously, if the time difference between them is less than a critical threshold (approximately 3 ms), the two stimuli are merged into a "virtual stimulus point" located in the middle of the line connecting them. The position of this virtual point depends on the ratio of the vibration intensities of the two actuators. Figure 4 As shown, an example is as follows:

[0044] Assuming the ring has four actuators located at 0°, 90°, 180°, and 270°, if the target navigation direction is... (For example, 45°), then the vibration intensity of the two actuators located at 0° and 90°. Distribute according to the following trigonometric function rules:

[0045]

[0046]

[0047] By continuous change The value allows the virtual vibration point to rotate smoothly around the finger, creating a continuous sense of direction.

[0048] To enhance directional awareness, this invention also incorporates an asymmetrical waveform characterized by rapid rise and slow decay. This asymmetrical waveform creates an illusion similar to "pulling" or "sliding." When this waveform is positioned to the right front of the ring using the aforementioned illusion perception algorithm, the user will instinctively feel a force pulling their finger to the right front. This transforms the navigation experience from "cognitive interpretation" (thinking about which way to turn) to "intuitive following" (the body being pulled).

[0049] The following is an example of adaptive interaction in cultural heritage exploration:

[0050] 1. Intuitive Navigation Phase: The user, wearing the ring, walks through the scenic area. The system sets the next exploration point 30 degrees to the user's right. The two LRAs on the right side of the ring activate, using a funnel illusion algorithm to synthesize a virtual vibration point at the 30-degree angle. The vibration uses a "traction waveform," making the user feel a gentle pull to the right, naturally turning right without needing to check their phone. When the user is facing the target (error < 5 degrees), all LRAs simultaneously emit a short, crisp "click" vibration, informing the user that they are on the right track.

[0051] 2. Content Triggering and Emotion Monitoring Phase: Upon reaching the target point, a suspenseful historical story plays in the headphones. The sensor array monitors in real time: EDA shows a surge in SCR (increased arousal), and PPG shows an increase in the LF / HF ratio (increased stress). The emotion computing engine determines that the user is in a state of "high arousal + negative valence," possibly feeling tense or afraid.

[0052] 3. AI Closed-Loop Feedback Stage: The system injects the emotion vector [V: -0.8, A: 0.9] into the cloud-based LLM model. The following adaptive content is generated: The AI ​​NPC virtual guide's voice softens: "This history is indeed quite thrilling, but don't worry, we're safe now. Look at that ancient tree ahead…", while the ring LRA array switches to low-frequency, gentle breathing rhythm vibrations (similar to a heartbeat), using tactile feedback to help the user calm their emotions.

[0053] 4. Physical Interaction Phase: The user touches the texture on the ancient tree as prompted. The NFC module inside the ring reads the passive tag on the ancient tree. The ring emits a special "energy flow" vibration pattern, and the phone screen unlocks a new illustration.

[0054] Example 2: In this example, the wearable biofeedback and tactile interaction device adopts a flexible smart bracelet architecture, such as... Figure 2 As shown, considering the large range of motion and prominent bones in the wrist, the following technical solution is adopted:

[0055] The wristband utilizes a combination of flexible circuit boards and elastic fabric to ensure a close fit between the sensors and the skin of the wrist. The wristband is composed of multiple functional islands and flexible bridges. Each functional island contains several rigid PCBs, housing the main control microcontroller (MCU), EDA sensors, PPG sensors, battery, and actuators. The flexible bridges are serpentine copper foil wires connecting the islands. When the wristband is stretched, the serpentine structure expands in-plane or bends out-of-plane, absorbing stress and ensuring unbroken circuit connections. The entire structure is encapsulated in skin-friendly silicone or elastic fabric.

[0056] The EDA sensor has 6-8 Ag / AgCl electrode contacts evenly distributed along the circumference on the inside of the wristband. Using an analog switching matrix, it scans the contact impedance between adjacent electrode pairs at a high frequency (e.g., 1Hz). The algorithm identifies the pair of electrodes with the lowest impedance (closest contact) in real time and instantly switches the EDA measurement channel to that pair. This allows the measurement point to dynamically "move" along the inside of the wristband as the wrist rotates, physically avoiding areas of suspension. The PPG sensors are located at the radial and ulnar arteries of the wrist.

[0057] The actuators are physically positioned to strictly avoid the styloid processes of the ulna and radius, conforming only to the soft tissue area of ​​the muscles to avoid the tingling sensation caused by bone conduction. To address the low tactile resolution of the wrist, a 20-50ms time delay is introduced when actuating adjacent actuators to enhance the illusion of movement direction.

[0058] Example 3: In the above examples, a high-precision thermistor can be used to monitor fingertip temperature instead of EDA. Stress-induced vasoconstriction can cause a drop in fingertip temperature ("cold hands and feet"), which can serve as an auxiliary indicator of stress, but its response speed (seconds to minutes) is much slower than EDA (milliseconds), making it only suitable for long-term emotion monitoring.

[0059] In addition, PPG sensors can also use only green LEDs, which will reduce hardware costs, but the signal quality will degrade under low temperature or low perfusion conditions, and it will be difficult to perform blood oxygen calibration.

[0060] Example 4: In the above examples, piezoelectric ceramic actuators, shape memory alloys, or pneumatic bellows can also be used instead of LRAs. Piezoelectric elements are thinner and have a faster response (<1ms), and can simulate richer physical textures (such as the feel of sand or silk), but their driving voltage usually needs to be boosted to 50V-100V, resulting in complex circuit design, high cost, and difficulty in achieving flexibility. Shape memory alloys or pneumatic bellows can be used to generate a "squeezing" sensation rather than a "vibration" sensation, providing a more realistic handshake or pressure feel, but pneumatic systems are bulky, slow to respond, and consume a lot of power, making them unsuitable for miniature devices like rings.

[0061] Example 5: In the above examples, the emotion feature extraction algorithm can also be run directly on the wearable biofeedback and haptic interaction device, outputting only emotion tags. This will significantly reduce Bluetooth bandwidth requirements and protect privacy, but will increase device power consumption.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An embodied interactive intelligent accessory system based on affective computing and haptic feedback, characterized in that, It consists of a wearable biofeedback and haptic interaction device, a mobile computing device, and a cloud device. The wearable biofeedback and haptic interaction device is responsible for collecting data such as electrodermal activity (EDA), photoplethysmography (PPG), and inertial measurement unit (IMU), and executing haptic rendering instructions. The mobile computing device is responsible for Bluetooth communication, signal preprocessing, emotional feature extraction, haptic rendering algorithm calculation, and communication with the cloud device. The cloud device is responsible for running a large language model, receiving emotional vectors, and generating personalized content. The wearable biofeedback and tactile interaction device includes a shell, several rigid motherboards, EDA sensors, PPG sensors, and actuators. The motherboards integrate a microcontroller unit, an inertial measurement unit, EDA sensor front-end circuits, PPG sensor front-end circuits, and a battery. The actuators execute tactile rendering instructions through a tactile rendering algorithm.

2. The embodied interactive intelligent accessory system based on affective computing and haptic feedback according to claim 1, characterized in that, The EDA sensor is a dry electrode made of Ag / AgCl coating or conductive ceramic, and is attached to the pad of the finger or the inside of the wrist; the PPG sensor integrates multi-wavelength LEDs and is attached to the arteries of the finger or wrist.

3. The embodied interactive intelligent accessory system based on affective computing and haptic feedback according to claim 2, characterized in that, The actuator is a spatialized tactile actuation array consisting of at least four miniature linear resonant actuators (LRAs).

4. The embodied interactive intelligent accessory system based on affective computing and haptic feedback according to claim 3, characterized in that, The haptic rendering algorithm specifically involves causing two adjacent actuators to vibrate simultaneously and merging them into a virtual stimulation point located in the middle of the line connecting the two actuators. The position of the virtual stimulation point depends on the ratio of the vibration intensity of the two actuators and is allocated according to the trigonometric function law.

5. The embodied interactive intelligent accessory system based on affective computing and haptic feedback according to claim 4, characterized in that, The actuator vibration uses an asymmetric waveform with rapid rise and slow decay characteristics.

6. The embodied interactive intelligent accessory system based on affective computing and haptic feedback according to claim 1, characterized in that, The wearable biofeedback and tactile interaction device takes the form of a wristband and consists of multiple functional islands and flexible bridges. The functional islands contain several rigid mainboards that avoid the styloid processes of the ulna and radius and only conform to the soft tissue area of ​​the muscles. The flexible bridges are serpentine copper foil wires that connect the islands. The entire structure is encapsulated in skin-friendly silicone or elastic fabric.

7. A personalized interactive intelligent accessory system based on affective computing and haptic feedback according to any one of claims 1-6, characterized in that, The mobile computing device and the cloud device perform emotion computing and generate personalized content through the following methods: Step 1: Using IMU data as a reference, the acceleration signal measured by the IMU is used as a reference noise source. The motion-induced component is subtracted from the original PPG and EDA signals. For the EDA signal, the skin conductance response (SCR) is separated by high-pass filtering, and the skin conductance level (SCL) is separated by low-pass filtering. Step 2: Extract the number and amplitude of SCR peaks per minute from the EDA signal, and extract the successive heartbeat interval and heart rate variability (HRV) from the PPG signal. The heart rate variability (HRV) includes the root mean square difference (RMSSD) and standard deviation normal to normal interval (SDNN) of the time domain indicators, and the low frequency to high frequency ratio (LF / HF) of the frequency domain indicators. Step 3: Construct an emotion coordinate system based on the Russell ring model. The X-axis represents valence (V), determined by the HRV index; the Y-axis represents arousal (A), mainly determined by the SCR frequency and amplitude of EDA. The output is a normalized emotion vector. ; Step 4: Input the emotion vector into the cloud-based large language model to generate personalized content, and trigger feedback through a haptic rendering algorithm.

Images