Interaction control method and device based on spiking neural network, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an interaction control method based on a spiking neural network, which comprises the following steps: acquiring a multi-modal signal, and preprocessing the multi-modal signal to generate a multi-modal data set; inputting the multi-modal data set into a pulse neural network, and outputting a motion execution instruction and a user intention feature; inputting the multi-modal data set into a pulse convolution capsule network, and outputting an emotion classification result; generating a multi-modal feedback strategy according to an emotion classification result and the user intention features through a pulse reinforcement learning engine; and controlling a robot and / or a mechanical arm to execute actions based on the motion execution instruction, and driving a virtual avatar module and a multi-sensory feedback module to execute an emotional operation in combination with a multi-modal feedback strategy. According to the method, EEG signals and robot sensor data are fused through the pulse neural network, emotion recognition, driving virtual avatar and multi-mode interaction are achieved, and high-emotional and low-burden accompanying experience is provided for the disabled and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an interaction control method and device based on a spiking neural network, an equipment and a storage medium. BACKGROUND

[0002] The mainstream BCI system faces multiple technical problems, the first being the balance dilemma between invasiveness and signal quality. Non-invasive EEG technology (such as attention training system) does not require traumatic operation, but has large signal noise and low resolution, making it difficult to capture fine intentions; invasive technology has good signal, but requires craniotomy surgery, which has the risk of infection and rejection; minimally invasive implant technology reduces the risk by implanting through blood vessels, but due to limited bandwidth, it only supports basic clicking and cannot analyze complex emotions or continuous motion intentions.

[0003] In terms of interaction capability, the existing BCI is mostly limited to two-dimensional instruction control, and some can only convert brain signals into "simulated mouse click" instructions, lacking three-dimensional interaction capability; although the AR-related BCI of some research teams has been extended to a three-dimensional scene, it still relies on preset markers and cannot adapt to complex environments, limiting its practicality.

[0004] Signal stability and algorithm adaptability are prominent. Long-term implanted electrodes are prone to signal attenuation due to tissue encapsulation, and LDA, CSP and other decoding algorithms rely on initial calibration data and cannot adapt to changes in neural plasticity online. If the traditional motor imagination classifier is not retrained regularly, the accuracy rate will decrease by more than 20%, affecting reliability.

[0005] Emotion recognition and companion fields have obvious defects. Most BCI related to virtual humans in the metaverse are based on shallow analysis of voice or expressions and are difficult to capture physiological emotions; CNN and other models have weak modeling of EEG signal time sequence characteristics, with an emotion recognition accuracy rate of less than 80% and poor cross-user generalization. Traditional virtual companions rely on visual and audio feedback, lack of tactile and olfactory integration, resulting in "empathy distortion"; some metaverse emotional companions based on neural networks need to pre-host content and cannot perform according to real-time interactions, which is mechanically repetitive in the long term.

[0006] Response delay and power consumption problems affect the experience. Brain-controlled Internet of Things BCI uploads raw EEG data to the cloud, with a round-trip delay of more than 500ms, which destroys the sense of immersion and may cause accidents when a brain-controlled wheelchair needs to avoid obstacles in an emergency. In terms of power consumption, multi-channel EEG caps based on GPU / CPU have a power consumption of watts, making it difficult to support all-weather wear; although FPGA systems optimize energy efficiency, they do not solve the sparsity problem of brain-like computing. Ultra-low-power neuro-morphic chips and SNN interface standards are not standardized, and SNN focuses on back-end processing, while traditional ADC is used in the front end, resulting in low coding efficiency and energy consumption of more than 60%, which restricts the endurance and portability. SUMMARY

[0007] The main purpose of the present application is to provide an interactive control method, device and equipment based on a spiking neural network and a storage medium, aiming to solve the technical bottlenecks of brain-computer interaction and accompanying systems in signal acquisition, emotional computing, interaction dimension, system architecture and personalized accompanying in the prior art.

[0008] To achieve the above-mentioned purpose, the present application provides an interactive control method based on a spiking neural network, comprising: acquiring a multi-modal signal, and preprocessing the multi-modal signal to generate a multi-modal data set; inputting the multi-modal data set into a spiking neural network to output a motion execution instruction and a user intention feature; inputting the multi-modal data set into a spiking convolution capsule network to output an emotional classification result; generating a multi-modal feedback strategy according to the emotional classification result and the user intention feature through a spiking reinforcement learning engine; controlling a robot and / or a mechanical arm to perform an action based on the motion execution instruction, and driving a virtual avatar and a multi-sensory feedback module to perform an empathetic operation in combination with the multi-modal feedback strategy.

[0009] Further, to achieve the above-mentioned purpose, the present application provides an interactive control device based on a spiking neural network, comprising: a signal module for acquiring a multi-modal signal and preprocessing the multi-modal signal to generate a multi-modal data set; an instruction module for inputting the multi-modal data set into a spiking neural network to output a motion execution instruction and a user intention feature; an emotional module for inputting the multi-modal data set into a spiking convolution capsule network to output an emotional classification result; a feedback module for generating a multi-modal feedback strategy according to the emotional classification result and the user intention feature through a spiking reinforcement learning engine; an interaction module for controlling a robot and / or a mechanical arm to perform an action based on the motion execution instruction, and driving a virtual avatar and a multi-sensory feedback module to perform an empathetic operation in combination with the multi-modal feedback strategy.

[0010] Further, to achieve the above-mentioned purpose, the present application also provides a computer device, which comprises a memory, a processor and an interactive control program based on a spiking neural network stored on the memory and executable on the processor, and the steps of the interactive control method based on a spiking neural network are implemented when the interactive control program based on a spiking neural network is executed by the processor.

[0011] Further, to achieve the above object, the present application also provides a computer readable storage medium, wherein the storage medium stores a pulse neural network based interaction control program, and the pulse neural network based interaction control program realizes the steps of the pulse neural network based interaction control method when executed by a processor.

[0012] Beneficial effects: The present application relates to the technical field of artificial intelligence, and discloses a pulse neural network based interaction control method, which comprises the following steps: acquiring a multi-modal signal, preprocessing the multi-modal signal, and generating a multi-modal data set; inputting the multi-modal data set into a pulse neural network, and outputting a motion execution instruction and a user intention feature; inputting the multi-modal data set into a pulse convolution capsule network, and outputting an emotion classification result; generating a multi-modal feedback strategy according to the emotion classification result and the user intention feature through a pulse reinforcement learning engine; controlling a robot and / or a mechanical arm to perform an action based on the motion execution instruction, and driving a virtual avatar to perform an empathetic operation with a multi-sensory feedback module in combination with the multi-modal feedback strategy. The present application fuses EEG signals and robot sensor data through a pulse neural network, realizes emotion recognition, drives a virtual avatar, and realizes multi-modal interaction, and provides a high-empathy and low-burden accompanying experience for the disabled and the like. BRIEF DESCRIPTION OF DRAWINGS

[0013] The present application will be further described below in conjunction with the drawings and embodiments, wherein: Figure 1 An application environment schematic diagram of the pulse neural network based interaction control method in an embodiment of the present application; Figure 2 A flow schematic diagram of the pulse neural network based interaction control method in an embodiment of the present application; Figure 3 A functional module schematic diagram of the pulse neural network based interaction control device in a preferred embodiment of the present application; Figure 4 A structure schematic diagram of a computer device in an embodiment of the present application; Figure 5 Another structure schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0014] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0015] The pulse neural network based interaction control method provided by the embodiments of the present application can be applied to, for example, Figure 1In an application environment of the application, a user terminal communicates with a server terminal through a network. The server terminal can acquire a multi-modal signal through the user terminal, preprocess the multi-modal signal to generate a multi-modal data set, input the multi-modal data set into a spiking neural network to output a motion execution instruction and a user intention feature, input the multi-modal data set into a spiking convolution capsule network to output an emotion classification result, generate a multi-modal feedback strategy according to the emotion classification result and the user intention feature through a spiking reinforcement learning engine, control a robot and / or a mechanical arm to perform an action based on the motion execution instruction, and drive a virtual avatar and a multi-sensory feedback module to perform an empathetic operation in combination with the multi-modal feedback strategy. The application fuses EEG signals and robot sensor data through a spiking neural network to realize emotion recognition, drive a virtual avatar, and multi-modal interaction, and provides a high-empathy and low-burden accompanying experience for the disabled and the like. In addition, a termination condition is used to control a generation process, improve reasoning efficiency, reduce computational overhead, and meet the needs of real-time speech synthesis. The user terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server terminal can be implemented by an independent server or a server cluster composed of multiple servers. The application will be described in detail below through specific embodiments.

[0016] Please refer to Figure 2 , Figure 2 The flowchart of an embodiment of the interactive control method based on the spiking neural network provided by the application is shown in the figure. It should be noted that although a logical sequence is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0017] As Figure 2 shown, the interactive control method based on the spiking neural network provided by the application includes the following steps: S100, acquiring a multi-modal signal and preprocessing the multi-modal signal to generate a multi-modal data set; S200, inputting the multi-modal data set into a spiking neural network to output a motion execution instruction and a user intention feature; S300, inputting the multi-modal data set into a spiking convolution capsule network to output an emotion classification result; S400, generating a multi-modal feedback strategy according to the emotion classification result and the user intention feature through a spiking reinforcement learning engine; S500, controlling a robot and / or a mechanical arm to perform an action based on the motion execution instruction, and driving a virtual avatar and a multi-sensory feedback module to perform an empathetic operation in combination with the multi-modal feedback strategy.

[0018] In this embodiment, in the immersive companion system based on pulse neural network (SNN, Spiking Neural Network, a kind of neural network simulating the pulse emission characteristics of biological neurons) and brain-computer interface (BCI, Brain-Computer Interface, a technology for direct interaction between brain and external device), the final execution action and empathy feedback are obtained from multi-modal signals, which needs to go through five core links of signal acquisition preprocessing, double network feature and result output, feedback strategy generation and execution control. Each link closely relies on system hardware design and algorithm logic to ensure data accuracy and interaction effectiveness.

[0019] First, the multi-modal signal acquisition and preprocessing generate a multi-modal data set. The system collects multi-modal physiological signals through two types of core devices: one is a Stentrode electrode array (a minimally invasive electroencephalogram collection electrode that can be implanted through blood vessels), which has 16 collection contacts with a contact impedance of less than 5kΩ and a spatial resolution of 2.3mm, and is specially designed to collect motor cortex neural signals; the second is a flexible electronic skin, which is used to monitor user hand skin conductance (EDA, Electrodermal Activity, a physiological indicator reflecting human sympathetic nervous activity, which can be associated with emotional changes) signals in real time. During the acquisition process, the two types of devices synchronously acquire signals with a 10ms sampling interval, and assign a unique timestamp to each frame of data, and establish "motor cortex signal-timestamp" and "skin conductance signal-timestamp" mapping tables respectively. The preprocessing stage is divided into two steps: hardware level and digital level. The hardware level is completed by the AFE (Analog Front-End, a front-end circuit responsible for analog signal amplification and filtering) chip, which first removes power frequency interference through a 50Hz notch filter, and then retains the effective signal band through a 0.5-100Hz bandpass filter. The digital level is executed by the edge layer SNN, which uses adaptive wiener filtering to reduce random noise and removes physiological artifacts such as electrooculogram and electromyogram through CCA (Canonical Correlation Analysis, a statistical method for removing signal artifacts). After preprocessing, the system aligns and integrates the two types of signals according to the timestamp, generating a multi-modal data set containing motor cortex signals and skin conductance signals with consistent timing, providing a pure and synchronous data basis for subsequent network input.

[0020] Then the multi-modal data set is input into the spiking neural network (SNN) to output the motion execution instruction and user intention feature. The motor cortex signals in the multi-modal data set are preferentially input into the edge computing layer of the SNN, and the continuous signals are converted into spatial pulse signals (reflecting the pulse distribution in different brain regions, associated with the spatial attributes of the motion) and temporal pulse signals (reflecting the temporal change of the pulse, associated with the dynamic process of the motion) through the biomimetic pulse coding algorithm (based on the membrane potential accumulation-threshold triggering mechanism, the membrane potential exceeding the threshold generates a pulse, and 0 is output if the threshold is not reached). Subsequently, the spatial and temporal convolution kernels (such as 3x1x5 specifications, considering both spatial and temporal dimension feature extraction) of the SNN extract the spatial features (such as the specific brain region pulse activation pattern corresponding to the "grabbing" intention) of the spatial pulse signals and the temporal features (such as the pulse frequency change trend corresponding to the "releasing" intention) of the temporal pulse signals, and then integrate the two types of features through the pulse firing rate pooling processing (statistical pooling window pulse firing rate, reduce the feature dimension) to output the motion intention categories such as "grabbing", "releasing" and "moving". The system combines the environmental information (such as target object coordinates, obstacle distribution) obtained by the robot sensors to convert the motion intention categories into motion execution instructions containing preset paths (such as the motion trajectory from the initial coordinates to the target coordinates) and preset speeds (such as 5 cm / s in the moving stage and 2 cm / s in the grabbing stage). At the same time, the SNN extracts the temporal correlation features of the multi-modal data pulse stream through the pulse LSTM layer (a long short-term memory network suitable for pulse signals, which can capture long-term temporal dependencies), and generates user intention features containing motion intention spatial positioning, temporal dynamics and emotional association information by combining spatial and temporal features. The motion intention recognition accuracy in this process reaches 78.3%, and the edge layer processing delay is less than 50 ms, meeting the real-time interaction requirements.

[0021] Then the multi-modal data set input pulse convolution capsule network, output emotion classification results. The edge layer of the pulse convolution capsule network (SCCapsNet, Spiking Convolutional Capsule Network, a sentiment recognition network integrating the characteristics of pulse neural network and capsule network) first converts the motor cortex signal and skin conductance signal in the multi-modal data set into motor cortex pulse stream (encoding the spatial and temporal characteristics of electroencephalogram activity) and skin conductance pulse stream (encoding the frequency characteristics related to emotional arousal) through the pulse coding algorithm. Subsequently, the leaky integrate-fire (LIF, Leaky Integrate-and-Fire, a pulse neuron model simulating the leakage characteristics of biological neuron membrane potential) neurons in the network extract the spatial and temporal characteristics of the motor cortex pulse stream (such as the decrease in pulse density in the right prefrontal lobe when anxious) and the frequency characteristics of the skin conductance pulse stream (such as EDA pulse frequency exceeding 25Hz when anxious) combined with the spatial and temporal convolution kernel. The extracted features are normalized by the tau-BN mechanism (a normalization method that dynamically adjusts the membrane decay constant of neurons) to eliminate the dimensional difference, and then fused by the pulse fusion layer (motor cortex spatial and temporal feature weight 60%, skin conductance frequency feature weight 40%) to generate emotional features. Finally, the system performs multi-dimensional similarity matching between the emotional features and the preset basic emotion templates (covering core emotions such as joy, anxiety, sadness, etc., labeling the typical feature range of each dimension), calculates the similarity using the Euclidean distance, and selects the emotion with the highest similarity and a confidence level ≥0.7 as the final emotion classification result. The emotion recognition accuracy of this classification method is over 97%, and the cross-user generalization is good.

[0022] Subsequently, a multi-modal feedback strategy is generated by the pulse reinforcement learning engine according to the emotion classification results and user intention features. The pulse reinforcement learning engine (a reinforcement learning model based on pulse signals, which optimizes the feedback strategy through the "state-action-reward" mechanism) first converts the emotion classification results into a multi-dimensional emotion feature vector (such as the weighted feature vector of "motor cortex alpha wave-motor cortex beta wave-skin conductance frequency-skin conductance fluctuation amplitude" corresponding to the anxiety emotion) according to the weight mapping rule, and sorts the user intention features according to the execution priority rule (safety-related intentions are the highest, basic life needs are the second, and interactive experience needs are the lowest) to generate an intention priority list. Next, the time stamp interpolation algorithm (based on the user intention feature timestamp, linear interpolation is used to complete the missing time points of the emotion data) is used to realize the time alignment of the emotion classification results and the user intention features, avoiding time sequence misalignment. Finally, the engine combines the two types of aligned data and the intention priority list to construct a multi-modal feedback strategy decision matrix, and matches the multi-channel feedback mode of the emotion for each intention from high to low priority. For example, the high-priority "grab the coffee machine" intention combined with the anxiety emotion, the strategy covers AR high-light marking target, haptic glove simulating warm feeling, and slowing down the speed of the mechanical arm; the low-priority "adjust the avatar expression" intention combined with the anxiety emotion, the strategy includes a virtual avatar with a gentle smile, a soothing tone of voice output, and adjusting the feedback intensity according to the emotion feature vector value (such as releasing a higher concentration of lavender fragrance in a high anxiety state).

[0023] Finally, the execution control based on the motion execution instruction and the multi-modal feedback strategy. At the motion execution level, the system drives the robot or mechanical arm action according to the motion execution instruction: taking the spatial coordinate sequence and speed parameter in the instruction as the reference, the current position and angle of the mechanical arm are collected in real time through the robot joint sensor, compared with the preset path, and the joint driving torque is corrected when the deviation is more than 3 cm; the speed control is realized through pulse frequency modulation (such as 50 Hz pulse corresponding to 5 cm / s speed in the moving stage, 20 Hz pulse corresponding to 2 cm / s speed in the grabbing stage), while combining with the environmental sensor to dynamically avoid obstacles, ensuring the safety and accuracy of the action, and the action success response rate in the "morning coffee making" scene reaches 96.2%. At the empathic feedback execution level, the system controls two parts of execution based on the multi-modal feedback strategy: one is the virtual avatar micro-expression adjustment, through the Facial Action Coding System (FACS, a system for labeling facial muscle movements) model, sending pulse signals to AU4 (brow muscle) and AU12 (masseter muscle) units to control the contraction intensity and update frequency (such as AU12 contraction intensity 60% and blink frequency 15 times per minute when anxious), and the expression realism reaches 4.7 / 5.0; the second is the multi-sensory feedback module driving, the haptic glove realizes 37℃ warm feeling + 50Hz vibration through 12 temperature control units and vibration motors, the micro-fluidic fragrance chip releases the corresponding concentration of fragrance, the voice module adjusts the speech rate and fundamental frequency, and the environment module adjusts the brightness and background sound effect, to build a full-dimensional empathic experience, and clinical verification shows that the feedback can reduce the Geriatric Depression Scale (GDS) score by 41%, effectively realizing emotional regulation and immersive companionship.

[0024] In one embodiment, the step S100 comprises: S101, collecting the motor cortex signal and the skin conductance signal according to a preset sampling interval, and labeling time stamps for the motor cortex signal and the skin conductance signal; S102, reading the time stamp of the motor cortex signal and the time stamp of the skin conductance signal, and establishing a motor cortex signal-time stamp mapping table and a skin conductance signal-time stamp mapping table; S103, performing hardware-level processing and data signal processing on the motor cortex signal and the skin conductance signal labeled with time stamps, to generate original multi-modal data with time stamps; S104, correcting the time stamp deviation of the skin conductance signal and the motor cortex signal based on the motor cortex signal-time stamp mapping table, according to the original multi-modal data with time stamps and the skin conductance signal-time stamp mapping table, through linear interpolation method, and outputting a multi-modal data set.

[0025] In this embodiment, in the immersive companion system based on SNN and brain-computer interface, the time synchronization of multi-modal signals is the core prerequisite to ensure the accuracy of subsequent emotional recognition, intention analysis and feedback execution, which needs to be realized through three key links of timestamp mapping establishment, signal preprocessing and time deviation correction.

[0026] First, the establishment of the motor cortex signal-time stamp and skin conductance signal-time stamp mapping table. The system uses two types of core acquisition devices to obtain physiological signals: one is the intravascular Stentrode electrode array, which has 16 signal acquisition contacts with a contact impedance of less than 5kΩ and a spatial resolution of 2.3mm, and is specially used for collecting motor cortex neural electrical signals. The acquisition interval is set to 10ms, and for each frame of motor cortex signal collected, the system will automatically assign a unique and accurate timestamp to the frame of data, forming a "motor cortex signal-time stamp" mapping table to ensure that each neural electrical signal data point can be accurately corresponded to the collection time. The second is a flexible electronic skin used to monitor the user's hand skin conductance (EDA) signal in real time. This device also follows the same timestamp generation rules as the Stentrode electrode, and after completing each EDA signal acquisition, the signal data is associated and stored with the corresponding timestamp to construct a "skin conductance signal-time stamp" mapping table. The establishment of the two types of mapping tables lays a data foundation for subsequent signal time synchronization and can effectively trace the collection time sequence of each physiological signal.

[0027] Next, the hardware level and digital signal processing of the two signals after the timestamp are marked to generate original multi-modal data with timestamps. The hardware level processing is mainly completed by the AFE (analog front end) chip at the front end of the system. For the motor cortex signal and skin conductance signal with marked timestamps, first, power frequency filtering is performed to remove power grid interference through a 50Hz notch filter to avoid power frequency noise pollution to physiological signals. Then, band-pass filtering is performed to limit the signal frequency range to 0.5-100Hz, which covers the effective neural electrical activity frequency of the motor cortex signal and the physiological response frequency of the skin conductance signal, and can filter out irrelevant high and low frequency noise. After hardware level processing, the signal enters the edge layer SNN (spiking neural network) for digital signal processing: adaptive wiener filtering is used to further reduce random noise and improve signal-to-noise ratio; and typical correlation analysis (CCA) technology is used to remove physiological artifacts such as electroencephalogram and electromyogram to ensure signal purity. In the entire processing process, the timestamp of each frame of signal is always bound with the data and will not lose the time sequence information due to filtering and noise reduction operations. Finally, original multi-modal data with complete timestamps are generated, including preprocessed motor cortex signals and skin conductance signals.

[0028] Finally, the timestamp deviation correction based on linear interpolation method is performed to output the multi-modal data set. Due to the slight differences in the collection mechanism and device response speed of the motor cortex signal and the skin conductance signal, even if the initial collection interval is consistent, there may still be a slight timestamp deviation (such as signal transmission delay, device sampling trigger time difference). To solve this problem, the system takes the "motor cortex signal-time stamp" mapping table as the reference timing reference, combines the original multi-modal data with timestamps and the "skin conductance signal-time stamp" mapping table, first identifies the timing deviation size and distribution rule of the two types of signals through timestamp comparison; then the linear interpolation method is used to correct the timestamp of the skin conductance signal. For the skin conductance data points with deviation, according to the timing relationship of the adjacent skin conductance signals with accurate timestamps and the corresponding motor cortex signals, a linear interpolation function is constructed to calculate the corrected timestamp of the data point, so that the timestamp of the skin conductance signal is completely aligned with the timestamp of the motor cortex signal. After correction, the time-synchronized motor cortex signal and skin conductance signal are integrated to form the final multi-modal data set, which can be directly used for multi-modal data fusion in the subsequent SNN and input analysis of the emotional decision engine, providing reliable timing consistent data support for the system to realize high-precision intent understanding and emotional interaction.

[0029] In one embodiment, the step S200 comprises: S2011, inputting the motor cortex signal in the multi-modal data set into the edge computing layer of the pulse neural network to convert the motor cortex signal into spatial pulse signals and time pulse signals; S2012, extracting spatial features of the spatial pulse signals through the spatio-temporal convolution kernel of the pulse neural network; S2013, extracting time features of the time pulse signals through the spatio-temporal convolution kernel of the pulse neural network; S2014, performing pooling processing on the spatial features and time features to output the motor intention category; S2015, determining the motor execution instruction according to the environmental information and the motor intention category.

[0030] In this embodiment, in the immersive accompanying system based on SNN and brain-computer interface, the motor cortex signal in the multi-modal data set is converted into a motor execution instruction, which needs to be realized through four core links of edge computing layer pulse conversion, spatio-temporal feature extraction, feature pooling and instruction decision, which closely rely on the characteristics of the pulse neural network (SNN) and the system design logic.

[0031] Firstly, the pulse conversion of motor cortex signals in the edge computing layer. The time-synchronized motor cortex signals in the multi-modal data set are preferentially input into the edge computing layer of the SNN, and the core function of this layer is to convert continuous motor cortex neural electrical signals into spatial and temporal pulse signals that conform to the SNN processing paradigm. The conversion process is based on the bionic pulse coding algorithm designed by the system: the edge computing layer first performs membrane potential accumulation operation on the motor cortex signals, initializes the neuron membrane potential (V_mem), and then adds each motor cortex signal sample and the gain factor according to the signal sampling order. When the membrane potential exceeds the preset threshold (V_th), a pulse signal represented by "1" is generated, and then the membrane potential is reset (V_reset); if it does not reach the threshold, "0" is generated, and finally a discrete pulse sequence is formed. Among them, the spatial pulse signal focuses on the signal differences of different regions of the motor cortex (such as specific brain regions that control hand grasping and releasing actions), and the signal distribution characteristics collected by the multi-channel electrode are converted into the spatial distribution pattern of the pulse, reflecting the spatial correlation information of the motor intention; the temporal pulse signal is based on the timing changes of the motor cortex signal (such as the fluctuation and rhythm changes of the signal intensity over time), and is converted into the timing characteristics such as the time interval and frequency of pulse emission, embodying the temporal dynamic information of the motor intention, both of which together constitute the pulse signal form that can be analyzed by the SNN, and the entire conversion process is completed in the edge computing layer, ensuring that the processing delay is controlled within 50 ms, meeting the real-time interaction requirements.

[0032] Then, the spatial and temporal features are extracted by the SNN spatio-temporal convolution kernel respectively. For the converted spatial pulse signal, the SNN calls a preset spatio-temporal convolution kernel (such as a 3x1x5 spatio-temporal convolution kernel) to extract spatial features. The convolution kernel captures the correlation pattern of the pulse signals in different channels by sliding and scanning the spatial distribution matrix of the spatial pulse signal. For example, when the motor cortex signal corresponds to the "grasp" intention, the pulse emission density of a specific brain region (such as the motor cortex region covered by the C3 / C4 electrode pair) is higher, and the spatio-temporal convolution kernel can enhance the features of this region by weight calculation and suppress the pulse noise of irrelevant regions, finally outputting a feature map reflecting the spatial attributes of the motor intention, such as the specific brain region pulse activation pattern corresponding to the "grasp" action. For the time pulse signal, the SNN also uses the spatio-temporal convolution kernel, but focuses on the timing dimension of the pulse sequence: the convolution kernel scans the emission sequence of the time pulse signal with time steps as the sliding unit, and extracts timing features such as pulse frequency changes and pulse interval rules. For example, the motor cortex signal corresponding to the "release" intention will show a change in pulse frequency that first increases and then decreases over time, and the spatio-temporal convolution kernel can capture this temporal dynamics to generate a feature map reflecting the temporal attributes of the motor intention, providing spatio-temporal double-dimensional feature support for subsequent intention classification.

[0033] Following this is the pooling processing of spatial and temporal features and the output of the motion intent category. After extracting the spatiotemporal features, the SNN performs pulse firing rate pooling on both the spatial and temporal feature maps. This pooling process uses a pooling window (e.g., a 2×2 window) to perform statistical calculations on the pulse feature values ​​within the window (e.g., taking the maximum or average pulse firing rate within the window). This reduces the dimensionality and data volume of the feature maps while preserving key feature information, thus reducing the computational burden on subsequent processing. For example, after pooling, the spatial feature map retains the pulse activation intensity of the core brain regions corresponding to the "grab" intent, while removing weak interference features from peripheral regions. After pooling, the temporal feature map focuses on the pulse change patterns of key time segments corresponding to the "release" intent, filtering out noise signals from irrelevant time points. After pooling, the SNN fuses the pooled spatial and temporal features and inputs them into the subsequent classification decision layer. The classification layer, based on a preset motion intent category (e.g., "grab," "release," "move," etc.), determines the final motion intent category through the activation of spiking neurons. The process achieves a motion intent recognition accuracy of up to 78.3% while consuming only 2.3mW, balancing recognition accuracy with the low energy consumption requirements of edge computing.

[0034] Finally, the system combines environmental information with the motion intent category to determine the motion execution command. After obtaining the motion intent category, the system does not directly generate the execution command, but rather needs to integrate real-time environmental information for decision-making. The environmental information mainly comes from data collected by robot sensors (such as joint angle sensors, vision sensors, etc.), such as the distribution of obstacles in the robot's current space, the position coordinates of the target object (such as a coffee cup), and the current joint state of the robotic arm. Taking the "virtual afternoon tea companionship" scenario as an example, if the motion intent category is "grasp," and the environmental information shows that the target coffee cup is within the reach of the robot's robotic arm (coordinates 120, 75) and there are no obstacles obstructing it, the system determines that the "grasp" intent has safe execution conditions and generates the motion execution command "the robotic arm moves to coordinates 120, 75 and performs the grasping action"; if the environmental information shows that there are obstacles next to the target object, the system will first adjust the robotic arm's movement path and then generate a grasping execution command that includes path planning. In addition, in scenarios such as assisting ALS patients with breakfast, if environmental information indicates that the robotic arm may cause risks by performing actions too fast, the system will also combine the type of motion intention and add speed adjustment parameters to the execution instructions (such as reducing the speed of the robotic arm by 20%) to ensure the safety and accuracy of motion execution and achieve collaborative decision-making that adapts motion intention to the environment.

[0035] In one embodiment, step S200 includes: S2021. The multimodal data set is converted into a multimodal data pulse stream through the edge computing layer of the spiking neural network; S2022. Extract spatial feature maps and temporal feature sequences of multimodal data pulse streams through the spatiotemporal convolutional layer of a spiking neural network; S2023. Extract the multimodal intent feature vector of the multimodal data pulse stream through the pulsed LSTM layer of the spiking neural network; S2024. Generate user intent features based on the spatial feature map, temporal feature sequence, and multimodal intent feature vector.

[0036] In this embodiment, in the immersive companion system based on SNN and brain-computer interface, the transformation of multimodal data sets into user intent features requires the gradual completion of signal conversion and feature extraction by relying on the edge computing layer, spatiotemporal convolutional layer, and spiking LSTM layer of the spiking neural network (SNN). Each link is closely connected and fully adapted to the characteristics of multimodal data.

[0037] The first step is the conversion of the multimodal dataset into a multimodal data pulse stream at the SNN edge computing layer. The multimodal dataset contains time-synchronized motor cortex signals (from the Stentrode electrode) and electrical conductance osmosis (EDA) signals (from flexible electronic skin). These continuous analog or digital physiological signals must first be converted into a pulse sequence that the SNN can process using a biomimetic pulse coding algorithm in the edge computing layer, forming a multimodal data pulse stream. The conversion process follows a membrane potential accumulation and threshold triggering mechanism: the edge computing layer assigns an independent pulse coding channel to each modality signal. For motor cortex signals, weighted accumulation is performed based on the intensity of neural electrical activity in different brain regions (e.g., the signal intensity of the C3 electrode controlling the "grasping" action). When the membrane potential exceeds a threshold, a pulse is generated, converting it into a pulse stream reflecting motor intention. For EDA signals, membrane potential calculations are performed based on the rate of change in EDA values ​​(e.g., signal fluctuations caused by a sudden increase in EDA in an anxious state), generating a pulse stream reflecting emotional state. Meanwhile, the edge computing layer adopts event-driven processing, generating pulses only when there is a valid change in the signal, avoiding redundant data (such as no pulse output during signal stability). The resulting multimodal data pulse stream not only retains the modal specificity of the motor cortex signal and the EDA signal, but also encodes the physiological meaning of the two types of signals through the characteristics of pulse firing frequency, time interval, and spatial distribution. Moreover, the delay of the entire conversion process is controlled within 50ms, which meets the real-time interaction requirements of the system.

[0038] Next, the SNN spatiotemporal convolutional layer extracts the spatial feature map and temporal feature sequence of the multimodal data pulse stream. As the core module of SNN feature extraction, the spatiotemporal convolutional layer extracts spatial and temporal features respectively by using a preset spatiotemporal convolutional kernel (such as a 3×1×5 convolutional kernel) to target the spatiotemporal dual attributes of the multimodal data pulse stream. For the extraction of spatial feature maps, the spatiotemporal convolution kernel slides and scans the multimodal data pulse stream in the dimension of "spatial channel × time step": In the motor cortex signal pulse channel, the convolution kernel captures the differences in pulse firing distribution in the brain regions corresponding to different electrodes (such as C3 / C4, Fp1 / Fp2). For example, under the intention of "grabbing", the pulse density of the C3 electrode is significantly higher than that of other electrodes. The spatial correlation feature is strengthened through convolution operation to generate a feature map reflecting the spatial localization of motor intention. In the EDA signal pulse channel, the convolution kernel focuses on the pulse distribution of different acquisition points of the flexible electronic skin (such as the difference in conductance changes in different areas of the hand), extracts emotion-related spatial features (such as stronger EDA pulses in the palm area of ​​the hand when anxious), and finally integrates the spatial features of the two modalities to form a multimodal spatial feature map. The extraction of time feature sequences focuses on the temporal dynamics of the pulse stream: the spatiotemporal convolution kernel slides with a fixed time step (e.g., 10ms / step) to capture the changing trend of pulse firing frequency (e.g., the pulse frequency of motor cortex signal increases from 20Hz to 80Hz, corresponding to the "action initiation" process) and the regularity of pulse interval (e.g., the pulse interval of EDA signal shortens from 500ms to 100ms, corresponding to increased emotional tension). The temporal features of different time windows are then concatenated in chronological order to generate a multimodal time feature sequence, fully preserving the evolution of the signal over time.

[0039] Next, the SNN-based pulsed LSTM layer extracts the multimodal intent feature vector. Since the temporal correlation between the motor cortex signal and the EDA signal in the multimodal data pulse stream (e.g., "a brief rise in EDA accompanies the initiation of a grasping action") is crucial for user intent parsing, and traditional convolutional layers struggle to capture long-term temporal dependencies, the system introduces a pulsed LSTM layer. The pulsed LSTM layer performs temporal modeling of the multimodal data pulse stream through a gating mechanism (input gate, forget gate, output gate): the input gate filters effective information from the current multimodal pulse (e.g., the "action initiation" feature of the motor cortex pulse and the "emotional stability" feature of the EDA pulse); the forget gate discards irrelevant historical temporal information (e.g., meaningless signal fluctuations from the previous moment); and the output gate combines the current effective information with the retained key historical information to generate a feature vector containing long-term temporal correlations. For example, in the "morning coffee making" scenario, the pulsed LSTM layer can associate the complete temporal chain of "gazing at the coffee machine (visual P300 pulse) → motor cortex 'grasping' pulse initiation → EDA pulse stabilization," recognizing the user's coherent intention to "actively make coffee," rather than an isolated "grasping" action. At the same time, the pulsed LSTM layer adapts to the pulse characteristics of SNN, updating the state only when the gating state changes or a key pulse is input, avoiding the high energy consumption problem of traditional LSTM. The final output multimodal intention feature vector can accurately encode the temporal coordination relationship between multimodal signals, providing core support for subsequent intention integration.

[0040] Finally, user intent features are generated based on spatial feature maps, temporal feature sequences, and multimodal intent feature vectors. The system first performs dimensional alignment and feature fusion on the three types of features: the spatial feature map is flattened into a one-dimensional spatial feature vector, dimensionally matched with the temporal feature sequence (already converted into a one-dimensional temporal vector according to the time step), and then element-wise weighted fusion is performed with the multimodal intent feature vector. During the fusion process, the system assigns weights according to modal importance (e.g., spatial features corresponding to motion cortex signals have higher weights than EDA signal spatial features, and multimodal intent feature vectors have the highest weights because they contain temporal correlation information). After fusion, pulse firing rate pooling further compresses the feature dimensions and removes noise (e.g., filtering weak pulse features in edge regions of the spatial feature map and meaningless temporal fluctuations in the temporal feature sequence), ultimately generating user intent features with simplified dimensions and complete information. This user intent feature not only includes the spatial localization of the motion intent (e.g., "the target to be grasped is the coffee machine") and temporal dynamics (e.g., "the grasping action will start in 100ms"), but also integrates the correlation between emotional state and motion intent (e.g., "the emotion is stable during the action, and no adjustment of the feedback strategy is needed"). It can be directly input into the subsequent emotional decision engine or motion control module, providing key feature support for the system to achieve accurate intent response and emotional interaction. For example, in the ALS patient assistance scenario, this intent feature can ensure that the system accurately recognizes the complete intent of "safely grasping the coffee cup" rather than just executing isolated mechanical actions.

[0041] In one embodiment, step S300 includes: S301. The multimodal data set is input into a pulse convolutional capsule network, and the edge layer of the pulse convolutional capsule network converts the multimodal data into a motor cortex pulse flow and a skin conductance pulse flow through a pulse coding algorithm. S302. Extract the spatiotemporal features of the motor cortex pulse flow and the frequency features of the skin conductance pulse flow through LIF neurons of the pulse convolution capsule network; S303. Normalize the spatiotemporal features and frequency features, and fuse the processed spatiotemporal features and frequency features to generate emotion features; S304. Perform multi-dimensional matching between the emotional features and the preset basic emotional template to determine the emotional classification result.

[0042] In this embodiment, in the immersive companionship system based on SNN and brain-computer interface, the spiking convolutional capsule network (SCCapsNet) is used to analyze the multimodal dataset and determine the user's emotion classification results. This is achieved through four core steps: multimodal spiking transformation, feature extraction, feature normalization fusion, and emotion matching. Each step is closely based on the hardware characteristics and algorithm logic of the system design.

[0043] The first step is the pulse stream conversion of the multimodal dataset at the edge layer of the pulse convolutional capsule network. The time-synchronized motor cortex signals and electrical conductance (EDA) signals in the multimodal dataset are first input into the edge layer of the pulse convolutional capsule network. This layer uses a system-customized biomimetic pulse coding algorithm to convert the two types of continuous physiological signals into motor cortex pulse streams and electrical conductance pulse streams, respectively. The conversion process follows a membrane potential accumulation-threshold triggering mechanism: the limbic layer is configured with an independent pulse coding unit for each modality signal. For motor cortex signals, after initializing the neuronal membrane potential (V_mem), the signal samples are weighted and superimposed with the gain factor at a 10ms sampling interval. When the membrane potential exceeds the preset threshold (V_th), a "1" pulse is generated, and the membrane potential is then reset (V_reset). If the threshold is not reached, "0" is output, ultimately forming a pulse stream that reflects the spatiotemporal distribution of neural electrical activity in the motor cortex. For EDA signals, the membrane potential is calculated based on the rate of change of skin conductance (such as signal fluctuations caused by a sudden increase in EDA during anxiety). When the rate of change of conductance exceeds the set threshold, a pulse is triggered, generating a pulse stream that reflects the physiological response of emotions. Meanwhile, the edge layer adopts event-driven processing, generating pulses only when there is a valid change in the signal, avoiding redundant data (such as no pulse output during signal stabilization). For example, when the motor cortex signal is stable, the pulse output frequency is below 5Hz, while when the intention to move is initiated, the pulse frequency suddenly rises to above 30Hz. The two types of pulse streams formed in the end completely preserve the physiological meaning and temporal characteristics of the original signal, and the conversion delay is controlled within 50ms, meeting the requirements of real-time emotion recognition.

[0044] Next, the LIF neurons in the pulsation convolutional capsule network extract features. The system calls the leak integration-firing (LIF) neurons in the pulsation convolutional capsule network to extract features from the motor cortex pulse flow and the skin conductance pulse flow, respectively. For the motor cortex pulse flow, the LIF neurons combine the network's pre-defined spatiotemporal convolutional kernels (such as 3×1×5 kernels) to perform multi-channel sliding scans of the spatial distribution and temporal sequence of the pulse flow: in the spatial dimension, they capture the differences in pulse firing in brain regions corresponding to different electrodes (such as C3 / C4, Fp1 / Fp2), for example, the pulse density in the right prefrontal cortex (Fp2 electrode) is significantly reduced in an anxious state; in the temporal dimension, they extract the trend of pulse firing frequency changes (such as an increase from 20Hz to 50Hz corresponding to an increase in emotional arousal), and finally integrate them to generate spatiotemporal features that include spatial localization and temporal dynamics. For skin conductance pulse flow, LIF neurons focus on the statistical characteristics of pulse firing frequency: the number of pulse firings is counted within a fixed time window (e.g., a 1-second window), converted into frequency values, and then the frequency variation patterns between different windows are analyzed. For example, the EDA pulse frequency is stable at 10-15Hz in a state of pleasure, while it surges to over 25Hz in a state of anger. Ultimately, frequency features reflecting emotional arousal are extracted. The leakage characteristics of LIF neurons (the membrane potential decays naturally over time) can also simulate the short-term memory effect of biological neurons, which is more in line with the dynamic evolution of physiological signals and improves the accuracy of feature extraction.

[0045] Next comes feature normalization and emotion feature generation. Because the spatiotemporal characteristics of motor cortical pulse flow (e.g., pulse density range 5-50Hz) and the frequency characteristics of skin conductance pulse flow (e.g., frequency range 5-30Hz) differ in numerical range and dimensions, direct fusion would lead to an imbalance in feature weights. Therefore, normalization is necessary first. The system employs a membrane potential normalization algorithm (Tau-BN mechanism) to dynamically adjust the numerical range of the two types of features: for spatiotemporal features, based on the distribution of pulse firing rates, they are mapped to the 0-1 interval, with the formula "Normalized spatiotemporal feature = (Original spatiotemporal feature - Minimum feature value) / (Maximum feature value - Minimum feature value)"; for frequency features, logarithmic compression and mean normalization are used to eliminate extreme value interference, with the formula "Normalized frequency feature = (log(Original frequency feature + 1) - Average logarithmic frequency) / Frequency standard deviation". After normalization, the system performs synergistic fusion of the two types of features through a pulse fusion layer: the fusion process uses spatiotemporal features of the motor cortex as the core (60% weight) and skin conductance frequency features as an auxiliary (40% weight). Through element-level multiplication and accumulation operations, it generates emotional features that contain information related to the correlation between EEG activity and peripheral physiological responses. For example, after fusing "right prefrontal cortex spatiotemporal feature value 0.3 (corresponding to a decrease in alpha wave power) + EDA frequency feature value 0.8 (corresponding to high arousal)", a feature vector representing anxiety is formed. This fusion method can effectively integrate the emotional correlation information of multimodal physiological signals, providing comprehensive support for subsequent classification.

[0046] Finally, the system determines the emotion classification results based on multi-dimensional matching using preset basic emotion templates. The system pre-constructs a basic emotion template library covering core emotions such as pleasure, anxiety, sadness, and anger. Each template includes multi-dimensional feature indicators: motor cortex dimension (e.g., prefrontal alpha wave power, motor cortex beta wave frequency) and skin conductance dimension (e.g., average pulse frequency, frequency fluctuation amplitude). Each dimension indicator is labeled with a typical numerical range for the corresponding emotion (e.g., in the anxiety template, the right prefrontal alpha wave power feature value ≤ 0.4, and the EDA frequency feature value ≥ 0.7). During the emotion matching stage, the system calculates the multi-dimensional similarity between the generated emotion features and each basic emotion template in the template library: using the Euclidean distance formula, the smaller the distance, the higher the similarity. For example, if the Euclidean distance between the emotion feature to be classified and the anxiety template is 0.15, and the distance with the pleasure template is 0.8, then the similarity of the anxiety template (0.87) is significantly higher than that of the pleasure template (0.56). The system selects the emotion corresponding to the template with the highest similarity as the initial classification result, and then verifies it with confidence (requiring a maximum similarity of ≥0.7). If the confidence score is met, the final emotion classification result (e.g., "anxiety") is output; if not, a secondary feature extraction is triggered (increasing the time window length to 3 seconds), and matching is performed again to ensure classification accuracy. This classification method achieves an emotion recognition accuracy of over 97% and exhibits good cross-user generalization, effectively adapting to the physiological signal differences of different users. This provides accurate emotional basis for the system to dynamically adjust the interactive behavior of virtual avatars and generate emotional feedback strategies. For example, after recognizing "anxiety," a feedback instruction of "reducing ambient brightness + releasing lavender fragrance" is triggered.

[0047] In one embodiment, step S400 includes: S401. Convert the emotion classification result into a multi-dimensional emotion feature vector according to the weight mapping rule; S402. Sort the user intent features according to the execution priority rules to generate an intent priority list; S403. Perform data alignment processing on the emotion classification results and user intent features using a timestamp interpolation algorithm; S404. Generate a multimodal feedback strategy based on the aligned emotion classification results, user intent features, emotion feature vectors, and intent priority list.

[0048] In this embodiment, in the immersive companionship system based on SNN and brain-computer interface, the generation of multimodal feedback strategies from emotion classification results and user intention features requires four key steps: multidimensional emotion feature vector transformation, intention priority sorting, data alignment processing, and strategy integration. Each step strictly follows the system's preset rules and technical logic.

[0049] The first step is the conversion of emotion classification results into multidimensional emotion feature vectors based on weighted mapping rules. The system pre-defines weighted mapping rules for different emotion classification results (such as pleasure, anxiety, sadness, and anger). These rules assign differentiated weights to the motor cortex signal dimension and the electrodermal conductance signal dimension based on the importance of the physiological signal features corresponding to each emotion, ensuring that the vectors can accurately quantify the multimodal attributes of emotions. For example, for the emotion classification result "anxiety," the weighted mapping rules stipulate that in the motor cortex dimension, the right prefrontal cortex alpha wave power feature accounts for 35% of the weight (because alpha wave power in this brain region decreases significantly during anxiety), and the motor cortex beta wave frequency feature accounts for 25% of the weight (beta wave frequency increases during anxiety); in the electrodermal conductance dimension, the pulse frequency feature accounts for 25% of the weight (EDA pulse frequency exceeds 25Hz during anxiety), and the frequency fluctuation amplitude feature accounts for 15% of the weight (frequency fluctuations are severe during anxiety). During the conversion process, the system first extracts the original feature values ​​of each dimension corresponding to the emotion classification results (e.g., the right prefrontal cortex alpha wave power feature value of 0.3 and the EDA pulse frequency feature value of 0.8 during anxiety). Then, each feature value is multiplied by its corresponding weight to obtain the weighted value of each dimension. Finally, a multidimensional emotion feature vector is generated by combining the features in a fixed dimensional order of "motor cortex alpha wave - motor cortex beta wave - skin conductance frequency - skin conductance fluctuation amplitude" (e.g., the vector corresponding to anxiety is [0.3×0.35, 0.7×0.25, 0.8×0.25, 0.9×0.15] = [0.105, 0.175, 0.2, 0.135]). This vector not only retains the multimodal feature information of emotion, but also reflects the contribution of different features to the emotion representation through weights, providing a quantitative basis for the subsequent generation of feedback strategies.

[0050] Next, user intent features are sorted and an intent priority list is generated based on execution priority rules. The system's preset execution priority rules are constructed around three core dimensions: user safety, fulfillment of core needs, and naturalness of interaction. Different user intent features correspond to different priority scores: intents directly related to user safety (such as "emergency obstacle avoidance" and "stop dangerous actions") have the highest priority (score 9-10 points), intents that meet basic user needs (such as "grab a water cup" and "start assisted eating") have the next highest priority (score 7-8 points), and non-essential intents that enhance the interactive experience (such as "adjust virtual avatar expressions" and "switch environmental backgrounds") have the lowest priority (score 4-6 points). During the sorting process, the system first extracts the core intent type from the user intent features (such as identifying the core intent "grab a coffee machine" from the intent feature "grab a coffee machine to make coffee"), and then scores each core intent according to the execution priority rules; if there are multiple user intent features (such as the simultaneous existence of the intents "grab a coffee machine" and "adjust the virtual avatar's smile"), they are sorted from high to low priority scores to generate an intent priority list. For example, "grab the coffee machine" (basic living need, score 7.5) has a higher priority than "adjust the virtual avatar's smile" (interaction experience need, score 5.2), and the list order is ["grab the coffee machine" (7.5 points), "adjust the virtual avatar's smile" (5.2 points)]. This list clarifies the order in which the system's execution intentions are prioritized, ensuring that key user needs are responded to first, and avoiding resource waste or delayed fulfillment of core needs.

[0051] Next, the system aligns the emotion classification results with user intent feature data using a timestamp interpolation algorithm. Because the generation cycle of emotion classification results (typically 1-3 seconds, requiring sufficient physiological signal accumulation for judgment) differs from the generation cycle of user intent features (typically 100-200ms, with faster response for motion intent parsing), their timestamps may misalign, requiring correction via timestamp interpolation. The system uses the timestamps of user intent features as a benchmark (due to the greater real-time requirement for motion intent response), first comparing the timestamp sequences of emotion classification results and user intent features to identify missing time segments in the emotion classification results (e.g., user intent features have data at 100ms, 200ms, and 300ms, while emotion classification results only have data at 0ms, 200ms, and 400ms, missing emotion data at 100ms and 300ms). Then, using linear interpolation, the system calculates the emotion feature vector for the missing time point based on the multi-dimensional emotion feature vectors corresponding to the emotion classification results adjacent to the missing time points. For example, the emotion feature vector at 100ms is obtained by interpolating the 0ms vector ([0.1, 0.15, 0.18, 0.12]) and the 200ms vector ([0.105, 0.175, 0.2, 0.135]) according to their time proportions. The formula is "Missing vector = Previous moment vector × (Next moment time - Missing time) / (Next moment time - Previous moment time) + Next moment vector × (Missing time - Previous moment time) / (Next moment time - Previous moment time)". Finally, the interpolated emotion data is matched one-to-one with the user intent features according to the timestamp to complete the data alignment. This ensures that when subsequent strategies are generated, each intent can be matched with the emotional state of the same period, avoiding feedback bias caused by "intent and emotion mismatch".

[0052] Finally, a multimodal feedback strategy is generated based on the aligned data and priority list. The system first integrates the aligned emotion classification results (e.g., "anxiety"), multidimensional emotion feature vectors (e.g., [0.105, 0.175, 0.2, 0.135]), and an intent priority list (e.g., ["grab the coffee machine", "adjust the virtual avatar's expression"]) to construct a feedback strategy decision matrix. The matrix rows correspond to intent priorities, and the columns correspond to emotion regulation dimensions (visual, tactile, olfactory, and vocal). During the decision-making process, the system matches the appropriate feedback method to the emotional state for each intent, from high to low priority: for the high-priority "grab the coffee machine" intent, combined with the "anxiety" emotion, the feedback strategy needs to consider both action assistance and emotion relief. Visually, a highlighted mark is overlaid on the coffee machine using an AR module (reducing the difficulty of user location and alleviating anxiety); tactilely, a tactile glove is controlled to simulate a warm touch (comfortable touch can calm anxiety); in terms of action control, the movement speed of the robotic arm is slowed down (to avoid operational errors caused by anxiety). For the low-priority intent of "adjusting the virtual avatar's expression," and considering anxiety, the feedback strategy is as follows: the virtual avatar's mouth angle is adjusted to 0.3 (a gentle smile, avoiding over-excitement), the blinking frequency is reduced to 15 times per minute (slow blinking conveys calmness), and the voice uses a 10% slower, soothing tone (e.g., "I'm preparing your coffee, don't rush me"). Furthermore, the system adjusts the feedback intensity based on the values ​​of multi-dimensional emotional feature vectors. For example, an EDA frequency feature value of 0.2 (high arousal) corresponds to the fragrance module releasing 50% concentration of lavender fragrance (concentration increases with arousal level), ensuring the feedback accurately matches the emotional intensity. The resulting multimodal feedback strategy encompasses multiple channels including vision, touch, smell, and speech. It not only meets the user's core intention needs but also dynamically regulates emotional state, achieving the dual goals of "intention assistance and emotion relief." For example, in the breakfast assistance scenario for ALS patients, this strategy allows patients to safely complete the eating process while effectively alleviating their anxiety. Validation data showing a 41% decrease in the Geriatric Depression Scale (GDS) score also confirms the emotion regulation effect of this strategy.

[0053] In one embodiment, step S500 includes: S501. Based on the motion execution command, control the robot and / or robotic arm to perform actions according to a preset path and a preset speed; S502: Based on a multimodal feedback strategy, the facial micro-expressions of the virtual avatar are controlled by the facial expression motion unit, and the multisensory feedback module is driven to perform empathic operations.

[0054] In this embodiment, the implementation of motion execution commands and multimodal feedback strategies in the immersive companionship system based on SNN and brain-computer interface needs to be achieved through three core execution links: robot / mechanical arm motion control, virtual avatar micro-expression adjustment, and multi-sensory feedback module drive. Each link strictly relies on the system hardware design and preset rules to ensure operational accuracy and emotional empathy effect.

[0055] Firstly, the system implements motion control for robots / robotic arms based on motion execution commands. The system generates motion execution commands that include the target action type (e.g., grasping, moving, releasing), preset path parameters (e.g., spatial coordinate sequence, path curvature), and preset speed parameters (e.g., constant speed, variable speed adjustment thresholds). During execution, the timestamp and spatial coordinates in the command serve as the core reference, driving the robot or robotic arm to complete the action. Taking the command "grab the coffee machine" in an ALS patient breakfast assistance scenario as an example, the preset path in the command is "move along a straight line from the initial coordinates (0,0) to the coffee machine coordinates (120,75), then perform the grasping action," and the preset speed is "a constant speed of 5cm / s during the moving phase, and decelerating to 2cm / s during the grasping phase." During execution, the robot's joint sensors collect the current angle and position data of the robotic arm in real time (e.g., elbow joint angle, end effector coordinates) and compare it with the preset path in the command. If the actual position of the robotic arm deviates from the preset coordinates by more than 3cm (the system's maximum allowable deviation), the joint driving torque is corrected through the feedback adjustment module of the spiking neural network, adjusting the motion trajectory to the preset path; speed control is achieved through pulse frequency modulation. During the movement phase, a 50Hz pulse signal is sent to the joint motors to maintain a speed of 5cm / s. During the grasping phase, the pulse frequency is reduced to 20Hz to ensure that the robotic arm approaches the target slowly, avoiding collisions or operational errors due to excessive speed. Simultaneously, the system dynamically optimizes the execution process based on environmental information. For example, when environmental sensors detect obstacles on the robotic arm's preset path, path replanning is immediately triggered, generating an obstacle avoidance path and updating speed parameters (e.g., reducing the speed to 3cm / s during obstacle avoidance). This ensures the safety and accuracy of the action execution. Verification data from the "morning coffee making" scenario shows that the robot's action response rate reaches 96.2% under this control method, with no safety incidents occurring.

[0056] Next is the virtual avatar facial micro-expression control based on a multimodal feedback strategy. The system uses a Facial Action Characteristic (FACS) model as its core, transforming the "micro-expression adjustment requirements" in the multimodal feedback strategy into control parameters for specific facial action characters (AUs), enabling dynamic adjustment of the virtual avatar's expressions. The feedback strategy clearly identifies the AU types to be activated (e.g., AU4 corrugator supercilii, AU12 zygomaticus major, AU25 orbicularis oris) and their corresponding contraction intensities (0-100%) and update frequencies (e.g., 50Hz real-time update). During execution, the micro-expression control SNN architecture of the spiking neural network outputs pulse signals to drive the coordinated actions of each AU. For example, when the feedback strategy requires the virtual avatar to present a "gentle smile to alleviate user anxiety," the system sends a control signal to AU12 (zygomaticus major) with a pulse frequency positively correlated with its contraction intensity. The AU12 contraction intensity was set to 60%, corresponding to a pulse frequency of 30Hz, achieving a mouth corner curvature of 0.3 (meeting the standard for a gentle smile). Simultaneously, a 15Hz pulse signal was sent to AU25 (orbicularis oris muscle) to control its slight contraction (intensity 20%), preventing a stiff expression. The blinking frequency was adjusted via the pulse interval of AU43 (orbicularis oculi muscle), set to 15 times / minute (lower than the normal blinking frequency of 20 times / minute), conveying a sense of calm. During control, the system ensured the naturalness of micro-expressions through expression update delay monitoring (requiring a delay <20ms) and AU intensity error calibration (allowing an error <±5%). FACS expert scoring showed that the virtual avatar's expression realism reached 4.7 / 5.0 under this control method, significantly better than the traditional system's 3.2 / 5.0, and there were no mechanical repetition reports during 72 hours of continuous interaction, fully meeting the visual needs for emotional empathy.

[0057] Finally, there's the empathic operation driven by a multi-sensory feedback module based on a multimodal feedback strategy. This module encompasses sub-modules for touch, smell, voice, and environmental adjustment. Based on the "multi-channel empathic needs" outlined in the feedback strategy, the system sends specific control commands to each sub-module, achieving a multi-sensory collaborative empathic experience. For tactile feedback, a tactile glove serves as the core execution device. The feedback strategy specifies the temperature adjustment range (e.g., 36-38℃, simulating the warmth of human skin) and vibration frequency (e.g., a subtle 50Hz vibration, simulating the tactile details of human contact). Execution is achieved through 12 temperature control units (Peltier elements) and a vibration motor built into the glove. For example, a feedback strategy to alleviate anxiety requires "delivering the feel of a warm teacup." The system sends a command to the temperature control unit, setting the temperature to 37℃ (with an error of ±0.5℃), and simultaneously activates the vibration motor to output a 50Hz vibration signal, allowing the user to perceive a realistic "cup-holding" experience through tactile sensation. Olfactory feedback is achieved through a microfluidic fragrance chip. The feedback strategy specifies the fragrance type (e.g., lavender to relieve anxiety) and release concentration (e.g., 50% concentration corresponds to a high anxiety state). The chip generates olfactory signals that meet the user's needs by precisely controlling the mixing ratio and release amount of different fragrance reagents. For example, it releases a 50% concentration of lavender fragrance when anxious and a 30% concentration of citrus fragrance when happy. Voice feedback is achieved through tone and speed adjustment. The feedback strategy is set to "reduce speech speed by 10% and reduce fundamental frequency by 5Hz when anxious." The system calls the speech synthesis module to process the preset voice content (e.g., "I will always be with you, let's complete this slowly") according to the adjustment parameters and output it, ensuring that the emotion conveyed by the voice matches the user's state. The environmental adjustment module adjusts the brightness (e.g., reduce brightness to 70% of normal when anxious) and background sound effects (e.g., play 30-decibel white noise) according to the feedback strategy to create an interactive atmosphere that matches the user's emotions. The synergistic operation of these multi-sensory feedback modules constructs a full-dimensional empathic experience encompassing "visual (micro-expression) + tactile + olfactory + voice + environment." This empathic operation can reduce the Geriatric Depression Scale (GDS) score by 41%, fully validating its emotional regulation effect and providing users with an immersive companionship experience that surpasses traditional visual-voice interaction.

[0058] In one embodiment, an interactive control device based on a spiking neural network is provided, which corresponds one-to-one with the interactive control method based on a spiking neural network described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the interactive control device based on a spiking neural network of the present invention. The modules include a signal module 10, an instruction module 20, an emotion module 30, a feedback module 40, and an interaction module 50. Detailed descriptions of each functional module are as follows: Signal module 10 is used to acquire multimodal signals and preprocess the multimodal signals to generate a multimodal data set; Instruction module 20 is used to input the multimodal data set into the spiking neural network and output motion execution instructions and user intent features; The emotion module 30 is used to input the multimodal data set into the pulse convolutional capsule network and output the emotion classification result; Feedback module 40 is used to generate a multimodal feedback strategy based on the emotion classification results and the user intent features through a pulse reinforcement learning engine; The interaction module 50 is used to control the robot and / or robotic arm to perform actions based on the motion execution instructions, and to drive the virtual avatar and multi-sensory feedback module to perform empathic operations in conjunction with a multimodal feedback strategy.

[0059] In one embodiment, the signal module 10 includes: The signal acquisition unit is used to acquire motor cortex signals and skin conductance signals according to a preset sampling interval, and to mark the motor cortex signals and skin conductance signals with timestamps; The mapping table unit is used to read the timestamps of the motor cortex signal and the skin conductance signal, and to establish the motor cortex signal-timestamp mapping table and the skin conductance signal-timestamp mapping table. The data processing unit is used to perform hardware-level processing and data signal processing on the time-stamped motor cortex signals and skin conductance signals to generate raw multimodal data with timestamps. The multimodal data set unit is used to correct the timestamp deviation between the skin conductance signal and the motor cortex signal based on the motor cortex signal-timestamp mapping table, according to the original multimodal data with timestamps and the skin conductance signal-timestamp mapping table, and output the multimodal data set by linear interpolation.

[0060] In one embodiment, the instruction module 20 includes: The pulse signal unit is used to input the motion cortex signal from the multimodal dataset into the edge computing layer of the spiking neural network, and convert the motion cortex signal into spatial pulse signal and temporal pulse signal; Spatial feature unit, used to extract spatial features of spatial pulse signals through the spatiotemporal convolution kernel of spiking neural network; The temporal feature unit is used to extract the temporal features of the temporal pulse signal through the spatiotemporal convolution kernel of the spiking neural network; A pooling processing unit is used to perform pooling processing on the spatial features and temporal features, and output the motion intention category; The motion execution instruction unit is used to determine the motion execution instruction based on environmental information and the type of motion intent.

[0061] In one embodiment, the instruction module 20 includes: A pulse stream unit is used to convert the multimodal data set into a multimodal data pulse stream through the edge computing layer of a spiking neural network; The spatiotemporal feature unit is used to extract the spatial feature map and temporal feature sequence of multimodal data pulse streams through the spatiotemporal convolutional layer of the spiking neural network; The feature vector unit is used to extract the multimodal intent feature vector of the multimodal data pulse stream through the pulsed LSTM layer of the spiking neural network; The intent feature unit is used to generate user intent features based on the spatial feature map, the temporal feature sequence, and the multimodal intent feature vector.

[0062] In one embodiment, the emotion module 30 includes: The pulse flow unit is used to input the multimodal data set into the pulse convolutional capsule network, and the edge layer of the pulse convolutional capsule network converts the multimodal data into motor cortical pulse flow and skin conduction pulse flow through the pulse coding algorithm; A frequency feature unit is used to extract the spatiotemporal features of the motor cortex pulse flow and the frequency features of the skin conductance pulse flow through LIF neurons of the pulse convolution capsule network; The feature processing unit is used to normalize the spatiotemporal features and frequency features, and then fuse the processed spatiotemporal features and frequency features to generate emotion features. The emotion classification unit is used to perform multi-dimensional matching between the emotion features and the preset basic emotion template to determine the emotion classification result.

[0063] In one embodiment, the feedback module 40 includes: The feature vector unit is used to convert the emotion classification result into a multi-dimensional emotion feature vector according to the weight mapping rule; A priority unit is used to sort the user intent features according to the execution priority rules and generate an intent priority list; The data processing unit is used to perform data alignment processing on the emotion classification results and user intent features using a timestamp interpolation algorithm; The feedback strategy unit is used to generate a multimodal feedback strategy based on the aligned emotion classification results, user intent features, emotion feature vectors, and intent priority lists.

[0064] In one embodiment, the interaction module 50 includes: The motion unit is used to control the robot and / or robotic arm to perform actions according to a preset path and a preset speed based on the motion execution command; The emotion unit is used to control the facial micro-expressions of the virtual avatar through the facial expression motion unit based on the multimodal feedback strategy, and to drive the multi-sensory feedback module to perform empathic operations.

[0065] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side interactive control method based on a spiking neural network.

[0066] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a user-side interactive control method based on a spiking neural network. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire multimodal signals and preprocess the multimodal signals to generate a multimodal data set; The multimodal data set is input into a spiking neural network, which outputs motion execution commands and user intent features. The multimodal dataset is input into a pulsed convolutional capsule network, which outputs the emotion classification result. A multimodal feedback strategy is generated using a spurious reinforcement learning engine based on the emotion classification results and the user intent features. Based on the motion execution commands, the robot and / or robotic arm are controlled to perform actions, and a multimodal feedback strategy is combined to drive the virtual avatar and multisensory feedback module to perform empathic operations.

[0067] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire multimodal signals and preprocess the multimodal signals to generate a multimodal data set; The multimodal data set is input into a spiking neural network, which outputs motion execution commands and user intent features. The multimodal dataset is input into a pulsed convolutional capsule network, which outputs the emotion classification result. A multimodal feedback strategy is generated using a spurious reinforcement learning engine based on the emotion classification results and the user intent features. Based on the motion execution commands, the robot and / or robotic arm are controlled to perform actions, and a multimodal feedback strategy is combined to drive the virtual avatar and multisensory feedback module to perform empathic operations.

[0068] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0071] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An interactive control method based on a spiking neural network, characterized in that, Includes the following steps: Acquire multimodal signals and preprocess the multimodal signals to generate a multimodal data set; The multimodal data set is input into a spiking neural network, which outputs motion execution commands and user intent features. The multimodal dataset is input into a pulsed convolutional capsule network, which outputs the emotion classification result. A multimodal feedback strategy is generated using a spurious reinforcement learning engine based on the emotion classification results and the user intent features. Based on the motion execution commands, the robot and / or robotic arm are controlled to perform actions, and a multimodal feedback strategy is combined to drive the virtual avatar and multisensory feedback module to perform empathic operations.

2. The interactive control method based on a spiking neural network as described in claim 1, characterized in that, The process of acquiring multimodal signals and preprocessing the multimodal signals to generate a multimodal data set includes: Motor cortex signals and skin conductance signals are acquired according to a preset sampling interval, and timestamps are marked on the motor cortex signals and skin conductance signals; Read the timestamps of motor cortex signals and skin conductance signals, and establish a mapping table of motor cortex signals-timestamps and a mapping table of skin conductance signals-timestamps. Hardware-level processing and data signal processing are performed on the time-stamped motor cortex signals and skin conductance signals to generate raw multimodal data with timestamps. Based on the motor cortex signal-timestamp mapping table, the timestamp deviation between the skin conductance signal and the motor cortex signal is corrected by linear interpolation according to the original multimodal data with timestamps and the skin conductance signal-timestamp mapping table, and the multimodal data set is output.

3. The interactive control method based on a spiking neural network as described in claim 1, characterized in that, The step of inputting the multimodal data set into a spiking neural network and outputting motion execution instructions includes: The motion cortex signals from the multimodal dataset are input into the edge computing layer of the spiking neural network, and the motion cortex signals are converted into spatial pulse signals and temporal pulse signals. Spatial features of spatial pulse signals are extracted using spatiotemporal convolution kernels of spiking neural networks; Temporal features of time pulse signals are extracted using spatiotemporal convolution kernels of spiking neural networks; The spatial and temporal features are pooled to output the motion intent category; The movement execution instructions are determined based on environmental information and the type of movement intention.

4. The interactive control method based on a spiking neural network as described in claim 1, characterized in that, The step of inputting the multimodal data set into a spiking neural network and outputting user intent features includes: The multimodal data set is converted into a multimodal data pulse stream through the edge computing layer of a spiking neural network; Spatial feature maps and temporal feature sequences of multimodal data pulse streams are extracted using the spatiotemporal convolutional layers of a spiking neural network. Multimodal intent feature vectors of multimodal data pulse streams are extracted using the pulsed LSTM layer of a spiking neural network. User intent features are generated based on the spatial feature map, temporal feature sequence, and multimodal intent feature vector.

5. The interactive control method based on a spiking neural network as described in claim 1, characterized in that, The step of inputting the multimodal dataset into a pulse convolutional capsule network and outputting emotion classification results includes: The multimodal data set is input into a pulse convolutional capsule network, and the edge layer of the pulse convolutional capsule network converts the multimodal data into motor cortical pulse streams and skin conductance pulse streams through a pulse coding algorithm; The spatiotemporal features of the motor cortex pulse flow and the frequency features of the skin conductance pulse flow are extracted using LIF neurons of the pulse convolution capsule network. The spatiotemporal and frequency features are normalized, and the processed spatiotemporal and frequency features are fused to generate emotion features; The emotional features are matched with a preset basic emotional template in multiple dimensions to determine the emotional classification result.

6. The interactive control method based on a spiking neural network as described in claim 1, characterized in that, The generation of a multimodal feedback strategy using a spurious reinforcement learning engine based on emotion classification results and user intent features includes: The emotion classification results are converted into a multidimensional emotion feature vector according to the weight mapping rule; The user intent features are sorted according to the execution priority rules to generate an intent priority list; The emotion classification results and user intent features are aligned using a timestamp interpolation algorithm. A multimodal feedback strategy is generated based on the aligned emotion classification results, user intent features, emotion feature vectors, and intent priority lists.

7. The interactive control method based on a spiking neural network as described in claim 1, characterized in that, The process of controlling the robot and / or robotic arm to perform actions based on the motion execution commands, and driving the virtual avatar and multi-sensory feedback module to perform empathic operations in conjunction with a multimodal feedback strategy, includes: Based on the motion execution commands, the robot and / or robotic arm are controlled to perform actions according to a preset path and a preset speed; Based on a multimodal feedback strategy, the facial micro-expressions of the virtual avatar are controlled by the facial expression motion unit, and the multi-sensory feedback module is driven to perform empathic operations.

8. An interactive control device based on a spiking neural network, characterized in that, The interactive control device based on a spiking neural network includes: A signal module is used to acquire multimodal signals and preprocess the multimodal signals to generate a multimodal data set; The instruction module is used to input the multimodal data set into the spiking neural network and output motion execution instructions and user intent features; The emotion module is used to input the multimodal dataset into the pulse convolutional capsule network and output the emotion classification result; The feedback module is used to generate a multimodal feedback strategy based on the emotion classification results and the user intent features using a pulse reinforcement learning engine. The interaction module is used to control the robot and / or robotic arm to perform actions based on the motion execution instructions, and to drive the virtual avatar and multi-sensory feedback module to perform empathic operations in conjunction with a multimodal feedback strategy.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and an interactive control program based on a spiking neural network stored in the memory and executable on the processor. When executed by the processor, the interactive control program based on the spiking neural network implements the steps of the interactive control method based on a spiking neural network as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores an interactive control program based on a spiking neural network, which, when executed by a processor, implements the steps of the interactive control method based on a spiking neural network as described in any one of claims 1-7.

Images