A multi-modal brain-like fusion perception system and method

By simulating the collaborative working mode of multiple brain regions in the human brain, a multimodal brain-like fusion perception system has solved the problems of insufficient spatiotemporal alignment accuracy, low degree of neuromorphic fusion, and insufficient robustness in multimodal fusion technology, and has achieved high-precision, low-power real-time multisensory collaborative perception and decision-making.

CN122388949APending Publication Date: 2026-07-14
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610793027.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Filing Date
2026-06-03
Publication Date
2026-07-14

AI Technical Summary

Technical Problem

Existing multimodal fusion technologies suffer from insufficient spatiotemporal alignment accuracy, low neuromorphic fusion degree, weak robustness and adaptability, and difficulty in balancing energy efficiency and real-time performance, thus failing to meet the multi-sensory collaborative perception needs of biomimetic embodied intelligent agents.

Method used

Employing a multimodal brain-like fusion perception system, which simulates the collaborative working mode of multiple brain regions in the human brain, the system achieves precise alignment, neuromorphic fusion, and adaptive decision-making of visual, auditory, and tactile signals through a closed-loop architecture consisting of a perception layer, alignment layer, fusion layer, decision-making layer, and feedback layer. This includes time synchronization, spatial calibration, multi-brain region collaborative fusion, adaptive weight allocation, and feedback optimization.

Benefits of technology

It achieves high-precision spatiotemporal alignment, improves the anthropomorphism of perception and the naturalness of interaction, enhances the robustness and adaptability of the system, reduces power consumption, and meets the real-time interaction needs of the biomimetic embodied intelligent agent.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122388949A_ABST
    Figure CN122388949A_ABST
Patent Text Reader

Abstract

The application discloses a kind of multi-modal brain-like fusion perception systems and methods, it is related to multi-modal sensing signal processing technical field.System uses perception layer synchronous acquisition vision, hearing, tactile biomimetic nerve signal and completes pre-processing;Alignment layer realizes multi-modal signal high-precision space-time alignment and unified feature representation;Fusion layer realizes brain-like collaborative fusion, combines signal confidence dynamic distribution weight, and sets up single mode failure detection and cross-modal compensation mechanism;Decision layer completes task intention recognition and human-like action instruction generation;Feedback layer realizes parameter adaptive optimization and optimal parameter memory storage.The application solves the problems of low precision of traditional multi-modal fusion space-time alignment, poor brain-like mimicry, weak robustness and insufficient real-time performance, and can be widely applied to humanoid robots, rehabilitation bionic devices, industrial bionic manipulators and other bionic embodied agents, with high perception accuracy, strong adaptive ability and good engineering landing performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of multimodal sensing signal processing technology, specifically to a multimodal brain-like fusion sensing system and method. Background Technology

[0002] As biomimetic embodied intelligent agents develop towards "humanization, intelligence, and scenario-based applications," their perception systems need to simulate the core human capabilities of coordinated perception through "seeing, hearing, and touching" in order to adapt to complex and dynamic real-world interactive scenarios. Currently, traditional multimodal fusion perception technologies still have many inherent defects and have not effectively solved the key problems of brain-inspired biomimetic fusion, including: (i) Insufficient spatiotemporal alignment accuracy, unable to meet the needs of brain-like perception. Visual signal acquisition frequencies are typically 30-60Hz, focusing on spatial features; auditory signal acquisition frequencies are typically 16-48kHz, focusing on temporal features; and tactile signal acquisition frequencies are typically 100-1000Hz, focusing on contact features. The frequency differences among these three modalities can reach three orders of magnitude, and the installation positions of the sensors for each modality exhibit spatial offsets. For example, visual sensors are typically placed on the head, auditory sensors on the ears, and tactile sensors on the extremities. Existing technologies often employ simple time interpolation and fixed coordinate mapping, failing to consider the dynamic posture changes of the biomimetic embodied intelligent agent, such as head rotation and limb movement. This results in spatiotemporal alignment errors typically greater than or equal to 50ms, severe signal misalignment, and an inability to simulate the brain-like characteristics of simultaneous perception and real-time response of human multi-sensory systems, thus affecting the accuracy of subsequent fusion and the timeliness of decision-making.

[0003] (ii) The degree of neuromimicry integration is low, and there is a lack of brain-like collaborative mechanisms. Existing multimodal fusion technologies mostly adopt the traditional approach of "feature splicing + weighted summation," which does not simulate the collaborative interaction mechanism of the human brain's visual cortex (V1-V5 areas), auditory cortex (A1 area), and somatosensory cortex (S1-S2 areas). Nor does it introduce the peak temporal characteristics of brain-like neural signals. It only treats the signals of each modality as ordinary data for fusion, lacking the coupling and associative learning capabilities at the neural level. It cannot achieve the brain-like perception effect of "multi-sensory complementarity and redundancy inhibition," resulting in poor anthropomorphism of fused features, rigid interactive behavior of intelligent agents, and significant differences from human perceptual logic.

[0004] (iii) It has weak robustness and adaptability, making it difficult to cope with complex scenarios. Existing systems lack a robust single-modal failure detection and compensation mechanism. When a modal signal fails due to interference—such as visual obstruction, strong noise interference with hearing, or damage to tactile sensors—the system cannot quickly identify the failure state or dynamically adjust the fusion weights of each modality, leading to decision interruption or misjudgment. Furthermore, existing fusion strategies typically use fixed parameters, failing to adapt to environmental changes such as variations in light intensity, noise levels, and contact force. This results in insufficient robustness and difficulty in adapting to diverse and complex scenarios such as home, industrial, and medical settings.

[0005] (iv) Energy efficiency and real-time performance are difficult to balance, limiting engineering applications. Traditional multimodal fusion relies on high-performance computing platforms and employs deep neural networks for feature extraction and fusion. This approach is computationally complex and power-intensive, making it unsuitable for the edge computing needs of bionic embodied intelligent agents, such as portable bionic devices and mobile robots. Furthermore, the cumbersome data transfer process results in signal processing delays typically greater than or equal to 100ms, failing to meet the real-time interaction requirements of intelligent agents, such as grasping, obstacle avoidance, and emotional responses, thus limiting its engineering implementation and large-scale application.

[0006] In summary, existing technologies cannot achieve multimodal signal processing that is "brain-like, precise, adaptive, efficient and low-power". There is an urgent need for a fusion processing system and method that simulates the multisensory collaborative mechanism of the biological brain to solve the above-mentioned technical pain points and improve the perception and decision-making capabilities of bionic embodied intelligent agents. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal brain-like fusion perception system and method that simulates the collaborative working mode of multiple brain regions in the human brain, and achieves precise alignment, neuromorphic fusion, adaptive decision-making, and closed-loop optimization of visual, auditory, and tactile multimodal brain-like bionic neural signals.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A multimodal brain-like fusion perception system includes a perception layer, an alignment layer, a fusion layer, a decision layer, and a feedback layer that form a closed-loop operation. The perception layer is used to collect visual, auditory, and tactile brain-like bionic neural signals, perform preliminary preprocessing, and output original features that conform to the characteristics of brain-like neural signals; the perception layer includes a visual bionic sensor, an auditory bionic sensor, a tactile bionic sensor, and a signal preprocessing unit; The alignment layer is connected to the perception layer and is used to perform time synchronization, spatial calibration and unified representation of the visual, auditory and tactile brain-like bionic neural signals, eliminate spatiotemporal misalignment, and output aligned multimodal brain-like feature vectors; the alignment layer includes a timestamp synchronization unit, a spatial calibration module and a cross-modal feature mapping unit. The fusion layer is connected to the alignment layer and is used to simulate the collaborative interaction mechanism of the human brain's visual cortex, auditory cortex, and somatosensory cortex. It performs feature-level brain-like fusion and decision-level brain-like compensation on the aligned multimodal brain-like feature vector to generate a fused feature vector. The fusion layer includes a multi-brain region collaborative fusion module, an adaptive weight allocation unit, a neural entrainment feature extraction unit, and a single-modal failure detection unit. The decision layer is connected to the fusion layer and is used to simulate the decision-making function of the prefrontal cortex of the human brain. It identifies task intent and generates human-like action decision instructions based on the fusion feature vector. The decision layer includes a deep reinforcement learning decision model, a task intent recognition unit, and an action execution interface. The feedback layer is connected to the fusion layer and the decision layer respectively, and is used to simulate the learning and memory functions of the cerebellum and hippocampus of the human brain. It collects feedback data after the action is executed, dynamically adjusts the weight parameters of the fusion layer and the model parameters of the decision layer, and realizes the adaptive iterative optimization of the system. The feedback layer includes a state feedback acquisition unit, a parameter dynamic adjustment module, and a memory storage unit.

[0009] Furthermore, the perception layer configuration is limited as follows: The visual bionic sensor is an eye-like CMOS camera with a resolution of 1920×1080, a frame rate of 60Hz, a focal length of 3.6mm, and supports RGB-D depth acquisition with a depth measurement range of 0.5-5m. The auditory bionic sensor is an 8-channel ear-like microphone array with a sampling rate of 48kHz, a frequency response range of 20Hz-20kHz, and a signal-to-noise ratio of ≥60dB. The tactile bionic sensor is a 400-point flexible skin-like pressure-temperature sensor array with a sampling rate of 1000Hz, a pressure measurement range of 0-50N, a temperature measurement range of 0-50℃, and a response time of ≤1ms. The signal preprocessing unit integrates Gaussian filtering, wavelet denoising, histogram equalization, and normalization modules to remove signal noise and unify signal scale.

[0010] Furthermore, the alignment layer configuration is limited to: The timestamp synchronization unit adopts hardware clock synchronization and software linear interpolation compensation. Based on the tactile signal, the visual and auditory signals are uniformly interpolated to a sampling frequency of 1000Hz, with a time alignment accuracy of ≤10ms. The spatial calibration module constructs a coordinate system for the intelligent agent based on IMU attitude sensor data, and uses a checkerboard calibration method and laser positioning method to complete the coordinate calibration of each sensor, with a spatial calibration error ≤0.5mm; The cross-modal feature mapping unit is a fully connected neural network with 256 hidden layer nodes and ReLU activation function, which maps the feature vectors of each modality to a 256-dimensional unified brain-like feature space.

[0011] Furthermore, the fusion layer configuration is limited as follows: The multi-brain region collaborative fusion module adopts the peak temporal neural network STDP with a learning rate of 0.001 and a peak firing threshold of 0.8 to simulate the synaptic plasticity and collaborative firing mechanism of neurons in the cerebral cortex. The adaptive weight allocation unit calculates the fusion weight based on the confidence level of each modal signal. The weight adjustment range is 0-1, the response time is ≤5ms, and the confidence level is calculated from the signal-to-noise ratio and feature variance. The single-mode failure detection unit sets the feature confidence threshold to 0.6. When the feature confidence of a certain mode is <0.6, it is determined that the mode is failed, and the compensation mechanism is automatically activated to reduce the weight of the failed mode to below 0.1 and increase the weight of other modes.

[0012] Furthermore, the decision-making layer configuration is limited as follows: The deep reinforcement learning decision model is the DDPG model. The Actor network takes a 256-dimensional fused feature vector as input and outputs a 6-dimensional action instruction. The Critic network evaluates the action reward, and the reward function is R = 0.6 × task completion degree - 0.3 × action energy consumption + 0.1 × interaction naturalness. The task intent recognition unit uses a CNN-LSTM hybrid model, with a task recognition accuracy of ≥95%. The action execution interface supports CAN bus communication, outputs voltage 0-24V, current 0-5A, and signal delay ≤10ms.

[0013] Furthermore, the feedback layer configuration is limited as follows: The status feedback acquisition unit integrates environmental sensors and motion posture sensors, with a sampling rate of 100Hz, and is used to collect environmental status, task completion status, and action execution effect. The parameter dynamic adjustment module uses the gradient descent algorithm with a learning rate of 0.0005 and an iteration period of 100ms to optimize the weights of the fusion layer and the model parameters of the decision layer. The memory storage unit has a capacity of ≥16GB and supports real-time storage and retrieval of optimal parameters, simulating the memory function of the hippocampus.

[0014] A multimodal brain-inspired fusion sensing method, applied to the aforementioned multimodal brain-inspired fusion sensing system, includes the following steps: S1: The perception layer synchronously collects visual, auditory, and tactile brain-like bionic neural signals, performs noise reduction, filtering, and normalization preprocessing on the original signals of each modality, and outputs visual, auditory, and tactile brain-like feature vectors. S2: The alignment layer uses tactile signals as a reference, eliminates the spatiotemporal misalignment of multimodal signals through time synchronization and spatial calibration, and then maps the feature vectors of each modality to a unified feature space through cross-modal feature mapping, outputting the aligned multimodal brain-like feature vectors. S3: The fusion layer learns the neural coupling relationship of multimodal signals through the multi-brain region collaborative fusion module, extracts neural entrainment feature vectors, and then dynamically adjusts the fusion weights of each modality through the adaptive weight allocation unit to achieve feature-level brain-like fusion; at the same time, the single-modal failure detection unit monitors the signal quality of each modality. When a single-modal failure is detected, the decision-level brain-like compensation mechanism is activated and the fusion feature vector is output. S4: The decision layer identifies task requirements based on the fused feature vector through the task intent recognition unit, and then generates human-like action decision instructions through the deep reinforcement learning decision model, which are then output to the execution mechanism of the bionic embodied intelligent agent through the action execution interface. S5: The feedback layer collects feedback data such as the environmental state and task completion status after the action is executed. It optimizes the weights of the fusion layer and the model parameters of the decision layer through the parameter dynamic adjustment module, and stores the optimal parameters in the memory storage unit to realize the adaptive iterative optimization of the system. S6: Repeat steps S1-S5 to achieve real-time fusion and decision optimization of multimodal brain-like bionic neural signals, ensuring accurate perception and natural interaction of the bionic embodied intelligent agent.

[0015] Furthermore, the preprocessing steps in step S1 include the following specific procedures: After the visual signal is filtered by Gaussian with a standard deviation of σ=1.5, histogram equalization, and Zhang's distortion correction, 128-dimensional visual features are extracted by brain-like CNN. After the auditory signal is decomposed and denoised by 5 layers of db4 wavelet and converted by STFT time-frequency, 64-dimensional auditory features are extracted by LSTM. After the tactile signal is low-pass filtered at 100Hz and min-max normalized, 32-dimensional tactile features are extracted by a causal convolutional network.

[0016] Furthermore, the specific process of spatiotemporal alignment in step S2 includes: The time synchronization includes hardware clock synchronization and software delay compensation. When the posture of the bionic avatar changes, the signal delay of each sensor is calculated in real time, and the delay is eliminated by a time offset compensation algorithm. The spatial calibration includes coordinate system construction, coordinate calibration and dynamic calibration. When the posture of the avatar changes, the position coordinates of each sensor are updated in real time, and spatial alignment is achieved by coordinate transformation.

[0017] Furthermore, the adaptive iterative optimization in step S5 includes parameter adjustment, memory storage, and parameter retrieval. When encountering the same scenario again, the optimal parameters in the memory storage unit are directly called to improve processing efficiency.

[0018] Compared with existing technologies, this multimodal brain-inspired fusion perception system and method integrates visual, auditory, and tactile brain-inspired bionic neural signals, simulating the collaborative working mechanism of multiple brain regions in a biological brain. This enables precise perception, adaptive decision-making, and natural interaction in bionic embodied intelligent agents. The system and method can be widely applied to environmental perception and intelligent control in various embodied intelligent equipment such as bionic humanoid robots, bionic robotic arms, medical rehabilitation bionic equipment, industrial bionic manipulators, and home service bionics. The specific beneficial effects of this invention include: 1. High-precision spatiotemporal alignment: This invention adopts hardware and software joint time synchronization based on high-frequency tactile signals, combined with real-time dynamic spatial calibration of IMU posture. The time alignment accuracy is ≤10ms and the spatial calibration error is ≤0.5mm. It effectively solves the signal misalignment problem caused by frequency differences of multimodal sensors, installation offset and intelligent agent posture changes, which is in line with the brain-like characteristics of human multi-sensory synchronous perception.

[0019] 2. Highly realistic brain-like neuromorphic fusion: This invention replicates the multi-cortical collaborative mechanism of the brain's vision, hearing, and somatosensory senses. It introduces STDP peak temporal neural network to simulate neuronal synaptic plasticity and coordinated firing, and combines neural entrainment feature extraction to achieve true neural-level multimodal fusion. It abandons the traditional fixed weighting method and dynamically allocates fusion weights based on signal confidence, realizing multi-sensory complementarity and redundancy suppression, which greatly improves the anthropomorphism of perception and the naturalness of interaction.

[0020] 3. Strong robustness and failure compensation capability: This invention sets a single-modal failure detection threshold, which can identify failure scenarios such as visual occlusion, auditory noise interference, and tactile sensory abnormalities in real time. It automatically reduces the weight of the failure mode and increases the contribution of other modes to achieve brain-like sensory compensation, avoid decision-making misjudgment and task interruption, and significantly improve the adaptability to complex scenarios.

[0021] 4. Adaptive learning and closed-loop iterative optimization: Relying on the feedback layer to simulate the cerebellum parameter regulation and hippocampal memory mechanism, the system collects environmental and task feedback data in real time, continuously optimizes the fusion weights and decision model parameters through the gradient descent algorithm, and stores the optimal scene parameters. The system can be directly called in the same scene in the future, realizing the adaptive evolution capability of the system to become better with use.

[0022] 5. It features strong real-time performance and good engineering feasibility. The overall five-layer closed-loop architecture has a streamlined link, controllable processing timing of each module, and low overall signal processing latency. The hardware uses general-purpose, readily available, and mature components, and the software algorithm parameters are clear and programmable. It takes into account both low power consumption and real-time edge operation requirements, and can be quickly deployed in various humanoid robots, rehabilitation bionic devices, industrial bionic manipulators, and other body-bound intelligent equipment. Attached Figure Description

[0023] Figure 1 This is a block diagram of the overall system architecture of the present invention; Figure 2 This is a schematic diagram illustrating the principle of the multi-brain region collaborative fusion module of the present invention; Figure 3 This is a schematic diagram of the spatiotemporal alignment process for multimodal signals in this invention; Figure 4 This is a schematic diagram of the single-mode failure compensation mechanism of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see Figures 1-4 The present invention provides the following technical solutions: This embodiment provides a multimodal brain-like fusion perception system and method, which is applied to bionic embodied intelligent agents, including humanoid service robots, bionic robotic arms, medical rehabilitation bionic prostheses, industrial inspection bionic robots and other carriers. The system adopts a five-layer closed-loop brain-like architecture consisting of a perception layer, an alignment layer, a fusion layer, a decision layer and a feedback layer.

[0026] 1. System Implementation 1.1 Perception Layer The perception layer hardware includes a visual bionic sensor, an auditory bionic sensor, a tactile bionic sensor, and a signal preprocessing unit. The visual bionic sensor uses an eye-like CMOS camera with a resolution of 1920×1080, a frame rate of 60Hz, and a focal length of 3.6mm. It supports RGB-D depth acquisition, with a depth measurement range of 0.5–5m, and simulates the imaging and perception mechanism of the human eye's retina. The auditory bionic sensor uses a WM8978 8-channel microphone array with a sampling rate of 48kHz, a frequency response range of 20Hz–20kHz, and a signal-to-noise ratio of ≥60dB, simulating the frequency division and sound field perception characteristics of the human cochlea. The tactile bionic sensor uses a FlexiForce 400-point flexible skin-like pressure-temperature sensor array with a sampling rate of 1000Hz, a pressure measurement range of 0–50N, a temperature measurement range of 0–50℃, and a response time of ≤1ms, which conforms to the sensory characteristics of human skin tactile receptors. The signal preprocessing unit uses a high-performance microcontroller with built-in Gaussian filtering, wavelet denoising, histogram equalization, and normalization modules to simultaneously perform denoising, enhancement, and scale normalization preprocessing on the three-channel original signals.

[0027] 1.2 Alignment Layer The alignment layer consists of a timestamp synchronization unit, a spatial calibration module, and a cross-modal feature mapping unit. The timestamp synchronization unit is equipped with a hardware clock synchronization chip and a software linear interpolation compensation algorithm. Using the 1000Hz tactile signal as a reference, the 60Hz visual signal and the 48kHz auditory signal are uniformly interpolated to the 1000Hz timing reference to achieve a time alignment accuracy of ≤10ms. The spatial calibration module integrates an IMU attitude sensor and constructs a unified body coordinate system based on the skeletal structure of a bionic embodied intelligent body. It uses a checkerboard calibration method to complete the calibration of the visual sensor and a laser positioning method to complete the coordinate calibration of the microphone array and tactile sensor array. It dynamically updates the coordinate transformation matrix in combination with the real-time attitude changes of the intelligent body, and the spatial calibration error is controlled within ≤0.5mm. The cross-modal feature mapping unit adopts an embedded fully connected neural network with 256 hidden layer nodes and ReLU activation function. It maps 128-dimensional visual, 64-dimensional auditory, and 32-dimensional tactile features to a 256-dimensional brain-like feature space, thus completing the unified representation of multimodal features.

[0028] 1.3 Fusion Layer The fusion layer deploys a multi-brain region collaborative fusion module, an adaptive weight allocation unit, a neural entrainment feature extraction unit, and a single-modal failure detection unit. The multi-brain region collaborative fusion module is equipped with a neuromorphic computing unit, runs a peak-sequential neural network STDP, and is configured with a learning rate of 0.001 and a peak firing threshold of 0.8 to simulate the synaptic plasticity and collaborative firing mechanism of neurons in the visual cortex, auditory cortex and somatosensory cortex of the brain. The adaptive weight allocation unit embeds the weight calculation logic, calculates the feature confidence based on the signal-to-noise ratio and feature variance of each modal signal, and allocates the fusion weight in a confidence normalization manner. The weight adjustment range is 0–1 and the response time is ≤5ms, realizing multimodal dynamic complementarity. Let the confidence level of the visual signal be Cv, the confidence level of the auditory signal be Ca, and the confidence level of the tactile signal be Ct, and the modal weights be Wv, Wa, and Wt, respectively. Then: Wv=Cv / (Cv+Ca+Ct), Wa=Ca / (Cv+Ca+Ct), Wt=Ct / (Cv+Ca+Ct) The single-modal failure detection unit has a preset feature confidence threshold of 0.6 and monitors the feature confidence of three modalities in real time. If the confidence of any modality is lower than 0.6, it is determined to be a signal failure and automatically triggers the compensation logic to reduce the weight of the failed modality to below 0.1 and simultaneously increase the weight of the other effective modalities to achieve brain-like sensory compensation.

[0029] 1.4 Decision-making level The decision layer includes a deep reinforcement learning decision model, a task intent recognition unit, and an action execution interface. The task intent recognition unit adopts a CNN-LSTM hybrid model hardware deployment and performs spatial and temporal joint feature analysis on the 256-dimensional fused features. The task intent recognition accuracy is ≥95%, and it can accurately identify task types such as environmental detection, object grasping, obstacle avoidance, human-computer interaction, and rehabilitation action guidance. The deep reinforcement learning decision model adopts the DDPG architecture. The Actor network takes 256-dimensional fused features as input and outputs 6-dimensional joint action commands. The Critic network is responsible for action reward evaluation and has a fixed reward function: R = 0.6 × task completion degree − 0.3 × action energy consumption + 0.1 × interaction naturalness, which takes into account task completion efficiency, motion energy consumption and human-like interaction fluency. The motion execution interface adopts a CAN bus communication module, supports 0–24V voltage and 0–5A current output, and has a command transmission delay of ≤10ms. It can be directly connected to the actuators such as servos, DC motors, and joint actuators of bionic intelligent bodies.

[0030] 1.5 Feedback Layer The feedback layer includes a status feedback acquisition unit, a parameter dynamic adjustment module, and a memory storage unit. The status feedback acquisition unit integrates an environmental temperature and humidity sensor, an attitude sensor, and a contact force sensor. The sampling rate is fixed at 100Hz, and it collects environmental status, task completion status, smoothness of action execution, and contact force feedback data in real time. The parameter dynamic adjustment module has a built-in gradient descent optimization algorithm with a learning rate of 0.0005 and an iteration period of 100ms. It dynamically iterates and optimizes the adaptive weights of the fusion layer, the STDP network parameters, and the DDPG model parameters of the decision layer based on feedback data. The memory storage unit uses a flash memory module with a capacity of ≥16GB to store the optimal fusion weights, network model parameters and decision-making strategies in different scenarios in real time. When encountering the same application scenario, the system can directly call the historical optimal parameters to improve system response efficiency and scenario adaptation speed.

[0031] 2. Specific Implementation of System Software and Algorithms 2.1 Implementation of the Preprocessing Algorithm for the Perception Layer Visual signals: Gaussian filtering with standard deviation σ=1.5 is used for noise reduction, and histogram equalization is used to enhance illumination robustness. Lens distortion is corrected by Zhang's calibration method, and then 128-dimensional visual brain-like feature vectors are extracted through brain-like CNN network layers. Auditory signals: 5-level decomposition wavelet noise reduction is performed using the db4 wavelet basis, and time-domain to time-frequency domain feature conversion is completed through short-time Fourier transform (STFT). Then, 64-dimensional auditory brain-like feature vectors are output through time-series feature learning via LSTM network. Tactile signal: A 100Hz cutoff frequency low-pass filter is used to filter out high-frequency interference. The signal range scale is unified by min-max normalization. Spatial contact features are extracted using a causal convolutional network, and a 32-dimensional tactile brain-like feature vector is output.

[0032] 2.2 Implementation of the Spatiotemporal Alignment Algorithm for the Alignment Layer Time synchronization uses a 1000Hz tactile signal as a unified timing reference. Through hardware clock stamp binding and software linear interpolation, the timing of signals of different frequencies is aligned. When the bionic android undergoes posture changes such as head rotation, limb swinging, bending and turning, the transmission delay is calculated using real-time posture data from the IMU. The time offset compensation algorithm is used to dynamically eliminate the time delay error, ensuring that the time alignment accuracy is ≤10ms. Spatial alignment first establishes a body coordinate system with the center of the intelligent body's torso as the origin, completes offline calibration of each sensor to obtain the coordinate transformation matrix, and updates the rotation matrix and translation matrix in real time according to the posture changes during operation, and maps the head vision, ear hearing and limb end tactile signals to the body coordinate system in a unified manner to achieve a dynamic spatial calibration error ≤0.5mm.

[0033] 2.3 Implementation of Brain-Inspired Fusion and Failure Compensation at the Fusion Layer The aligned 256-dimensional unified features are input into the multi-brain region collaborative fusion module. The STDP neural network is used to learn the multimodal neural coupling associations and extract neural entrainment features matching the 10–40Hz physiological rhythm. Then, the weights are calculated and assigned based on the real-time confidence of each modality to achieve feature-level adaptive brain-like fusion. The system polls the confidence levels of each modality feature in real time. Once a single-modal failure scenario such as visual occlusion, strong environmental noise interference, or tactile sensor failure is detected, decision-level brain-like compensation is immediately initiated to reduce the weight of the failed modality and strengthen the perception contribution of other modalities, ensuring that the fused features remain continuous and do not fail, thus maintaining decision stability.

[0034] 2.4 Human-like decision generation and implementation at the decision-making level By relying on CNN-LSTM units to analyze and fuse features corresponding to task intent, and then using the DDPG deep reinforcement learning model to generate 6-dimensional joint action commands that conform to human behavioral logic, the task completion degree, motion energy consumption and interaction naturalness are constrained by the reward function. The output commands are sent to the actuator in real time via CAN bus, realizing the bionic intelligent agent's precise grasping, flexible obstacle avoidance and natural human-computer interaction.

[0035] 2.5 Implementation of Adaptive Iterative Learning in the Feedback Layer The feedback acquisition unit continuously samples environmental, action, and task result data. The parameter dynamic adjustment module iterates every 100ms, using a gradient descent algorithm to fine-tune the fusion weights and decision model parameters. The optimized parameters are stored in the memory storage unit in real time. Subsequent similar scenarios do not require re-iteration; the historical optimal parameters can be directly called, realizing a closed-loop brain-like learning mechanism that reuses hippocampal memory and continuously optimizes the cerebellum.

[0036] 3. Detailed Implementation of the Overall Work Process The workflow of this invention strictly follows steps S1 to S6 in a cyclical manner: S1: The perception layer synchronously collects visual, auditory, and tactile brain-like bionic neural signals, performs noise reduction, filtering, and normalization preprocessing on the original signals of each modality, and outputs visual, auditory, and tactile brain-like feature vectors. S2: The alignment layer uses tactile signals as a reference, eliminates the spatiotemporal misalignment of multimodal signals through time synchronization and spatial calibration, and then maps the feature vectors of each modality to a unified feature space through cross-modal feature mapping, outputting the aligned multimodal brain-like feature vectors. S3: The fusion layer learns the neural coupling relationship of multimodal signals through the multi-brain region collaborative fusion module, extracts neural entrainment feature vectors, and then dynamically adjusts the fusion weights of each modality through the adaptive weight allocation unit to achieve feature-level brain-like fusion; at the same time, the single-modal failure detection unit monitors the signal quality of each modality. When a single-modal failure is detected, the decision-level brain-like compensation mechanism is activated and the fusion feature vector is output. S4: The decision layer identifies task requirements based on the fused feature vector through the task intent recognition unit, and then generates human-like action decision instructions through the deep reinforcement learning decision model, which are then output to the execution mechanism of the bionic embodied intelligent agent through the action execution interface. S5: The feedback layer collects feedback data such as the environmental state and task completion status after the action is executed. It optimizes the weights of the fusion layer and the model parameters of the decision layer through the parameter dynamic adjustment module, and stores the optimal parameters in the memory storage unit to realize the adaptive iterative optimization of the system. S6: Repeat steps S1-S5 to achieve real-time fusion and decision optimization of multimodal brain-like bionic neural signals, ensuring accurate perception and natural interaction of the bionic embodied intelligent agent.

[0037] First, the perception layer synchronously acquires three brain-like bionic neural signals and performs noise reduction, filtering, and feature extraction. Then, the alignment layer completes time synchronization, spatial calibration, and unified feature representation. Subsequently, the fusion layer performs multi-brain region collaborative brain-like fusion, dynamic weight allocation, and single-modal failure compensation. The decision layer completes task intent recognition and human-like action command generation and output. Finally, the feedback layer collects operational feedback data, optimizes parameters online, and stores the data. The system continuously cycles through the above process to achieve full-time, adaptive, and brain-like multimodal fusion perception and intelligent decision control of the bionic embodied intelligent agent.

[0038] 4. Specific Implementation Scenarios The system in this embodiment can be directly implemented and applied to: 1) Home / commercial bionic humanoid service robots that enable environmental perception, human-machine voice and visual interaction, and flexible object grasping; 2) In the field of medical rehabilitation, bionic prostheses and rehabilitation training robots replicate the human body's visual, auditory, and tactile sensory processing and limb movement logic; 3) Bionic manipulators and inspection robots for industrial applications enable interference-resistant perception and precise operation under complex working conditions; 4) Special operation bionic intelligent agents rely on multimodal compensation to maintain reliable perception and decision-making in high-risk, occluded, and high-noise scenarios.

[0039] Contents not described in detail in this specification are prior art known to those skilled in the art. Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal brain-like fusion sensing system, characterized in that, This includes the perception layer, alignment layer, fusion layer, decision-making layer, and feedback layer that form a closed-loop operation. The perception layer is used to collect visual, auditory, and tactile brain-like bionic neural signals, perform preliminary preprocessing, and output original features that conform to the characteristics of brain-like neural signals; the perception layer includes a visual bionic sensor, an auditory bionic sensor, a tactile bionic sensor, and a signal preprocessing unit; The alignment layer is connected to the perception layer and is used to perform time synchronization, spatial calibration and unified representation of the visual, auditory and tactile brain-like bionic neural signals, eliminate spatiotemporal misalignment, and output the aligned multimodal brain-like feature vector. The alignment layer includes a timestamp synchronization unit, a spatial calibration module, and a cross-modal feature mapping unit; The fusion layer is connected to the alignment layer and is used to simulate the collaborative interaction mechanism of the human brain's visual cortex, auditory cortex, and somatosensory cortex. It performs feature-level brain-like fusion and decision-level brain-like compensation on the aligned multimodal brain-like feature vector to generate a fused feature vector. The fusion layer includes a multi-brain region collaborative fusion module, an adaptive weight allocation unit, a neural entrainment feature extraction unit, and a single-modal failure detection unit; The decision layer is connected to the fusion layer and is used to simulate the decision-making function of the prefrontal cortex of the human brain. It identifies task intent and generates human-like action decision instructions based on the fusion feature vector. The decision layer includes a deep reinforcement learning decision model, a task intent recognition unit, and an action execution interface. The feedback layer is connected to the fusion layer and the decision layer respectively, and is used to simulate the learning and memory functions of the cerebellum and hippocampus of the human brain. It collects feedback data after the action is executed, dynamically adjusts the weight parameters of the fusion layer and the model parameters of the decision layer, and realizes the adaptive iterative optimization of the system. The feedback layer includes a state feedback acquisition unit, a parameter dynamic adjustment module, and a memory storage unit.

2. A multimodal brain-like fusion perception method based on the system described in claim 1, characterized in that, Includes the following steps: S1: The perception layer synchronously collects visual, auditory, and tactile brain-like bionic neural signals, performs noise reduction, filtering, and normalization preprocessing on the original signals of each modality, and outputs visual, auditory, and tactile brain-like feature vectors. S2: The alignment layer uses tactile signals as a reference, eliminates the spatiotemporal misalignment of multimodal signals through time synchronization and spatial calibration, and then maps the feature vectors of each modality to a unified feature space through cross-modal feature mapping, outputting the aligned multimodal brain-like feature vectors. S3: The fusion layer learns the neural coupling relationship of multimodal signals through the multi-brain region collaborative fusion module, extracts neural entrainment feature vectors, and then dynamically adjusts the fusion weights of each modality through the adaptive weight allocation unit to achieve feature-level brain-like fusion; at the same time, the single-modal failure detection unit monitors the signal quality of each modality. When a single-modal failure is detected, the decision-level brain-like compensation mechanism is activated and the fusion feature vector is output. S4: The decision layer identifies task requirements based on the fused feature vector through the task intent recognition unit, and then generates human-like action decision instructions through the deep reinforcement learning decision model, which are then output to the execution mechanism of the bionic embodied intelligent agent through the action execution interface. S5: The feedback layer collects feedback data such as the environmental state and task completion status after the action is executed. It optimizes the weights of the fusion layer and the model parameters of the decision layer through the parameter dynamic adjustment module, and stores the optimal parameters in the memory storage unit to realize the adaptive iterative optimization of the system. S6: Repeat steps S1-S5 to achieve real-time fusion and decision optimization of multimodal brain-like bionic neural signals, ensuring accurate perception and natural interaction of the bionic embodied intelligent agent.

3. The multimodal brain-like fusion sensing system according to claim 1, characterized in that, In the perception layer: The visual bionic sensor is an eye-like CMOS camera with a resolution of 1920×1080, a frame rate of 60Hz, a focal length of 3.6mm, and supports RGB-D depth acquisition with a depth measurement range of 0.5-5m. The auditory bionic sensor is an 8-channel ear-like microphone array with a sampling rate of 48kHz, a frequency response range of 20Hz-20kHz, and a signal-to-noise ratio of ≥60dB. The tactile bionic sensor is a 400-point flexible skin-like pressure-temperature sensor array with a sampling rate of 1000Hz, a pressure measurement range of 0-50N, a temperature measurement range of 0-50℃, and a response time of ≤1ms. The signal preprocessing unit integrates Gaussian filtering, wavelet denoising, histogram equalization, and normalization modules to remove signal noise and unify signal scale.

4. The multimodal brain-like fusion sensing system according to claim 1, characterized in that, In the alignment layer: The timestamp synchronization unit adopts hardware clock synchronization and software linear interpolation compensation. Based on the tactile signal, the visual and auditory signals are uniformly interpolated to a sampling frequency of 1000Hz, with a time alignment accuracy of ≤10ms. The spatial calibration module constructs a coordinate system for the intelligent agent based on IMU attitude sensor data, and uses a checkerboard calibration method and laser positioning method to complete the coordinate calibration of each sensor, with a spatial calibration error ≤0.5mm; The cross-modal feature mapping unit is a fully connected neural network with 256 hidden layer nodes and ReLU activation function, which maps the feature vectors of each modality to a 256-dimensional unified brain-like feature space.

5. The multimodal brain-like fusion sensing system according to claim 1, characterized in that, In the fusion layer: The multi-brain region collaborative fusion module adopts the peak temporal neural network STDP with a learning rate of 0.001 and a peak firing threshold of 0.8 to simulate the synaptic plasticity and collaborative firing mechanism of neurons in the cerebral cortex. The adaptive weight allocation unit calculates the fusion weight based on the confidence level of each modal signal. The weight adjustment range is 0-1, the response time is ≤5ms, and the confidence level is calculated from the signal-to-noise ratio and feature variance. The single-mode failure detection unit sets the feature confidence threshold to 0.

6. When the feature confidence of a certain mode is <0.6, it is determined that the mode is failed, and the compensation mechanism is automatically activated to reduce the weight of the failed mode to below 0.1 and increase the weight of other modes.

6. The multimodal brain-like fusion sensing system according to claim 1, characterized in that, In the aforementioned decision-making level: The deep reinforcement learning decision model is the DDPG model. The Actor network takes a 256-dimensional fused feature vector as input and outputs a 6-dimensional action instruction. The Critic network evaluates the action reward, and the reward function is R = 0.6 × task completion degree - 0.3 × action energy consumption + 0.1 × interaction naturalness. The task intent recognition unit uses a CNN-LSTM hybrid model, with a task recognition accuracy of ≥95%. The action execution interface supports CAN bus communication, outputs voltage 0-24V, current 0-5A, and signal delay ≤10ms.

7. The multimodal brain-like fusion sensing system according to claim 1, characterized in that, In the feedback layer: The status feedback acquisition unit integrates environmental sensors and motion posture sensors, with a sampling rate of 100Hz, and is used to collect environmental status, task completion status, and action execution effect. The parameter dynamic adjustment module uses the gradient descent algorithm with a learning rate of 0.0005 and an iteration period of 100ms to optimize the weights of the fusion layer and the model parameters of the decision layer. The memory storage unit has a capacity of ≥16GB and supports real-time storage and retrieval of optimal parameters, simulating the memory function of the hippocampus.

8. The multimodal brain-like fusion perception method according to claim 2, characterized in that, In step S1, the preprocessing specifically includes: After the visual signal is filtered by Gaussian with a standard deviation of σ=1.5, histogram equalization, and Zhang's distortion correction, 128-dimensional visual features are extracted by brain-like CNN. After the auditory signal is decomposed and denoised by 5 layers of db4 wavelet and converted by STFT time-frequency, 64-dimensional auditory features are extracted by LSTM. After the tactile signal is low-pass filtered at 100Hz and min-max normalized, 32-dimensional tactile features are extracted by a causal convolutional network.

9. The multimodal brain-like fusion perception method according to claim 2, characterized in that, In step S2, the time synchronization includes hardware clock synchronization and software delay compensation. When the posture of the bionic avatar changes, the signal delay of each sensor is calculated in real time, and the delay is eliminated by a time offset compensation algorithm. The spatial calibration includes coordinate system construction, coordinate calibration and dynamic calibration. When the posture of the avatar changes, the position coordinates of each sensor are updated in real time, and spatial alignment is achieved by coordinate transformation.

10. A multimodal brain-like fusion perception method according to claim 2, characterized in that, In step S5, the adaptive iterative optimization includes parameter adjustment, memory storage, and parameter retrieval. When the same scenario is encountered again, the optimal parameters in the memory storage unit are directly called to improve processing efficiency.