Neuromorphic multi-modal perception method and apparatus, electronic device, and storage medium

Abstract

The application discloses a neuromorphic multi-modal perception method and device, electronic equipment and storage medium, and relates to the technical field of brain-inspired intelligent computing. The method comprises the following steps: acquiring multi-modal data, preprocessing the multi-modal data to form enhanced modal pairs that are aligned in space; processing each modal data in the enhanced modal pairs through a double-flow feature extraction network to obtain a plurality of semantic-level modal feature pairs; performing cross-modal interaction and fusion on the plurality of semantic-level modal feature pairs based on an event-driven pulse computer mechanism to generate multi-scale fusion features; performing global modeling on high-level fusion features in the multi-scale fusion features based on a neuromorphic attention mechanism to obtain global enhanced features; and processing the global enhanced features in an event-driven manner based on a hybrid network model to generate a target perception result, wherein the hybrid network model comprises an artificial neural network and a pulse neural network. The application realizes high-precision multi-modal perception under low-power consumption constraints.

Description

Technical Field

[0001] This application relates to the field of brain-like intelligent computing technology, and in particular to neuromorphic multimodal sensing methods, devices, electronic devices and storage media. Background Technology

[0002] With the continuous expansion of applications such as intelligent security, unmanned systems, and smart cities, multimodal perception technology, due to its ability to integrate complementary information, has become an important way to improve the reliability and robustness of target perception in complex environments. Multimodal perception technology can be deployed on edge devices, which are computing devices deployed near the data source or physical location, possessing certain local data acquisition, processing, decision-making, and response capabilities. Examples of edge devices include drone terminals, vehicle-mounted terminals, bionic robots, or roadside units. Roadside units refer to intelligent monitoring equipment deployed on road infrastructure, such as traffic light poles and gantries.

[0003] Currently, most mainstream multimodal perception methods are built upon deep learning models such as convolutional neural networks. While these models offer high computational efficiency in local feature extraction, they have limitations in modeling cross-modal global semantic relationships and long-range dependencies. Furthermore, existing models generally operate on traditional von Neumann computing architectures, whose inherent separation of storage and computation leads to frequent data migration, resulting in high access latency and dynamic energy consumption. When these models are deployed on edge devices with limited computing, storage, and power resources, the computationally intensive processes such as multimodal fusion and deep feature interaction further amplify these energy efficiency bottlenecks. This leads to a severe challenge for edge devices in balancing perception accuracy with energy consumption and latency, resulting in insufficient detection accuracy in practical applications.

[0004] Therefore, when targeting low-power edge devices, how to achieve high-precision multimodal sensing and improve the detection accuracy of edge devices in practical applications is a problem that urgently needs to be solved. Summary of the Invention

[0005] The main objective of this application is to provide a neuromorphic multimodal sensing method, device, electronic device, and storage medium, aiming to solve the technical problem of how to achieve high-precision multimodal sensing for low-power edge devices and improve the detection accuracy of edge devices in practical applications.

[0006] To achieve the above objectives, this application proposes a neuromorphic multimodal perception method, the method comprising: Acquire multimodal data and preprocess the multimodal data to form spatially aligned enhanced modal pairs; The modal data in the enhanced modal pair are processed by a dual-stream feature extraction network to obtain modal feature pairs at multiple semantic levels. Based on an event-driven pulse computing mechanism, cross-modal interaction and fusion are performed on the modal feature pairs of the multiple semantic levels to generate multi-scale fused features; Global enhancement features are obtained by globally modeling the high-level fusion features in the multi-scale fusion features based on the neuromorphic attention mechanism. Based on a hybrid network model, the global enhancement features are processed in an event-driven manner to generate target perception results. The hybrid network model includes artificial neural networks and spiking neural networks.

[0007] In one embodiment, the multimodal data includes visible light modal data and infrared modal data. The step of acquiring the multimodal data and preprocessing the multimodal data to form spatially aligned enhanced modal pairs includes: Acquire visible light modal data and infrared modal data synchronously collected by visible light sensor and infrared sensor; Scale normalization is performed on the visible light modal data and the infrared modal data respectively to adjust the visible light modal data and the infrared modal data to the same spatial resolution, thereby obtaining scale-normalized visible light modal data and infrared modal data. The scale normalization operation includes a spatial mapping function based on interpolation resampling, and bilinear interpolation is used to process the visible light modal data and the infrared modal data respectively to maintain the continuity of image structure and thermal radiation distribution characteristics. The scale-normalized visible light modal data and infrared modal data are subjected to data augmentation transformation to obtain spatially aligned augmented modal pairs. The data augmentation transformation includes geometric augmentation and illumination augmentation, wherein the geometric augmentation is applied simultaneously to the visible light modal data and the infrared modal data, and the illumination augmentation is applied to the visible light modal data.

[0008] In one embodiment, the step of processing each modality data in the modality pair using a dual-stream feature extraction network to obtain modality feature pairs at multiple semantic levels includes: The visible light modal data is input into the first branch of the dual-stream feature extraction network, which outputs multi-level visible light features. The infrared modal data is input into the second branch of the dual-stream feature extraction network, which outputs multi-level infrared features. Features corresponding to the same semantic level in the multi-level visible light features and the multi-level infrared features are selected to form the modal feature pairs of the multiple semantic levels.

[0009] In one embodiment, the event-driven pulse computing mechanism for performing cross-modal interaction and fusion of modal feature pairs at multiple semantic levels to generate multi-scale fused features includes: For each semantic level modal feature pair, the visible light feature and infrared feature of the current level are encoded into pulse feature sequences respectively; Construct a cross-modal attention computation unit that takes pulse feature sequences as input; In the cross-modal attention calculation unit, the pulse feature sequence of the first modality is used to guide the enhancement of the pulse feature sequence of the second modality. After the bidirectional guidance enhancement is completed, the enhanced features of the two modalities are fused to obtain the fused features of the current level. The first modality is a visible light mode or an infrared mode, and the second modality is another mode different from the first modality. The encoding, processing, and fusion operations are performed on multiple semantic levels respectively to generate multi-scale fusion features corresponding to each semantic level.

[0010] In one embodiment, the step of globally modeling the high-level fusion features in the multi-scale fusion features based on the neuromorphic attention mechanism to obtain the global enhanced features includes: The high-level fusion features in the multi-scale fusion features are mapped into a temporal pulse sequence using pulse coding. The time-series pulse sequence is input into the pulse-driven attention module to perform event-driven modeling of long-range spatial correlation and obtain the pulse attention output. The pulse attention output and the temporal pulse sequence are fused together using a residual structure to obtain the fused features; The fused features from multiple time steps are subjected to nonlinear enhancement processing to obtain globally enhanced features.

[0011] In one embodiment, the step of performing nonlinear enhancement processing on the fused features at multiple time steps to obtain globally enhanced features further includes: The fused features are input into a pulse-driven feedforward network module for nonlinear enhancement to obtain a pulse-feedforward output. The pulse feedforward output is then fused again with the fused features through a residual structure to obtain the pulse representation of the enhanced features; The global enhanced features are obtained by time accumulation and averaging of the pulse representations of the enhanced features at multiple time steps.

[0012] In one embodiment, the hybrid network model includes an artificial neural network and a spiking neural network. The step of processing the global augmentation features in an event-driven manner based on the hybrid network model to generate the target perception result includes: Based on the neuron activation distribution of the artificial neural network during the training phase, the firing threshold of the corresponding spiking neurons is calibrated. The weight parameters of the corresponding modules in the artificial neural network are mapped to the synaptic weights of the spiking neurons; The global enhancement features are input into the hybrid network model; In the spiking neurons of a spiking neural network, a spiking event is triggered when the membrane potential accumulated by the global enhancement features exceeds a preset threshold. The outputs of neurons corresponding to impulse events are weighted, accumulated, and nonlinearly transformed to obtain target perception results, which include target detection boxes or category labels.

[0013] Furthermore, to achieve the above objectives, this application also proposes a neuromorphic multimodal sensing device, which includes: A data acquisition module is used to acquire multimodal data and preprocess the multimodal data to form spatially aligned enhanced modal pairs; The feature extraction module is used to process the modal data in the enhanced modality pair respectively through a two-stream feature extraction network to obtain modal feature pairs at multiple semantic levels; The feature fusion module is used to perform cross-modal interaction and fusion of the modal feature pairs at multiple semantic levels based on an event-driven pulse computing mechanism to generate multi-scale fused features. The feature enhancement module is used to perform global modeling of the high-level fusion features in the multi-scale fusion features based on the neuromorphic attention mechanism to obtain global enhanced features; The modal perception module is used to process the global enhancement features in an event-driven manner based on a hybrid network model to generate target perception results. The hybrid network model includes artificial neural networks and spiking neural networks.

[0014] In addition, to achieve the above objectives, this application also proposes an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the neuromorphic multimodal perception method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a non-transitory storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the neuromorphic multimodal perception method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: Multimodal data is acquired and preprocessed to form spatially aligned enhanced modal pairs. By spatially registering multi-source heterogeneous images, the alignment of multimodal information at the pixel level is ensured, overcoming the information fusion failure problem caused by modal misalignment.

[0017] By using a dual-stream feature extraction network to process the modal data of each enhanced modality pair separately, multiple semantic-level modal feature pairs are obtained. By constructing a dual-stream feature extraction network, the independent and faithful extraction of visible light texture and infrared thermal structure features is achieved, effectively avoiding feature confusion caused by early fusion and preserving the unique advantages of each modality.

[0018] Based on an event-driven pulsation mechanism, cross-modal interaction and fusion of modal feature pairs at multiple semantic levels are performed to generate multi-scale fused features. By introducing an event-driven pulsation mechanism to achieve cross-modal interaction and fusion, bimodal information can be efficiently and complementaryly fused at multiple semantic scales. At the same time, the inherent sparse computation characteristics of this mechanism significantly reduce the computational complexity and energy consumption of the fusion process.

[0019] Globally enhanced features are obtained by globally modeling high-level fusion features in multi-scale fusion features based on neuromorphic attention mechanisms. A neuromorphic attention mechanism implemented with impulses is used to globally model high-level features, enabling the model to dynamically focus on key target regions and suppress background interference, thus enhancing the discrimination ability and robustness of targets in complex scenes.

[0020] Based on a hybrid network model, global augmentation features are processed in an event-driven manner to generate target perception results. The hybrid network model includes artificial neural networks and spiking neural networks. The entire system operates on a hybrid artificial neural network and spiking neural network architecture, with a continuous front-end and an event-driven core, maximizing the energy efficiency of spiking computation during inference. While maintaining high-precision multimodal perception performance, it achieves a significant reduction in computational energy consumption and latency. This enables the complex and efficient neuromorphic multimodal perception model to be practically deployed on edge devices with limited computing resources and power, improving the detection accuracy of edge devices in real-world applications and thus enabling real-time and reliable smart city traffic monitoring tasks. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the first embodiment of the neuromorphic multimodal perception method of this application; Figure 2 This is a schematic diagram of the hybrid network model of this application; Figure 3 This is a schematic diagram of the SDF module structure in this application; Figure 4 This is a schematic diagram of the STEN module structure in this application; Figure 5 This is a schematic diagram of the NAIFI module structure in this application; Figure 6 This is a schematic diagram of the module structure of the neuromorphic multimodal sensing device of this application; Figure 7 This is a schematic diagram showing the experimental verification results of the neuromorphic multimodal perception method of this application.

[0024] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] This application provides a neuromorphic multimodal perception method that can be deployed on edge devices. Specifically, refer to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the neuromorphic multimodal perception method of this application. In this embodiment, the neuromorphic multimodal perception method includes steps S10 to S40:

[0027] Step S10: Acquire multimodal data and preprocess the multimodal data to form spatially aligned enhanced modal pairs.

[0028] It should be noted that multimodal data refers to heterogeneous data collected by different types of sensors in the same scene. For example, multimodal data can include visible light modal data and infrared modal data, which can be visible light images and infrared images simultaneously acquired by visible light sensors and infrared sensors. Before inputting the raw data into the model for calculation, data standardization and enhancement preprocessing can be performed. Spatial alignment refers to ensuring, through sensor extrinsic parameter calibration and preprocessing, that the processed multimodal data, in the spatial coordinate system, that the same pixel position corresponds to the same physical point in the real world, forming a pixel-level one-to-one correspondence. The spatially aligned and diversity-enhanced pair of visible light and infrared images obtained after the above preprocessing process serves as the enhanced modal pair.

[0029] Step S20: The modal data in the enhanced modal pair are processed separately by a dual-stream feature extraction network to obtain modal feature pairs at multiple semantic levels.

[0030] It should be noted that a two-stream feature extraction network can be a convolutional neural network architecture containing two independent parameter branches. One branch processes visible light modal data, extracting visible light texture semantic features such as texture and color; the other branch processes infrared modal data, extracting infrared thermal radiation structural features such as target thermal radiation. A modal feature pair can be understood as a pair of visible light feature maps and infrared feature maps output by the two branches of the two-stream feature extraction network at a specific semantic level.

[0031] Step S30: Based on the event-driven pulse computing mechanism, cross-modal interaction and fusion are performed on modal feature pairs at multiple semantic levels to generate multi-scale fused features.

[0032] It should be noted that event-driven spiking computation can be understood as a biomimetic sparse computing paradigm, with spiking neurons as its core computational unit. Cross-modal interaction refers to the ability, within the spiking computation framework, for a sequence of spiking features from one modality to guide and enhance the feature representation of another modality, forming a bidirectional information exchange. Cross-modal fusion can be understood as the process of combining the enhanced visible light and infrared features into a unified feature representation through methods such as splicing and addition after completing bidirectional cross-modal interaction enhancement. Multi-scale fusion features refer to a set of multi-scale feature maps that fuse bimodal information, obtained after performing the aforementioned cross-modal interactions and fusions at different semantic levels.

[0033] Step S40: Based on the neuromorphic attention mechanism, perform global modeling of the high-level fusion features in the multi-scale fusion features to obtain global enhanced features.

[0034] It should be noted that the neuromorphic attention mechanism is an attention calculation method based on spiking neural networks. It can dynamically calculate the importance weights of different spatial locations in the feature map through spiking neurons, and reweight the features in an event-driven manner to achieve global modeling of key information. Globally enhanced features refer to the feature representation rich in global contextual and discriminative information obtained after processing the high-level features in the multi-scale fusion features through the above neuromorphic attention mechanism.

[0035] Step S50: Based on the hybrid network model, global enhancement features are processed in an event-driven manner to generate target perception results. The hybrid network model includes artificial neural networks and spiking neural networks.

[0036] It should be noted that the hybrid network model, used in the final deployment and inference of this embodiment, is a heterogeneous computing architecture composed of traditional artificial neural network units and spiking neural network units. Artificial neural network units can process continuous values, while spiking neural network units handle discrete pulse events. Event-driven processing means that during model inference, data is transmitted through the network in the form of pulse sequences, and computation is triggered only by arriving pulse events; if no pulse occurs, the corresponding unit remains in a quiet, power-saving state. The target perception result refers to the final application layer output. In actual vehicle detection tasks, this can be represented as the target bounding box's position coordinates, category label, and confidence level.

[0037] In this embodiment, sensor extrinsic calibration and synchronous data augmentation techniques are employed. Firstly, the spatial registration challenge between multi-source heterogeneous images is addressed, ensuring the alignment of multimodal information at the pixel level and overcoming information fusion failure caused by modal misalignment. By constructing a dual-stream feature extraction network, independent and faithful extraction of visible light texture and infrared thermal structure features is achieved, effectively avoiding feature confusion caused by early fusion and preserving the unique advantages of each modality. Furthermore, an event-driven spiking computation mechanism is introduced to achieve cross-modal interaction and fusion, enabling efficient and complementary deep fusion of bimodal information across multiple semantic scales. Simultaneously, the inherent sparse computation characteristics of this mechanism significantly reduce the computational complexity and energy consumption of the fusion process. Based on this, a spiking-based neuromorphic attention mechanism is used for global modeling of high-level features, allowing the model to dynamically focus on key target regions and suppress background interference, enhancing the ability and robustness to discriminate targets in complex scenes. Finally, the entire system is based on a hybrid artificial neural network and spiking neural network architecture, operating in a continuous front-end and event-driven core manner, maximizing the energy efficiency advantages of spiking computation during inference. The method in this embodiment maintains high-precision multimodal perception performance while significantly reducing computational energy consumption and latency. This enables complex and efficient neuromorphic multimodal perception models to be practically deployed on edge devices with limited computing resources and power, improving the detection accuracy of edge devices in practical applications and thus completing real-time and reliable smart city traffic monitoring tasks.

[0038] In one implementation, step S10 includes: Step S101: Acquire visible light modal data and infrared modal data synchronously collected by the visible light sensor and the infrared sensor.

[0039] It should be noted that a visible light sensor is an imaging device capable of capturing electromagnetic waves within the visible spectrum of the human eye. Its output is a visible light image reflecting the texture, color, and detail information of a scene, i.e., visible light modal data. An infrared sensor is an imaging device capable of capturing the infrared thermal radiation emitted by an object itself. Its output is an infrared image reflecting the temperature distribution and thermal differences on the object's surface, i.e., infrared modal data. Through hardware synchronization signals or precise timestamp control, visible light and infrared sensors can capture images of the same scene at the same moment or within a very short time interval, ensuring the consistency of the two modal data in the temporal dimension and avoiding information misalignment caused by target movement.

[0040] For example, the acquired visible light modal data and infrared modal data are the original visible light image and infrared image, respectively, represented as: in, and These represent the number of channels in the visible light and infrared images, respectively.

[0041] Step S102: Perform scale normalization operations on the visible light modal data and the infrared modal data respectively, and adjust the visible light modal data and the infrared modal data to the same spatial resolution to obtain scale-normalized visible light modal data and infrared modal data.

[0042] The scale normalization operation includes a spatial mapping function based on interpolation resampling, which uses bilinear interpolation to process the visible light modal data and infrared modal data separately to maintain the continuity of image structure and thermal radiation distribution characteristics.

[0043] It should be noted that digital image processing techniques can be used to uniformly adjust images with different original resolutions or sizes to the same preset spatial resolution, thereby eliminating differences in input size and providing standardized data for subsequent neural network processing that requires a fixed input size. During scale normalization, a spatial mapping function based on interpolation resampling is used to calculate the grayscale or intensity value of each target pixel at the new resolution. This spatial mapping function estimates the value based on the values ​​of neighboring pixels in the original image through weighted summation and other methods to preserve image content as much as possible after scaling. For each pixel in the target image, bilinear interpolation is performed within its corresponding neighborhood in the original image, first twice in the horizontal direction and then once in the vertical direction to finally determine the pixel value.

[0044] For example, by scaling operations, they can be uniformly adjusted to a fixed spatial resolution. The standardized input image pairs are obtained, represented as: in, This represents a spatial mapping function based on interpolation resampling, used to resample the pixel grid while preserving the structural continuity of the original image. Specifically, for a target spatial size... The pixel value at any pixel location in the image is calculated by a weighted combination of the corresponding neighboring pixels in the original image. This maintains a smooth transition of image brightness and structural information during spatial scaling, avoiding significant distortion caused by discrete sampling. By employing a scale normalization method based on bilinear interpolation, the texture continuity of visible light images and the thermal radiation distribution characteristics of infrared images can be effectively maintained without introducing additional computational complexity.

[0045] Step S103: Perform data augmentation transformation on the scale-normalized visible light mode data and infrared mode data to obtain spatially aligned augmented mode pairs.

[0046] It should be noted that data augmentation transformation includes geometric augmentation and illumination augmentation. Geometric augmentation is applied simultaneously to visible light modal data and infrared modal data, while illumination augmentation is applied to visible light modal data.

[0047] For example, geometric enhancement can simulate different viewing perspectives by changing the geometry or spatial relationships of an image. Specific operations can include random cropping, horizontal flipping, and scale perturbation. Random cropping is used to simulate different regions of interest or distances, horizontal flipping is used to simulate perspectives from opposite driving directions, and scale perturbation is used to simulate changes in target size. Illumination enhancement can simulate complex and variable lighting conditions in natural environments by changing image properties such as brightness and contrast, or by adding noise. Specific operations can include brightness adjustment, contrast variation, and noise perturbation. Brightness adjustment can simulate day / night cycles or changes in light intensity, contrast variation can simulate the effects of fog or shadows, and noise perturbation can simulate sensor noise.

[0048] For example, synchronized data augmentation can be performed on standardized visible-infrared image pairs to simulate the imaging changes of vehicles under different viewpoints, scales, and lighting conditions in smart city traffic scenarios, thereby improving the robustness of the model without disrupting the multimodal spatial correspondence. Data augmentation operations include two categories: geometric augmentation and illumination augmentation, and their processes can be uniformly represented as follows:

[0049] ; in, This represents the enhancement mapping function that synchronizes the two modes. To enhance the parameter set, a synchronous enhancement method is used to ensure the quality of the enhanced visible light image. With infrared images Strict alignment is maintained in the spatial dimension. After scale normalization and synchronization data augmentation, the enhanced visible-infrared multimodal input data pair is obtained. , This data serves as the input to the subsequent dual-stream backbone network, used to extract visible light texture semantic features and infrared thermal radiation structural features, and participates in multi-scale cross-modal fusion and event-driven neuromorphic computing processes in subsequent steps.

[0050] This implementation employs synchronous acquisition technology using visible light and infrared sensors. This ensures the consistency of dual-modal data over time from the outset, resolving the core challenge of positional shifts in moving targets between the two modal images caused by asynchronous acquisition. This lays a reliable data foundation for subsequent accurate spatial alignment and information fusion. By unifying heterogeneous input images to a fixed resolution, not only are the standardized input requirements of deep neural networks met, but the smooth interpolation method effectively maintains the continuity of texture edges in the visible light image and the authenticity of temperature distribution in the infrared image, avoiding distortion of key feature information or the introduction of structural artifacts due to coarse scaling. Furthermore, synchronous geometric enhancement and modality-specific illumination enhancement transformations are applied. While strictly maintaining the fusion premise of multimodal spatial alignment, this significantly expands the effective diversity of training data, enabling subsequent models to learn robust feature representations insensitive to changes in viewpoint, scale, and illumination. This significantly improves the overall adaptability and detection accuracy of the intelligent transportation perception system in complex and variable real-world environments.

[0051] In one implementation, step S20 includes: Step S201: Input the visible light modal data into the first branch of the dual-stream feature extraction network and output multi-level visible light features.

[0052] For example, multi-level visible light features mainly include detailed information such as the shape, texture, and color of an object.

[0053] Step S202: Input the infrared modal data into the second branch of the dual-stream feature extraction network and output multi-level infrared features.

[0054] For example, multi-level infrared features mainly include structural information such as the thermal radiation distribution and temperature profile of an object.

[0055] Step S203: Select features from the multi-level visible light features and multi-level infrared features that correspond to the same semantic level to form multiple semantic level modal feature pairs.

[0056] It's important to note that semantic level refers to the level of information abstraction corresponding to a feature. Lower semantic levels correspond to specific pixel-level information, such as edges, while higher semantic levels correspond to abstract concept-level information, such as "vehicle." In feature extraction networks, network layers of different depths naturally correspond to different semantic levels. A modal feature pair refers to a pairing at the same semantic level, consisting of a visible light feature map output from the first branch and an infrared feature map output from the second branch. For example, the feature maps output from the third convolutional block of each of the two branches constitute a mid-level semantic modal feature pair.

[0057] For example, such as Figure 2As shown, the dual-stream backbone network employs a consistent multi-level feature extraction network. The two branches maintain consistency in network hierarchy, convolution stride, and downsampling position, but their network parameters are independent to model the imaging characteristics of different modalities. Assuming the dual-stream backbone network comprises five feature extraction stages, corresponding to stages C1 to C5 of ResNet-50, then in the k-th feature extraction stage, visible light feature maps and infrared feature maps are obtained, as follows:

[0058] ; in, and These represent the feature extraction mappings of the visible light branch and the infrared branch at the k-th stage, respectively. Since the two branches are structurally consistent, the visible light features from the corresponding stages... infrared features Natural alignment in spatial resolution and pixel location provides a consistent spatial reference for subsequent cross-modal feature interactions. For example, features extracted at different stages have different semantic levels and spatial resolutions. The features output by the shallower stage C3 have higher spatial resolution, mainly containing edge, texture, and local structural information; the features output by the intermediate stage C4 begin to possess preliminary semantic expression capabilities while maintaining certain spatial details; the features output by the higher stage C5 have lower spatial resolution but contain more abstract semantic information and a representation of the overall target contour. Considering the differences in complementarity between visible light and infrared modes at different semantic levels, the visible light mode has richer texture and structural information in shallow and intermediate features, while the infrared mode has a more stable response to the target contour and salient regions in high-level features. Therefore, this application explicitly retains and outputs the first feature in the dual-stream backbone network stage. Visible light characteristics of each stage output infrared features This serves as the input for subsequent multi-scale progressive cross-modal fusion modules. By extracting and retaining feature representations at different semantic levels in the backbone stage, subsequent cross-modal fusion can proceed step by step from shallow to deep, effectively enhancing the complementarity and expressive power of multimodal features.

[0059] In this embodiment, a parameter-independent dual-stream feature extraction network is employed. Visible light and infrared data are input into their respective dedicated first and second branches for parallel processing. Firstly, the network structure enables isolated deep feature extraction of the two modalities, effectively solving the problems of mutual interference between heterogeneous modal information and the submergence of weak modal features by strong modal features, which are common in earlier or single-stream fusion methods. This maximizes the preservation and enhancement of the rich texture details of visible light and the unique temperature range structure information of infrared. Furthermore, by extracting multi-level features from each branch, the model can capture a complete information spectrum from local details to global semantics, addressing the limited perception capability of single-scale features for different targets and providing a hierarchical representation foundation for multi-scale target detection in complex scenes. Finally, by selecting and pairing features from the two branches at the same semantic level to form modal feature pairs, it ensures that subsequent cross-modal interaction and fusion occur within the spatial receptive field of information abstraction and alignment, resolving the information mismatch and inefficiency caused by blind fusion of features at different abstraction levels or spatial resolutions.

[0060] In one implementation, step S30 includes: Step S301: For each semantic level modal feature pair, the visible light feature and infrared feature of the current level are encoded into pulse feature sequences respectively.

[0061] For example, the two-dimensional features at the k-th scale can be mapped into a sequence. Let the visible light and infrared features be represented as follows:

[0062] Then, through feature segmentation and linear mapping operations, the corresponding sequence representation is obtained: in, This represents a mapping function that converts two-dimensional features into a sequence of tokens.

[0063] Step S302: Construct a cross-modal attention computation unit with pulse feature sequence as input.

[0064] For example, such as Figure 2 As shown, a cross-modal feature fusion module based on an event-driven pulse mechanism, namely the Spiking Dual-Modal Fusion (SDF) module, can be constructed at each feature scale. This module is used to respectively implement infrared feature-guided visible light feature enhancement and visible light feature-guided infrared feature enhancement, thereby forming a bidirectional cross-modal information interaction structure. The SDF structure is shown in the figure. Figure 3 .

[0065] In step S302, in the cross-modal attention calculation unit, the pulse feature sequence of the first modality is used to guide and enhance the pulse feature sequence of the second modality. After completing the bidirectional guidance enhancement, the enhanced features of the two modalities are fused to obtain the fused features of the current level.

[0066] The first mode is either the visible light mode or the infrared mode, and the second mode is a different mode from the first mode.

[0067] For example, for any scale k, the bidirectional cross-modal enhancement process can be uniformly represented as: ; in, and These represent cross-modal enhancement mappings with visible light or infrared features as the dominant modality, respectively. These cross-modal enhancement mappings can be implemented based on an event-driven pulse computation mechanism. In the specific implementation, the sequence features are expanded into multiple discrete time steps in the time dimension, and continuous feature activations are converted into a sequence of pulse events through multi-threshold spiking neurons. The corresponding pulse is triggered only when the feature activation exceeds a preset threshold, thereby reducing invalid computations and lowering overall energy consumption. In the above cross-modal enhancement process, Spiking Transformers for Enhancement of cross-modal features (STEN) can be introduced to model the correlation between different modal features under the event-driven computation paradigm. The STEN structure is referenced... Figure 4 Taking visible light-guided enhanced infrared as an example, at time step t, the visible light pulse features are used as the query input, and the infrared pulse features are used as the key and value input. The cross-modal attention calculation process is constructed as follows:

[0068] Based on query Q, key K, and value V, a cross-modal correlation map is generated. The enhanced cross-modal response at time step t is obtained through event-driven correlation accumulation and normalization operations. ; in, This indicates a correlation accumulation operation based on impulse events. This represents the desired compensation normalization operation used for stabilizing the pulse representation. This represents the corresponding matrix multiplication-addition mapping. Subsequently, the cross-modal augmentation responses obtained across multiple time steps are accumulated along the time dimension and averaged to obtain the augmented sequence feature representation:

[0069] ; The sequence features are restored to a two-dimensional space by reverse mapping to obtain the enhanced infrared features. Symmetrically, the same structure can be used to achieve the infrared-guided visible light feature enhancement process, resulting in enhanced visible light features. After completing bidirectional cross-modal enhancement, such as Figure 2 As shown, the enhanced visible light features Compared with enhanced infrared features The fusion process is performed within the same scale.

[0070] Step S303: Encode, process and fuse multiple semantic levels respectively to generate multi-scale fusion features corresponding to each semantic level.

[0071] For example, this can be achieved by feature concatenation to obtain fused features at scale k: The aforementioned cross-modal enhancement and fusion process based on the event-driven impulse mechanism is executed step by step from shallow to deep at the three scales of C3, C4 and C5, forming a multi-scale progressive cross-modal feature fusion structure.

[0072] In one implementation, step S40 includes: Step S401: The high-level fusion features in the multi-scale fusion features are mapped into a temporal pulse sequence through pulse coding.

[0073] It should be noted that a temporal pulse sequence refers to a sequence of spiking neuron states at each discrete time step after data is expanded in the time dimension. It simulates the dynamic firing pattern of biological neurons over time and is the primary form of information representation and transmission in spiking neural networks. It can transform continuous, real-valued feature vectors or feature maps into a temporal pulse sequence composed of discrete 0s and 1s. Pulse coding, as a key interface connecting traditional artificial neural networks and spiking neural networks, provides input for subsequent event-driven computation.

[0074] For example, high semantic layer features from multi-scale fusion features can be used as input, and a Neuromorphic Attention-based Intra-scale Feature Interaction (NAIFI) module can be introduced to model the global dependencies of the fused features. The NAIFI structure is referenced. Figure 5 . Fusing features Mapped to a temporal representation that can be processed by a spiking neural network, in discrete time steps The above forms a pulse input sequence:

[0075] in, This represents a pulse-code mapping used to transform continuous-value fused features into event-driven pulse representations. The pulse-code mapping can be accomplished by introducing spiking neurons. Let the membrane potential of a neuron at time step t be... and set threshold set Then its pulse output at that time step is defined as:

[0076] ; Among them, pulse output As a basic computational unit in subsequent calculations, it participates in the data flow. Through a multi-threshold discharge mechanism, the NAIFI module can distinguish different activation intensities in fused features within a single time step, enabling global attention modeling to maintain its sparse event-driven characteristics while providing a more refined expression of the strength differences of high semantic features.

[0077] Step S402: Input the temporal pulse sequence into the pulse-driven attention module to perform event-driven modeling of spatial long-range correlation and obtain the pulse attention output.

[0078] It should be noted that the spiking-driven attention module can be implemented using spiking neurons and an attention mechanism constructed based on spiking event-driven computation rules. This allows for the dynamic allocation of importance between different locations in the feature map through sparse spiking event propagation and computation. Spatially long-range correlation refers to the semantic or structural relationship between two or more spatially distant locations in the feature map. For example, although the front and rear of a vehicle are geographically separated in an image, they belong to the same object and are strongly correlated. Event-driven modeling means that the computation process is not executed at fixed times, but is triggered by input spiking events. When a spiking event arrives, the relevant neuron state is updated, and new output spiking events may be generated; if no event occurs, the relevant computational units remain silent, thus conserving energy. Spiking attention output refers to the sequence of feature spiking events that has been processed by the spiking-driven attention module and incorporates attention weights.

[0079] For example, after the pulse representation is formed, the pulse characteristics along Figure 2 The data flow path shown enters the pulse-driven attention modeling unit, yielding the global modeling response at time step t: in, express Figure 2 The "Pulse-Driven Attention" module models the long-range correlation of fused features in the spatial dimension. This process is executed under an event-driven mechanism, accumulating correlation only at the activated pulse locations.

[0080] Step S403: The pulse attention output and the temporal pulse sequence are fused using a residual structure to obtain the fused features.

[0081] For example, the pulse attention output is fused with the original pulse input through residual connections and a normalization structure to form intermediate pulse features: in, This represents the normalization process used to stabilize the distribution of impulse features over time. This residual structure ensures that the fused features do not lose their original semantic information due to attention modulation during global modeling.

[0082] Step S404: Perform nonlinear enhancement processing on the fused features from multiple time steps to obtain globally enhanced features.

[0083] It should be noted that step S404 includes: inputting the fused features into the pulse-driven feedforward network module for nonlinear enhancement to obtain the pulse feedforward output; fusing the pulse feedforward output with the fused features again through the residual structure to obtain the pulse representation of the enhanced features; and performing time accumulation and averaging on the pulse representations of the enhanced features at multiple time steps to obtain the global enhanced features.

[0084] Nonlinear enhancement processing refers to further transforming features through network layers containing nonlinear activation functions, such as feedforward neural networks, to increase the model's expressive power and enable it to fit more complex functional relationships. The spiking-driven feedforward network module can be understood as a network module composed of multiple layers of spiking neurons connected in a feedforward manner, used to perform nonlinear transformations and feature enhancement in the spiking domain. The spiking feedforward output refers to the output spiking sequence after the features have been processed by the spiking-driven feedforward network module. The spiking representation of the enhanced features refers to the spiking state presented at the current time step after the two-step enhancement process of the attention module and the feedforward network module. Since spiking information unfolds over time, the spiking at a single time step may contain noise or incomplete information. By accumulating and averaging the spiking activities of multiple consecutive time steps, a more stable and robust continuous-valued feature representation can be obtained for the final decision.

[0085] Exemplary intermediate pulse characteristics along Figure 2 The data stream enters the pulse-driven feedforward modeling unit to further enhance the nonlinear expressive power of the features: ; in, express Figure 2The "Pulse-Driven Multilayer Perceptron" module is also used. This module employs an event-driven computation paradigm, performing operations only on neurons that have undergone pulse activation. After pulse feedforward modeling, the feedforward output and intermediate pulse features are fused again through residual connections and normalization structures to obtain the output pulse features of the NAIFI module at time step t.

[0086] After completing pulse modeling at multiple time steps, the pulse modeling results obtained at each time step are accumulated and averaged over the time dimension to obtain a stable feature representation, thus yielding the final output feature representation of the NAIFI module. It serves as a high-level global enhancement feature input to subsequent modules, used to complete cross-scale feature fusion between different scales.

[0087] In this implementation, high-level semantic information is first converted into a sparse pulse representation suitable for event-driven computation, laying the foundation for processing complex semantic tasks within a low-power spiking neural network framework. By inputting these pulse sequences into the pulse-driven attention module, the spatial long-range correlations within the feature map are dynamically modeled using event-driven computation rules. This allows the model to capture and enhance global semantic relationships between target components with extremely low energy consumption, solving the high energy consumption problem caused by dense matrix operations in traditional continuous-value attention mechanisms when calculating global relationships, making it particularly suitable for resource-constrained edge scenarios. Subsequently, the pulse attention output is fused with the original temporal pulse sequence through a residual structure, ensuring that key details in the original features are preserved while introducing global contextual information. This effectively alleviates the gradient vanishing or information loss problems that may exist in deep spiking networks, enhancing the stability of network training and the robustness of feature representation. Furthermore, the fused features are nonlinearly enhanced through a pulse-driven feedforward network module, and the residual structure is used again after its output, deepening the discriminative ability of the features layer by layer, enabling the model to fit more complex decision boundaries. Finally, by performing time accumulation and averaging on the enhanced pulse representations of multiple time steps, the dynamic pulse event stream is integrated into a stable and continuous global enhanced feature vector. This not only smooths out the random noise of pulse firing, but also integrates information from the time domain, thus obtaining an information-rich and robust high-level feature representation, achieving a balance between high precision and high energy efficiency.

[0088] In one implementation, before step S50, the following steps are included: during the model training phase, an initial network model can be constructed and optimized using a continuous value calculation method. The initial network model includes a two-stream feature extraction network, a cross-modal interaction and fusion module, and a neuromorphic attention module to obtain initial network parameters; during the model inference phase, based on the initial network parameters, the cross-modal interaction and fusion module and the neuromorphic attention module are converted into an event-driven calculation form based on spiking neurons, while keeping the two-stream feature extraction network in a continuous value calculation form to obtain a hybrid network model.

[0089] It's important to note that hybrid network models include Artificial Neural Networks (ANNs) and Spiking Neural Networks (SNNs). The model training phase refers to the process of using labeled training datasets and optimization algorithms, such as backpropagation, to adjust the internal parameters of the neural network, enabling it to learn the mapping relationship between input data and target output. Continuous value computation can be understood as a neural network computation paradigm where neuron activation values, weight parameters, and gradients during computation are all continuous real numbers. This facilitates training using gradient-based optimization methods and achieves higher task accuracy. The initial network model refers to the complete network prototype built during the training phase for parameter learning. It may include a two-stream feature extraction network, a cross-modal interaction and fusion module, and a neuromorphic attention module, but these modules are implemented as traditional continuous value artificial neural networks during training. Initial network parameters refer to all learnable components in the initial network model after the training phase, such as convolutional kernel weights, fully connected layer weights, and biases. The model inference phase refers to the process of using the trained model to predict or perceive new, unseen input data after training.

[0090] Based on this, step S50 includes: calibrating the firing threshold of the corresponding spiking neuron based on the neuron activation distribution of the artificial neural network during the training phase; mapping the weight parameters of the corresponding module in the artificial neural network to the synaptic weights of the spiking neuron; inputting the global enhancement features into the hybrid network model; triggering a spiking event in the spiking neuron of the spiking neural network when the membrane potential accumulated by the global enhancement features exceeds a preset threshold; and performing weighted accumulation and nonlinear transformation on the neuron output corresponding to the spiking event to obtain the target perception result, which includes a target detection box or category label.

[0091] It should be noted that neuron activation distribution refers to the statistical distribution of the output activation values ​​of neurons in a specific layer on a given dataset, such as mean, variance, and maximum value. Based on the activation distribution of neurons in a continuous-value network, one or more suitable firing thresholds can be set for the transformed spiking neurons through analysis or calculation, so that the firing frequency of the spiking neurons can approximately simulate the activation intensity of the original continuous neurons. Weight parameter mapping refers to the process of assigning the weight parameters of a specific module in a trained artificial neural network, either directly or after scaling, to the synaptic connection weights in the corresponding spiking neural network, thereby enabling the spiking neural network to inherit the knowledge learned by the artificial neural network. In a spiking neural network, synaptic weights are the strength values ​​of the edges connecting two neurons, used to modulate the influence of the signal from the previous neuron on the signal from the next neuron. Membrane potential is a state variable inside a spiking neuron, simulating the transmembrane voltage of a biological neuron, which can accumulate and increase or slowly decay with the weighted input of input pulses. A pulse event refers to a discrete, all-or-nothing output signal generated when the membrane potential of a spiking neuron exceeds the firing threshold; it is the basic unit of information transmission in a spiking neural network. In a spiking neural network or subsequent processing layer, the output pulse signals from multiple spiking neurons are summed according to their corresponding synaptic weights, and then processed through a pulse firing function to finally generate continuous target detection box coordinates or class probabilities that can be used for decision-making.

[0092] For example, the firing threshold range of corresponding spiking neurons can be determined based on the activation distribution of neurons in each layer during the training phase, establishing a mapping relationship between continuous value activation and spiking frequency. The continuous value weight parameters obtained during training are mapped to the synaptic weights of the spiking neurons, making the spiking calculation results statistically approximate the output of the original continuous value network. The attention calculation unit and feedforward calculation unit inside the STEN module are converted into spiking neuron structures, enabling them to complete cross-modal feature enhancement under spiking event triggering conditions. In the neuromorphic attention module, the attention calculation and feedforward network in the NAIFI module are implemented in spiking form, while the dual-stream backbone network and some linear mapping layers maintain continuous value calculation form, thus forming a hybrid ANN-SNN architecture that only spiks high-energy-consuming modules. The hybrid ANN-SNN network performs feature calculation in an event-driven manner. Specifically, a spiking event is triggered and participates in subsequent calculations only when the cumulative input value of a neuron exceeds the corresponding firing threshold; when the input does not reach the threshold, no spiking is generated and no related calculation operations are performed, thereby avoiding invalid calculations and redundant energy consumption. This event-driven computation mechanism significantly reduces the computational complexity and energy consumption of the inference phase while maintaining perceptual accuracy. Finally, the high-level features obtained in the inference phase are input into the output module to generate the corresponding target perception results, which are output as detection boxes or category labels, thus completing the multimodal perception task.

[0093] In this embodiment, by employing continuous-value computation to construct and optimize the initial network model containing all core modules during the model training phase, the inherent problems of spiking neural networks (SNNs)—such as training difficulties, slow convergence, and typically lower accuracy compared to artificial neural networks—due to discrete pulse signals and complex dynamics are first addressed. This allows for the full utilization of mature artificial neural network training frameworks to obtain high-precision initial network parameters. During the model inference phase, based on these trained parameters, the computationally intensive cross-modal interaction and fusion module and the neuromorphic attention module are converted into an event-driven computation form based on spiking neurons, while retaining the continuous-value computation form of the two-stream feature extraction network. This constructs a hybrid network model, thus resolving the core contradiction between high computational power consumption and limited resources when deploying complex models on edge devices. Furthermore, by calibrating the firing threshold of spiking neurons based on the neuron activation distribution during artificial neural network training, a reasonable correspondence is established between continuous-value activation intensity and pulse firing frequency, resolving the significant performance degradation problem during knowledge transfer from artificial neural networks to spiking neural networks. By directly mapping the weight parameters learned by the artificial neural network to the synaptic weights of spiking neurons, the spiking neural network module can inherit and utilize the rich feature representations and fusion logic it has learned, solving the problem of low data efficiency when spiking neural networks need to learn complex tasks from scratch. During inference, the accumulation of membrane potential and threshold triggering mechanism of spiking neurons ensures that computation only occurs when the input features are effectively activated, achieving true event-driven processing and solving the energy waste caused by the full, intensive computation required by traditional artificial neural network architectures at every inference step. Finally, by weighting and nonlinearly transforming the outputs of the event-driven neurons, the sparse pulse sequence is decoded into continuous and stable target perception results. While ensuring that the model can be trained to a high level of performance through standard procedures, event-driven sparsity is introduced into the critical computation path during the inference stage. This achieves for the first time a balance between high precision and extremely high energy efficiency in complex multimodal perception tasks in smart city edge perception scenarios, enabling advanced models that were previously limited by power consumption to run in real time on resource-constrained terminal devices.

[0094] For example, such as Figure 6As shown, road traffic and urban monitoring scenarios with complex lighting changes, occlusion interference, and multiple target categories can be selected as experimental environments. The proposed method is tested on the FLIR and DroneVeh datasets, respectively. To compare the impact of multimodal fusion on perception accuracy, the target detection performance is evaluated using only visible light modality input, only infrared modality input, and simultaneous input of visible light (Vis) and infrared (IR) modalities. Experimental results show that on the FLIR dataset, when only visible light modality is used as input, the mean average precision (mAP@0.5) is approximately 57%, and when only infrared modality is used as input, mAP@0.5 is approximately 75%. However, after using the visible light-infrared multimodal (Vis+IR) fusion method of this application, mAP@0.5 is improved to approximately 81%. Under more stringent evaluation metrics, the proposed method also achieved significant improvements over single-modal input in mAP@0.75 and mAP@0.5:0.95, demonstrating that multimodal fusion can effectively enhance target localization accuracy and detection stability in complex environments. Experimental results on the DroneVeh dataset show a consistent trend. Compared to using only visible light or only infrared modal input, the multimodal fusion method described in this application achieves a detection accuracy of approximately 78% on mAP@0.5, an improvement of more than ten percentage points compared to single-modal input. It also exhibits superior detection performance on mAP@0.75 and mAP@0.5:0.95, especially in scenarios with dense small targets and occlusion, enabling more accurate differentiation of vehicle categories.

[0095] Based on the same inventive concept, such as Figure 7 As shown, this application also provides a neuromorphic multimodal sensing device, which includes: The data acquisition module 10 is used to acquire multimodal data and preprocess the multimodal data to form spatially aligned enhanced modal pairs; The feature extraction module 20 is used to process the modal data in the enhanced modality pair respectively through a dual-stream feature extraction network to obtain modal feature pairs at multiple semantic levels; The feature fusion module 30 is used to perform cross-modal interaction and fusion on the modal feature pairs of multiple semantic levels based on an event-driven pulse computing mechanism to generate multi-scale fused features. Feature enhancement module 40 is used to perform global modeling of high-level fusion features in the multi-scale fusion features based on neuromorphic attention mechanism to obtain global enhanced features; The modal perception module 50 is used to process the global enhancement features in an event-driven manner based on a hybrid network model to generate target perception results. The hybrid network model includes artificial neural networks and spiking neural networks.

[0096] It should be noted that the neuromorphic multimodal sensing device and the neuromorphic multimodal sensing method provided in this application are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned neuromorphic multimodal sensing method, and the repeated parts will not be described again.

[0097] In some embodiments, an electronic device provided in this application includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the above-described neuromorphic multimodal perception method.

[0098] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0099] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0100] This application also provides a non-transitory storage medium storing a computer program that, when executed by a processor, implements the aforementioned neuromorphic multimodal perception method. This storage medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned non-transitory storage medium carries one or more programs, which, when executed, implement the method as described in the embodiments or implementations of this application.

[0101] According to embodiments of this application, a non-transitory storage medium can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. A computer-readable signal medium can also be any storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0103] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A neuromorphic multimodal sensing method, characterized in that, The method includes: Acquire multimodal data and preprocess the multimodal data to form spatially aligned enhanced modal pairs; The modal data in the enhanced modal pair are processed by a dual-stream feature extraction network to obtain modal feature pairs at multiple semantic levels. Based on an event-driven pulse computing mechanism, cross-modal interaction and fusion are performed on the modal feature pairs of the multiple semantic levels to generate multi-scale fused features; Global enhancement features are obtained by globally modeling the high-level fusion features in the multi-scale fusion features based on the neuromorphic attention mechanism. Based on a hybrid network model, the global enhancement features are processed in an event-driven manner to generate target perception results. The hybrid network model includes artificial neural networks and spiking neural networks.

2. The method as described in claim 1, characterized in that, The multimodal data includes visible light modal data and infrared modal data. The step of acquiring the multimodal data and preprocessing the multimodal data to form spatially aligned enhanced modal pairs includes: Acquire visible light modal data and infrared modal data synchronously collected by visible light sensor and infrared sensor; Scale normalization is performed on the visible light modal data and the infrared modal data respectively to adjust the visible light modal data and the infrared modal data to the same spatial resolution, thereby obtaining scale-normalized visible light modal data and infrared modal data. The scale normalization operation includes a spatial mapping function based on interpolation resampling, and bilinear interpolation is used to process the visible light modal data and the infrared modal data respectively to maintain the continuity of image structure and thermal radiation distribution characteristics. The scale-normalized visible light modal data and infrared modal data are subjected to data augmentation transformation to obtain spatially aligned augmented modal pairs. The data augmentation transformation includes geometric augmentation and illumination augmentation, wherein the geometric augmentation is applied simultaneously to the visible light modal data and the infrared modal data, and the illumination augmentation is applied to the visible light modal data.

3. The method as described in claim 2, characterized in that, The step of processing each modality data in the modality pair using a dual-stream feature extraction network to obtain modality feature pairs at multiple semantic levels includes: The visible light modal data is input into the first branch of the dual-stream feature extraction network, which outputs multi-level visible light features. The infrared modal data is input into the second branch of the dual-stream feature extraction network, which outputs multi-level infrared features. Features corresponding to the same semantic level in the multi-level visible light features and the multi-level infrared features are selected to form the modal feature pairs of the multiple semantic levels.

4. The method as described in claim 1, characterized in that, The event-driven pulse computing mechanism, which performs cross-modal interaction and fusion of modal feature pairs at multiple semantic levels to generate multi-scale fused features, includes the following steps: For each semantic level modal feature pair, the visible light feature and infrared feature of the current level are encoded into pulse feature sequences respectively; Construct a cross-modal attention computation unit that takes pulse feature sequences as input; In the cross-modal attention calculation unit, the pulse feature sequence of the first modality is used to guide the enhancement of the pulse feature sequence of the second modality. After the bidirectional guidance enhancement is completed, the enhanced features of the two modalities are fused to obtain the fused features of the current level. The first modality is a visible light mode or an infrared mode, and the second modality is another mode different from the first modality. The encoding, processing, and fusion operations are performed on multiple semantic levels respectively to generate multi-scale fusion features corresponding to each semantic level.

5. The method as described in claim 1, characterized in that, The step of globally modeling the high-level fusion features in the multi-scale fusion features based on the neuromorphic attention mechanism to obtain the global enhanced features includes: The high-level fusion features in the multi-scale fusion features are mapped into a temporal pulse sequence using pulse coding. The time-series pulse sequence is input into the pulse-driven attention module to perform event-driven modeling of long-range spatial correlation and obtain the pulse attention output. The pulse attention output and the temporal pulse sequence are fused together using a residual structure to obtain the fused features; The fused features from multiple time steps are subjected to nonlinear enhancement processing to obtain globally enhanced features.

6. The method as described in claim 5, characterized in that, The step of performing nonlinear enhancement processing on the fused features from multiple time steps to obtain globally enhanced features further includes: The fused features are input into a pulse-driven feedforward network module for nonlinear enhancement to obtain a pulse-feedforward output. The pulse feedforward output is then fused again with the fused features through a residual structure to obtain the pulse representation of the enhanced features; The global enhanced features are obtained by time accumulation and averaging of the pulse representations of the enhanced features at multiple time steps.

7. The method as described in claim 1, characterized in that, The hybrid network model includes artificial neural networks and spiking neural networks. The step of processing the global enhancement features in an event-driven manner based on the hybrid network model to generate target perception results includes: Based on the neuron activation distribution of the artificial neural network during the training phase, the firing threshold of the corresponding spiking neurons is calibrated. The weight parameters of the corresponding modules in the artificial neural network are mapped to the synaptic weights of the spiking neurons; The global enhancement features are input into the hybrid network model; In the spiking neurons of a spiking neural network, a spiking event is triggered when the membrane potential accumulated by the global enhancement features exceeds a preset threshold. The outputs of neurons corresponding to impulse events are weighted, accumulated, and nonlinearly transformed to obtain target perception results, which include target detection boxes or category labels.

8. A neuromorphic multimodal sensing device, characterized in that, The neuromorphic multimodal sensing device includes: A data acquisition module is used to acquire multimodal data and preprocess the multimodal data to form spatially aligned enhanced modal pairs; The feature extraction module is used to process the modal data in the enhanced modality pair respectively through a two-stream feature extraction network to obtain modal feature pairs at multiple semantic levels; The feature fusion module is used to perform cross-modal interaction and fusion of the modal feature pairs at multiple semantic levels based on an event-driven pulse computing mechanism to generate multi-scale fused features. The feature enhancement module is used to perform global modeling of the high-level fusion features in the multi-scale fusion features based on the neuromorphic attention mechanism to obtain global enhanced features; The modal perception module is used to process the global enhancement features in an event-driven manner based on a hybrid network model to generate target perception results. The hybrid network model includes artificial neural networks and spiking neural networks.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the neuromorphic multimodal perception method as described in any one of claims 1 to 7.

10. A non-transitory storage medium, characterized in that, The non-transitory storage medium stores a computer program, which, when executed by a processor, implements the steps of the neuromorphic multimodal perception method as described in any one of claims 1 to 7.

Images