Image event multimodal semantic segmentation method, device and equipment

Abstract

The present application relates to the field of computer vision and artificial intelligence, in particular to an image event multi-modal semantic segmentation method, device and equipment, which can be applied to automatic driving, robot perception and intelligent transportation scenes; by fixed time window division time slice and accumulation of event information, asynchronous event stream is converted into TxHxW voxel tensor; by using the selective state scanning mechanism of Mamba architecture, the self-attention mechanism of traditional Transformer is replaced, the calculation complexity is reduced while modeling the global feature dependence, the memory and delay problems of Transformer in high resolution scene are solved; in addition, by cross-space interaction, image texture and event edge are aligned, cross-time interaction captures event dynamic timing, and residual connection is used to retain the inherent characteristics of the mode, effectively avoiding feature degradation caused by excessive fusion; finally, the image segmentation precision is effectively improved, and the processing efficiency and model robustness are improved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and artificial intelligence, and in particular to image event multimodal semantic segmentation methods, devices and equipment, which can be applied to scenarios such as autonomous driving, robot perception and intelligent transportation. Background Technology

[0002] Semantic segmentation is a crucial task in computer vision, widely applied in scenarios such as autonomous driving, intelligent robots, video surveillance, augmented reality, and intelligent transportation. It achieves accurate understanding of target objects and scenes by performing pixel-level classification on images or multimodal inputs. However, in real-world applications, segmentation algorithms not only need high accuracy but also must meet real-time performance and robustness requirements.

[0003] Existing semantic segmentation methods are mainly based on RGB images and employ convolutional neural networks (CNNs) or Transformers to achieve pixel-level prediction. Typical methods include SegFormer and SegNeXt. These methods achieve high accuracy on standard datasets through multi-scale feature extraction and decoder fusion. However, they are heavily reliant on traditional frame-based images and are easily affected by motion blur and exposure limitations under complex conditions such as high-speed motion, low lighting, and high dynamic range, leading to significant performance degradation.

[0004] To overcome these limitations, event cameras have gained attention as a novel visual sensor. They can asynchronously capture pixel intensity changes with microsecond-level temporal resolution, offering high temporal resolution, low latency, and wide dynamic range, demonstrating advantages in dynamic or low-light environments. However, because event streams only record brightness changes and lack rich semantic information such as color and texture, their performance is insufficient when used alone in intensive prediction tasks such as semantic segmentation.

[0005] To fully leverage the complementary characteristics of image and event modalities, multimodal fusion semantic segmentation methods have emerged in recent years. Early approaches were primarily based on convolutional neural networks (such as EDCNet and ESS), achieving basic modality fusion by jointly encoding event fluidized data with image features. However, due to the limited receptive field of convolutional networks, these methods are insufficient in modeling long-range dependencies and struggle to capture complex spatial relationships. Subsequently, Transformer-based cross-modal interaction methods (such as CMX, CMNeXt, and EISNet) introduced self-attention mechanisms, enabling global feature modeling and improving cross-modal consistency. However, the computational complexity of self-attention operations increases quadratically with the input length, leading to excessive memory consumption and inference latency in high-resolution scenarios, limiting real-time deployment. Other studies employ hybrid neural network architectures (such as HALSIE), combining spiking neural networks with conventional networks to utilize the temporal characteristics of event data and reduce energy consumption; however, overall accuracy remains limited, and the fusion effect is still unsatisfactory.

[0006] While these methods have improved upon the shortcomings of single-modality approaches to varying degrees, existing technologies still generally suffer from two core problems. First, feature fusion relies too heavily on the spatial dimension, failing to adequately model the inherent temporal dynamics of event data, thus hindering the full realization of the advantages of event modalities in high-speed motion or long-term time-series scenarios. Second, cross-modal feature alignment is not refined enough; simple fusion in the spatial domain easily introduces semantic ambiguity and fails to achieve sufficient complementarity. Furthermore, the computational overhead of Transformer-type methods is too high, severely limiting their application on embedded and edge devices.

[0007] Therefore, there is an urgent need for a novel multimodal semantic segmentation scheme that can achieve fine spatial-temporal fusion while maintaining high efficiency. Summary of the Invention

[0008] To address issues such as excessive computational complexity, insufficient utilization of temporal dynamic information, and inadequate cross-modal fusion accuracy, this application provides a method, apparatus, and device for multimodal semantic segmentation of image events.

[0009] Firstly, this application provides a method for multimodal semantic segmentation of image events: The image event multimodal semantic segmentation method includes: S1. Input Data and Preprocessing: Acquire image frame data from a regular camera and asynchronous event stream data from an event camera; process the image frame data into a C×H×W tensor with C channels, H height, and W width; using a voxel mesh representation method, divide the asynchronous event stream into multiple time slices within a fixed time window, accumulate event information in a three-dimensional mesh structure, and form a voxel tensor with dimensions T×H×W, where T is the number of time slices; S2. Feature extraction from dual-branch Mamba encoders: The preprocessed image frame data in C×H×W tensor form and the voxel tensor in T×H×W are respectively input into two parallel Mamba encoders. Each Mamba encoder consists of multiple layers of visual state space blocks. Multi-scale feature extraction is performed through a selective state scanning mechanism to obtain image branch features and event branch features respectively. S3. Feature Fusion of the Two-Dimensional Interaction Module: The two-dimensional interaction module is invoked to fuse the image branch features and the event branch features. The two-dimensional interaction module includes a cross-spatial interaction module and a cross-temporal interaction module. The cross-spatial interaction module aligns image texture information with event edge features through a cross-modal spatial attention mechanism and combines two-dimensional selective scanning to model long-distance spatial dependencies to obtain spatial fusion features. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism and uses bidirectional temporal selective scanning to model long-range temporal relationships to obtain temporal fusion features. S4. Modality-aware residual update: The spatial fusion feature and the temporal fusion feature are respectively connected back to the corresponding image branch feature channel and the event branch feature channel through modality-aware residuals to obtain the updated image branch feature and the updated event branch feature; S5. Multi-scale decoding and prediction: The updated image branch features and the updated event branch features are fused and input into a lightweight decoder. Spatial resolution is restored through stepwise upsampling, and a probability distribution tensor of size K×H×W is output, where K is the number of predefined semantic categories. The category label corresponding to the maximum probability at each pixel position is taken from the probability distribution tensor to obtain the semantic segmentation result.

[0010] By adopting the above technical solution, time slices are divided by a fixed time window and event information is accumulated, transforming the asynchronous event stream into a T×H×W voxel tensor. This preserves the temporal dynamic characteristics of the event data while achieving dimensional adaptation with the image frame (C×H×W tensor), laying a data foundation for cross-modal fusion. Furthermore, the selective state scanning mechanism of the Mamba architecture is used to replace the traditional Transformer's self-attention mechanism, reducing computational complexity from O(N^2) while modeling global feature dependencies. 2 The computational complexity is reduced to O(N), addressing memory and latency issues in high-resolution Transformer scenarios. Furthermore, by aligning image textures and event edges through cross-spatial interaction and capturing dynamic event sequences through cross-temporal interaction, and then preserving the inherent characteristics of the modality through residual connections, feature degradation caused by over-fusion is effectively avoided. Ultimately, this effectively improves image segmentation accuracy while also enhancing processing efficiency and model robustness.

[0011] Optionally, in step S1, each event in the asynchronous event stream data is represented as a quadruple (x, y, t, p), where (x, y) are pixel coordinates, t is the timestamp that triggered the event, and p∈{+1, -1} represents the polarity of the brightness change; when the brightness of a pixel in the scene increases, the pixel generates an event with a polarity of +1; when the brightness of a pixel in the scene decreases, the pixel generates an event with a polarity of -1.

[0012] By adopting the above technical solution, the event camera samples independently at the pixel level. When the pixel brightness change exceeds the threshold, the event is output asynchronously. The spatial location (x,y), timestamp (t), and brightness change direction (p∈{+1,-1}) of the event are accurately recorded through a quadruple (x,y,t,p). Here, +1 corresponds to brightness enhancement and -1 corresponds to brightness reduction. Essentially, it transforms continuous brightness changes into a discrete event sequence, realizing data representation with microsecond-level time resolution.

[0013] Optionally, in step S2, the Mamba encoder adopts a VMamba-T structure, which is pre-trained on the ImageNet-1K dataset and contains four stages of visual state space blocks. Specifically, the first stage visual state space block maintains the original input resolution and outputs a feature dimension of 96; the second stage visual state space block downsamples the input features by a factor of 2, resulting in an output feature dimension of 192; the third stage visual state space block downsamples the input features by a factor of 2, resulting in an output feature dimension of 384; and the fourth stage visual state space block downsamples the input features by a factor of 2, resulting in an output feature dimension of 768.

[0014] By adopting the above technical solution, a VMamba-T structure pre-trained on ImageNet-1K is used as the Mamba encoder. Through four stages of progressive downsampling (2x / stage), the feature dimension is simultaneously improved. This not only preserves local detail features (such as edges and textures) in the lower stages, but also models the global context through high-dimensional features in the higher stages, providing a rich feature foundation for subsequent fusion. At the same time, the selective scanning of the VSS Block maintains linear complexity, avoiding the high computational cost of traditional Transformers. The number of parameters is controlled within 25.44M, reducing the dependence on dataset annotation in scenarios such as autonomous driving.

[0015] Optionally, in step S3, the specific process of feature fusion performed by the cross-space interaction module includes: S31a. Add the image branch features and the event branch features element by element to obtain shallow fusion features; S32a. Average pooling and max pooling are performed on the image branch features, the event branch features and the shallow fusion features respectively to generate 6 spatial description maps; S33a. Perform convolution operation on the 6 spatial description maps to generate spatial attention weights, which include image branch spatial attention weights, event branch spatial attention weights, and shallow fusion feature spatial attention weights. S34a. Multiply the image branch features element-wise with the event branch space attention weights and the shallow fusion feature space attention weights to obtain the image branch space optimized features; multiply the event branch features element-wise with the image branch space attention weights and the shallow fusion feature space attention weights to obtain the event branch space optimized features. S35a. The image branch space optimization feature and the event branch space optimization feature are fused to obtain the initial spatial fusion feature; S36a. Input the initial spatial fusion features into the two-dimensional selective scanning module for four-way sequential scanning, and output the spatial fusion features.

[0016] By adopting the above technical solution, the Cross-Spatial Interaction Module (CSIM) addresses the problem of insufficient spatial alignment between image texture and event edge. It initially establishes cross-modal spatial association by adding the feature elements of the image and event branches, and then obtains three types of spatial attention weights (image, event, and shallow fusion) through convolution, realizing local weight allocation based on global spatial information. Finally, it models long-range spatial dependencies through two-dimensional selective scanning (four-way sequence), thereby solving the problem of limited receptive field in traditional CNNs.

[0017] Optionally, in step S3, the specific process of feature fusion performed by the cross-time interaction module includes: S31b. According to the temporal sequence of the event branch features, the image branch features are interleaved into the event branch features to obtain temporally interleaved features, wherein the dimension of the temporally interleaved features is 2T×H×W; S32b. Perform max pooling and average pooling on the time dimension of the temporal interleaving features to obtain temporal pooling features. S33b: Perform a convolution operation on the temporal pooling features to generate temporal attention weights; S34b. Multiply the image branch features element-wise with the temporal attention weights to obtain image branch temporal optimization features; multiply the event branch features element-wise with the temporal attention weights to obtain event branch temporal optimization features. S35b: The image branch time optimization feature and the event branch time optimization feature are fused to obtain the initial time fusion feature; S36b. After flattening the initial time fusion feature into a feature sequence, input it into the bidirectional selective scanning module to perform forward sequence scanning and reverse sequence scanning, and output the time fusion feature.

[0018] By adopting the above technical solution, image features are inserted according to the event time sequence to form a 2T×H×W time-series interleaved feature, which forces the establishment of a time correlation between images and events. Global time-series information is extracted from the time dimension by pooling, and then time weights are obtained by convolution. Feature filtering based on time-series changes is realized. After flattening the initial time fusion features, forward and backward scanning are performed to fully capture the long-range time-series relationship of past → present → future, which solves the problem of insufficient modeling of event stream time dynamic information.

[0019] Optionally, in step S4, the specific process of updating the modality-aware residual includes: S41. The spatial fusion feature is divided into spatial image sub-features and spatial event sub-features, and the temporal fusion feature is divided into temporal image sub-features and temporal event sub-features; S42. Call the spatial attention module and the temporal attention module to refine the spatial image sub-features, spatial event sub-features, temporal image sub-features and temporal event sub-features to obtain refined spatial image features, refined spatial event features, refined temporal image features and refined temporal event features; S43. The refined features of the spatial image and the refined features of the temporal image are concatenated and fused by convolution, and the residuals are added to the image branch features obtained in step S2 to obtain the updated image branch features; the refined features of the spatial event and the refined features of the temporal event are concatenated and fused by convolution, and the residuals are added to the event branch features obtained in step S2 to obtain the updated event branch features. S44. The updated image branch features and the updated event branch features are concatenated and fused to obtain fused multi-scale features.

[0020] By adopting the above technical solution, residual connection preserves the core information such as color texture of the image and the temporal dynamics of the event, which solves the problem of loss of modal characteristics caused by over-fusion (such as loss of texture details in the image and loss of temporal advantage in the event). The refined features complement the original features, so that the updated branch features have both the advantages of the original modality and the cross-modal fusion gain, providing higher quality input for decoding and prediction.

[0021] Optionally, in step S5, the lightweight decoder is designed with reference to the SegFormer decoder. When the lightweight decoder performs the decoding operation, it first performs channel compression processing on the updated image branch features and the updated event branch features, then performs multi-scale stitching on the channel compressed features, and finally restores the spatial resolution to the same as the image frame data through an upsampling operation.

[0022] By adopting the above technical solutions, channel compression significantly reduces the number of decoder parameters. Combined with the low complexity of the Mamba encoder, it can better meet real-time requirements. The combination of multi-scale stitching avoids the loss of details caused by upsampling, ensuring segmentation accuracy (such as clear edges of small target pedestrians), and effectively improving the accuracy and completeness of the final segmentation results.

[0023] Optionally, in step S1, the duration of the fixed time window is 50ms, the number of time slices T=10, the resolution of the image frame data is 346×260, and the resolution of the voxel tensor is 346×260.

[0024] Secondly, this application provides an image event multimodal semantic segmentation apparatus; the apparatus includes modules for performing the method in the first aspect or any possible implementation of the first aspect: The device includes: The data acquisition and preprocessing module is used to acquire image frame data captured by a regular camera and asynchronous event stream data captured by an event camera; process the image frame data into a C×H×W tensor with C channels, H height, and W width; and use a voxel mesh representation method to divide the asynchronous event stream into multiple time slices within a fixed time window, accumulate event information in a three-dimensional mesh structure, and form a voxel tensor with dimensions T×H×W, where T is the number of time slices. A dual-branch Mamba encoder is used to input the preprocessed image frame data in the form of C×H×W tensor and the voxel tensor of T×H×W into two parallel Mamba encoders respectively. Each Mamba encoder consists of multiple layers of visual state space blocks and performs multi-scale feature extraction through a selective state scanning mechanism to obtain image branch features and event branch features respectively. A two-dimensional interaction module is used to fuse the image branch features and the event branch features. The two-dimensional interaction module includes a cross-spatial interaction module and a cross-temporal interaction module. The cross-spatial interaction module aligns image texture information with event edge features through a cross-modal spatial attention mechanism and combines two-dimensional selective scanning to model long-distance spatial dependencies to obtain spatial fusion features. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism and uses bidirectional temporal selective scanning to model long-range temporal relationships to obtain temporal fusion features. The modality-aware residual update module is used to connect the spatial fusion features and the temporal fusion features back to the corresponding image branch feature channels and event branch feature channels through modality-aware residuals, respectively, to obtain updated image branch features and updated event branch features; The multi-scale decoding and prediction module is used to fuse the updated image branch features and the updated event branch features and input them into the lightweight decoder. The spatial resolution is restored by upsampling at each level, and the output is a probability distribution tensor of size K×H×W, where K is the number of predefined semantic categories. The semantic segmentation result is obtained by taking the category label corresponding to the maximum probability at each pixel position of the probability distribution tensor.

[0025] Thirdly, this application provides a computer device including a processor, a memory, and a communication bus, wherein the communication bus is used to realize a communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement the method described in any of the preceding claims.

[0026] Fourthly, this application also provides a computer-readable storage medium storing a computer program; the computer program can be executed by a processor to implement the method described above.

[0027] Fifthly, this application also provides a computer program product, including a computer program that can be executed by a processor to implement the method described above.

[0028] This application includes at least the following beneficial technical effects: By introducing a state-space modeling mechanism with linear complexity, the overall computational cost is reduced and inference efficiency is improved. Simultaneously, this invention designs a two-dimensional interactive fusion strategy, achieving fine alignment of color texture information and edge structures in the spatial dimension, and modeling the dynamic characteristics of event flow in the temporal dimension, ensuring the advantages of the fusion result in temporal consistency and semantic integrity. This invention not only improves the robustness of the model in complex scenarios such as dynamic and low-light conditions, but also guarantees high-precision semantic segmentation performance while meeting real-time requirements, making it suitable for various application scenarios such as autonomous driving, unmanned systems, and intelligent monitoring. Attached Figure Description

[0029] Figure 1 This is a schematic flowchart of an image event multimodal semantic segmentation method provided in an embodiment of this application; Figure 2 A diagram illustrating an image event multimodal semantic segmentation architecture provided in this application embodiment; Figure 3 This is a schematic diagram of a two-dimensional interactive module structure provided in an embodiment of this application; Figure 4 A comparison chart of the performance, number of parameters, and computational load of the method proposed in this invention and existing methods is provided for embodiments of this application. Figure 5 A visualization comparison diagram of the proposed method of this invention and existing segmentation methods provided for embodiments of this application; Figure 6 A schematic diagram of an image event multimodal semantic segmentation device provided in this application embodiment; Figure 7 This is a schematic diagram of a computer device structure provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items. The term “exemplary” means “serving as an example, embodiment, or illustration,” and any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. The terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, “a plurality” means two or more.

[0032] The purpose of this invention is to address three major problems in existing image-event multimodal semantic segmentation methods: excessively high computational complexity, insufficient utilization of temporal dynamic information, and inadequate cross-modal fusion accuracy, and to propose a new solution. While existing Transformer-based fusion methods can achieve global feature modeling, the quadratic computational complexity of the self-attention mechanism makes them difficult to apply in real-time scenarios. Furthermore, some methods over-rely on spatial feature alignment, failing to fully exploit the temporal information of event data, resulting in performance degradation in dynamic environments such as high-speed motion and low-light conditions. In addition, fusion strategies often only perform simple weighting or attention interactions in a single dimension, leading to incomplete resolution of semantic differences between cross-modal features, thus affecting segmentation accuracy.

[0033] To address the aforementioned issues, this invention proposes a multimodal semantic segmentation method for image events based on the Mamba architecture. By introducing a state-space modeling mechanism with linear complexity, the overall computational cost is reduced, and inference efficiency is improved. Simultaneously, this invention designs a two-dimensional interactive fusion strategy, achieving fine alignment of color texture information and edge structures in the spatial dimension, and modeling the dynamic characteristics of the event flow in the temporal dimension, ensuring the advantages of the fusion result in temporal consistency and semantic integrity. This method not only improves the robustness of the model in complex scenarios such as dynamic and low-light conditions, but also ensures high-precision semantic segmentation performance while meeting real-time requirements, making it suitable for various application scenarios such as autonomous driving, unmanned systems, and intelligent monitoring.

[0034] refer to Figure 1 This paper presents a multimodal semantic segmentation method for image events, aiming to achieve efficient cross-modal feature fusion and accurate pixel-level prediction. The overall framework is as follows: Figure 2 As shown.

[0035] This method mainly includes five core steps: input data preprocessing, feature extraction from a dual-branch Mamba encoder, feature fusion from a two-dimensional interactive module (DDIM), modality-aware residual update, and decoding prediction. S1: Input Data and Preprocessing Acquire image frame data captured by a regular camera and asynchronous event stream data captured by an event camera; Image frame data refers to two-dimensional intensity images captured by a regular camera at a fixed frame rate. Each frame contains complete scene brightness and texture information, such as common grayscale images or RGB color images. The image frame data is processed into a C×H×W tensor with C channels, H height, and W width. The asynchronous event stream is a sparse data stream obtained by the event camera through independent sampling at the pixel level. When the brightness change of a pixel exceeds a set threshold, the event camera immediately outputs an event without waiting for the global frame to be updated. Each event in the asynchronous event stream data is represented as a quadruple (x, y, t, p), where (x, y) are the pixel coordinates, t is the timestamp that triggered the event, and p∈{+1, -1} represents the polarity of the brightness change; when the brightness of a pixel in the scene increases, that pixel generates an event with a polarity of +1; when the brightness of a pixel in the scene decreases, that pixel generates an event with a polarity of -1.

[0036] For event data, this invention employs a voxel grid representation method, which divides the asynchronous event stream into multiple time slices within a fixed time window, accumulates event information in a three-dimensional grid structure, and forms a voxel tensor with dimensions T×H×W, where T is the number of time slices. This preprocessing process preserves the spatial distribution and temporal dynamics of the event data, ensuring the integrity of subsequent time series modeling.

[0037] S2: Feature Extraction from a Dual-Branch Mamba Encoder The preprocessed image frame data in C×H×W tensor form and the event voxel data in T×H×W form are input into two parallel Mamba encoders. Each Mamba encoder consists of multiple layers of visual state space blocks. Multi-scale feature extraction is performed through a selective state scanning mechanism to obtain image branch features and event branch features respectively.

[0038] like Figure 2As shown, image frame data and event voxel data are input into two parallel Mamba encoders, each consisting of multiple layers of visual state space blocks to achieve multi-scale feature extraction. Unlike the Transformer, Mamba employs a selective state scanning mechanism, which can reduce computational complexity to a linear level while maintaining global dependency modeling capabilities, thereby significantly improving inference efficiency and meeting real-time processing requirements.

[0039] Continue to refer to Figure 2 In an optional embodiment of this application, the Mamba encoder adopts a VMamba-T structure, which is pre-trained on the ImageNet-1K dataset and contains four stages of visual state space blocks. Specifically, the first stage visual state space block maintains the original input resolution and outputs a feature dimension of 96; the second stage visual state space block downsamples the input features by a factor of 2, resulting in an output feature dimension of 192; the third stage visual state space block downsamples the input features by a factor of 2, resulting in an output feature dimension of 384; and the fourth stage visual state space block downsamples the input features by a factor of 2, resulting in an output feature dimension of 768.

[0040] S3: Feature Fusion of Two-Dimensional Interactive Modules refer to Figure 3 The system invokes a dual-dimensional interaction module (DDIM) to fuse the image branch features and the event branch features. The dual-dimensional interaction module includes a cross-spatial interaction module (CSIM) and a cross-temporal interaction module (CTIM). The cross-spatial interaction module aligns image texture information with event edge features through a cross-modal spatial attention mechanism and combines two-dimensional selective scanning to model long-distance spatial dependencies, thereby obtaining spatial fusion features. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism and uses bidirectional temporal selective scanning to model long-range temporal relationships, thereby obtaining temporal fusion features.

[0041] The cross-spatial interaction module achieves fine alignment of image texture information and event edge features through a cross-modal spatial attention mechanism, and combines 2D selective scanning (SS2D) to model long-range spatial dependencies, enhancing the fusion effect between local structure and global context. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism, and uses bidirectional temporal selective scanning (BTSS) to model long-range temporal relationships, thereby improving semantic consistency in dynamic scenes. CSIM and CTIM work alternately in the multi-scale feature extraction stage, progressively enhancing cross-modal consistency.

[0042] Continue to refer to Figure 3 The specific process of feature fusion performed by the cross-space interaction module includes: S31a. Add the image branch features and event branch features element by element to obtain shallow fusion features; S32a. Average pooling and max pooling are performed on the image branch features, event branch features and shallow fusion features respectively to generate 6 spatial description maps; S33a. Perform convolution operation on 6 spatial description maps to generate spatial attention weights, which include image branch spatial attention weights, event branch spatial attention weights and shallow fusion feature spatial attention weights. S34a. Multiply the image branch features element-wise with the event branch space attention weights and the shallow fusion feature space attention weights to obtain the image branch space optimized features; multiply the event branch features element-wise with the image branch space attention weights and the shallow fusion feature space attention weights to obtain the event branch space optimized features. S35a. The image branch space optimization feature and the event branch space optimization feature are fused to obtain the initial spatial fusion feature; S36a. Input the initial spatial fusion features into the two-dimensional selective scanning module for four-way sequential scanning, and output the spatial fusion features.

[0043] Continue to refer to Figure 3 The specific process of feature fusion performed by the cross-time interaction module includes: S31b. According to the temporal sequence of the event branch features, the image branch features are interleaved into the event branch features to obtain temporally interleaved features, wherein the dimension of the temporally interleaved features is 2T×H×W; S32b. Perform max pooling and average pooling on the time dimension of the temporal interleaving features to obtain temporal pooling features. S33b: Perform a convolution operation on the temporal pooling features to generate temporal attention weights; S34b. Multiply the image branch features element-wise with the temporal attention weights to obtain image branch temporal optimization features; multiply the event branch features element-wise with the temporal attention weights to obtain event branch temporal optimization features. S35b: The image branch time optimization feature and the event branch time optimization feature are fused to obtain the initial time fusion feature; S36b. After flattening the initial time fusion feature into a feature sequence, input it into the bidirectional selective scanning module to perform forward sequence scanning and reverse sequence scanning, and output the time fusion feature.

[0044] S4: Modality-aware residual update In each interaction phase, the spatial fusion features and temporal fusion features output by DDIM are respectively connected back to the corresponding image branch feature channels and event branch feature channels through modality-aware residuals to obtain updated image branch features and updated event branch features; this achieves information supplementation and stable training, while preserving the characteristics of each modality and avoiding feature degradation problems caused by over-fusion.

[0045] Specifically, the modality-aware residual update process includes: S41. Decompose the spatial fusion features into spatial image sub-features and spatial event sub-features, and decompose the temporal fusion features into temporal image sub-features and temporal event sub-features; Specifically, before CSIM fusion, image branch features Event branching features Dimension alignment has been performed, unifying the dimensions to the same T×H×W (T is the number of time slices, H is the height, and W is the width); spatial fusion features. The dimensions of the split feature are completely consistent with the dimensions of the aligned feature, ensuring that the sub-features after splitting can match the original branch features. Spatial integration features The information originates from the texture features of the image branch and the edge features of the event branch. During splitting, features are separated according to their modality source, and the original image branch features are included in the fused features. The feature part is split into spatial image sub-features. ; The features corresponding to the original event branches in the fused features The feature part is broken down into spatial event sub-features. ; Spatial image sub-features obtained after splitting Spatial event sub-features Their dimensions are all maintained as T×H×W, consistent with the original image branch features. Event branch characteristics The dimensions are completely consistent, providing a dimensional basis for subsequent residual connections back to the original feature channels.

[0046] Similarly, before CTIM fusion, image branch features By event branch characteristics The temporal sequences are interleaved, forming a temporal interleaving feature with dimensions 2T×H×W. Temporal fusion features subsequently generated by bidirectional selective scanning Although its dimensions are flattened and scanned, they will eventually be restored to the T×H×W dimensions that match the original branch features. Temporal fusion features The information originates from the temporal correlation features of image branches and the dynamic temporal features of event branches. During splitting, features are separated according to temporal affiliation, and the original image branch features are included in the fused features. The temporal features are broken down into temporal image sub-features. ; The features corresponding to the original event branches in the fused features The dynamic time-series features are broken down into time event sub-features. ; Split temporal image sub-features Time event sub-features Their dimensions are all maintained as T×H×W, consistent with the original image branch features. Event branch characteristics Dimensional consistency ensures that updates are superimposed on the original features, preserving the inherent characteristics of each modality.

[0047] S42. The spatial attention module (SA) is invoked to refine the spatial image sub-features and spatial event sub-features respectively, obtaining the corresponding refined spatial sub-features; the temporal attention module (TA) is invoked to refine the temporal image sub-features and temporal event sub-features, obtaining the corresponding refined temporal sub-features; the specific process includes: Invoking the Spatial Attention Module (SA) to process spatial image sub-features Refinement is performed to obtain refined sub-features of the spatial image. ; Invoking the Spatial Attention Module (SA) to process spatial event sub-features Refinement, spatial event refinement sub-features ; Invoking the Temporal Attention (TA) module to process temporal image sub-features Refinement is performed on the time-series image, refining sub-features. ; Invoking the Temporal Attention (TA) module to analyze the sub-features of temporal events Refinement, time-event refinement sub-features ; After refinement, image branch features + spatial image refined sub-features, event branch features + spatial event refined sub-features, image branch features + temporal image refined sub-features, and event branch features + temporal event refined sub-features are used to obtain four outputs (e.g., Figure 3 (CSIM has two outputs, CTIM has two outputs) In the formula, Represents refined features of spatial images. Representing the refined characteristics of spatial events, Represents refined features of a time-based image. This indicates the refined characteristics of time-related events.

[0048] S43. Refine the spatial features of CSIM images Refined features of CTIM temporal images Perform convolutional fusion and add the residuals from the image branch features obtained in step S2 to obtain the updated image branch features; refine the spatial event features of CSIM. Refined features of CTIM's time events Perform convolutional fusion and add the residuals from the event branch features obtained in step S2 to obtain the updated event branch features; specifically including: In the formula, This represents the updated image branch features. This indicates the updated event branch characteristics.

[0049] S44. Update the image branch features With updated event branch features By concatenating and fusing convolutional features, multi-scale features can be obtained. : S5: Multi-scale Decoding and Prediction Integrating multi-scale features The input is a lightweight decoder, which recovers the spatial resolution through stepwise upsampling and outputs a probability distribution tensor of size K×H×W, where K is the number of predefined semantic categories. The semantic segmentation result is obtained by taking the category label corresponding to the maximum probability at each pixel position of the probability distribution tensor.

[0050] Multiscale features after fusion After step-by-step upsampling by a lightweight decoder, the spatial resolution is restored and pixel-level semantic segmentation results are output. The lightweight decoder structure is designed with reference to SegFormer. When the decoder performs decoding operations, it first performs channel compression processing on the updated image branch features and the updated event branch features, then performs multi-scale stitching on the channel-compressed features, and finally restores the spatial resolution to the same as the image frame data through upsampling operations.

[0051] The semantic segmentation result is a pixel-by-pixel classification mapping of the input image and event fusion data. Specifically, the decoder outputs a probability distribution tensor of size K×H×W, where H and W are the spatial dimensions of the input image, and K is the predefined number of semantic categories. The semantic segmentation map is obtained by taking the category label corresponding to the highest probability at each pixel location from this tensor. In other words, the semantic segmentation result is essentially a pixel-level classification matrix that assigns a clear semantic category to each pixel in the input scene, such as "road," "vehicle," or "pedestrian," thereby achieving a fine-grained understanding of the scene.

[0052] The proposed image event multimodal semantic segmentation method based on the Mamba state space model significantly outperforms existing technologies in terms of accuracy and efficiency, and maintains strong robustness in complex scenes. Specifically, it offers the following advantages: (1) Significantly improve segmentation accuracy Table 1. Performance comparison of the proposed method (MagbaSeg) with existing methods on the DDD17 and DSEC datasets. As shown in Table 1, the test results on the DDD17 and DSEC public datasets demonstrate that the proposed method achieves state-of-the-art performance in cross-modal semantic segmentation tasks. On the DDD17 dataset, the proposed method achieves a mean Intersection over Union (mIoU) of 77.56%, and on the DSEC dataset, it achieves 75.10%, representing improvements of approximately 2.5 percentage points over the current state-of-the-art methods. This advantage stems from the joint modeling of the Two-Dimensional Interaction Module (DDIM) in both spatial and temporal dimensions, which fully leverages the dynamic characteristics of event flows and the textural details of images to achieve refined modal fusion.

[0053] (2) Reduce computational complexity and improve real-time performance Compared with the Transformer-based cross-modal fusion method, the Mamba state-space model used in this invention has linear time complexity, avoiding the quadratic computational overhead of the self-attention mechanism.

[0054] Table 2. Comparison of performance, number of parameters, and computational cost between the method proposed in this invention and existing methods. As shown in Table 2 and Figure 4 As shown, the model parameters of this invention on the DDD17 dataset are only 25.44M, and the multiply-accumulate operation is 15.59G, which is far lower than the computational cost required by typical Transformer models (such as CMX and EISNet), while maintaining higher accuracy. This characteristic enables this invention to better adapt to the real-time processing requirements in autonomous driving and edge computing scenarios.

[0055] (3) Enhance robustness in dynamic scenarios This invention exhibits higher stability in dynamic environments and low-light scenes. By using the CTIM module to perform bidirectional selective scanning of the temporal characteristics of the event stream, the method of this invention can maintain prediction consistency and reduce semantic missegmentation problems under conditions of rapid movement and frequent changes in lighting. Figure 5 The qualitative results show that the present invention can more accurately segment small targets such as pedestrians and traffic signs in complex traffic scenarios, while existing methods are prone to edge blurring or missed detection in these scenarios.

[0056] (4) The fusion strategy is significantly better than the traditional method. Table 3 Comparison between the fusion strategy proposed in this invention and existing fusion strategies As shown in Table 3, the comparative experimental results demonstrate that the Two-Dimensional Interaction Module (DDIM) proposed in this invention significantly outperforms existing methods in cross-modal feature fusion performance. Under the same encoder and decoder conditions, DDIM achieves an mIoU of 77.56%, exceeding the baseline additive fusion (74.38%) and other representative fusion strategies (such as FFM, MRFM, and CSF). This indicates that the present invention possesses stronger expressive power in feature interaction modeling and can effectively alleviate cross-modal semantic ambiguity issues.

[0057] (5) The spatial and temporal interaction modules work together to improve performance. Table 4 Ablation results of different modules in the method proposed in this invention According to the ablation experiments in Table 4, the CSIM and CTIM modules designed in this invention play a complementary role in feature fusion. Removing either module leads to a performance decrease; for example, removing CTIM reduces mIoU to 76.32%, and removing CSIM reduces it to 76.20%, while using both modules together improves performance to 77.56%. This result demonstrates that the dual-dimensional interaction mechanism of this invention can achieve information complementarity in both spatial and temporal dimensions, thereby significantly improving the accuracy and robustness of semantic segmentation.

[0058] This embodiment combines Figure 1 and Figure 2 This document details the optimal implementation of the present invention in autonomous vehicles. Using the DDD17 dataset as an example, this embodiment simulates a semantic segmentation task in a real-world driving environment, including the entire process of data acquisition, preprocessing, model building, training, and inference.

[0059] Step 1: Data Acquisition and Preprocessing The autonomous vehicle is equipped with a DAVIS event camera (capable of simultaneously acquiring image frames and event streams). Both the image frames and the event stream have a resolution of 346×260. The event stream is an asynchronous sequence of events, with each event containing coordinates (x, y), a timestamp t, and polarity p∈(+1,-1). The acquisition time window is fixed at 50ms, and the acquired event sequence serves as the raw input.

[0060] Step 2: Feature Extraction from the Dual-Branch Mamba Encoder like Figure 2 As shown, the event stream E and image I pass through the Voxel Grid module and Stem Block, respectively, before entering the Mamba encoder. The Voxel Grid module is responsible for representing the original asynchronous event stream using a voxel grid: a given time window is divided into T=10 time slices, and events are accumulated in each time slice to obtain a voxel grid tensor. in Indicates the event coordinates, For timestamps, It is polar. For the first This module not only completes the voxelization representation, but also maps the 3D voxel tensor into a feature sequence through patch embedding operations, which is convenient for subsequent input into the encoder.

[0061] Image input First, the Stem Block is used. This module is essentially a convolutional patch embedding layer, which divides the original two-dimensional image into several patches and maps them to a high-dimensional feature space, resulting in an initial feature sequence of uniform dimension. In this way, the image branch and the event branch complete a unified representation process before entering the Mamba encoder.

[0062] Subsequently, voxelization event characteristics and image features Two parallel Mamba encoders are input separately. Each encoder uses a VMamba-T architecture (pre-trained on ImageNet-1K) and contains four stages of visual state space blocks (VSSBlock): Phase 1: The input resolution remains the same size, and the output feature dimension is C1=96.

[0063] Second stage: Downsample by 2 times, output dimension C2=192.

[0064] Third stage: Downsample by 2 times, output dimension C3=384.

[0065] Fourth stage: Downsampled by 2 times, output dimension C4=768.

[0066] Each VSS Block models long-range dependencies through a selective scan mechanism (S6 Block), with a computational complexity of O(n log n). Far lower than Transformer Event branches capture dynamic edges, while image branches extract texture details.

[0067] Step 3: Feature Fusion of Two-Dimensional Interactive Module (DDIM) After each encoding stage, DDIM is applied for cross-modal fusion, such as... Figure 3 As shown. DDIM includes the Cross-Spatial Interaction Module (CSIM) and the Cross-Temporal Interaction Module (CTIM).

[0068] CSIM processing: Event branching characteristics and image branch features (Dimensional alignment is) First, add the elements to obtain a shallow fusion. Then, six spatial description maps are generated through average / max pooling, and spatial attention weights are produced after convolution. (divided into) Applying attention: The initial spatial fusion features are obtained through fusion. .then, Input SS2D module (four-way sequence scan), output spatial fusion features .

[0069] CTIM Processing: Event Branching Features and image branch features Temporal interleaving features are obtained by interleaving event characteristics in a temporal sequence. ( The temporally interleaved features are subjected to max pooling and average pooling along the time dimension to obtain temporally pooled features, which are then convolved to generate temporal attention weights. The image branching features are multiplied element-wise with the temporal attention weights to obtain the image branching temporal optimization features. The event branch features are multiplied element-wise with the time attention weights to obtain the event branch time optimization features. Image branching time optimization features Event branch time optimization features The initial temporal fusion features are obtained by performing fusion. Flattened into a bidirectional S6 scan (forward + backward), the output temporal fusion features are obtained. .

[0070] Step 4: Modality-aware residual update Fusion features and The feature is split into event branches and image branches, and refined using a spatial / temporal attention module (SA / TA) to obtain multi-scale fused features. Modal characteristics are preserved through residual connections to avoid feature degradation.

[0071] Step 5: Multi-scale decoding and prediction Multi-scale fusion features are upsampled step-by-step through an MLP decoder: compressed channels are then concatenated to restore the original resolution. The final output is a pixel-level semantic map covering six categories of labels (road, background, vegetation, pedestrian, vehicle, and others). Cross-entropy loss is used for training.

[0072] The specific training and reasoning process includes: The model was trained for 60 epochs on an NVIDIA RTX-4090D using PyTorch and the AdamW optimizer, with an initial learning rate of 2e-4 and a batch size of 12. At inference, the model had 25.44M parameters and 15.59G MACs. The measured mIoU was 77.56%, as shown in Table 2.

[0073] Based on the above method embodiments, this embodiment also provides an image event multimodal semantic device, which can be used to implement the steps of the above method.

[0074] refer to Figure 6 The device includes: The data acquisition and preprocessing module 61 is used to acquire image frame data captured by a regular camera and asynchronous event stream data captured by an event camera; process the image frame data into a C×H×W tensor with C channels, H height, and W width; and use a voxel mesh representation method to divide the asynchronous event stream into multiple time slices within a fixed time window, accumulate event information in a three-dimensional mesh structure, and form a voxel tensor with dimensions T×H×W, where T is the number of time slices. The dual-branch Mamba encoder 62 is used to input the preprocessed image frame data in the form of C×H×W tensor and the voxel tensor of T×H×W into two parallel Mamba encoders respectively. Each Mamba encoder consists of multiple layers of visual state space blocks and performs multi-scale feature extraction through a selective state scanning mechanism to obtain image branch features and event branch features respectively. The two-dimensional interaction module 63 is used to fuse the image branch features and the event branch features. The two-dimensional interaction module includes a cross-spatial interaction module and a cross-temporal interaction module. The cross-spatial interaction module aligns image texture information with event edge features through a cross-modal spatial attention mechanism and combines two-dimensional selective scanning to model long-distance spatial dependencies to obtain spatial fusion features. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism and uses bidirectional temporal selective scanning to model long-range temporal relationships to obtain temporal fusion features. The modality-aware residual update module 64 is used to connect the spatial fusion feature and the temporal fusion feature back to the corresponding image branch feature channel and the event branch feature channel through modality-aware residuals, respectively, to obtain the updated image branch feature and the updated event branch feature; The multi-scale decoding and prediction module 65 is used to fuse the updated image branch features and the updated event branch features and input them into the lightweight decoder. The spatial resolution is restored by upsampling at each level, and the output is a probability distribution tensor of size K×H×W, where K is the number of predefined semantic categories. The semantic segmentation result is obtained by taking the category label corresponding to the maximum probability at each pixel position of the probability distribution tensor.

[0075] Various variations and specific examples of the methods provided in the above embodiments are also applicable to the apparatus of this embodiment. Through the foregoing detailed description of the methods, those skilled in the art can clearly understand the implementation method of the apparatus in this embodiment. For the sake of brevity, they will not be described in detail here.

[0076] To better execute the above methods, embodiments of this application also provide a computer device, such as... Figure 7As shown, the computer device includes a processor 71, a memory 72, and a communication bus 73 for enabling communication between the processor 71 and the memory 72.

[0077] The computer device can be implemented in various forms, including in-vehicle computers, mobile phones, tablets, handheld computers, laptops, and desktop computers.

[0078] The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the methods provided in the above embodiments; the data storage area may store data involved in the methods provided in the above embodiments.

[0079] Optionally, the memory may be a read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), optical disc (including compact disc read-only memory (CD-ROM), compressed optical disc, laser disc, digital versatile optical disc, Blu-ray disc, etc.), magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory exists independently and is connected to the processor via a communication bus, or the memory is integrated with the processor.

[0080] A processor may include one or more processing cores. The processor executes instructions, programs, code sets, or instruction sets stored in memory, and calls data stored in memory to perform various functions and process data as described in this application. The processor may be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the above-described processor functions may also be other types, and the embodiments of this application do not specifically limit this.

[0081] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0082] In an alternative embodiment, the computer device may also include a communication interface (not shown) for communication with other devices.

[0083] This application provides a computer-readable storage medium, including, for example, various media capable of storing program code such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. This computer-readable storage medium stores a computer program that can be loaded by a processor and execute the methods of the above embodiments.

[0084] This application also provides a computer program product comprising a computer program tangibly embodied on a readable medium thereof, the computer program containing program code for performing any of the methods described in any of the embodiments of this application, the computer program being downloadable and installable over a network, and / or installed from a removable medium (such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc.).

[0085] The above description of the embodiments is only used to provide a detailed introduction to the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this application, and should not be construed as a limitation of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for multimodal semantic segmentation of image events, characterized in that, The method includes: S1. Input Data and Preprocessing: Acquire image frame data from a regular camera and asynchronous event stream data from an event camera; process the image frame data into a C×H×W tensor with C channels, H height, and W width; using a voxel mesh representation method, divide the asynchronous event stream into multiple time slices within a fixed time window, accumulate event information in a three-dimensional mesh structure, and form a voxel tensor with dimensions T×H×W, where T is the number of time slices; S2. Feature extraction from dual-branch Mamba encoders: The preprocessed image frame data in C×H×W tensor form and the voxel tensor in T×H×W are respectively input into two parallel Mamba encoders. Each Mamba encoder consists of multiple layers of visual state space blocks. Multi-scale feature extraction is performed through a selective state scanning mechanism to obtain image branch features and event branch features respectively. S3. Feature Fusion of the Two-Dimensional Interaction Module: The two-dimensional interaction module is invoked to fuse the image branch features and the event branch features. The two-dimensional interaction module includes a cross-spatial interaction module and a cross-temporal interaction module. The cross-spatial interaction module aligns image texture information with event edge features through a cross-modal spatial attention mechanism and combines two-dimensional selective scanning to model long-distance spatial dependencies to obtain spatial fusion features. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism and uses bidirectional temporal selective scanning to model long-range temporal relationships to obtain temporal fusion features. S4. Modality-aware residual update: The spatial fusion feature and the temporal fusion feature are respectively connected back to the corresponding image branch feature channel and the event branch feature channel through modality-aware residuals to obtain the updated image branch feature and the updated event branch feature; S5. Multi-scale decoding and prediction: The updated image branch features and the updated event branch features are fused and input into a lightweight decoder. Spatial resolution is restored through stepwise upsampling, and a probability distribution tensor of size K×H×W is output, where K is the number of predefined semantic categories. The category label corresponding to the maximum probability at each pixel position is taken from the probability distribution tensor to obtain the semantic segmentation result.

2. The method as described in claim 1, characterized in that, In step S1, each event in the asynchronous event stream data is represented as a quadruple (x, y, t, p), where (x, y) are pixel coordinates, t is the timestamp that triggered the event, and p∈{+1, -1} represents the polarity of the brightness change; When the brightness of a pixel in the scene increases, that pixel generates an event with a polarity of +1; When the brightness of a pixel in the scene decreases, that pixel generates an event with a polarity of -1.

3. The method as described in claim 1, characterized in that, In step S2, the Mamba encoder adopts a VMamba-T structure, which is pre-trained on the ImageNet-1K dataset and contains four stages of visual state space blocks. The first stage visual state space block maintains the original input resolution and outputs a feature dimension of 96. The second stage visual state space block downsamples the input features by a factor of 2 and outputs a feature dimension of 192. The third stage visual state space block downsamples the input features by a factor of 2 and outputs a feature dimension of 384. The fourth stage visual state space block downsamples the input features by a factor of 2 and outputs a feature dimension of 768.

4. The method as described in claim 1, characterized in that, In step S3, the specific process of feature fusion performed by the cross-space interaction module includes: S31a. Add the image branch features and the event branch features element by element to obtain shallow fusion features; S32a. Average pooling and max pooling are performed on the image branch features, the event branch features and the shallow fusion features respectively to generate 6 spatial description maps; S33a. Perform convolution operation on the 6 spatial description maps to generate spatial attention weights, which include image branch spatial attention weights, event branch spatial attention weights, and shallow fusion feature spatial attention weights. S34a. Multiply the image branch features element-wise with the event branch space attention weights and the shallow fusion feature space attention weights to obtain the image branch space optimized features; multiply the event branch features element-wise with the image branch space attention weights and the shallow fusion feature space attention weights to obtain the event branch space optimized features. S35a. The image branch space optimization feature and the event branch space optimization feature are fused to obtain the initial spatial fusion feature; S36a. Input the initial spatial fusion features into the two-dimensional selective scanning module for four-way sequential scanning, and output the spatial fusion features.

5. The method as described in claim 1, characterized in that, In step S3, the specific process of feature fusion performed by the cross-time interaction module includes: S31b. According to the temporal sequence of the event branch features, the image branch features are interleaved into the event branch features to obtain temporally interleaved features, wherein the dimension of the temporally interleaved features is 2T×H×W; S32b. Perform max pooling and average pooling on the time dimension of the temporal interleaving features to obtain temporal pooling features. S33b: Perform a convolution operation on the temporal pooling features to generate temporal attention weights; S34b. Multiply the image branch features element-wise with the temporal attention weights to obtain image branch temporal optimization features; multiply the event branch features element-wise with the temporal attention weights to obtain event branch temporal optimization features. S35b: The image branch time optimization feature and the event branch time optimization feature are fused to obtain the initial time fusion feature; S36b. After flattening the initial time fusion feature into a feature sequence, input it into the bidirectional selective scanning module to perform forward sequence scanning and reverse sequence scanning, and output the time fusion feature.

6. The method as described in claim 1, characterized in that, In step S4, the specific process of updating the modality-aware residual includes: S41. The spatial fusion feature is divided into spatial image sub-features and spatial event sub-features, and the temporal fusion feature is divided into temporal image sub-features and temporal event sub-features; S42. Call the spatial attention module and the temporal attention module to refine the spatial image sub-features, spatial event sub-features, temporal image sub-features and temporal event sub-features to obtain refined spatial image features, refined spatial event features, refined temporal image features and refined temporal event features; S43. The refined features of the spatial image and the refined features of the temporal image are concatenated and fused by convolution, and the residuals are added to the image branch features obtained in step S2 to obtain the updated image branch features; the refined features of the spatial event and the refined features of the temporal event are concatenated and fused by convolution, and the residuals are added to the event branch features obtained in step S2 to obtain the updated event branch features. S44. The updated image branch features and the updated event branch features are concatenated and fused to obtain fused multi-scale features.

7. The method as described in claim 1, characterized in that, In step S5, the lightweight decoder is designed with reference to the SegFormer decoder. When the lightweight decoder performs the decoding operation, it first performs channel compression processing on the updated image branch features and the updated event branch features, then performs multi-scale stitching on the channel compressed features, and finally restores the spatial resolution to the same as the image frame data through upsampling operation.

8. The method according to any one of claims 1-7, characterized in that, In step S1, the duration of the fixed time window is 50ms, the number of time slices T=10, the resolution of the image frame data is 346×260, and the resolution of the voxel tensor is 346×260.

9. An image event multimodal semantic segmentation device, characterized in that, include: The data acquisition and preprocessing module is used to acquire image frame data captured by a regular camera and asynchronous event stream data captured by an event camera; process the image frame data into a C×H×W tensor with C channels, H height, and W width; and use a voxel mesh representation method to divide the asynchronous event stream into multiple time slices within a fixed time window, accumulate event information in a three-dimensional mesh structure, and form a voxel tensor with dimensions T×H×W, where T is the number of time slices. A dual-branch Mamba encoder is used to input the preprocessed image frame data in C×H×W tensor form and the voxel tensor in T×H×W into two parallel Mamba encoders respectively. Each Mamba encoder consists of multiple layers of visual state space blocks and performs multi-scale feature extraction through a selective state scanning mechanism to obtain image branch features and event branch features respectively. A two-dimensional interaction module is used to fuse the image branch features and the event branch features. The two-dimensional interaction module includes a cross-spatial interaction module and a cross-temporal interaction module. The cross-spatial interaction module aligns image texture information with event edge features through a cross-modal spatial attention mechanism and combines two-dimensional selective scanning to model long-distance spatial dependencies to obtain spatial fusion features. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism and uses bidirectional temporal selective scanning to model long-range temporal relationships to obtain temporal fusion features. The modality-aware residual update module is used to connect the spatial fusion features and the temporal fusion features back to the corresponding image branch feature channels and event branch feature channels through modality-aware residuals, respectively, to obtain updated image branch features and updated event branch features; The multi-scale decoding and prediction module is used to fuse the updated image branch features and the updated event branch features and input them into the lightweight decoder. The spatial resolution is restored by upsampling at each level, and the output is a probability distribution tensor of size K×H×W, where K is the number of predefined semantic categories. The semantic segmentation result is obtained by taking the category label corresponding to the maximum probability at each pixel position of the probability distribution tensor.

10. A computer device, characterized in that, It includes a processor, a memory, and a communication bus, wherein the communication bus is used to establish a communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement the method as described in any one of claims 1-8.

Images