Image event multi-mode semantic segmentation method, device and equipment

Abstract

The invention relates to the field of computer vision and artificial intelligence, in particular to an image event multi-mode semantic segmentation method, device and equipment, which can be applied to scenes such as automatic driving, robot perception and intelligent traffic. Time slices are divided through a fixed time window, event information is accumulated, and asynchronous event streams are converted into T * H * W voxel tensors; a selective state scanning mechanism of a Mama framework is used for replacing a self-attention mechanism of a traditional Transform, the calculation complexity is reduced while modeling global feature dependence is achieved, and the problems of video memory and delay in a Transform high-resolution scene are solved; besides, image textures and event edges are aligned through cross-space interaction, an event dynamic time sequence is captured through cross-time interaction, and modal inherent characteristics are retained through residual connection, so that feature degradation caused by excessive fusion is effectively avoided; finally, the image segmentation precision is effectively improved, and the processing efficiency and the model robustness are improved at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and artificial intelligence, and particularly to an image event multi-modal semantic segmentation method, device and equipment, which can be applied to automatic driving, robot perception and intelligent transportation scenes. BACKGROUND

[0002] Semantic segmentation is an important task in the field of computer vision, widely used in automatic driving, intelligent robots, video surveillance, augmented reality, intelligent transportation and other scenarios. Through pixel-level classification of images or multi-modal inputs, it realizes accurate understanding of target objects and scenes. However, in real application scenarios, segmentation algorithms not only need to have high precision, but also must meet the requirements of real-time and robustness.

[0003] Existing semantic segmentation methods are mainly based on RGB images and use convolutional neural networks (CNN) or Transformer structure to realize pixel-level prediction. Typical methods include SegFormer, SegNeXt, etc. These methods can achieve high accuracy on standard datasets through multi-scale feature extraction and decoder fusion. However, they are strongly dependent on traditional frame images and are easily affected by motion blur and exposure limitations in complex conditions such as high-speed motion, low light and high dynamic range, resulting in significant performance degradation.

[0004] To overcome the above limitations, event cameras as a new type of visual sensor have attracted attention. They can asynchronously capture pixel intensity changes with microsecond-level time resolution, with high time resolution, low latency and wide dynamic range, and perform well in dynamic or low-light environments. However, since the event stream only records brightness changes, it lacks rich semantic information such as color and texture, resulting in poor performance when used alone in dense prediction tasks such as semantic segmentation.

[0005] To fully utilize the complementary characteristics of both image and event modalities, multi-modal fusion based semantic segmentation methods have emerged in recent years. Early approaches mainly based on convolutional neural networks (such as EDCNet, ESS) achieve basic modality fusion by jointly encoding the event stream voxelized with image features. However, due to the limited receptive field of convolutional networks, these methods are insufficient in modeling long-range dependencies and are difficult to capture complex spatial relationships. Subsequently, cross-modal interaction methods based on Transformer (such as CMX, CMNeXt, EISNet, etc.) introduce self-attention mechanisms, which can model global features and improve cross-modal consistency. However, the computational complexity of self-attention operations grows quadratically with the input length, resulting in high memory consumption and inference delay in high-resolution scenarios, which limits real-time deployment. Other researches adopt hybrid neural network architectures (such as HALSIE) to combine spiking neural networks with conventional networks to utilize the temporal characteristics of event data and reduce energy consumption, but the overall accuracy is limited and the fusion effect is still not ideal.

[0006] Although these methods have improved the shortcomings of single modalities to varying degrees, existing technologies still generally have two core problems. First, feature fusion relies too much on spatial dimensions and does not adequately model the inherent temporal dynamic information of event data, resulting in the inability to fully utilize the advantages of the event modality in high-speed motion or long-time sequence scenarios. Second, cross-modal feature alignment is not fine enough, and simple fusion in the spatial domain can easily introduce semantic ambiguity, making it difficult to achieve sufficient complementarity. In addition, the high computational overhead of Transformer-based methods severely restricts their application on embedded and edge devices.

[0007] Therefore, there is an urgent need for a new multi-modal semantic segmentation scheme that can achieve fine spatial-temporal fusion while maintaining efficiency. SUMMARY

[0008] To address the problems of high computational complexity, insufficient utilization of temporal dynamic information, and insufficient cross-modal fusion accuracy, the present application provides an image event multi-modal semantic segmentation method, device, and equipment.

[0009] In a first aspect, the present application provides an image event multi-modal semantic segmentation method: The image event multi-modal semantic segmentation method comprises: S1, input data and preprocessing: obtaining image frame data collected by a normal camera and asynchronous event stream data collected by an event camera; processing the image frame data into a CxHxW tensor form with a channel number C, a height H, and a width W; using a voxel grid representation method, dividing the asynchronous event stream into multiple time slices within a fixed time window, accumulating event information in a three-dimensional grid structure, and forming a voxel tensor with dimensions TxHxW, where T is the number of time slices; S2, double-branch Mamba encoder feature extraction: input the pre-processed image frame data in the form of CxHxW tensor and the TxHxW voxel tensor into two parallel Mamba encoders respectively, each of which is composed of multiple visual state space blocks, and multi-scale feature extraction is performed through a selective state scanning mechanism, to obtain image branch features and event branch features respectively; S3, dual-dimensional interaction module feature fusion: call a dual-dimensional interaction module to fuse the image branch features and the event branch features, the dual-dimensional interaction module includes a cross-space interaction module and a cross-time interaction module; the cross-space interaction module aligns image texture information and event edge features through a cross-modal space attention mechanism, and combines a two-dimensional selective scanning to model long-distance spatial dependence, to obtain spatial fusion features; the cross-time interaction module captures event flow dynamic characteristics through a cross-modal time attention mechanism, and adopts a bidirectional time selective scanning to model long-range temporal relationships, to obtain time fusion features; S4, modal perception residual update: pass the spatial fusion features and the time fusion features through a modal perception residual connection back to the corresponding image branch feature channels and event branch feature channels respectively, to obtain updated image branch features and updated event branch features; S5, multi-scale decoding and prediction: fuse the updated image branch features and the updated event branch features, and input them into a lightweight decoder, recover the spatial resolution through step-by-step upsampling, and output a probability distribution tensor with a size of KxHxW, where K is a pre-defined number of semantic categories; take the category label corresponding to the maximum probability at each pixel position of the probability distribution tensor, to obtain a semantic segmentation result.

[0010] By adopting the above technical solution, the asynchronous event stream is converted into a TxHxW voxel tensor by dividing the time window into time slices and accumulating event information, which not only preserves the time dynamic characteristics of event data, but also realizes the dimension adaptation with image frames (CxHxW tensor), laying a data foundation for cross-modal fusion; the selective state scanning mechanism of the Mamba architecture is used to replace the self-attention mechanism of the traditional Transformer, which reduces the computational complexity from O (N 2 ) to O (N) while modeling global feature dependencies, solving the memory and delay problems of Transformer in high-resolution scenarios; in addition, the image texture and event edge are aligned through cross-space interaction, the event dynamic time sequence is captured through cross-time interaction, and the modal inherent characteristics are preserved through residual connection, effectively avoiding feature degradation caused by excessive fusion; ultimately, the image segmentation accuracy is effectively improved, and the processing efficiency and model robustness are also improved.

[0011] Optionally, in step S1, each event in the asynchronous event stream data is represented as a quadruple (x, y, t, p), where (x, y) is the pixel coordinates, t is the timestamp of triggering the event, and p∈{+1, -1} represents the polarity of the brightness change; when the brightness of a certain pixel in the scene increases, the pixel generates an event with a polarity of +1; when the brightness of a certain pixel in the scene decreases, the pixel generates an event with a polarity of -1.

[0012] By adopting the above technical solution, the event camera independently samples at the pixel level, and asynchronously outputs events when the brightness of a pixel changes by more than a threshold value. The spatial position (x, y), timestamp (t), and brightness change direction (p∈{+1, -1}) of the event are accurately recorded by a quadruple (x, y, t, p), where +1 corresponds to brightness increase and -1 corresponds to brightness decrease. The essence is to convert continuous brightness changes into a discrete event sequence, achieving a data representation with microsecond-level time resolution.

[0013] Optionally, in step S2, the Mamba encoder adopts a VMamba-T structure, which is pre-trained on an ImageNet-1K dataset and includes 4 stages of visual state space blocks; wherein the input resolution of the first stage of visual state space blocks remains the original size, and the output feature dimension is 96; the second stage of visual state space blocks downsample the input features by 2 times, and the output feature dimension is 192; the third stage of visual state space blocks downsample the input features by 2 times, and the output feature dimension is 384; the fourth stage of visual state space blocks downsample the input features by 2 times, and the output feature dimension is 768.

[0014] By adopting the above technical solution, the VMamba-T structure pre-trained on ImageNet-1K is used as the Mamba encoder, which gradually down-samples (2 times per stage) through 4 stages and synchronously increases the feature dimension. This not only preserves the local detail features (such as edges and textures) of low stages, but also models the global context through high-stage high-dimensional features, providing a rich feature basis for subsequent fusion. At the same time, relying on the selective scanning of VSS Block maintains linear complexity, avoiding the high computational problem of traditional Transformer, with parameter quantity controlled within 25.44M, reducing the dependence on dataset labeling in autonomous driving and other scenarios.

[0015] Optionally, in step S3, the cross-space interaction module performs the specific process of feature fusion, which includes: S31a, element-wise addition of the image branch feature and the event branch feature to obtain a shallow fusion feature; S32a, average pooling and maximum pooling are respectively performed on the image branch feature, the event branch feature and the shallow fusion feature to generate 6 spatial description maps; S33a, a convolution operation is performed on the 6 spatial description maps to generate spatial attention weights, including image branch spatial attention weights, event branch spatial attention weights and shallow fusion feature spatial attention weights; S34a, the image branch feature is multiplied element by element with the event branch spatial attention weights and the shallow fusion feature spatial attention weights to obtain image branch spatial optimization features; the event branch feature is multiplied element by element with the image branch spatial attention weights and the shallow fusion feature spatial attention weights to obtain event branch spatial optimization features; S35a, the image branch spatial optimization features and the event branch spatial optimization features are fused to obtain initial spatial fusion features; S36a, the initial spatial fusion features are input into a two-dimensional selective scanning module for four-way sequence scanning to output the spatial fusion features.

[0016] By adopting the above technical solution, the cross-space interaction module (CSIM) can preliminarily establish cross-modal spatial correlation by adding image and event branch feature elements, and then obtain three types of spatial attention weights (image, event and shallow fusion) through convolution to realize local weight distribution based on global spatial information; and then long-range spatial dependence is modeled through two-dimensional selective scanning (four-way sequence) to solve the problem of limited receptive field of traditional CNN.

[0017] Optionally, in step S3, the specific process of the cross-time interaction module performing feature fusion includes: S31b, the image branch feature is inserted into the event branch feature in time sequence to obtain time-interleaved features, and the dimension of the time-interleaved features is 2T×H×W; S32b, maximum pooling and average pooling are respectively performed on the time-interleaved features in the time dimension to obtain time-pooled features; S33b, a convolution operation is performed on the time-pooled features to generate time attention weights; S34b, the image branch feature is multiplied element by element with the time attention weights to obtain image branch time optimization features; and the event branch feature is multiplied element by element with the time attention weights to obtain event branch time optimization features; S35b, fuse the image branch time optimization feature and the event branch time optimization feature to obtain an initial time fusion feature; S36b, after flattening the initial time fusion feature into a feature sequence, input the time fusion feature into a bidirectional selective scanning module for forward sequence scanning and reverse sequence scanning, and output the time fusion feature.

[0018] By adopting the above technical solution, the image feature is inserted according to the event time sequence, a 2T x H x W time sequence interleaved feature is formed, the time association between the image and the event is forcedly established, the global time sequence information is extracted through time dimension pooling, the time weight is obtained through convolution, the feature screening based on the time sequence change is realized, the forward + reverse scanning is performed after the initial time fusion feature is flattened, the long-range time sequence relationship of past->current->future is completely captured, and the problem of insufficient dynamic information modeling of the event stream is solved.

[0019] Optionally, in step S4, the specific process of the modal perception residual error update includes: S41, split the spatial fusion feature into a spatial image sub-feature and a spatial event sub-feature, and split the time fusion feature into a time image sub-feature and a time event sub-feature; S42, call a spatial attention module and a time attention module to refine the spatial image sub-feature, the spatial event sub-feature, the time image sub-feature, and the time event sub-feature, to obtain a spatial image refined feature, a spatial event refined feature, a time image refined feature, and a time event refined feature; S43, perform convolution fusion on the spatial image refined feature and the time image refined feature, and add the residual error to the image branch feature obtained in step S2 to obtain the updated image branch feature; perform convolution fusion on the spatial event refined feature and the time event refined feature, and add the residual error to the event branch feature obtained in step S2 to obtain the updated event branch feature; S44, perform convolution fusion on the updated image branch feature and the updated event branch feature to obtain a fusion multi-scale feature.

[0020] By adopting the above technical solution, the residual connection retains the core information such as the color texture of the image and the time dynamics of the event, solves the problem of loss of modal characteristics caused by excessive fusion (such as loss of texture details of the image and loss of time sequence advantages of the event), the refined features and the original features are complementary, the updated branch features have both the original modal advantages and the cross-modal fusion gain, and higher quality input is provided for decoding prediction.

[0021] Optionally, in step S5, the lightweight decoder refers to a SegFormer decoder design, and when performing a decoding operation, first performs channel compression processing on the updated image branch feature and the updated event branch feature, then performs multi-scale splicing on the channel-compressed features, and finally restores the features to the same spatial resolution as the image frame data through an upsampling operation.

[0022] By adopting the above technical solution, channel compression greatly reduces the parameter quantity of the decoder, and in combination with the low complexity of the Mamba encoder, the real-time requirement can be better met; in combination with multi-scale splicing, loss of details caused by upsampling is avoided, segmentation accuracy (such as clear edges of small target pedestrians) is ensured, and the accuracy and integrity of the final segmentation result are effectively improved.

[0023] Optionally, in step S1, the length of the fixed time window is 50 ms, the number of time slices T=10, the resolution of the image frame data is 346x260, and the resolution of the voxel tensor is 346x260.

[0024] In a second aspect, the present application provides an image event multi-modal semantic segmentation device; the device comprises various modules for executing the method in the first aspect or any possible implementation manner of the first aspect: The device comprises: A data acquisition and preprocessing module is configured to acquire image frame data collected by a normal camera and asynchronous event stream data collected by an event camera, process the image frame data into a CxHxW tensor form with a channel number C, a height H, and a width W, and divide the asynchronous event stream into a plurality of time slices within a fixed time window by using a voxel grid representation method, accumulate event information in a three-dimensional grid structure, and form a voxel tensor with a dimension of TxHxW, where T is the number of time slices. A double-branch Mamba encoder is configured to input the preprocessed image frame data in the CxHxW tensor form and the TxHxW voxel tensor into two parallel Mamba encoders, respectively, each of the Mamba encoders is composed of a plurality of visual state space blocks, multi-scale feature extraction is performed through a selective state scanning mechanism, and image branch features and event branch features are obtained, respectively. a two-dimensional interaction module configured to fuse the image branch feature and the event branch feature, the two-dimensional interaction module comprising a cross-space interaction module and a cross-time interaction module; the cross-space interaction module is configured to align image texture information and event edge features through a cross-modal space attention mechanism, and combine a two-dimensional selective scanning to model long-distance spatial dependence, to obtain a spatial fusion feature; the cross-time interaction module is configured to capture event flow dynamic characteristics through a cross-modal time attention mechanism, and adopt a bidirectional time selective scanning to model long-range temporal relationships, to obtain a time fusion feature; a modal-aware residual update module configured to connect the spatial fusion feature and the time fusion feature back to the image branch feature channel and the event branch feature channel, respectively, to obtain updated image branch features and updated event branch features; a multi-scale decoding and prediction module configured to fuse the updated image branch features and the updated event branch features, and input the fused features into a lightweight decoder, to recover spatial resolution through progressive upsampling, and output a probability distribution tensor with a size of KxHxW, where K is a pre-defined number of semantic categories; and obtain a semantic segmentation result by taking a category label corresponding to a maximum probability at each pixel position of the probability distribution tensor.

[0025] In a third aspect, a computer device is provided, comprising a processor, a memory, and a communication bus configured to realize communication connection between the processor and the memory, and the processor is configured to execute a computer program stored in the memory to realize the method according to any one of the above aspects.

[0026] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program; the computer program can be executed by a processor to realize the method according to the above aspects.

[0027] In a fifth aspect, a computer program product is provided, comprising a computer program, which can be executed by a processor to realize the method according to the above aspects.

[0028] The present application has at least the following beneficial technical effects: By introducing a state space modeling mechanism with linear complexity, the overall calculation cost is reduced, and the inference efficiency is improved. At the same time, the invention designs a double-dimensional interactive fusion strategy, realizes fine alignment of color texture information and edge structure in spatial dimension, and models the dynamic characteristics of event flow in time dimension, ensuring the advantages of fusion results in terms of timing consistency and semantic integrity. The invention not only improves the robustness of the model in complex scenes such as dynamic and low light, but also meets the real-time demand while ensuring high-precision semantic segmentation performance, and is suitable for various application scenarios such as autonomous driving, unmanned system and intelligent monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A flowchart of an image event multi-modal semantic segmentation method is provided for the embodiments of the present application. Figure 2 A block diagram of an image event multi-modal semantic segmentation architecture is provided for the embodiments of the present application. Figure 3 A double-dimensional interactive module structure diagram is provided for the embodiments of the present application. Figure 4 A comparison chart of performance, parameter quantity and calculation quantity of the method proposed by the invention and existing methods is provided for the embodiments of the present application. Figure 5 A comparison chart of visual effects of the method proposed by the invention and existing segmentation methods is provided for the embodiments of the present application. Figure 6 A block diagram of an image event multi-modal semantic segmentation device structure is provided for the embodiments of the present application. Figure 7 A block diagram of a computer device structure is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0031] The terminology used in the following description of the embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the description of the embodiments and the appended claims herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It also will be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration," any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The term "first", "second", etc. are used herein only to describe one of the features, and do not imply relative importance or a number of the features. Thus, a feature defined with "first", "second", etc. can explicitly or implicitly include one or more of the features, and the meaning of "a plurality" is two or more in the description of the embodiments herein unless otherwise specified.

[0032] The present application aims to solve the three problems existing in the existing image-event multi-modal semantic segmentation method, i.e. high computational complexity, insufficient use of time dynamic information, and insufficient cross-modal fusion precision. Although the existing fusion method based on Transformer can realize global feature modeling, the square-level computational complexity of the self-attention mechanism makes it difficult to apply to real-time scenarios. At the same time, some methods excessively rely on spatial dimension feature alignment and do not fully exploit the time information of event data, resulting in performance degradation in dynamic environments such as high-speed motion and low light. In addition, the fusion strategy often only performs simple weighting or attention interaction in a single dimension, resulting in incomplete elimination of semantic differences between cross-modal features, thereby affecting segmentation accuracy.

[0033] To solve the above problems, the present application proposes an image-event multi-modal semantic segmentation method based on Mamba architecture, which reduces the overall computational cost and improves the inference efficiency by introducing a state space modeling mechanism with linear complexity. At the same time, the present application designs a dual-dimensional interactive fusion strategy to realize fine alignment of color texture information and edge structure in the spatial dimension, and to model the dynamic characteristics of event flow in the time dimension, ensuring the advantages of the fusion result in terms of temporal consistency and semantic integrity. This method not only improves the robustness of the model in dynamic, low light and other complex scenarios, but also meets the real-time requirements while ensuring high-precision semantic segmentation performance, and is suitable for various application scenarios such as autonomous driving, unmanned systems, and intelligent monitoring.

[0034] Reference Figure 1 An image-event multi-modal semantic segmentation method aims to realize efficient cross-modal feature fusion and accurate pixel-level prediction, and the overall framework is as followsFigure 2 as shown.

[0035] The method mainly includes five core steps of input data preprocessing, double-branch Mamba encoder feature extraction, double-dimensional interaction module (DDIM) feature fusion, modal perception residual update and decoding prediction: S1: input data and preprocessing Obtain image frame data collected by a general camera and asynchronous event stream data collected by an event camera; The image frame data refers to a two-dimensional intensity image collected by a general camera at a fixed frame rate, each frame containing complete scene brightness and texture information, such as a common grayscale image or an RGB color image, and the image frame data is processed into a CxHxW tensor form with a channel number C, a height H and a width W; The asynchronous event stream is a sparse data stream obtained by independent sampling of the event camera at the pixel level. When the brightness of a certain pixel point changes by more than a certain threshold, the event camera will immediately output an event without waiting for the update of the global frame. Each event in the asynchronous event stream data is represented as a four-tuple (x, y, t, p), where (x, y) are the pixel coordinates, t is the timestamp of triggering the event, and p∈{+1, -1} represents the polarity of the brightness change; when the brightness of a certain pixel point in the scene increases, the pixel point generates an event with a polarity of +1; when the brightness of a certain pixel point in the scene decreases, the pixel point generates an event with a polarity of -1.

[0036] For event data, the application adopts a voxel grid (Voxel Grid) representation method, divides the asynchronous event stream into multiple time slices within a fixed time window, accumulates event information in a three-dimensional grid structure, and forms a voxel tensor with dimensions TxHxW, where T is the number of time slices; this preprocessing process preserves the spatial distribution and temporal dynamic characteristics of the event data, ensuring the integrity of subsequent time series modeling.

[0037] S2: double-branch Mamba encoder feature extraction The preprocessed image frame data in the form of CxHxW tensor and the event voxel data TxHxW are input into two parallel Mamba encoders, each Mamba encoder is composed of multiple visual state space blocks, and multi-scale feature extraction is performed through a selective state scanning mechanism to obtain image branch features and event branch features, respectively.

[0038] As Figure 2As shown, the image frame data and the event voxel data are respectively input into two parallel Mamba encoders, each of which is composed of multiple visual state space blocks to realize multi-scale feature extraction. Unlike the Transformer, Mamba adopts a selective state scanning mechanism, which can reduce the computational complexity to a linear level while maintaining the ability to model global dependencies, thereby significantly improving inference efficiency and meeting real-time processing requirements.

[0039] With reference to the foregoing Figure 2 In optional embodiments of the present application, the Mamba encoder adopts a VMamba-T structure, which is pre-trained on the ImageNet-1K dataset and includes four stages of visual state space blocks. The input resolution of the visual state space block in the first stage remains the original size, and the output feature dimension is 96. The input feature of the visual state space block in the second stage is down-sampled by 2, and the output feature dimension is 192. The input feature of the visual state space block in the third stage is down-sampled by 2, and the output feature dimension is 384. The input feature of the visual state space block in the fourth stage is down-sampled by 2, and the output feature dimension is 768.

[0040] S3: Feature fusion of dual-dimensional interaction module Referring to the foregoing Figure 3 The image branch feature and the event branch feature are fused by calling a dual-dimensional interaction module (DDIM), which includes a cross-spatial interaction module (CSIM) and a cross-temporal interaction module (CTIM). The cross-spatial interaction module realizes the alignment of image texture information and event edge features through a cross-modal spatial attention mechanism, and combines a two-dimensional selective scanning mechanism to model long-range spatial dependencies, thereby obtaining spatial fusion features. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism, and adopts a bidirectional time selective scanning mechanism to model long-range temporal relationships, thereby obtaining temporal fusion features.

[0041] The cross-space interaction module realizes fine alignment of image texture information and event edge features through a cross-modal space attention mechanism, and combines two-dimensional selective scanning (SS2D) to model long-distance spatial dependencies, thereby enhancing the fusion effect of local structures and global context. The cross-time interaction module captures the dynamic characteristics of the event stream through a cross-modal time attention mechanism, and uses bidirectional time selective scanning (BTSS) to model long-range temporal relationships, thereby improving semantic consistency in dynamic scenes. The CSIM and CTIM alternately act on the multi-scale feature extraction stage, gradually enhancing cross-modal consistency.

[0042] With reference to Figure 3 , the specific process of feature fusion performed by the cross-space interaction module includes: S31a, element-wise addition of image branch features and event branch features to obtain shallow fusion features; S32a, average pooling and maximum pooling processing are respectively performed on the image branch features, event branch features and shallow fusion features to generate 6 spatial description maps; S33a, convolution operation is performed on the 6 spatial description maps to generate spatial attention weights, including image branch spatial attention weights, event branch spatial attention weights and shallow fusion feature spatial attention weights; S34a, element-wise multiplication of the image branch features and the event branch spatial attention weights, the shallow fusion feature spatial attention weights to obtain image branch spatial optimization features; element-wise multiplication of the event branch features and the image branch spatial attention weights, the shallow fusion feature spatial attention weights to obtain event branch spatial optimization features; S35a, fusion of the image branch spatial optimization features and the event branch spatial optimization features to obtain initial spatial fusion features; S36a, inputting the initial spatial fusion features into a two-dimensional selective scanning module for four-way sequence scanning to output the spatial fusion features.

[0043] With reference to Figure 3 , the specific process of feature fusion performed by the cross-time interaction module includes: S31b, according to the time sequence of the event branch features, the image branch features are interleaved into the event branch features to obtain time interleaved features, the dimension of the time interleaved features is 2T×H×W; S32b, maximum pooling and average pooling processing are respectively performed on the time interleaved features in the time dimension to obtain time pooling features; S33b, convolution operation is performed on the time pooling features to generate time attention weights; S34b, element-wise multiplication of the image branch feature and the time attention weight is performed to obtain an image branch time-optimized feature; element-wise multiplication of the event branch feature and the time attention weight is performed to obtain an event branch time-optimized feature; S35b, the image branch time-optimized feature and the event branch time-optimized feature are fused to obtain an initial time fusion feature; S36b, after the initial time fusion feature is flattened into a feature sequence, the time fusion feature is input into a bidirectional selective scanning module for forward sequence scanning and reverse sequence scanning.

[0044] S4: modal-aware residual update In each interaction stage, the spatial fusion feature and the time fusion feature output by the DDIM are respectively connected back to the corresponding image branch feature channel and the event branch feature channel through the modal-aware residual, to obtain updated image branch features and updated event branch features; information supplement and stable training are realized, while the characteristics of the respective modal are retained, avoiding the feature degradation problem caused by excessive fusion.

[0045] Specifically, the specific process of the modal-aware residual update includes: S41, the spatial fusion feature is split into a spatial image sub-feature and a spatial event sub-feature, and the time fusion feature is split into a time image sub-feature and a time event sub-feature; Specifically, before CSIM fusion, the image branch feature and the event branch feature have been processed through dimension alignment, and are unified to the same dimension of T×H×W (T is the number of time slices, H is the height, and W is the width); the dimension of the spatial fusion feature is completely consistent with the dimension of the aligned feature, ensuring that the sub-features after splitting can match the original branch features; The information source of the spatial fusion feature comes from the texture feature of the image branch and the edge feature of the event branch, and when splitting, the features are separated according to the modal source. The feature part corresponding to the original image branch feature in the fusion feature is split into a spatial image sub-feature ; the feature part corresponding to the original event branch feature in the fusion feature is split into a spatial event sub-feature ; The spatial image sub-feature and the spatial event sub-feature obtained after splitting both maintain the dimension of T×H×W, which is consistent with the original image branch feature and the event branch feature The dimensions are completely consistent, providing a dimensional basis for subsequent residual connection back to the original feature channel.

[0046] Similarly, before CTIM fusion, the image branch features are interleaved according to the timing of the event branch features to form time interleaved features with a dimension of 2T×H×W ; the time fusion features generated by subsequent bidirectional selective scanning, although their dimensions are flattened and processed by scanning, will eventually recover to the T×H×W dimension matching the original branch features; The time fusion features obtain information from the time sequence correlation features of the image branch and the dynamic time sequence features of the event branch. When split by time, the time sequence feature part corresponding to the original image branch feature in the fusion feature is split into a time image sub-feature ; the dynamic time sequence feature part corresponding to the original event branch feature in the fusion feature is split into a time event sub-feature ; The time image sub-feature , time event sub-feature after splitting, both maintain a dimension of T×H×W, consistent with the original image branch feature , event branch feature , ensuring that the superposition update with the original feature is achieved, preserving the inherent characteristics of each modality.

[0047] S42, call the spatial attention module (SA) to refine the spatial image sub-feature and the spatial event sub-feature respectively to obtain the corresponding refined spatial sub-feature; call the time attention module (TA) to refine the time image sub-feature and the time event sub-feature to obtain the corresponding refined time sub-feature; the specific process includes: Call the spatial attention module (SA) to refine the spatial image sub-feature to obtain the spatial image refined sub-feature ; Call the spatial attention module (SA) to refine the spatial event sub-feature , the spatial event refined sub-feature ; Call the time attention module (TA) to refine the time image sub-feature , the time image refined sub-feature ; Call the time attention module (TA) to refine the time event sub-feature , the time event refined sub-feature ; After refining, using image branch feature + spatial image refining sub-feature, event branch feature + spatial event refining sub-feature, image branch feature + temporal image refining sub-feature, event branch feature + temporal event refining sub-feature, four outputs are obtained (such as Figure 3 two outputs of CSIM, two outputs of CTIM): wherein, denotes the spatial image refining feature, denotes the spatial event refining feature, denotes the temporal image refining feature, denotes the temporal event refining feature.

[0048] S43, the spatial image refining feature of CSIM is spliced and convolved with the temporal image refining feature of CTIM , and residual addition is performed with the image branch feature obtained in step S2 to obtain an updated image branch feature; the spatial event refining feature of CSIM is spliced and convolved with the temporal event refining feature of CTIM , and residual addition is performed with the event branch feature obtained in step S2 to obtain an updated event branch feature; specifically including: wherein, denotes the updated image branch feature, denotes the updated event branch feature.

[0049] S44, the updated image branch feature is spliced and convolved with the updated event branch feature to obtain a fused multi-scale feature . S5: multi-scale decoding and prediction The fused multi-scale feature is input into a lightweight decoder, and the spatial resolution is restored through step-by-step upsampling, and a probability distribution tensor with a size of KxHxW is output, wherein K is a pre-defined number of semantic categories; the class label corresponding to the maximum probability at each pixel position of the probability distribution tensor is taken to obtain a semantic segmentation result.

[0050] Fused multi-scale features The spatial resolution is recovered and a pixel-level semantic segmentation result is output after step-by-step upsampling by the lightweight decoder. The lightweight decoder structure is referenced from the SegFormer design. When the decoder performs the decoding operation, the updated image branch features and the updated event branch features are first subjected to channel compression processing, then the channel-compressed features are subjected to multi-scale splicing, and finally the upsampling operation is performed to recover to the same spatial resolution as the image frame data.

[0051] The semantic segmentation result is a pixel-by-pixel classification mapping of the input image and event fusion data. Specifically, the decoder outputs a probability distribution tensor with a size of K×H×W, where H and W are the spatial dimensions of the input image, and K is the number of predefined semantic categories. Taking the category label corresponding to the maximum probability at each pixel position of the tensor, a semantic segmentation map can be obtained. In other words, the semantic segmentation result is essentially a pixel-level classification result matrix, which assigns a clear semantic category, such as "road", "vehicle", "pedestrian", etc., to each pixel in the input scene, thereby achieving fine understanding of the scene.

[0052] The image event multi-modal semantic segmentation method based on the Mamba state space model proposed in the present application can significantly outperform the prior art in terms of accuracy and efficiency, and maintain strong robustness in complex scenes, and has the following beneficial effects: (1) Significantly improve segmentation accuracy Table 1 Performance comparison of the method (MagbaSeg) proposed in the present application and existing methods on DDD17 and DSEC data sets As shown in Table 1, the test results on the DDD17 and DSEC public data sets show that the method of the present application achieves leading performance in the cross-modal semantic segmentation task. On the DDD17 data set, the average intersection over union (mIoU) of the method of the present application reaches 77.56%, and on the DSEC data set, it reaches 75.10%, which is about 2.5 percentage points higher than the current best method. This advantage is due to the joint modeling of the dual-dimensional interaction module (DDIM) in the spatial and temporal dimensions, which can fully utilize the dynamic characteristics of the event stream and the texture details of the image to achieve fine modal fusion.

[0053] (2) Reduce computational complexity and improve real-time performance Compared with the cross-modal fusion method based on Transformer, the Mamba state space model adopted in the present application has linear time complexity, avoiding the quadratic operation overhead of the self-attention mechanism.

[0054] Table 2 Performance, parameter quantity, and calculation quantity comparison of the proposed method and existing methods As shown in Table 2 and Figure 4 The model parameter quantity of the present application on the DDD17 dataset is only 25.44M, and the multiply-add operation quantity is 15.59G, which is much lower than the operation quantity required by typical Transformer models (such as CMX, EISNet), while maintaining higher accuracy. This feature makes the present application better adapt to real-time processing requirements in autonomous driving and edge computing scenarios.

[0055] (3) Strengthen the robustness in dynamic scenarios The present application shows higher stability in dynamic environments and low light scenes. Through the bidirectional selective scanning of the time characteristics of the event stream by the CTIM module, the present application method can maintain prediction consistency in the case of fast motion and frequent light changes, reducing semantic missegmentation problems. As shown in the qualitative results in Figure 5 The present application can more accurately segment small targets such as pedestrians and traffic signs in complex traffic scenarios, while existing methods are prone to edge blurring or missed detection in these scenarios.

[0056] (4) The fusion strategy is significantly better than traditional methods Table 3 Comparison of the proposed fusion strategy and existing fusion strategies As shown in the comparison experiment results in Table 3, the dual-dimensional interaction module (DDIM) proposed by the present application is significantly better than existing methods in cross-modal feature fusion performance. Under the same encoder and decoder conditions, the mIoU achieved by DDIM is 77.56%, which exceeds the baseline addition fusion (74.38%) and other representative fusion strategies (such as FFM, MRFM, CSF). This shows that the present application has stronger expression ability in feature interaction modeling, which can effectively alleviate the cross-modal semantic ambiguity problem.

[0057] (5) Spatial and temporal interaction modules cooperate to improve performance Table 4 Ablation results of different modules of the proposed method According to the ablation experiment in Table 4, the CSIM and CTIM modules designed in the application play a complementary role in feature fusion. Removing any module will cause performance degradation, for example, mIoU drops to 76.32% after removing CTIM, and drops to 76.20% after removing CSIM, while the performance is improved to 77.56% when both are used together. The results show that the dual-dimension interaction mechanism of the application can achieve information complementation in both spatial and temporal dimensions, thereby significantly improving the accuracy and robustness of semantic segmentation.

[0058] The embodiment combines Figure 1 and Figure 2 to explain the best implementation of the application on an autonomous vehicle. This embodiment takes the DDD17 dataset as an example to simulate the semantic segmentation task in a real driving environment, including the whole process of data acquisition, preprocessing, model construction, training and inference.

[0059] Step 1: Data acquisition and preprocessing On the autonomous vehicle, a DAVIS event camera is equipped (which can synchronously collect image frames and event streams). The resolution of image frames and event streams is 346x260, and the event stream is an asynchronous event sequence, each event containing coordinates (x, y), timestamp t and polarity p∈(+1, -1). The acquisition time window is fixed at 50ms, and the collected event sequence is used as the original input.

[0060] Step 2: Dual-branch Mamba encoder feature extraction As shown in Figure 2 , the event stream E and the image I are processed by the Voxel Grid module and the Stem Block, respectively, before entering the Mamba encoder. The Voxel Grid module is responsible for voxel grid representation of the original asynchronous event stream: divide the given time window into T=10 time slices, and accumulate events in each time slice to obtain a voxel grid tensor: where represents the event coordinates, is the timestamp, is the polarity, is the time slice. This module not only completes the voxelization representation, but also maps the three-dimensional voxel tensor to a feature sequence through the Patch embedding operation, which is convenient for subsequent input to the encoder.

[0061] Image input First, the Stem Block is used. This module is essentially a convolutional patch embedding layer, which divides the original two-dimensional image into several patches and maps them to a high-dimensional feature space, resulting in an initial feature sequence of uniform dimension. In this way, the image branch and the event branch complete a unified representation process before entering the Mamba encoder.

[0062] Subsequently, voxelization event characteristics and image features Two parallel Mamba encoders are input separately. Each encoder uses a VMamba-T architecture (pre-trained on ImageNet-1K) and contains four stages of visual state space blocks (VSSBlock): Phase 1: The input resolution remains the same size, and the output feature dimension is C1=96.

[0063] Second stage: Downsample by 2 times, output dimension C2=192.

[0064] Third stage: Downsample by 2 times, output dimension C3=384.

[0065] Fourth stage: Downsampled by 2 times, output dimension C4=768.

[0066] Each VSS Block models long-range dependencies through a selective scan mechanism (S6 Block), with a computational complexity of O(n log n). Far lower than Transformer Event branches capture dynamic edges, while image branches extract texture details.

[0067] Step 3: Feature Fusion of Two-Dimensional Interactive Module (DDIM) After each encoding stage, DDIM is applied for cross-modal fusion, such as... Figure 3 As shown. DDIM includes the Cross-Spatial Interaction Module (CSIM) and the Cross-Temporal Interaction Module (CTIM).

[0068] CSIM processing: Event branching characteristics and image branch features (Dimensional alignment is) First, add the elements to obtain a shallow fusion. Then, six spatial description maps are generated through average / max pooling, and spatial attention weights are produced after convolution. (divided into) Applying attention: The initial spatial fusion features are obtained through fusion. .then, Input SS2D module (four-way sequence scan), output spatial fusion features .

[0069] CTIM processing: event branch feature and image branch feature Temporal interleaving feature is obtained by temporal interleaving insertion according to event features ( ). The time pooling feature is obtained by maximum pooling and average pooling of the temporal interleaving feature from the time dimension, and then the time attention weight is generated by convolution ; the image branch time optimization feature is obtained by element-wise multiplication of the image branch feature and the time attention weight ; the event branch time optimization feature is obtained by element-wise multiplication of the event branch feature and the time attention weight ; the image branch time optimization feature and the event branch time optimization feature are fused to obtain the initial time fusion feature , which is flattened into a sequence and then bidirectional S6 scanning (forward + reverse) is performed to output the time fusion feature .

[0070] Step 4: modal perception residual update fusion feature and are respectively split into event branch and image branch, refined by spatial / time attention module (SA / TA) to obtain multi-scale fusion feature. The modal characteristics are preserved by residual connection to avoid feature degradation.

[0071] Step 5: multi-scale decoding and prediction The multi-scale fusion feature is upsampled by the MLP decoder: after channel compression, the features are concatenated to restore the original resolution. Finally, the pixel-level semantic map is output, covering 6 labels (including road, background, vegetation, pedestrian, vehicle, and other). Cross-entropy loss is used for training.

[0072] The specific training and inference process includes: PyTorch is used to train on NVIDIA RTX-4090D for 60 rounds, AdamW optimizer, initial learning rate 2e-4, batch size 12. When inferring, the model parameters are 25.44M, and the MACs are 15.59G. The test mIoU reaches 77.56%, as shown in Table 2.

[0073] On the basis of the above method embodiment, the present embodiment further provides an image event multi-modal semantic device, which can be used to realize the steps of the above method.

[0074] Referring to Figure 6 , the device comprises: The data acquisition and preprocessing module 61 is configured to acquire image frame data collected by a general camera and asynchronous event stream data collected by an event camera; process the image frame data into a CxHxW tensor form with a channel number C, a height H and a width W; divide the asynchronous event stream into a plurality of time slices in a fixed time window by using a voxel grid representation method, accumulate event information in a three-dimensional grid structure, and form a voxel tensor with a dimension of TxHxW, where T is a number of time slices; The dual-branch Mamba encoder 62 is configured to input the preprocessed image frame data in the CxHxW tensor form and the voxel tensor with the dimension of TxHxW into two parallel Mamba encoders respectively, each of the Mamba encoders is composed of a plurality of visual state space blocks, and multi-scale feature extraction is performed through a selective state scanning mechanism to obtain image branch features and event branch features respectively. The dual-dimension interaction module 63 is configured to fuse the image branch features and the event branch features, and the dual-dimension interaction module includes a cross-space interaction module and a cross-time interaction module; the cross-space interaction module aligns image texture information and event edge features through a cross-modal space attention mechanism, and combines a two-dimensional selective scanning to model long-distance spatial dependence, to obtain spatial fusion features; the cross-time interaction module captures dynamic characteristics of the event stream through a cross-modal time attention mechanism, and adopts a bidirectional time selective scanning to model long-range temporal relationships, to obtain time fusion features. The modal perception residual update module 64 is configured to connect the spatial fusion features and the time fusion features back to the image branch feature channels and the event branch feature channels respectively through a modal perception residual connection, to obtain updated image branch features and updated event branch features. The multi-scale decoding and prediction module 65 is configured to fuse the updated image branch features and the updated event branch features, input the fused features into a lightweight decoder, recover spatial resolution through progressive upsampling, and output a probability distribution tensor with a size of KxHxW, where K is a number of predefined semantic categories; take a category label corresponding to a maximum probability at each pixel position of the probability distribution tensor, to obtain a semantic segmentation result.

[0075] The various changes and specific examples in the method provided in the above embodiments are also applicable to the device of the present embodiment. By the foregoing detailed description of the method, those skilled in the art can clearly understand the implementation method of the device in the present embodiment. For the sake of brevity of the description, the implementation method of the device is not described in detail here.

[0076] In order to better execute the program of the above method, the embodiment of the present application further provides a computer device, such as Figure 7As shown, the computer device includes a processor 71, a memory 72, and a communication bus 73 to enable communication between the processor 71 and the memory 72.

[0077] The computer device can be implemented in various forms, including a vehicle-mounted computer, a mobile phone, a tablet computer, a palm computer, a notebook computer, and a desktop computer, etc.

[0078] The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the method provided in the above embodiments, etc.; the data storage area can store data involved in the method provided in the above embodiments, etc.

[0079] Optionally, the memory is a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disk (including a compact disc read-only memory (CD-ROM), a compressed disk, a laser disk, a digital universal disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory exists independently and is connected to the processor through the communication bus, or the memory is integrated with the processor.

[0080] The processor can include one or more processing cores. The processor invokes data stored in the memory by running or executing instructions, programs, code sets or instruction sets stored in the memory, performs various functions and processes data of the present application. The processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that the electronic device for implementing the functions of the processor described above can also be other devices for different devices, and the embodiments of the present application are not limited specifically.

[0081] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience, only one thick line is used to represent the communication bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0082] In an optional embodiment, the computer device can further include a communication interface (not shown) for communication with other devices.

[0083] The embodiments of the present application provide a computer readable storage medium, for example, including: a U disk, a mobile hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes. The computer readable storage medium stores a computer program capable of being loaded by the processor and executing the method of the above embodiments.

[0084] The embodiments of the present application further provide a computer program product, which comprises a computer program tangibly embodied on a computer readable medium, the computer program comprising program codes for executing any of the methods of the embodiments of the present application. The computer program can be downloaded and installed on the network, and / or installed from a detachable medium (such as a disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc.).

[0085] The above embodiments are only used to specifically introduce the technical solutions of the present application, and the above descriptions are only used to help understand the method and the core idea of the present application, and should not be understood as limitations on the present application. Those skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application.

Claims

1. A method for multimodal semantic segmentation of image events, characterized in that, The method includes: S1. Input Data and Preprocessing: Acquire image frame data from a regular camera and asynchronous event stream data from an event camera; process the image frame data into a C×H×W tensor with C channels, H height, and W width; using a voxel mesh representation method, divide the asynchronous event stream into multiple time slices within a fixed time window, accumulate event information in a three-dimensional mesh structure, and form a voxel tensor with dimensions T×H×W, where T is the number of time slices; S2. Feature extraction from dual-branch Mamba encoders: The preprocessed image frame data in C×H×W tensor form and the voxel tensor in T×H×W are respectively input into two parallel Mamba encoders. Each Mamba encoder consists of multiple layers of visual state space blocks. Multi-scale feature extraction is performed through a selective state scanning mechanism to obtain image branch features and event branch features respectively. S3. Feature Fusion of the Two-Dimensional Interaction Module: The two-dimensional interaction module is invoked to fuse the image branch features and the event branch features. The two-dimensional interaction module includes a cross-spatial interaction module and a cross-temporal interaction module. The cross-spatial interaction module aligns image texture information with event edge features through a cross-modal spatial attention mechanism and combines two-dimensional selective scanning to model long-distance spatial dependencies to obtain spatial fusion features. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism and uses bidirectional temporal selective scanning to model long-range temporal relationships to obtain temporal fusion features. S4. Modality-aware residual update: The spatial fusion feature and the temporal fusion feature are respectively connected back to the corresponding image branch feature channel and the event branch feature channel through modality-aware residuals to obtain the updated image branch feature and the updated event branch feature; S5. Multi-scale decoding and prediction: The updated image branch features and the updated event branch features are fused and input into a lightweight decoder. Spatial resolution is restored through stepwise upsampling, and a probability distribution tensor of size K×H×W is output, where K is the number of predefined semantic categories. The category label corresponding to the maximum probability at each pixel position is taken from the probability distribution tensor to obtain the semantic segmentation result.

2. The method as described in claim 1, characterized in that, In step S1, each event in the asynchronous event stream data is represented as a quadruple (x, y, t, p), where (x, y) are pixel coordinates, t is the timestamp that triggered the event, and p∈{+1, -1} represents the polarity of the brightness change; When the brightness of a pixel in the scene increases, that pixel generates an event with a polarity of +1; When the brightness of a pixel in the scene decreases, that pixel generates an event with a polarity of -1.

3. The method as described in claim 1, characterized in that, In step S2, the Mamba encoder adopts a VMamba-T structure, which is pre-trained on the ImageNet-1K dataset and contains four stages of visual state space blocks. The first stage visual state space block maintains the original input resolution and outputs a feature dimension of 96. The second stage visual state space block downsamples the input features by a factor of 2 and outputs a feature dimension of 192. The third stage visual state space block downsamples the input features by a factor of 2 and outputs a feature dimension of 384. The fourth stage visual state space block downsamples the input features by a factor of 2 and outputs a feature dimension of 768.

4. The method as described in claim 1, characterized in that, In step S3, the specific process of feature fusion performed by the cross-space interaction module includes: S31a. Add the image branch features and the event branch features element by element to obtain shallow fusion features; S32a. Average pooling and max pooling are performed on the image branch features, the event branch features and the shallow fusion features respectively to generate 6 spatial description maps; S33a. Perform convolution operation on the 6 spatial description maps to generate spatial attention weights, which include image branch spatial attention weights, event branch spatial attention weights, and shallow fusion feature spatial attention weights. S34a. Multiply the image branch features element-wise with the event branch space attention weights and the shallow fusion feature space attention weights to obtain the image branch space optimized features; multiply the event branch features element-wise with the image branch space attention weights and the shallow fusion feature space attention weights to obtain the event branch space optimized features. S35a. The image branch space optimization feature and the event branch space optimization feature are fused to obtain the initial spatial fusion feature; S36a. Input the initial spatial fusion features into the two-dimensional selective scanning module for four-way sequential scanning, and output the spatial fusion features.

5. The method as described in claim 1, characterized in that, In step S3, the specific process of feature fusion performed by the cross-time interaction module includes: S31b. According to the temporal sequence of the event branch features, the image branch features are interleaved into the event branch features to obtain temporally interleaved features, wherein the dimension of the temporally interleaved features is 2T×H×W; S32b. Perform max pooling and average pooling on the time dimension of the temporal interleaving features to obtain temporal pooling features. S33b: Perform a convolution operation on the temporal pooling features to generate temporal attention weights; S34b. Multiply the image branch features element-wise with the temporal attention weights to obtain image branch temporal optimization features; multiply the event branch features element-wise with the temporal attention weights to obtain event branch temporal optimization features. S35b: The image branch time optimization feature and the event branch time optimization feature are fused to obtain the initial time fusion feature; S36b. After flattening the initial time fusion feature into a feature sequence, input it into the bidirectional selective scanning module to perform forward sequence scanning and reverse sequence scanning, and output the time fusion feature.

6. The method as described in claim 1, characterized in that, In step S4, the specific process of updating the modality-aware residual includes: S41. The spatial fusion feature is divided into spatial image sub-features and spatial event sub-features, and the temporal fusion feature is divided into temporal image sub-features and temporal event sub-features; S42. Call the spatial attention module and the temporal attention module to refine the spatial image sub-features, spatial event sub-features, temporal image sub-features and temporal event sub-features to obtain refined spatial image features, refined spatial event features, refined temporal image features and refined temporal event features; S43. The refined features of the spatial image and the refined features of the temporal image are concatenated and fused by convolution, and the residuals are added to the image branch features obtained in step S2 to obtain the updated image branch features; the refined features of the spatial event and the refined features of the temporal event are concatenated and fused by convolution, and the residuals are added to the event branch features obtained in step S2 to obtain the updated event branch features. S44. The updated image branch features and the updated event branch features are concatenated and fused to obtain fused multi-scale features.

7. The method as described in claim 1, characterized in that, In step S5, the lightweight decoder is designed with reference to the SegFormer decoder. When the lightweight decoder performs the decoding operation, it first performs channel compression processing on the updated image branch features and the updated event branch features, then performs multi-scale stitching on the channel compressed features, and finally restores the spatial resolution to the same as the image frame data through upsampling operation.

8. The method according to any one of claims 1-7, characterized in that, In step S1, the duration of the fixed time window is 50ms, the number of time slices T=10, the resolution of the image frame data is 346×260, and the resolution of the voxel tensor is 346×260.

9. An image event multimodal semantic segmentation device, characterized in that, include: The data acquisition and preprocessing module is used to acquire image frame data captured by a regular camera and asynchronous event stream data captured by an event camera; process the image frame data into a C×H×W tensor with C channels, H height, and W width; and use a voxel mesh representation method to divide the asynchronous event stream into multiple time slices within a fixed time window, accumulate event information in a three-dimensional mesh structure, and form a voxel tensor with dimensions T×H×W, where T is the number of time slices. A dual-branch Mamba encoder is used to input the preprocessed image frame data in C×H×W tensor form and the voxel tensor in T×H×W into two parallel Mamba encoders respectively. Each Mamba encoder consists of multiple layers of visual state space blocks and performs multi-scale feature extraction through a selective state scanning mechanism to obtain image branch features and event branch features respectively. A two-dimensional interaction module is used to fuse the image branch features and the event branch features. The two-dimensional interaction module includes a cross-spatial interaction module and a cross-temporal interaction module. The cross-spatial interaction module aligns image texture information with event edge features through a cross-modal spatial attention mechanism and combines two-dimensional selective scanning to model long-distance spatial dependencies to obtain spatial fusion features. The cross-temporal interaction module captures the dynamic characteristics of event flow through a cross-modal temporal attention mechanism and uses bidirectional temporal selective scanning to model long-range temporal relationships to obtain temporal fusion features. The modality-aware residual update module is used to connect the spatial fusion features and the temporal fusion features back to the corresponding image branch feature channels and event branch feature channels through modality-aware residuals, respectively, to obtain updated image branch features and updated event branch features; The multi-scale decoding and prediction module is used to fuse the updated image branch features and the updated event branch features and input them into the lightweight decoder. The spatial resolution is restored by upsampling at each level, and the output is a probability distribution tensor of size K×H×W, where K is the number of predefined semantic categories. The semantic segmentation result is obtained by taking the category label corresponding to the maximum probability at each pixel position of the probability distribution tensor.

10. A computer device, characterized in that, It includes a processor, a memory, and a communication bus, wherein the communication bus is used to establish a communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement the method as described in any one of claims 1-8.

Images