An event-RGB semantic segmentation method based on large model adaptation

Abstract

This invention discloses a multimodal semantic segmentation method based on large model adaptation. First, for each pair of input data, the event information is divided into different time scales based on the exposure timestamps of the RGB data. Then, the events are processed into voxels, and these voxels, along with video frames, are input to a multi-temporal-scale event embedding module and the SAM backbone, respectively. Next, the temporal features extracted by the multi-temporal-scale event embedding module are input into the SAM event adapter, interacting with the image information in the SAM to obtain features rich in high-quality motion information. Finally, these features are input into a simple semantic segmentation head to generate semantic segmentation results. This invention is end-to-end, allowing for direct training of the entire system. The trained model can then be used to handle semantic segmentation problems in event-RGB scenes. This invention solves the problem of information loss due to RGB information in extreme scenes and achieves superior semantic segmentation results compared to previous works.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically involving the adaptation of large-scale semantic analysis models (SAMs) and the efficient semantic segmentation in multimodal motion scenarios. Background Technology

[0002] Semantic segmentation, a fundamental visual task, is widely used in fields such as transportation, robotics, and healthcare, and has consistently attracted research attention. With the rapid development of large-scale model research, the foundational model for semantic segmentation—SAM (Segment Anything Model)—has been proposed. This model essentially addresses the poor generalization problem of previous segmentation models and the need to retrain the entire model on different datasets.

[0003] Nevertheless, segmentation models developed based on SAM are still constrained by the inherent limitations of RGB sensors, especially in scenes with complex lighting conditions and high-speed motion. Based on these motivations, a natural remedy is to introduce additional visual modalities into SAM without compromising its high generalization capabilities. Compared to frame-based RGB sensors, event cameras offer higher temporal resolution, dynamic range, and lower power consumption, making them suitable for deployment on platforms such as vehicles and drones. Therefore, incorporating event information into SAM is undoubtedly a good choice.

[0004] While some studies have incorporated event information into semantic segmentation tasks, their encoding methods often involve coarsely compressing high temporal resolution event information into a frame-like representation and processing it in the same way as RGB information. This approach ignores the characteristics of events. First, it neglects the temporal information of events; second, event information only represents changes in light and is more sensitive to motion information, necessitating a separate processing method for event information. Therefore, how to process event information and fuse it with RGB information to guide segmentation tasks is a pressing problem that needs to be solved. Summary of the Invention

[0005] Based on the above analysis, this invention designs a multimodal semantic segmentation method based on large model adaptation, taking advantage of the complementary nature of event information in motion scenes and RGB information, as well as an event encoding method that leverages the high temporal resolution and asynchronous characteristics of event information. This invention implements an event-RGB semantic segmentation method based on large model adaptation. This method utilizes the complementary nature of event information in motion scenes and RGB information to guide the segmentation of SAM (Semantic Aspect-Oriented Semantic Array) in motion scenes.

[0006] The event-RGB semantic segmentation method based on large model adaptation disclosed in this invention specifically includes the following steps:

[0007] Step 1: Data processing. For each pair of input data, the event information is divided into different time scales based on the exposure timestamp of the RGB data, and then processed into the format of network input.

[0008] Step 2: Construct and train the network model. The network model includes a SAM backbone with frozen parameters, an event adapter (SE_Adapter), and a multi-spatiotemporal scale patch embedding module (MSP). Except for SAM, all network parts are learnable. The trained model is used to process image and event pairs. Image data is input into the SAM backbone, and event data is input into the MSP and then into the SE_Adapter to interact with the SAM backbone, thereby achieving high-precision and high-efficiency semantic segmentation.

[0009] Step 3: Semantic segmentation result generation. Input images and event pairs into the trained network model to generate semantic segmentation results.

[0010] The specific details of each step are as follows:

[0011] The set of input data in step 1 is an event-image set, which is represented as follows:

[0012]

[0013] Among them, I i For a specific frame in the dataset, This corresponds to events at three different time scales. We perform a series of scaling and cropping operations on the frame and its corresponding event information to achieve data augmentation and normalization.

[0014] in, The specific representation format is as follows Where t is the exposure time of the frame in the corresponding event-image group, x(t)∈[1,W], y(t)∈[1,H] are the generation positions of the generated event at time t, where W and H represent the maximum length of the row and column, respectively. p(t)∈{+1,-1} represents the polarity of the generated event at time t, where +1 represents an increase of one threshold in brightness, and -1 represents a decrease of one threshold in brightness.

[0015] Then it is necessary to The time dimension is discretized into B consecutive time bins, and they are encoded into a 3D voxel grid (E∈R). B×H×W The specific formula is as follows:

[0016]

[0017] Where t0 and N represents the start and end times of the event data, respectively.e The number of event data points is represented by m, which ranges from [0, B-1]. The event data after voxelization is represented as follows: And serve as subsequent network input.

[0018] The SAM backbone involved in step 2 is a classic transformer structure, including several self-attention layers, feedforward neural network layers, residual modules, and batch normalization layers. The event adapter (SE_Adapter) includes a cross-attention module and a gating unit composed of feedforward neural network layers. The multi-spatiotemporal scale patch embedding module (MSP) includes several convolutional layers, spiking neurons, and linear layers with different downsampling scales.

[0019] After constructing the network model, the data processing flow in the model is as follows: First, for the integrated event voxels... We expand it as The form is denoted as Where T is a fixed time step, and its value is the same as B in step 1. Then, we will... The input is fed into a ConvLIF block for spatiotemporal feature learning. Each ConvLIF module consists of multiple convolutions, batch normalization, and LIF neurons, where ConvLIF contains ConvLIF... 4X and ConvLIF 2X Two variants. Specifically, ConvLIF 4X It also includes a max pooling layer. A ConvLIF 4X The specific operation of the module is as follows:

[0020] The LIF(Batch norm(Conv3*3(V'))) part loops three times, with the strides of the convolution kernels being 2, 1, and 1, respectively.

[0021] A ConvLIF 2X The specific operation of the module is as follows:

[0022] The stride of the convolution kernel is 2.

[0023] The LIF neuron in the above formula is a type of spiking neuron, and its principle can be expressed as follows:

[0024]

[0025] S(t-1)=Heaviside(H(t-1)-V th ),

[0026] V(t) = H(t)(1-S(t)) + V resetS(t),

[0027] Where X(t) is the input at time t, and H(t) and V(t) represent the membrane potentials of a neuron after it fires a pulse and after it is charged, respectively. th V is the pulse emission threshold, Heaviside(·) is the Heaviside step function, and V reset The reset potential is given by τ, which is the membrane potential time constant.

[0028] Subsequently, the event voxel with the largest time range Only through a ConvLIF 4X Modules, and event voxels with the smallest time range. Through 1 ConvLIF 4X Module and 2 ConvLIF 2X The modules are then used to align the features output by these modules through a linear layer with dimension 1. Then flatten it into a patch format in spatial dimensions, with dimensions of N = H * W, and then concatenate them together. The specific operation process is as follows:

[0029] Concat(·) represents a concatenation operation on N.

[0030] Next we will F event In the input event adapter (SE_Adapter), the image features F in the SAM backbone are... image To interact, taking one layer as an example, we will use F in the SAM backbone. image As F in MSP event As and Where N I and N E This refers to the number of patches, determined by the resolution of the model's input. Cross-attention is then applied to the aforementioned features, and the output is... To control the influence of information from different modalities, we implemented a gating unit, which involves adjusting the attention matrix in the cross-attention mechanism. Pass it through a linear layer and normalize it to a probability matrix of 0 to 1 using the Sigmoid function. Then Γ and the obtained Perform element-wise multiplication, then pass it through a linear layer and add it back to the SAM backbone to obtain the output F'. image The specific steps of the above process are as follows:

[0031]

[0032]

[0033]

[0034] Where ⊙ represents the element-wise multiplication of the matrix.

[0035] The semantic segmentation result in step 3 It is provided by a simple semantic segmentation head, which consists of two MLPs. The specific operation is as follows:

[0036]

[0037] This method leverages the complementary nature of events in motion scenarios and RGB information. It designs a model that utilizes the rich motion information within events to guide a large-scale semantic segmentation (SAM) model for high-efficiency and robust semantic segmentation. Furthermore, to better preserve the temporal sequence of event information, a novel event information encoding method is devised. This approach overcomes the limitations of traditional semantic segmentation methods imposed by the inherent constraints of RGB sensors, as well as the challenges of processing event information and fusing it with RGB information to guide the segmentation task. Attached Figure Description

[0038] Figure 1 This is an overall structural diagram of the model of the present invention.

[0039] Figure 2 This is a structural diagram of the MSP in this invention.

[0040] Figure 3 This figure shows the comparison results of the present invention with current state-of-the-art semantic segmentation algorithms on the event-RGB semantic segmentation dataset DSEC-SEMANTIC. Detailed Implementation

[0041] The purpose of this invention is to propose an event-RGB semantic segmentation method based on large model adaptation. This invention is designed specifically for the characteristics of event data and performs semantic segmentation on a trained deep learning model.

[0042] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0043] The workflow of this invention is divided into a training phase and a prediction phase. In the training phase, we applied several data augmentation techniques to the data in the dataset and paired them into event-image groups according to the exposure timestamps of the RGB images before feeding them into the model for training. The parameters of the SAM backbone are frozen. In the prediction phase, based on the trained model parameters, we tested semantic segmentation tasks in different scenarios.

[0044] The specific data augmentation techniques are as follows:

[0045] (1) Random flipping: Horizontal and vertical flipping are performed randomly according to probability.

[0046] (2) Random rotation: Rotate clockwise or counterclockwise by 90, 180 or 270 degrees according to probability.

[0047] (3) Random cropping: randomly crop the region of interest of the image. In the experiment, the size was set to 256 pixels in length and width.

[0048] The above data augmentation techniques are applied to both image frames and event streams. Furthermore, since event data lacks a frame structure, it is first voxelized before the corresponding operations are performed.

[0049] The input data in the invention is a set of event-image data, represented as follows:

[0050]

[0051] Among them, I i For a specific frame in the dataset, This corresponds to events at three different time scales. We perform a series of scaling and cropping operations on the frame and its corresponding event information to achieve data augmentation and normalization.

[0052] in, The specific representation format is as follows Where t is the exposure time of the frame in the corresponding event-image group, x(t)∈[1,W], y(t)∈[1,H] are the generation positions of the generated event at time t, where W and H represent the maximum length of the row and column, respectively. p(t)∈{+1,-1} represents the polarity of the generated event at time t, where +1 represents an increase of one threshold in brightness, and -1 represents a decrease of one threshold in brightness.

[0053] Then it is necessary to The time dimension is discretized into B consecutive time bins, and they are encoded into a 3D voxel grid (E∈R). B×H×W The specific formula is as follows:

[0054]

[0055] Where t0 and N represents the start and end times of the event data, respectively. e The number of event data points is represented by m, which ranges from [0, B-1]. The event data after voxelization is represented as follows: And serve as subsequent network input.

[0056] After the event preprocessing process is completed, according to Figure 1 , 2 The algorithm network of this invention comprises a SAM backbone, an event adapter (SE_Adapter), and a multi-spatiotemporal scale patch embedding module (MSP). The SAM backbone is a classic transformer structure, including several self-attention layers, feedforward neural network layers, residual modules, and batch normalization layers. The event adapter (SE_Adapter) includes a cross-attention module and a gating unit composed of feedforward neural network layers. The multi-spatiotemporal scale patch embedding module (MSP) includes several convolutional layers with different downsampling scales, spiking neurons, and linear layers.

[0057] After constructing the network model, the data processing flow in the model is as follows: First, for the integrated event voxels... We expand it as The form is denoted as Where T is a fixed time step, and its value is the same as B in step 1. Then, we will... The input is fed into a ConvLIF block for spatiotemporal feature learning. Each ConvLIF module consists of multiple convolutions, batch normalization, and LIF neurons, where ConvLIF contains ConvLIF... 4X and ConvLIF 2X Two variants. Specifically, ConvLIF 4X It also includes a max pooling layer. A ConvLIF 4X The specific operation of the module is as follows:

[0058] The LIF(Batch norm(Conv3*3(V'))) part loops three times, with the strides of the convolution kernels being 2, 1, and 1, respectively.

[0059] A ConvLIF 2X The specific operation of the module is as follows:

[0060] The stride of the convolution kernel is 2.

[0061] The LIF neuron in the above formula is a type of spiking neuron, and its principle can be expressed as follows:

[0062]

[0063] S(t-1)=Heaviside(H(t-1)-V th ),

[0064] V(t) = H(t)(1-S(t)) + V reset S(t),

[0065] Where X(t) is the input at time t, and H(t) and V(t) represent the membrane potentials of a neuron after it fires a pulse and after it is charged, respectively. th V is the pulse emission threshold, Heaviside(·) is the Heaviside step function, and V reset The reset potential is given by τ, which is the membrane potential time constant.

[0066] Subsequently, the event voxel with the largest time range Only through a ConvLIF 4X Modules, and event voxels with the smallest time range. Through 1 ConvLIF 4X Module and 2 ConvLIF 2X The modules are then used to align the features output by these modules through a linear layer with dimension 1. Then flatten it into a patch format in spatial dimensions, with dimensions of N = H * W, and then concatenate them together. The specific operation process is as follows:

[0067]

[0068] Concat(·) represents a concatenation operation on N.

[0069] Next we will F event In the input event adapter (SE_Adapter), the image features F in the SAM backbone are... image To interact, taking one layer as an example, we will use F in the SAM backbone. image As F in MSP event As and Where N I and N E This refers to the number of patches, determined by the resolution of the model's input. Cross-attention is then applied to the aforementioned features, and the output is... To control the influence of information from different modalities, we implemented a gating unit, which involves adjusting the attention matrix in the cross-attention mechanism. Pass it through a linear layer and normalize it to a probability matrix of 0 to 1 using the Sigmoid function. Then Γ and the obtained Perform element-wise multiplication, then pass it through a linear layer and add it back to the SAM backbone to obtain the output F'. image The specific steps of the above process are as follows:

[0070]

[0071]

[0072]

[0073] Where ⊙ represents the element-wise multiplication of the matrix.

[0074] The original event-image set is processed by the model to obtain fine feature information, and then we need to obtain the semantic segmentation result. It is provided by a simple semantic segmentation head, which consists of two MLPs. The specific operation is as follows:

[0075]

[0076] Throughout the process, we F image , Supervision is performed using the mean intersection-union ratio (mIoU) as the loss function to constrain the quality of the segmented image, as shown in the following formula:

[0077]

[0078] Where M represents the number of categories, y c It is a one-hot vector where each element has only two values: 0 and 1. If the class and the sample class are the same, the element is set to 1; otherwise, it is set to 0. As for p... c This represents the probability that the predicted sample belongs to class c.

[0079] Appendix Figure 3 This visualization shows the comparison results of our method with state-of-the-art semantic segmentation algorithms on the DSEC-SEMANTIC event-RGB semantic segmentation dataset (column 6 for our method, and columns 4 and 5 for state-of-the-art semantic segmentation algorithms). We compare the following semantic segmentation algorithms on the dataset:

[0080] (1) EV-SegNet: The first event semantic segmentation model that uses event monomodality.

[0081] (2) E2ViD: An event semantic segmentation method based on image reconstruction, which reconstructs events into images and then segments them.

[0082] (3)ESS: A domain-adaptive event semantic segmentation method that transfers mature segmentation priors from image modalities to event modalities for segmentation.

[0083] (4) EDCNet-S2D: A semantic segmentation method based on images and events, which uses a U-Net-like method to fuse events and images.

[0084] (5) CMX: A multimodal semantic segmentation method for RGB-X that uses channel fusion to process event and image information.

[0085] (6)CMNeXt: A multimodal semantic segmentation method for RGB-X that uses query fusion to process event and image information.

[0086] (7) SE_Adapter(Ours): A semantic segmentation method based on images and events, which performs spatiotemporal encoding of events and uses events and images for motion estimation.

[0087] The Accuracy and mIoU metrics are used for evaluation on different datasets, and their specific definitions are as follows:

[0088] Where P correct ,P total These represent the number of correctly predicted pixels and the total number of pixels, respectively.

[0089] Where c represents a certain category, and M represents the total number of categories.

[0090] Higher Accuracy and mIoU values ​​indicate better segmentation results. Accuracy is more pixel-level in its evaluation, while mIoU is more macroscopic and better represents the visualization effect observed by the human eye.

[0091] Table 1 shows the test results on the DSEC-SEMANTIC dataset:

[0092] Table 1: Evaluation of frame interpolation effect on synthetic dataset

[0093]

[0094] Table 2 then presents the test results on the DDD17 dataset:

[0095] Table 2: Evaluation of frame interpolation effect on real datasets

[0096]

[0097] As shown in Tables 1 and 2, the present invention demonstrates superior performance in semantic segmentation compared to other related fields.

[0098] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal semantic segmentation method based on large model adaptation, characterized in that: Includes the following steps: Step (1): Data processing. For each pair of input data, the event information is divided into different time scales based on the exposure timestamp of the RGB data and processed into the format of network input in sequence. Step (2): Construct a network model based on the data obtained after data processing in step (1). The network model includes the SAM backbone with frozen parameters and the event adapter SE_Adapter. Step (3): Based on the data obtained in step (1) and the architecture of the network model obtained in step (2), construct a specific data encoding module. The specific data encoding module is a multi-temporal scale patch embedding module (MSP), which includes several convolutional layers, spiking neurons, pooling layers and linear layers. The parameters of the specific data encoding module can be learned. Step (4): Construct a semantic segmentation detection head based on the network model in step (2), which consists of several linear layers; the structure and process of the semantic segmentation detection head mentioned in step (4) are as follows: Step (4-1), the semantic segmentation head consists of five linear layers; Step (4-2): Based on the semantic segmentation head constructed in step (4-1), first extract four multimodal features at different scales from SAM; Step (4-3) involves inputting the multimodal features of different scales obtained in step (4-2) into the first four linear layers respectively, and then concatenating the results in the patch dimension. Step (4-4): Input the features obtained in step (4-3) into the last linear layer to obtain the semantic segmentation result; Step (5): Input the data obtained in step (1) into the model obtained in steps (2) and (3) for training. Image data is input into the SAM backbone, and event data is input into MSP and then into SE_Adapter to interact with the SAM backbone. Step (6): Use the network model in step (4) to perform multimodal semantic segmentation. Input the paired event-RGB multimodal information into the network model to achieve semantic segmentation.

2. The multimodal semantic segmentation method based on large model adaptation according to claim 1, characterized in that, The specific operation method for data processing in step (1) is as follows: Step (1-1): First, read a set of input data as an event-image group, which includes an image frame and an event cluster aggregated from three different time scales; Step (1-2) involves performing a series of scaling and cropping operations on the input data obtained in step (1-1), an image frame, and the corresponding event information, in order to achieve data augmentation and normalization. Steps (1-3) are followed by discretizing the time dimension of the event information from steps (1-2) into several consecutive time boxes and encoding them into a 3D voxel mesh, which is then used as input to the subsequent network.

3. The multimodal semantic segmentation method based on large model adaptation according to claim 2, characterized in that, The method for 3D voxel mesh encoding in steps (1-3) is as follows: Step (1-3-1) divides the time dimension of the event information into B consecutive time boxes, each containing a certain number of events, and normalizes the timestamps of the event information; Step (1-3-2) involves taking a coordinate in a timebox obtained in step (1-3-1) and performing a time-weighted sum of all events in that timebox at that coordinate. The closer the timestamp of an event is to the timestamp of the timebox, the greater its weight. Step (1-3-3) uses step (1-3-2) to process the entire event information and obtain the most primitive event representation.

4. The multimodal semantic segmentation method based on large model adaptation according to claim 1, characterized in that, The construction method and processing flow of the data encoding module -- multi-temporal scale patch embedding module MSP built in step (3) are as follows: Step (3-1): The MSP contains several ConvLIF modules of different specifications, including 4x ConvLIF modules of specification. 4X The module consists of a strided 3x3 convolution, two 3x3 convolutions, three batch normalizations, three LIF neurons, and a max pooling operation, representing a 2x ConvLIF module. 2X The module consists of a strided 3x3 convolution, a batch normalization, and a LIF neuron; Step (3-2): Based on the MSP constructed in step (3-1), the event information encoded in step (1-3) is received as input; for the integrated event voxel, it is expanded into a four-channel form, which explicitly contains time information; The data expanded in steps (3-3) and (3-2) includes three different time ranges, with the event voxel with the largest time range passing through only one ConvLIF. 4X Modules, while event voxels with the smallest time range are passed through 1 ConvLIF 4X Module and 2 ConvLIF 2X Module; Step (3-4): The event features processed in step (3-3) will be aligned through a linear layer, flattened into a patch form in the spatial dimension, and spliced ​​together to obtain the input of the event adapter SE_Adapter.

5. The multimodal semantic segmentation method based on large model adaptation according to claim 1, characterized in that, The construction method and processing flow of the network model constructed in step (2) are as follows: Step (2-1) The SAM backbone is a transformer structure, including several self-attention layers, feedforward neural network layers, residual modules and batch normalization layers. The event adapter SE_Adapter includes a cross-attention module and a gating unit composed of feedforward neural network layers. In step (2-2), the SAM backbone constructed in step (2-1) receives the enhanced image information obtained in step (1-2) and performs patch embedding to obtain the original image features. The event adapter SE_Adapter receives the event features obtained in step (3-4). Step (2-3): The image features obtained in step (2-2) interact with the event features in SE_Adapter, using the image features as the query and the event features as the key and value, and then performing cross-attention on the above features; In step (2-4), to control the influence of information from different modalities on the cross-attention results obtained in step (2-3), a gating unit is set up to pass the attention matrix in the cross-attention through a linear layer and normalize it to a probability matrix of 0 to 1 using the Sigmoid function. Then, the probability matrix is ​​multiplied element-wise with the obtained cross-attention results, and then passed through a linear layer and added back to the SAM backbone to obtain the output.

6. The multimodal semantic segmentation method based on large model adaptation according to claim 5, characterized in that, The concept and processing flow of patch embedding mentioned in step (2-2) are as follows: Step (2-2-1), patch embedding refers to the process of dividing an image into several sub-images evenly and then projecting each sub-image into a vector; Step (2-2-2), for the concept mentioned in step (2-2-1), the method is as follows: for a given sub-image size N*N, set a convolutional layer with a kernel size of N*N and a stride of N to process the image, and unfold the three-dimensional output into a two-dimensional form in the spatial layer.

7. The multimodal semantic segmentation method based on large model adaptation according to claim 5, characterized in that, The SAM processing flow mentioned in step (2-1) is as follows: Step (2-1-1): For the input image information, first use the method in step (2-2-2) to embed it into patch form; In step (2-1-2), for the patch obtained in step (2-1-1), SAM first retains the original patch as a residual connection, and then performs self-attention operation on it. In step (2-1-3), for the residual connections and self-attention results obtained in step (2-1-2), SAM projects the self-attention results through a linear layer and adds residual connections to obtain the result of one of the transformer layers. Step (2-1-4) involves repeating the operation mentioned in step (2-1-3) several times to obtain the output of SAM.

8. The multimodal semantic segmentation method based on large model adaptation according to claim 5, characterized in that, The attention operation process mentioned in steps (2-3) is as follows: Step (2-3-1) first determines the features to be used as queries, keys, and values; Step (2-3-2): Based on the features to be used as queries, keys and values ​​determined in step (2-3-1), project them through different linear layers to obtain the queries, keys and values ​​for this attention operation. Step (2-3-3): Based on the query and key obtained in step (2-3-2), the transposes of the query and key are multiplied by matrix to obtain the attention matrix; Step (2-3-4): Based on the attention matrix obtained in step (2-3-4), multiply it with the value obtained in step (2-3-2) to obtain the attention result for this step. The self-attention operations mentioned in steps (2-3-5) and (2-1-2) are similar to the steps above, except that they are different as features of queries, keys, and values.

9. The multimodal semantic segmentation method based on large model adaptation according to claim 1, characterized in that, The model training process mentioned in step (5) is as follows: Step (5-1) First, input the paired event-RGB data into the model for inference; Step (5-2): Input the result obtained in step (5-1) into the loss function to calculate the loss; Step (5-3): Input the loss obtained in step (5-2) into the AdamW optimizer for backpropagation, update the network parameters, and update the learning rate; Step (5-4): Repeat steps (5-1) to (5-3), and test at regular intervals. If the effect reaches the current optimal level, save the parameters. The semantic segmentation process mentioned in step (6) is as follows: Step (6-1) involves loading the parameters obtained in step (5-4) of the network. Step (6-2): Input the data into the network obtained in step (6-1) to obtain the inference results; Step (6-3) normalizes the pixel values ​​of the inference results to 0-255 to obtain the visualized semantic segmentation results.

10. The multimodal semantic segmentation method based on large model adaptation according to claim 1, characterized in that, It includes a data augmentation module for the data preprocessing stage, a 3D voxel mesh module, a multi-spatiotemporal scale patch embedding module (MSP) for event encoding, an event adapter (SE_Adapter) for generating semantic features, and a semantic segmentation detection head for generating semantic results; among which, The data augmentation module is used to enhance the diversity of data samples, thereby indirectly expanding the dataset; The 3D voxel mesh module is used to aggregate discrete four-dimensional event data into a three-dimensional space to satisfy the input conditions of the model. The Multi-Spatiotemporal Scale Patch Embedding Module (MSP) is responsible for efficiently encoding the aggregated event information into patch form while preserving its temporality and asynchronicity. The role of the event adapter SE_Adapter is to inject specific information about downstream tasks into the SAM backbone, guide it to converge to the downstream tasks, and generate high-quality image semantic features. The semantic segmentation detection head is responsible for decoding multi-scale semantic features and generating semantic segmentation results.

Images