A method and system for video semantic segmentation based on detail-refined state-space models

Abstract

This invention discloses a video semantic segmentation method and system based on a detailed refined state-space model, belonging to the field of video semantic segmentation. The method includes: constructing a video semantic segmentation dataset; constructing a video semantic segmentation model, including a first state-space model and a second state-space model; performing channel-level amplitude sensing on the second state-space model to obtain spectral features; adaptively flipping and refining the forgetting gate parameter matrix of the second state-space model according to the spectral features and the channel-level forgetting degree to obtain a refined forgetting gate parameter matrix, which serves as the parameter matrix of the first state-space model; training the video semantic segmentation model using the video semantic segmentation dataset; and obtaining the semantic segmentation result using the trained video semantic segmentation model. This invention effectively alleviates the forgetting phenomenon of spatiotemporal details in state-space models during state compression, and improves the model's ability to segment detailed information during pixel-level semantic segmentation.

Description

Technical Field

[0001] This invention belongs to the field of video semantic segmentation technology, and more specifically, relates to a video semantic segmentation method and system based on a state-space model with detailed refinement. Background Technology

[0002] Existing video semantic segmentation models primarily utilize the global context modeling mechanism of attention models, relying on simultaneous computation across all frames of the video. However, in practical applications such as autonomous driving and video surveillance, the increasing prevalence of long video streams due to factors like urban development and the widespread use of surveillance equipment necessitates models capable of efficiently processing these streams. Since state-space models offer the advantage of processing input data with linear complexity, researching video semantic segmentation methods based on state-space models is crucial for promoting the application of efficient video semantic segmentation models.

[0003] In recent years, video semantic segmentation has received increasing attention. However, existing patents mainly use attention models for video semantic segmentation. For example, Zhejiang University, in its patent application "A Video Semantic Segmentation Method Based on Temporal Cross-Attention Mechanism" (patent application number CN202510912693.X, patent publication number CN120411866B), constructs a multi-level encoder-decoder network for feature extraction and fusion, enhancing feature extraction through a temporal cross-attention module. Furthermore, the University of Electronic Science and Technology of China, in its patent application "A Semantic Segmentation Method for Major Components of Flying Targets in Video Streams Based on Deep Learning" (patent application number CN202310881500.X, patent publication number CN116883663A), utilizes a local temporal attention mechanism to introduce information from the preceding and following frames of the current frame for prediction, achieving better segmentation results. However, these patents neglect the problem of high computational cost for long video data in video semantic segmentation tasks, lacking research on improving the efficiency of models in processing long videos in video semantic segmentation scenarios.

[0004] In summary, state-space model-based video semantic segmentation is an important approach to address the computationally intensive nature of video data processing in practical applications. However, existing technologies have not yet resolved the issue of spatiotemporal detail loss in state-space models. Therefore, the technical solution of this invention is groundbreaking, advanced, and has high application value. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a video semantic segmentation method and system based on a detailed refined state space model. The purpose is to restore and refine the spatiotemporal details lost during state space compression by adaptively flipping and refining the forget gate parameter matrix of the state space model, thereby enhancing the video semantic segmentation performance of the model.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A video semantic segmentation method based on a detailed state-space model includes the following steps: Construct a video semantic segmentation dataset; Construct a video semantic segmentation model, which includes a first state space model and a second state space model; Channel-level amplitude sensing is performed on the second state-space model to obtain the spectral characteristics of each channel; Based on the spectral characteristics and the degree of forgetting at the channel level, the forgetting gate parameter matrix of the second state space model is adaptively flipped and refined to obtain the refined forgetting gate parameter matrix, which is then used as the parameter matrix of the first state space model. The video semantic segmentation model is trained using a video semantic segmentation dataset; Given a video to be processed, the semantic segmentation result is obtained using the trained video semantic segmentation model.

[0007] Furthermore, the video semantic segmentation model includes a video feature extraction model, a first state space model, a second state space model, a multilayer perceptron, and a semantic segmentation head.

[0008] Furthermore, the channel-level amplitude sensing of the second state-space model includes: The hidden states of the second state space model are converted into frequency domain features using a two-dimensional Fourier transform. The amplitude characteristics are calculated based on the real and imaginary parts of the frequency domain features; Based on the spatial dimension of the amplitude features, the hidden state of each frame is divided into multiple frequency bands after calculating the frequency radius, thus obtaining the frequency band mask, and the energy distribution of each frequency band is calculated. The energy distribution of each frequency band is normalized, and the sum of the energy of the first few high-frequency bands is calculated based on the normalized energy distribution as the spectral characteristics of each channel.

[0009] Furthermore, the adaptive flipping and refinement of the forgetting gate parameter matrix of the second state-space model based on spectral characteristics and channel-level forgetting degree includes: Obtain the forget gate parameter matrix of the second state space model and calculate the importance of each channel. The lower the importance of a channel, the higher the degree of forgetting of the channel. The range of the forget gate parameter matrix of the second state space model is flipped along the channel dimension to obtain the flipped forget gate parameter matrix; Given a batch of data containing multiple video segments, calculate the spectral features of the entire batch based on the spectral features of each frame, and calculate the adaptive gating by combining the importance of each channel. Based on the forget gate parameter matrix of the second state space model, the flipped forget gate parameter matrix, and the adaptive gating, the refined forget gate parameter matrix is ​​calculated.

[0010] Furthermore, the refined forget gate parameter matrix is ​​generated by adaptive gating weighting of the forget gate parameter matrix and the flipped forget gate parameter matrix, thereby achieving decoupling and complementarity between global semantics and spatiotemporal details at the model parameter level.

[0011] Furthermore, training the video semantic segmentation model using the video semantic segmentation dataset includes: Each frame of the video in the video semantic segmentation dataset is input into the video feature extraction model for feature extraction; The extracted features are input into the first state space model and the second state space model. The outputs of the first state space model and the second state space model are fused using a multilayer perceptron to obtain a feature map. Input the feature map into the semantic segmentation head to obtain the semantic segmentation mask.

[0012] Furthermore, given the video to be processed, the semantic segmentation result is obtained using the trained video semantic segmentation model, including: Each frame of the video to be processed is input into the video feature extraction model for feature extraction; The extracted features are input into the first state space model and the second state space model. The outputs of the first state space model and the second state space model are fused using a multilayer perceptron to obtain a feature map. Input the feature map into the semantic segmentation head to obtain the semantic segmentation mask.

[0013] A video semantic segmentation system based on a detailed state-space model, comprising: The dataset building module is used to build video semantic segmentation datasets; The model building module is used to build a video semantic segmentation model, which includes a first state space model and a second state space model. The channel-level amplitude sensing module is used to perform channel-level amplitude sensing on the second state space model; The forget gate information refinement module is used to refine the forget gate information of the second state space model based on the channel-level amplitude sensing results, and obtain the refined forget gate parameter matrix. The refined forget gate parameter matrix is ​​then used as the parameter matrix of the first state space model. The model training module is used to train the video semantic segmentation model using the video semantic segmentation dataset; The semantic segmentation module is used to obtain semantic segmentation results from the trained video semantic segmentation model, given a video to be processed.

[0014] The present invention also provides a computer device including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.

[0015] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a computer, implements the above-described method.

[0016] The innovations and beneficial effects of this invention are as follows, with the key focus being "parameter-level refinement guided by the frequency domain": 1) Innovative "Cross-Domain" Perception Mechanism (Frequency Domain Guided Spatial Domain). This invention does not use conventional convolution or attention mechanisms to perceive details, but instead innovatively introduces frequency domain analysis into the state-space model. It transforms the hidden state to the frequency domain using FFT, and quantitatively calculates the "detail information content" of each channel by utilizing the energy distribution of high-frequency bands. This method can more accurately identify which channels carry texture details at a physical level, rather than relying solely on learnable weights.

[0017] 2) Unique "Parameter-Level" Forget Gate Flipping Strategy. This is the most significant differentiating feature of this invention. Existing detail enhancement typically involves adding to or concatenating output features (ResNet concept), while this invention modifies the "valve" (forget gate) that controls the flow of information. Specifically, this invention performs a domain flipping operation on the forget gate parameter matrix unique to the state-space model. This design is based on a reverse thinking—if the original model determines that certain channels should be "forgotten," then these channels often contain low-level spatiotemporal details suppressed by high-level semantics. By flipping the parameter matrix, the model is forced to... Pay attention to these Ignored information.

[0018] 3) Adaptive parameter generation and complementary modeling. This invention constructs a set of complementary state-space models ( and They are not trained independently, but rather through adaptive gating. Dynamic association. The refined Forgetting Gate. It is composed of the original matrix and flip matrix It is generated based on weighted spectral features. This approach achieves decoupling and complementarity between "global semantics" and "spatiotemporal details" at the model parameter level, rather than simply fusion at the result level. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the steps of a video semantic segmentation method based on a detailed refinement state-space model, according to an embodiment of the present invention.

[0020] Figure 2 This is a block diagram of a video semantic segmentation system based on a detailed state-space model, according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] In one embodiment, the present invention provides a video semantic segmentation method based on a detailed state-space model, such as... Figure 1 As shown, it includes the following steps: Construct a video semantic segmentation dataset; Construct a video semantic segmentation model, which includes a first state space model and a second state space model; Channel-level amplitude sensing is performed on the second state-space model to obtain the spectral characteristics of each channel; Based on the spectral characteristics and the degree of forgetting at the channel level, the forgetting gate parameter matrix of the second state space model is adaptively flipped and refined to obtain the refined forgetting gate parameter matrix, which is then used as the parameter matrix of the first state space model. The video semantic segmentation model is trained using a video semantic segmentation dataset; Given a video to be processed, the semantic segmentation result is obtained using the trained video semantic segmentation model.

[0023] In one embodiment, the video semantic segmentation model includes a video feature extraction model, a first state space model, a second state space model, a multilayer perceptron, and a semantic segmentation head.

[0024] In one embodiment, the channel-level amplitude sensing of the second state-space model includes: The hidden states of the second state space model are converted into frequency domain features using a two-dimensional Fourier transform. The amplitude characteristics are calculated based on the real and imaginary parts of the frequency domain features; Based on the spatial dimension of the amplitude features, the hidden state of each frame is divided into multiple frequency bands after calculating the frequency radius, thus obtaining the frequency band mask, and the energy distribution of each frequency band is calculated. The energy distribution of each frequency band is normalized, and the sum of the energy of the first few high-frequency bands is calculated based on the normalized energy distribution as the spectral characteristics of each channel.

[0025] In one embodiment, the adaptive flipping and refinement of the forgetting gate parameter matrix of the second state-space model based on spectral characteristics and channel-level forgetting degree includes: Obtain the forget gate parameter matrix of the second state space model and calculate the importance of each channel. The lower the importance of a channel, the higher the degree of forgetting of the channel. The range of the forget gate parameter matrix of the second state space model is flipped along the channel dimension to obtain the flipped forget gate parameter matrix; Given a batch of data containing multiple video segments, calculate the spectral features of the entire batch based on the spectral features of each frame, and calculate the adaptive gating by combining the importance of each channel. Based on the forget gate parameter matrix of the second state space model, the flipped forget gate parameter matrix, and the adaptive gating, the refined forget gate parameter matrix is ​​calculated.

[0026] In one embodiment, the refined forget gate parameter matrix is ​​generated by adaptive gating weighting of the forget gate parameter matrix and the flipped forget gate parameter matrix, thereby achieving decoupling and complementarity between global semantics and spatiotemporal details at the model parameter level.

[0027] In one embodiment, training the video semantic segmentation model using a video semantic segmentation dataset includes: Each frame of the video in the video semantic segmentation dataset is input into the video feature extraction model for feature extraction; The extracted features are input into the first state space model and the second state space model. The outputs of the first state space model and the second state space model are fused using a multilayer perceptron to obtain a feature map. Input the feature map into the semantic segmentation head to obtain the semantic segmentation mask.

[0028] In one embodiment, given a video to be processed, obtaining a semantic segmentation result using the trained video semantic segmentation model includes: Each frame of the video to be processed is input into the video feature extraction model for feature extraction; The extracted features are input into the first state space model and the second state space model. The outputs of the first state space model and the second state space model are fused using a multilayer perceptron to obtain a feature map. Input the feature map into the semantic segmentation head to obtain the semantic segmentation mask.

[0029] In one embodiment, the present invention provides a video semantic segmentation method based on a detailed state-space model, comprising the following steps: S1. Construct a video semantic segmentation dataset. Collect target data from multiple cameras with different perspectives at different locations. Label the semantic category of each frame pixel-by-pixel.

[0030] S2. Model Building. Constructing a video feature extraction model. Semantic segmentation head One of each, construct a state-space model and One group each.

[0031] S3. Channel-level amplitude sensing. This step includes the following sub-steps: Feature extraction based on frequency domain transform: The state-space model in step S2 is transformed using a two-dimensional Fourier transform. Hidden state Transform to frequency domain features .

[0032] Amplitude characteristic calculation: based on frequency domain characteristics real part and the virtual part Calculate amplitude characteristics .

[0033] Band decomposition and energy calculation: based on amplitude characteristics Spatial Dimensions and The hidden state of each frame After calculating the frequency radius, it is divided into Each frequency band is used to obtain the frequency band mask. And further calculate the energy distribution of each frequency band. .

[0034] Spectral feature extraction: energy distribution for each frequency band Normalization is performed to obtain the normalized energy distribution. And further statistics on the previous The sum of energy in each high-frequency band represents the content of detailed information in each channel, serving as the spectral characteristic of each channel. In this context, "channel" refers to the embedding dimension of a feature vector in a neural network, for example... If the embedding dimension has C dimensions, then it is called There are C channels.

[0035] S4. Refine the information at the forget gate. This step includes the following sub-steps: Channel-level forgetting perception: from the original state-space model Obtain its forget gate parameter matrix And calculate the importance of each channel. The lower the importance level, the higher the degree of forgetting of that channel.

[0036] State-space model forget gate matrix flipping: given a state-space model Forget gate parameter matrix Flipping its range along the channel dimension yields This is in preparation for refining the Gate of Oblivion.

[0037] Adaptive gating based on spectral features: Each frame of image has several channels, the first... c The spectral characteristics of each channel are as follows Spectral characteristics of all channels The spectral features of each frame of the image are obtained by stitching them together. Given a A batch of data for each video segment, where each video segment contains Frame images, based on the spectral characteristics of each frame image Calculate the spectral characteristics for the entire batch. And further related to the importance of each channel. Calculation yields adaptive gating .

[0038] State-space model forget gate matrix refinement: given a state-space model Forget gate parameter matrix Its value range inverted version Given adaptive gating The refined forget gate parameter matrix was calculated. The refined forget gate parameter matrix This will be for models with high detail and information content in the original state space. Channels with higher forgetting rates are given higher attention, and the refined forgetting gate parameter matrix is ​​used to... As a model The parameter matrix is ​​beneficial to the model. Refinement of detailed information.

[0039] S5. Model Training: Using the video semantic segmentation dataset constructed in step S1, given each frame of the video as input image... After video feature extraction model Feature maps are obtained after encoding. Then it is input into a set of stacked state space models. and middle, and Receive the same input, but only Using the refined forget gate parameter matrix For the first Layer state space model group and The input feature map is denoted as First, it is projected to a hidden state through a linear layer. Then, they are input into the state space model respectively. and Then, the corresponding output passes through a multilayer perceptron. Feature map obtained by fusion Finally, it goes through the semantic segmentation head. Obtain the semantic segmentation mask. .

[0040] S6. Model Testing: Given a specified video, test each video... The frame is divided into segments, and these segments are sequentially input into the video feature extraction model. After feature extraction, the data is input into the state space model group. , The corresponding output passes through a multilayer perceptron. After fusion, the semantic segmentation head is then input. The semantic segmentation mask is obtained as the semantic segmentation result.

[0041] In one embodiment, this invention proposes a video semantic segmentation method based on a detailed state-space model, comprising six parts: constructing a video semantic segmentation dataset, model construction, channel-level amplitude sensing, forget gate information refinement, model training, and model testing. In the video semantic segmentation dataset construction stage, target data from different locations and perspectives are collected, and the samples are semantically labeled pixel-by-pixel. In the model construction stage, the network structure and prototype required for model training are initialized. In the channel-level amplitude sensing stage, the detail information content in each channel of the state space is quantified through frequency domain analysis. In the forget gate information refinement stage, the forget gate parameter matrix of the state space is adaptively flipped and refined based on the channel-level forgetting degree and the detail information content. In the model training stage and the model testing stage, the video semantic segmentation function is completed through input data feature extraction. The channel-level amplitude sensing method based on frequency domain analysis in this invention can effectively quantify the content of detailed information in different channels in the hidden state of the model. In addition, the forget gate information refinement method proposed in this invention uses the content of detailed information in different channels and the degree of forgetting to adaptively flip and refine the forget gate parameter matrix in the state space model, so as to enable the model to effectively model lost details. Furthermore, the state space model video semantic segmentation method with refined details uses global semantic information and refined spatiotemporal detailed information for adaptive fusion to improve the pixel-level video semantic segmentation capability.

[0042] This embodiment provides a video semantic segmentation method with a refined state-space model, comprising the following steps: S1. Construct a video semantic segmentation dataset. Collect target data from multiple cameras with different perspectives at different locations. Label the semantic category of each frame pixel-by-pixel.

[0043] S2. Model Building. Constructing a video feature extraction model. Semantic segmentation head One of each, construct a state-space model and One group each.

[0044] S3. Channel-level amplitude sensing.

[0045] S31. Feature extraction based on frequency domain transform. The state-space model from step S2 is extracted using a two-dimensional Fourier transform. Hidden state Transform to frequency domain features ; S32. Amplitude characteristic calculation. Based on frequency domain characteristics. real part and the virtual part Calculate amplitude characteristics ; S33. Bandwidth decomposition and energy calculation. Based on amplitude characteristics. Spatial Dimensions and First, calculate the frequency radius: in, Indicates the height of the feature map. Indicates the width of the feature map. It is the coordinate index of a pixel on the feature map. It is the row index in the vertical direction (0 to H-1). It is a horizontal column index (0 to W-1).

[0046] Then, according to the following formula, it is divided into Each frequency band is used to obtain the frequency band mask. : in, This is an indicator function that outputs 1 when the condition is met, and 0 otherwise.

[0047] Further calculations yielded the energy distribution for each frequency band. : And normalize it: in, express The values ​​in the channel dimension c and the spatial dimension (h, w) are indices of the corresponding dimensions.

[0048] S34. Spectral Feature Extraction. Energy distribution for each frequency band. Normalization is performed to obtain the normalized energy distribution. And further statistics on the previous The sum of energy in each high-frequency band represents the content of detailed information in each channel, serving as the spectral characteristic of each channel. : S4. Forgot Gate Information Refinement.

[0049] S41. Channel-level forgetting perception. From the original state-space model. Obtain its forget gate parameter matrix And calculate the importance of each channel. The lower the importance level, the higher the degree of forgetting of that channel. in, Represents the numerical stability constant, with a value of .

[0050] S42. Forget Gate Matrix Flipping in State-Space Models. Given a state-space model... Forget gate parameter matrix Flipping its range along the channel dimension yields In preparation for refining the Gate of Oblivion; in, Represents the forget gate parameter matrix The maximum value in the channel dimension. Represents the forget gate parameter matrix Minimum value in the channel dimension.

[0051] S43. Adaptive gating based on spectral features. Given a given... A batch of data for each video segment, where each video segment contains Frame images, based on the spectral characteristics of each frame image Calculate the spectral characteristics for the entire batch. And further related to the importance of each channel. Calculation yields adaptive gating . in, Indicates the importance of each channel. splicing in the channel dimension.

[0052] S44. Refining the Forget Gate Matrix of a State-Space Model. Given a state-space model... Forget gate parameter matrix Its value range inverted version Given adaptive gating The refined forget gate parameter matrix was calculated. The refined forget gate parameter matrix This will be for models with high detail and information content in the original state space. Channels that are more easily forgotten receive more attention, which is why they are used as models. The parameter matrix is ​​beneficial to the model. Refinement of detailed information; S5. Model Training. Given an input image for each frame of a video. After video feature extraction model Feature maps are obtained after encoding. Then it is input into a set of stacked state space models. and middle, and Receive the same input, but only Using the refined forget gate parameter matrix For the first Layer state space model group and Its input feature map is denoted as First, it is projected to a hidden state through a linear layer. Then, they are input into the state space model respectively. and Then, the corresponding output passes through a multilayer perceptron. Feature map obtained by fusion : Feature map Finally, after semantic segmentation head Obtain the semantic segmentation mask. .

[0053] Ultimately, for the total The video input of the first frame, the The semantic segmentation mask of the frame is The baseline segmentation mask is The model training uses the following loss function: in, Let cross-entropy be the loss function. This is a hyperparameter (set to 0.5).

[0054] S6. Model Testing. Given a specified video, test each video... The frame is divided into segments, and these segments are sequentially input into the video feature extraction model. After feature extraction, the data is input into the state space model group. , The corresponding output passes through a multilayer perceptron. After fusion, the semantic segmentation head is then input. The semantic segmentation mask is obtained as the result.

[0055] To verify the effectiveness of the proposed technology, performance tests were conducted on the video semantic segmentation datasets VSPW, NYUv2, CamVid, and Cityscapes. The experimental results are shown in Table 1.

[0056] Table 1 As can be seen from the results in Table 1, the method of this invention improves the average intersection-union ratio (mIoU) of the segmentation results on the public video semantic segmentation datasets VSPW, NYUv2, CamVid and Cityscapes by 1.7, 1.7, 1.9 and 2.7 respectively compared with the baseline method, with an average performance improvement of 2.0. This proves that the detailed refined state space model video semantic segmentation method proposed in this invention can significantly improve the performance of the model in video semantic segmentation scenarios.

[0057] Another embodiment of the present invention provides a video semantic segmentation system based on a detailed state-space model, such as... Figure 2 As shown, it includes: The dataset building module is used to build video semantic segmentation datasets; The model building module is used to build a video semantic segmentation model, which includes a first state space model and a second state space model. The channel-level amplitude sensing module is used to perform channel-level amplitude sensing on the second state space model; The forget gate information refinement module is used to refine the forget gate information of the second state space model based on the channel-level amplitude sensing results, and obtain the refined forget gate parameter matrix. The refined forget gate parameter matrix is ​​then used as the parameter matrix of the first state space model. The model training module is used to train the video semantic segmentation model using the video semantic segmentation dataset; The semantic segmentation module is used to obtain semantic segmentation results from the trained video semantic segmentation model, given a video to be processed.

[0058] The above division of modules is merely illustrative. In practical applications, the functions described above can be assigned to different functional modules as needed to complete all or part of the functions described in the aforementioned method. The specific working process of each module can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here. Each of the above modules can be implemented entirely or partially through software, hardware, or a combination thereof.

[0059] Another embodiment of the present invention provides a computer device (computer, server, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing steps of the method of the present invention.

[0060] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) that stores a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0061] Another embodiment of the present invention provides a computer program product, the computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0062] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video semantic segmentation method based on a detailed state-space model, characterized in that, Includes the following steps: Construct a video semantic segmentation dataset; Construct a video semantic segmentation model, which includes a first state space model and a second state space model; Channel-level amplitude sensing is performed on the second state-space model to obtain the spectral characteristics of each channel; Based on the spectral characteristics and the degree of forgetting at the channel level, the forgetting gate parameter matrix of the second state space model is adaptively flipped and refined to obtain the refined forgetting gate parameter matrix, which is then used as the parameter matrix of the first state space model. The video semantic segmentation model is trained using a video semantic segmentation dataset; Given a video to be processed, the semantic segmentation result is obtained using the trained video semantic segmentation model.

2. The method according to claim 1, characterized in that, The video semantic segmentation model includes a video feature extraction model, a first state space model, a second state space model, a multilayer perceptron, and a semantic segmentation head.

3. The method according to claim 1, characterized in that, The channel-level amplitude sensing of the second state-space model includes: The hidden states of the second state space model are converted into frequency domain features using a two-dimensional Fourier transform. The amplitude characteristics are calculated based on the real and imaginary parts of the frequency domain features; Based on the spatial dimension of the amplitude features, the hidden state of each frame is divided into multiple frequency bands after calculating the frequency radius, thus obtaining the frequency band mask, and the energy distribution of each frequency band is calculated. The energy distribution of each frequency band is normalized, and the sum of the energy of the first few high-frequency bands is calculated based on the normalized energy distribution as the spectral characteristics of each channel.

4. The method according to claim 1, characterized in that, The adaptive flipping and refinement of the forgetting gate parameter matrix of the second state-space model based on spectral characteristics and channel-level forgetting degree includes: Obtain the forget gate parameter matrix of the second state space model and calculate the importance of each channel. The lower the importance of a channel, the higher the degree of forgetting of the channel. The range of the forget gate parameter matrix of the second state space model is flipped along the channel dimension to obtain the flipped forget gate parameter matrix; Given a batch of data containing multiple video segments, the spectral features of the entire batch are calculated based on the spectral features of each frame, and adaptive gating is calculated by combining the importance of each channel. Based on the forget gate parameter matrix of the second state space model, the flipped forget gate parameter matrix, and the adaptive gating, the refined forget gate parameter matrix is ​​calculated.

5. The method according to claim 4, characterized in that, The refined forget gate parameter matrix is ​​generated by adaptive gating weighting of the forget gate parameter matrix and the flipped forget gate parameter matrix, thereby achieving decoupling and complementarity between global semantics and spatiotemporal details at the model parameter level.

6. The method according to claim 1, characterized in that, The step of training the video semantic segmentation model using a video semantic segmentation dataset includes: Each frame of the video in the video semantic segmentation dataset is input into the video feature extraction model for feature extraction; The extracted features are input into the first state space model and the second state space model. The outputs of the first state space model and the second state space model are fused using a multilayer perceptron to obtain a feature map. Input the feature map into the semantic segmentation head to obtain the semantic segmentation mask.

7. The method according to claim 1, characterized in that, Given a video to be processed, the semantic segmentation result is obtained using the trained video semantic segmentation model, including: Each frame of the video to be processed is input into the video feature extraction model for feature extraction; The extracted features are input into the first state space model and the second state space model. The outputs of the first state space model and the second state space model are fused using a multilayer perceptron to obtain a feature map. Input the feature map into the semantic segmentation head to obtain the semantic segmentation mask.

8. A video semantic segmentation system based on a detailed state-space model, characterized in that, include: The dataset building module is used to build video semantic segmentation datasets; The model building module is used to build a video semantic segmentation model, which includes a first state space model and a second state space model. The channel-level amplitude sensing module is used to perform channel-level amplitude sensing on the second state space model; The forget gate information refinement module is used to refine the forget gate information of the second state space model based on the channel-level amplitude sensing results, and obtain the refined forget gate parameter matrix. The refined forget gate parameter matrix is ​​then used as the parameter matrix of the first state space model. The model training module is used to train the video semantic segmentation model using the video semantic segmentation dataset; The semantic segmentation module is used to obtain semantic segmentation results from the trained video semantic segmentation model, given a video to be processed.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images