A method and system for target region segmentation based on spatiotemporal event pulse streams

Abstract

This invention discloses a target region segmentation method and system for spatiotemporal event pulse streams, belonging to the field of video segmentation. This invention effectively adapts to the input of traditional neural networks by transforming continuous event pulse streams into specific input representations; it improves the model's feature extraction capability for continuous event pulse inputs by extracting information and memorizing past features through recurrent neural networks; it improves the model's query matching capability by matching targets and updating hidden states simultaneously through a recurrent feature encoder to model spatiotemporal relationships; it improves the model's robustness to feature matching by obtaining the attention relationship matrix of the target at the current and past moments through a feature decoder; and it improves the target region segmentation model's feature fusion capability for coarse-grained and fine-grained information by inputting intermediate features into the segmentation head through skip connections, thereby improving the accuracy of predicting target regions. This invention is adaptable to low-light and high-speed scenes and can segment target regions at any time.

Description

Technical Field

[0001] This invention relates to a target region segmentation method and system for spatiotemporal event pulse streams, and more particularly to a target region segmentation method under low light and high speed environments, belonging to the field of video object segmentation. Background Technology

[0002] Video object segmentation is a fundamental task in computer vision and has been widely applied in many fields such as human-computer interaction, autonomous driving, and object tracking. In recent years, deep learning methods for video object segmentation have made great strides and have become the mainstream solution for this type of problem. However, existing methods are based on traditional visual sensors to obtain image modal data, which is increasingly unable to meet the needs of current visual tasks, especially in low-light, high-dynamic, and high-speed scenes where good segmentation results cannot be produced. This is mainly due to two reasons: (1) These sensors usually sample the scene completely at a preset fixed frequency, and this sampling method cannot adapt to the dynamic changes of the scene, resulting in insufficient information being captured in low-light scenes. (2) The acquired images are not continuous in time, and the motion information between frames is not recorded. In high-speed scenes, traditional visual sensors often cause severe dynamic blur, which has an adverse effect on object segmentation.

[0003] Based on the visual sampling mechanism of the biological retina, a series of novel cameras that acquire pulse array signals have gradually come into view. These include sensors that emit pulse signals based on changes in light intensity, such as Dynamic Vision Sensors (DVS), Asynchronous Time-based Image Sensors (ATIS), and Dynamic and Active Pixel Vision Sensors (DAVIS); and sensors that emit signals based on the cumulative intensity of light, such as pulse cameras (Vidar). Unlike the sampling methods of traditional cameras, these cameras acquire light signal information within a certain time and area, offering advantages such as low motion blur, high dynamic range, and high temporal resolution.

[0004] In recent years, spatiotemporal event pulse streams have been applied to various computer vision tasks, such as object classification, target detection, semantic segmentation, and depth estimation, with some success. This data modality still has great potential and is expected to solve scenarios that traditional image modalities cannot adapt to. Summary of the Invention

[0005] To address the limitations of existing video object segmentation methods in handling low-light and high-speed scenes, this invention aims to provide a method and system for accurate target region segmentation in such environments. Based on the data characteristics of spatiotemporal event pulse streams, the invention transforms continuous event pulse streams into specific input representations to adapt to the input of traditional neural networks. Recurrent neural networks extract information and memorize past features, improving the feature extraction capability of the target region segmentation model for continuous event pulse inputs. A recurrent feature encoder matches the target while updating the hidden state, modeling spatiotemporal relationships and enhancing the model's ability to query and match targets. A feature decoder obtains the attention relationship matrix of the target at the current and past moments, improving the model's robustness to feature matching. Skip connections input intermediate features into the segmentation head enhance the model's ability to fuse coarse-grained and fine-grained information, thereby improving the accuracy of target region prediction. This invention not only achieves accurate segmentation results in low-light and high-speed scenes but also segments target regions at any given time.

[0006] The objective of this invention is achieved through the following technical solution.

[0007] This invention discloses a target region segmentation method for spatiotemporal event pulse streams, comprising the following steps:

[0008] Step 101: Acquire spatiotemporal event pulse stream E through an event pulse camera. In order to adapt to the input form of the neural network, take advantage of the fact that voxels can retain the information of events in the time dimension and can represent the spatial distribution of events. According to the event voxel conversion relationship, the continuous spatiotemporal event pulse stream E is converted into an input representation V that adapts to the neural network.

[0009] The continuous spatiotemporal event pulse stream E includes, but is not limited to: event data generated by the event camera and pulse data generated by the pulse camera.

[0010] The continuous spatiotemporal event stream E is an N×4 matrix, where N is the number of events, and each event is represented by 4-dimensional data (timestamp, x, y, polarity), where timestamp is the timestamp when the event occurs, 1≤x≤W, 1≤y≤H are the pixel positions corresponding to the event, H×W is the spatial resolution of the event, and polarity∈{-1,1} is the polarity of the event. In order to adapt to the input form of the neural network, the continuous spatiotemporal event pulse stream E is transformed into the input representation V adapted to the neural network according to the event voxel conversion relationship shown in formula (1).

[0011]

[0012] Where t0,t n t represents the start and end times of the entire event flow. iThe time of each event's occurrence. B is the specified voxel dimension, ΔT is the time interval of the entire event stream, and p i For the event polarity, x l ,y m These are spatial coordinates.

[0013] Step 102: Use a recurrent neural network to represent the input V at time t. t Perform feature extraction to obtain features for each layer. Simultaneously update the hidden state of the recurrent neural network. Improve the feature extraction capability of the target region segmentation model for continuous event pulse inputs.

[0014] The recurrent neural network for feature extraction includes, but is not limited to, ConvLSTM, ConvGRU, and SNN. Preferably, ConvLSTM is used to extract voxel features.

[0015] Event data reflects the change in light intensity of pixels at a certain moment. However, for cases with small changes in light intensity and insignificant object motion, sufficient event signals will not be emitted, leading to sparse spatial distribution of events and insufficient information in some voxel data V. Based on the highly continuous nature of events in the temporal dimension, utilizing past event information can improve the feature extraction capability of the target region segmentation model. Therefore, a recurrent neural network (ConvLSTM) is used to extract voxel features, and a hidden state (state) is used to record past event information. Features are adaptively extracted from sparse or dense voxels, ensuring model robustness while enhancing the information memory capacity of the target region segmentation model and improving its segmentation accuracy. The specific formula for ConvLSTM voxel feature extraction is as follows:

[0016]

[0017] Where i is the layer index of ConvLSTM, V t Let be the voxel at time t. When t = 1, It is initialized as a matrix of all zeros; when t≥1, For each layer of ConvLSTM, the hidden state is from the previous iteration. To be hidden after the update. The intermediate features of each voxel are defined.

[0018] Step 103: Given the label of the target region to be detected, use the label fusion feature extraction recurrent neural network to extract the input representation V at time t-1. t-1 and Perform feature fusion to obtain label fusion features Simultaneously update the hidden status.

[0019] The label of the target region to be detected, the label acquisition method includes but is not limited to: manually specifying the target region and category, and using a salient region detection model to obtain the target label; the recurrent neural network for label fusion feature extraction includes but is not limited to: ConvLSTM, ConvGRU, SNN, and preferably, ConvLSTM is selected as the recurrent neural network for label fusion feature extraction.

[0020] Since the acquisition of event data depends on changes in light intensity, the event density in each input voxel varies considerably. To adapt to different input densities, a ConvLSTM is used to memorize past feature and label information. When fusing label and voxel features, a background convolution ConvLSTM is used. bg Conv and foreground convolution fg The labels are processed, and the resulting features are added to the voxel features through summation to obtain the fused label-based features. For voxel V at time t-1... t-1 and tags Feature fusion is performed using the following formula:

[0021]

[0022] When t=1, For the given object label, It is initialized as a matrix of all zeros; when t≥2, The object labels generated from the previous prediction. This represents the hidden state for each ConvLSTM iteration.

[0023] Step 104: Combine the last layer of features from steps 102 and 103. Output of the recurrent attention module The concatenation is used as the input Z of the feature encoder. t Modeling target features in spatial and temporal dimensions Improve the model's ability to match queries against the target.

[0024] Feature encoders include, but are not limited to, CNN convolutional neural networks, fully connected neural networks, and Transformer. Preferably, the Transformer Encoder in the Transformer model is used as the feature encoder.

[0025] The Transformer Encoder's self-attention mechanism models the relationships between each pixel, combining the last layer features of the ConvLSTM in steps 102 and 103: label fusion features. Voxel characteristics and features of the recurrent attention module The data is concatenated to form the input Z. t The specific formula is as follows:

[0026]

[0027] in Z t ∈R 3HW×N×d H and W represent the feature resolution, and N is the number of target objects in the sequence. For Z... t The system employs a self-attention mechanism to establish information relationships between pixels in the temporal dimension and queries target object features in the spatial dimension. Utilizing a multi-head attention mechanism to extract feature information from multiple dimensions prevents overfitting. Finally, a feedforward neural network (FFN) is used to extract target features. The specific formula is as follows:

[0028]

[0029]

[0030] d is the dimension of the hidden layer, M is the number of multi-head attention mechanisms, and d k =d / M, This is a dimension mapping matrix that maps features back to dimension d. The feature vector for each multi-head attention mechanism is determined by formula (7):

[0031]

[0032] Learnable parameters for each multi-head attention module.

[0033] Step 105: Use the output of step 104 The input V at time t is represented by V. t Features As input to the recurrent attention module, obtain partial input for the next step 104. Simultaneously update the hidden status. Improve the ability of the target region segmentation model to detect target motion.

[0034] Use a recurrent attention module to memorize the output features of the feature encoder and truncate them. The input representation V belongs to t Features As input, obtain the attention matrix. Output of the recurrent attention module Update via residual join And obtain the input of the feature encoder for the next iteration.

[0035]

[0036]

[0037]

[0038]

[0039] When t=1, It is initialized as a matrix of all zeros; when t≥1, Let σ be the hidden state of the recurrent attention module in the previous iteration, and σ be the elu() function. and These are the learnable parameters of the multi-head attention module for the input branch and the hidden state branch, respectively. This is the combined output of the two branches. (Output of the recurrent attention module) Update via residual join It also obtains a portion of the input from the feature encoder for the next iteration. LN stands for LayerNorm, and its specific implementation formula is as follows:

[0040]

[0041]

[0042] Step 106: Use and a trainable query vector Q emb As input to the feature decoder, obtain the target query features. Use and Calculate the relation matrix AM t AM t and The concatenated data serves as the input S for the segmentation head. t This improves the model's robustness to feature matching.

[0043] Feature decoders include, but are not limited to: CNN convolutional neural networks, fully connected neural networks, and Transformer. Preferably, the Transformer Decoder in Transformer is used as the feature decoder.

[0044] Using Transformer Decoder and query vector Q emb ∈R d Query the characteristics of the target object, and assign Q based on the number of objects N contained in the sequence. emb Copy N times to obtain vector Q dec ∈RN×d Q dec and As input to the TransformerDecoder, query features are obtained through cross-attention operations. The calculation formulas are the same as those in formulas (5), (6), and (7).

[0045] at this time The data contains rich semantic information, but it also includes voxel data from time t-1 and feature information from the recurrent attention module. Therefore, a relation extraction operation is still needed to obtain the target object information at time t. The target attention module is used to compute... and The relationships between pixels are analyzed using a multi-head attention mechanism to extract information from different dimensions, resulting in an attention relationship matrix (AM). i,t Where i is the index of the multi-head attention module, and the specific formula is:

[0046]

[0047] Attention Matrix AM i,t It includes The target object information is obtained from the two, so they are concatenated together and denoted as S. t ∈R HW×N×(d+M) This can enhance the target information contained in the features. The specific formula is as follows:

[0048]

[0049] Step 107: Intermediate features of Step 102 By using skip connections to input into the segmentation head, and upsampling operations to improve feature resolution, the model's ability to fuse features from both coarse and fine-grained information is enhanced, thereby improving the accuracy of predicting the target region.

[0050] S is converted through convolution operation t Mapped to Then, upsampling and convolution operations are used to adjust the feature size to 1 / 4 of the original image. The specific formula is as follows:

[0051]

[0052] By linear interpolation Enlarged to the original image size, denoted as O t , for O t Calculate the softmax value to obtain the category corresponding to each pixel, and obtain V at time t. t The predicted That is, the segmented region corresponding to the target.

[0053] Step 108: Repeat steps 101-107, using the predicted values. As a given label for the next iteration, features are fused through a recurrent neural network to improve the feature extraction capability for continuous time pulse inputs; by modeling spatiotemporal relationships through a recurrent feature encoder and decoder, the query capability and robustness of feature matching are improved, thereby improving the accuracy of predicting the target region and achieving the goal of segmenting the target region at any time in low-light and high-speed scenes.

[0054] This invention discloses a target region segmentation system for spatiotemporal event pulse streams, implemented based on a target region segmentation method for spatiotemporal event pulse streams. The target region segmentation system for spatiotemporal event pulse streams includes a spatiotemporal event stream acquisition module, a feature extraction module, a label fusion feature extraction module, a target feature matching module, and a target region generation module.

[0055] The spatiotemporal event stream acquisition module acquires a spatiotemporal event stream E through an event pulse camera, and converts the continuous spatiotemporal event pulse stream E into an input representation V for an adaptive neural network based on the event voxel conversion relationship. The module then converts the current voxel representation V... t The input is fed into the feature extraction module, which takes the voxel representation V from the previous time step. t-1 Input to the label fusion feature extraction module.

[0056] The feature extraction module is used to extract spatial and temporal information from continuous event data. It achieves long-term information memory through a recurrent neural network, updating the internal hidden state while outputting features containing the target information at the current moment. Improve the feature extraction capability for continuous-time pulse input. Input to the target feature matching module.

[0057] The label fusion feature extraction module is used to fuse a given target label with the target features from the previous time step, thereby obtaining the fused label information and target features. Improve the utilization of a given target label. Input to the target feature matching module.

[0058] The target feature matching module is used to match target features Features fused with labels To query the target location, the self-attention and cross-attention operations of the recurrent feature encoder-decoder are used to complete the target query between two features, obtaining the target relationship matrix S. t Improve the query capability and robustness of feature matching, S t Input into the target area generation module.

[0059] The target region generation module is used to generate the predicted target region at the current time, based on the input S. t The target relation matrix is ​​used to determine the target category of each pixel using the Softmax function, thus obtaining the predicted region at the current time step. then As the given label for the next label fusion feature module, it is input into the label fusion feature module for the next loop iteration, until the spatiotemporal event stream acquisition module stops outputting data representation V.

[0060] Beneficial effects:

[0061] 1. The present invention discloses a target region segmentation method and system based on spatiotemporal event pulse streams. Using spatiotemporal event pulse streams as input to the target region segmentation model, it can not only obtain accurate segmentation results in low-light and high-speed scenes, but also segment target regions at any time.

[0062] 2. The present invention discloses a target region segmentation method and system for spatiotemporal event pulse streams, which improves the feature extraction capability of the target region segmentation model for continuous event pulse input by extracting information and memorizing past features through a recurrent neural network;

[0063] 3. The present invention discloses a target region segmentation method and system for spatiotemporal event pulse streams, which matches targets and models spatiotemporal relationships through a cyclic feature encoder; and obtains the attention relationship matrix of target features at the current and past moments through a feature decoder, thereby improving the target region segmentation model's query matching ability and feature matching robustness.

[0064] 4. The present invention discloses a target region segmentation method and system for spatiotemporal event pulse streams. By using skip connections to input intermediate features into the segmentation head, the feature fusion capability of the target region segmentation model for coarse-grained and fine-grained information is improved, thereby improving the accuracy of predicting the target region. Attached Figure Description

[0065] Figure 1 This is a flowchart of a target region segmentation method and system for spatiotemporal event pulse streams disclosed in this invention.

[0066] Figure 2 This is a system framework diagram of a target region segmentation method and system for spatiotemporal event pulse streams disclosed in this invention.

[0067] Figure 3 This invention discloses a target region segmentation method and system for spatiotemporal event pulse streams, and the result diagram of the input of a real event pulse stream.

[0068] Figure 4This is a segmentation result diagram of a target region segmentation method and system for spatiotemporal event pulse streams disclosed in this invention in low-light and high-speed scenes.

[0069] Figure 5 This invention discloses a method and system for segmenting target regions of spatiotemporal event pulse streams, and the segmentation result diagram of an event pulse stream at any time. Detailed Implementation

[0070] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0071] Example 1:

[0072] Figure 1 This invention illustrates a target region segmentation framework provided by an embodiment of the invention, such as... Figure 1 As shown in the figure, this embodiment discloses a target region segmentation method for spatiotemporal event pulse streams, which includes the following steps:

[0073] Step 101: Acquire spatiotemporal event pulse stream E through an event pulse camera. In order to adapt to the input form of the neural network, take advantage of the fact that voxels can retain the information of events in the time dimension and can represent the spatial distribution of events. According to the event voxel conversion relationship, the continuous spatiotemporal event pulse stream E is converted into an input representation V that adapts to the neural network.

[0074] The continuous spatiotemporal event stream E is an N×4 matrix, where N is the number of events within the time interval. Each event is represented by 4-dimensional data (timestamp, x, y, polarity), where timestamp is the timestamp when the event occurs, 1≤x≤260, 1≤y≤346 are the pixel positions corresponding to the event, 260×346 is the spatial resolution of the event, and polarity∈{-1,1} is the polarity of the event. To adapt to the input format of the neural network, the continuous spatiotemporal event pulse stream E is converted into the input representation V adapted to the neural network according to the event voxel conversion relationship shown in formula (1).

[0075]

[0076] Where t0,t n t represents the start and end times of the entire event flow. i The time of each event's occurrence. B is the specified voxel dimension. In this embodiment, B is set to 5, resulting in voxels V∈R.5×260×346 ΔT is the time interval of the entire event stream, p i For the event polarity, x l ,y m These are spatial coordinates.

[0077] Step 102: Use a recurrent neural network to represent the input V at time t. t Perform feature extraction to obtain features for each layer. Simultaneously update the hidden state of the recurrent neural network. Improve the feature extraction capability of the target region segmentation model for continuous event pulse inputs.

[0078] Event data reflects the change in light intensity of pixels at a certain moment. However, for cases with small changes in light intensity and insignificant object motion, sufficient event signals will not be emitted, leading to sparse spatial distribution of events and insufficient information in some voxel data V. Based on the highly continuous nature of events in the temporal dimension, utilizing past event information can improve the feature extraction capability of the target region segmentation model. Therefore, a recurrent neural network (ConvLSTM) is used to extract voxel features, and a hidden state (state) is used to record past event information. Features are adaptively extracted from sparse or dense voxels, ensuring model robustness while enhancing the information memory capacity of the target region segmentation model and improving its segmentation accuracy. The specific formula for ConvLSTM voxel feature extraction is as follows:

[0079]

[0080] In this embodiment, a 3-layer ConvLSTM is used as the recurrent neural network for feature extraction, where i is the layer index of the ConvLSTM, and V... t Let be the voxel at time t. When t = 1, It is initialized as a matrix of all zeros; when t≥1, For each layer of ConvLSTM, the hidden state is from the previous iteration. To be hidden after the update. The intermediate features of each voxel are defined.

[0081] Step 103: Given the label of the target region to be detected, use the label fusion feature extraction recurrent neural network to extract the input representation V at time t-1. t-1 and Perform feature fusion to obtain label fusion features Simultaneously update the hidden status.

[0082] Since the acquisition of event data depends on changes in light intensity, the event density in each input voxel varies considerably. To adapt to different input densities, a ConvLSTM is used to memorize past feature and label information. When fusing label and voxel features, a background convolution ConvLSTM is used. bg Conv and foreground convolution fg The labels are processed, and the resulting features are added to the voxel features through summation to obtain the fused label-based features. For voxel V at time t-1... t-1 and tags Feature fusion is performed using the following formula:

[0083]

[0084] In this embodiment, a 3-layer ConvLSTM is used as a recurrent neural network for label fusion feature extraction, where when t=1, For a given target label, It is initialized as a matrix of all zeros; when t≥2, The target label generated from the previous prediction. This represents the hidden state of each ConvLSTM layer in the previous iteration.

[0085] Step 104: Combine the last layer of features from steps 102 and 103. Output of the recurrent attention module The concatenation is used as the input Z of the feature encoder. t Modeling target features in spatial and temporal dimensions Improve the model's ability to match queries against the target.

[0086] The Transformer Encoder's self-attention mechanism models the relationships between each pixel, combining the last layer features of the ConvLSTM in steps 102 and 103: label fusion features. Voxel characteristics and features of the recurrent attention module The data is concatenated to form the input Z. t The specific formula is as follows:

[0087]

[0088] in Z t ∈R 1122×N×256 N is the number of target objects in the sequence. For Z... tThe system employs a self-attention mechanism to establish information relationships between pixels in the temporal dimension and queries target object features in the spatial dimension. Utilizing a multi-head attention mechanism to extract feature information from multiple dimensions prevents overfitting. Finally, a feedforward neural network (FFN) is used to extract target features. The specific formula is as follows:

[0089]

[0090]

[0091] This embodiment uses a 6-layer Transformer Encoder as the feature encoder, where each layer contains 8 multi-head attention modules. k =32 is the vector dimension of each multi-head attention module. For dimension mapping matrix, The feature vector for each multi-head attention mechanism is determined by formula (7):

[0092]

[0093] Learnable parameters for each multi-head attention module.

[0094] Step 105: Use the output of step 104 The input V at time t is represented by V. t Features As input to the recurrent attention module, obtain partial input for the next step 104. Simultaneously update the hidden status. Improve the ability of the target region segmentation model to detect target motion.

[0095] Use a recurrent attention module to memorize the output features of the feature encoder and truncate them. The input representation V belongs to t Features As input, obtain the attention matrix. Output of the recurrent attention module Update via residual join And obtain the input of the feature encoder for the next iteration.

[0096]

[0097]

[0098]

[0099]

[0100] When t=1, It is initialized as a matrix of all zeros; when t≥1, Let σ be the hidden state of the recurrent attention module in the previous iteration, and σ be the elu() function. and These are the learnable parameters of the multi-head attention module for the input branch and the hidden state branch, respectively. This is the combined output of the two branches. (Output of the recurrent attention module) Update via residual join And obtain part of the input of the feature encoder for the next iteration. LN stands for LayerNorm operation, and the specific implementation formula is as follows:

[0101]

[0102]

[0103] Step 106: Use and a trainable query vector Q emb As input to the feature decoder, obtain the target query features. Use and Calculate the relation matrix AM t AM t and The concatenated data serves as the input S for the segmentation head. t This improves the model's robustness to feature matching.

[0104] Using a 6-layer Transformer Decoder and query vector Q emb ∈R 256 Query the characteristics of the target object, and assign Q based on the number of objects N contained in the sequence. emb Copy N times to obtain vector Q dec ∈R N×256 Q dec and As input to the TransformerDecoder, query features are obtained through cross-attention operations. The calculation formulas are the same as those in formulas (5), (6), and (7).

[0105] at this time The data contains rich semantic information, but it also includes voxel data from time t-1 and feature information from the recurrent attention module. Therefore, a relation extraction operation is still needed to obtain the target object information at time t. The target attention module is used to compute... and The relationships between pixels were analyzed using eight multi-head attention modules to extract information from different dimensions, resulting in an attention relationship matrix (AM). i,t ∈R 374×8×256 The specific formula is as follows:

[0106]

[0107] in for The feature vector, divided into 8 parts, is used as the input to the multi-head attention module. At this point, the attention matrix AM... i,t It includes The target object information is obtained from the two, so they are concatenated together and denoted as S. t ∈R 374×N×264 This can enhance the target information contained in the features. The specific formula is as follows:

[0108]

[0109] Step 107: Intermediate features of Step 102 By using skip connections to input into the segmentation head, and upsampling operations to improve feature resolution, the model's ability to fuse features from both coarse and fine-grained information is enhanced, thereby improving the accuracy of predicting the target region.

[0110] S is converted through convolution operation t Mapped to The feature size is then adjusted to 65×87 using two more upsampling and convolution operations, as shown in the following formula:

[0111]

[0112] By linear interpolation Enlarged to the original image size, denoted as O t ∈R N×260×346 , for O t Calculate the softmax value to obtain the category corresponding to each pixel, and obtain V at time t. t The predicted That is, the segmented region corresponding to the target.

[0113] Step 108: Repeat steps 101-107, using the predicted values. As a given label for the next iteration, features are fused through a recurrent neural network to improve the feature extraction capability for continuous time pulse inputs; by modeling spatiotemporal relationships through a recurrent feature encoder and decoder, the query capability and robustness of feature matching are improved, thereby improving the accuracy of predicting the target region and achieving the goal of segmenting the target region at any time in low-light and high-speed scenes.

[0114] This embodiment discloses a target region segmentation system for spatiotemporal event pulse streams, which is implemented based on the target region segmentation method for spatiotemporal event pulse streams. Figure 2 The system modules of the embodiment are shown, such as Figure 2 As shown, the target region segmentation system for spatiotemporal event pulse streams includes the following modules:

[0115] The spatiotemporal event stream acquisition module acquires a spatiotemporal event stream E through an event pulse camera, and converts the continuous spatiotemporal event pulse stream E into an input representation V for an adaptive neural network based on the event voxel conversion relationship. The module then converts the current voxel representation V... t The input is fed into the feature extraction module, which takes the voxel representation V from the previous time step. t-1 Input to the label fusion feature extraction module.

[0116] The feature extraction module is used to extract spatial and temporal information from continuous event data. It achieves long-term information memory through a recurrent neural network, updating the internal hidden state while outputting features containing the target information at the current moment. Improve the feature extraction capability for continuous-time pulse input, and then Input to the target feature matching module.

[0117] The label fusion feature extraction module is used to fuse a given target label with the target features from the previous time step, thereby obtaining the fused label information and target features. Improve the utilization of a given target label, and then Input to the target feature matching module.

[0118] The target feature matching module is used to match target features Features fused with labels To query the target location, the self-attention and cross-attention operations of the recurrent feature encoder-decoder are used to complete the target query between two features, obtaining the target relationship matrix S. t Improve the query capability and robustness of feature matching, S t Input into the target area generation module.

[0119] The target region generation module is used to generate the predicted target region at the current time, based on the input S. t The target relation matrix is ​​used to determine the target category of each pixel using the Softmax function, thus obtaining the predicted region at the current time step. then As the given label for the next label fusion feature module, it is input into the label fusion feature module for the next loop iteration, until the spatiotemporal event stream acquisition module stops outputting data representation V.

[0120] This embodiment will illustrate the effectiveness of the present invention through the results of target area prediction.

[0121] 1. Experimental conditions

[0122] The hardware testing conditions for this experiment were: Intel Xeon Gold 6240C, 32GB RAM, NVIDIA GeForce RTX 3090 GPU with 24GB VRAM and CUDA version 11.1; and a DAVIS 346 event camera. To ensure fairness in the comparative experiments, in addition to the spatiotemporal event pulse stream data input, E2Vid was used to reconstruct grayscale images of continuous event pulses to adapt the image modality input for the comparison methods. All comparison methods used the original training hyperparameters and were retrained on the EOS event segmentation dataset for the same number of iterations.

[0123] 2. Experimental Results

[0124] To quantitatively measure the quality of target segmentation results, region similarity (J) and contour accuracy (F) are used to measure the similarity of predicted target regions and the accuracy of contours. At the same time, based on the characteristic that event pulse streams reflect changes in light intensity, the scene is divided into two scenarios: camera motion and target object motion.

[0125] Table 1 shows the target region segmentation index results of the method using input image modal data and voxel modal data on the EOS event segmentation dataset after training. As can be seen from the table, the method of the present invention has a significant improvement over the comparison method, and can achieve good segmentation accuracy regardless of camera movement or target movement. Figure 3 The diagram illustrates the results of this method on an event stream input. Regardless of whether it is a single-target or multi-target scenario, the method of this invention can accurately segment the target region.

[0126] Table 1. Comparison of Segmentation Index Results of Different Methods

[0127]

[0128] Figure 4 The diagram shows the segmentation results of the method of this invention in low-light and high-speed scenes. As can be seen from the diagram, the target segmentation method of this invention can effectively detect target regions in discrete and sparse event streams; it also adapts well to low-light and high-speed environments, greatly expanding the application scenarios of video object segmentation methods and possessing high application value. Furthermore, the event stream is continuous in the time dimension, allowing for the extraction of data within any time interval. Figure 5 The results of segmenting events at any given time are shown.

[0129] In summary, the target region segmentation method and system proposed in this embodiment for spatiotemporal event pulse streams significantly improves the utilization capability of continuous spatiotemporal pulse streams by using data representation transformation, recurrent neural networks to memorize past information, and recurrent feature codecs to query target features. This has led to its successful application in the field of video target segmentation. The target region segmentation method and system for spatiotemporal event pulse streams proposed in this embodiment not only broadens the application scenarios of video target segmentation but also has significant application value in low-light, high-speed scenes, human-computer interaction, target tracking, and autonomous driving.

[0130] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A target region segmentation method for spatiotemporal event pulse streams, characterized in that: Includes the following steps, Step 101: Acquire a continuous spatiotemporal event pulse stream E using an event pulse camera. In order to adapt to the input form of the neural network, take advantage of the fact that voxels can retain the information of events in the time dimension and can represent the spatial distribution of events. According to the event voxel conversion relationship, the continuous spatiotemporal event pulse stream E is converted into an input representation V that adapts to the neural network. Step 102: Use a recurrent neural network to represent the input V at time t. t Perform feature extraction to obtain features for each layer. Simultaneously update the hidden state of the recurrent neural network. Improve the feature extraction capability of the target region segmentation model for continuous event pulse inputs; Step 103: Given the label of the target region to be detected, use the label fusion feature extraction recurrent neural network to extract the input representation V at time t-1. t-1 and Perform feature fusion to obtain label fusion features Simultaneously update the hidden status. Step 104: Combine the last layer of features from steps 102 and 103. Output of the recurrent attention module The concatenation is used as the input Z of the feature encoder. t Modeling target features in spatial and temporal dimensions Improve the model's ability to match queries against the target; Step 105: Use the output of step 104 The input V at time t is represented by... t Features As input to the recurrent attention module, obtain partial input for the next step 104. Simultaneously update the hidden status. Improve the ability of target region segmentation models to detect target motion; Step 106: Use And a trainable query vector Q emb As input to the feature decoder, obtain the target query features. Use and Calculate the relation matrix AM t AM t and The concatenated data serves as the input S for the segmentation head. t This improves the model's robustness to feature matching; Step 107: Intermediate features of Step 102 By using skip connections to input into the segmentation head, and upsampling operations to improve feature resolution, the model's ability to fuse features from both coarse and fine-grained information is enhanced, thereby improving the accuracy of predicting the target region. Step 108: Repeat steps 101-107, using the predicted values. As a given label for the next iteration, features are fused through a recurrent neural network to improve the feature extraction capability for continuous time pulse inputs; by modeling spatiotemporal relationships through a recurrent feature encoder and decoder, the query capability and robustness of feature matching are improved, thereby improving the accuracy of predicting the target region and achieving the goal of segmenting the target region at any time in low-light and high-speed scenes.

2. The target region segmentation method for spatiotemporal event pulse streams as described in claim 1, characterized in that: The continuous spatiotemporal event pulse stream E mentioned in step 101 includes, but is not limited to: event data generated by the event camera and pulse data generated by the pulse camera; The continuous spatiotemporal event pulse stream E is an N×4 matrix, where N is the number of events, and each event is represented by 4-dimensional data (timestamp, x, y, polarity), where timestamp is the timestamp when the event occurs, 1≤x≤W, 1≤y≤H are the pixel positions corresponding to the event, H×W is the spatial resolution of the event, and polarity∈{-1,1} is the polarity of the event; in order to adapt to the input form of the neural network, the continuous spatiotemporal event pulse stream E is converted into the input representation V of the neural network according to the event voxel conversion relationship shown in formula (1); Where t0,t n t represents the start and end times of the entire event flow. i The time of each event's emission, B is the specified voxel dimension, ΔT is the time interval of the entire event stream, and p i For the event polarity, x l ,y m These are spatial coordinates.

3. The target region segmentation method for spatiotemporal event pulse streams as described in claim 2, characterized in that: The label of the target region to be detected in step 102, the label acquisition method includes but is not limited to: manually specifying the target region and category, and using a salient region detection model to obtain the target label; the feature extraction recurrent neural network includes but is not limited to: ConvLSTM, ConvGRU, SNN, and the recurrent neural network uses ConvLSTM to extract voxel features; Event data reflects the change in light intensity of a pixel at a certain moment. However, for cases where the change in light intensity is small and the motion of the object is not obvious, there will not be enough event signals, which will lead to the sparse spatial distribution of events and insufficient information in some voxel data V. Based on the highly continuous nature of events over time, utilizing past event information can improve the feature extraction capability of target region segmentation models. Therefore, a recurrent neural network (ConvLSTM) is used to extract voxel features, and a hidden state is used to record past event information. Features are adaptively extracted from sparse or dense voxels, which enhances the information memory capability of the target region segmentation model while ensuring model robustness, thereby improving the segmentation accuracy. The specific formula for ConvLSTM to extract voxel features is as follows: Where i is the layer index of ConvLSTM, V t Let be the voxel at time t. When t = 1, It is initialized as a matrix of all zeros; when t≥1, For each layer of ConvLSTM, the hidden state is from the previous iteration. To be hidden after the update. The intermediate features of each voxel are defined.

4. The target region segmentation method for spatiotemporal event pulse streams as described in claim 3, characterized in that: Step 103 provides the recurrent neural network for label fusion feature extraction, which includes, but is not limited to, ConvLSTM, ConvGRU, and SNN. The recurrent neural network for label fusion feature extraction uses ConvLSTM to fuse features. Since the acquisition of event data depends on changes in light intensity, the event density in each input voxel varies considerably. To adapt to different input densities, a ConvLSTM is used to memorize past feature and label information. When fusing label and voxel features, a background convolution ConvLSTM is used. bg Conv and foreground convolution fg The labels are processed, and the resulting features are added to the voxel features in a summation manner to obtain the fused label features. Vocillary values ​​at time t-1 t-1 and tags Feature fusion is performed using the following formula: When t=1, For the given object label, It is initialized as a matrix of all zeros; when t≥2, The object labels generated from the previous prediction. This represents the hidden state for each ConvLSTM iteration.

5. The target region segmentation method for spatiotemporal event pulse streams as described in claim 4, characterized in that: The feature encoder mentioned in step 104 includes, but is not limited to: CNN convolutional neural network, fully connected neural network, and Transformer, with the Transformer Encoder in the Transformer model used as the feature encoder; The Transformer Encoder's self-attention mechanism models the relationships between each pixel, combining the last layer features of the ConvLSTM in steps 102 and 103: label fusion features. voxel characteristics and features of the recurrent attention module The input Z is constructed by concatenating the two parts. t The specific formula is as follows: in H and W are the resolutions of the features, and N is the number of target objects in the sequence; For Z t The system employs a self-attention mechanism to establish information relationships between pixels in the temporal dimension and queries target object features in the spatial dimension. It then utilizes a multi-head attention mechanism to extract feature information from multiple dimensions, preventing overfitting. Finally, a feedforward neural network (FFN) is used to extract target features. The specific formula is as follows: d represents the dimension of the hidden layer, and M represents the number of multi-head attention mechanisms. This is a dimension mapping matrix that maps features back to dimension d. The feature vector for each multi-head attention mechanism is determined by formula (7): Learnable parameters for each multi-head attention module.

6. The target region segmentation method for spatiotemporal event pulse streams as described in claim 5, characterized in that: Step 105 uses the recurrent attention module to memorize the output features of the feature encoder and extracts them. The input representation V belongs to t Features As input, obtain the attention matrix. Output of the recurrent attention module Update stat via residual join And obtain the input of the feature encoder for the next iteration. When t=1, It is initialized as a matrix of all zeros; when t≥1, Let σ be the hidden state of the recurrent attention module in the previous iteration, and σ be the elu() function. and These are the learnable parameters of the multi-head attention module for the input branch and the hidden state branch, respectively. The combined output of the two branches; the output of the recurrent attention module. Update via residual join It also obtains a portion of the input from the feature encoder for the next iteration. LN stands for LayerNorm, and its specific implementation formula is as follows:

7. The target region segmentation method for spatiotemporal event pulse streams as described in claim 6, characterized in that: The feature decoder mentioned in step 106 includes, but is not limited to: CNN convolutional neural network, fully connected neural network, Transformer, and using Transformer Decoder in Transformer as the feature decoder; Using Transformer Decoder and query vector Q emb ∈R d Query the characteristics of the target object, and assign Q based on the number of objects N contained in the sequence. emb Copy N times to obtain vector Q dec ∈R N×d Q dec and As input to the Transformer Decoder, query features are obtained through cross-attention operations. The calculation formulas are the same as those in formulas (5), (6), and (7); at this time It contains rich semantic information, but it is mixed with the feature information of the voxel at time t-1 and the recurrent attention module. Therefore, a relation extraction operation is still needed to obtain the target object information at time t. Calculated using the target attention module and The relationships between pixels are analyzed using a multi-head attention mechanism to extract information from different dimensions, resulting in an attention relationship matrix (AM). i,t Where i is the index of the multi-head attention module, and the specific formula is: Attention Matrix AM i,t It includes The target object information is obtained from the two, so they are concatenated together and denoted as S. t ∈R HW ×N×(d+M) This can enhance the target information contained in the features. The specific formula is as follows:

8. The target region segmentation method for spatiotemporal event pulse streams as described in claim 7, characterized in that: In step 107, S is converted through a convolution operation. t Mapped to Then, upsampling and convolution operations are used to adjust the feature size to 1 / 4 of the original image. The specific formula is as follows: By linear interpolation Enlarged to the original image size, denoted as O t , for O t Calculate the softmax value to obtain the category corresponding to each pixel, and obtain V at time t. t The predicted That is, the segmented region corresponding to the target.

9. The target region segmentation method for spatiotemporal event pulse streams as described in claim 8, characterized in that: In step 108, the target segmentation region at any time of the continuous spatiotemporal event pulse stream is obtained through iterative looping.

10. A target region segmentation system for spatiotemporal event pulse streams, implemented based on the target region segmentation method for spatiotemporal event pulse streams as described in any one of claims 1 to 9, characterized in that: It includes a spatiotemporal event stream acquisition module, a feature extraction module, a label fusion feature extraction module, a target feature matching module, and a target region generation module; The spatiotemporal event stream acquisition module acquires a spatiotemporal event stream E through an event pulse camera, and converts the continuous spatiotemporal event pulse stream E into an input representation V for an adaptive neural network based on the event voxel conversion relationship, and converts the voxel representation V at the current moment into... t The input is fed into the feature extraction module, which takes the voxel representation V from the previous time step. t-1 Input to the label fusion feature extraction module; The feature extraction module is used to extract spatial and temporal information from continuous event data. It achieves long-term information memory through a recurrent neural network, updating the internal hidden state while outputting features containing the target information at the current moment. Improve the feature extraction capability for continuous-time pulse input. Input to the target feature matching module; The label fusion feature extraction module is used to fuse a given target label with the target features from the previous time step, thereby obtaining the fused label information and target features. Improve the utilization of a given target label. Input to the target feature matching module; The target feature matching module is used to match target features Features fused with labels To query the target location, the self-attention and cross-attention operations of the recurrent feature encoder-decoder are used to complete the target query between two features, obtaining the target relationship matrix S. t Improve the query capability and robustness of feature matching, S t Input into the target region generation module; The target region generation module is used to generate the predicted target region at the current time, based on the input S. t The target relation matrix is ​​used to determine the target category of each pixel using the Softmax function, thus obtaining the predicted region at the current time step. then As the given label for the next label fusion feature module, it is input into the label fusion feature module for the next loop iteration, until the spatiotemporal event stream acquisition module stops outputting data representation V.

Images