An event semantic segmentation method and an event semantic segmentation network

Abstract

The application provides an event semantic segmentation method and an event semantic segmentation network, and relates to computer vision technology. The event semantic segmentation method provided by the embodiment of the application comprises the following steps: training a teacher network by using a first data set; training a first student network and a second student network by using a second data set and the output of the teacher network, the second data set having multiple pairs of gray image-event data, the input of the first student network being event data, and the input of the second student network being pseudo image data obtained by converting the event data; and fusing the outputs of the first student network and the second student network to generate a semantic segmentation result. The input of the first student network is different from the input of the second student network, so that the first student network and the second student network learn knowledge in different ways, the knowledge of the first student network and the second student network is diverse and complementary, and the semantic segmentation result obtained by fusing the first student network and the second student network is more effective.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to an event semantic segmentation method and an event semantic segmentation network. Background Technology

[0002] Semantic segmentation is a fundamental task in computer vision and plays an indispensable role in the field of autonomous driving. Currently, many representative methods have achieved breakthroughs in high accuracy with clear daytime images. However, traditional cameras, due to their small dynamic range, cannot output clear images in low-light scenes. In contrast, event cameras are a novel type of sensor that can asynchronously detect changes in brightness per pixel and output a series of events, thus offering several advantages over traditional cameras: high dynamic range (140dB vs 60dB) and high temporal resolution (μs level). Utilizing this high dynamic range property, event data can achieve effective semantic segmentation in low-light scenes.

[0003] Currently, there are two main methods for semantic segmentation using event data: direct training based on pseudo-labels and cross-modal knowledge distillation based on teacher-student networks. The direct training method based on pseudo-labels primarily uses pseudo-labels provided in the dataset as supervision signals and trains the event semantic segmentation network using the cross-entropy loss function.

[0004] Compared to direct training methods based on pseudo-labels, the cross-modal knowledge distillation method based on teacher-student networks provides soft labels from the teacher network, and the intermediate features of its output also play a significant role, thus providing a more powerful supervisory signal. If the input consists of paired event grayscale image data, the output of the event-based student network should be consistent with the output of the teacher network. By applying loss constraints to the outputs, an event semantic segmentation network can be trained.

[0005] The inventors' research found that the effectiveness of semantic segmentation in current event semantic segmentation methods that employ cross-modal knowledge distillation based on teacher-student networks still needs further improvement. Summary of the Invention

[0006] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0007] The present invention aims to provide an event semantic segmentation method that can improve the effectiveness of semantic segmentation.

[0008] The present invention also aims to provide an event semantic segmentation network that can improve the effectiveness of semantic segmentation.

[0009] The event semantic segmentation method and event semantic segmentation network provided in the embodiments of the present invention can be implemented in the following ways:

[0010] An event semantic segmentation method, comprising:

[0011] The teacher network is trained using a first dataset; wherein the first dataset contains multiple image-label pairs;

[0012] The first and second student networks are trained using a second dataset and supervised by the output of the teacher network; wherein the second dataset has multiple pairs of grayscale image-event data; the input of the teacher network is the grayscale image, the input of the first student network is the event data, and the input of the second student network is pseudo-image data converted from the event data;

[0013] The outputs of the first student network and the second student network are combined to generate a semantic segmentation result.

[0014] Optionally, the method further includes a step of converting the event data into pseudo-image data, and this step includes:

[0015] Construct a mode transition network;

[0016] The event data is converted into pseudo-image data using the modality conversion network.

[0017] The mode conversion network is trained synchronously with the second student network.

[0018] Optionally, the transformation loss function of the mode conversion network is:

[0019]

[0020]

[0021] Among them, L EFIT λ is the conversion loss of the mode conversion network. pw The hyperparameter for balancing the loss term; x i This is grayscale image data; x e For event data; D is the Euclidean distance function used to calculate the distance between two features; L mckd-fi The loss from the teacher network to the second student network.

[0022] Optionally, in the step of supervising the training of the first student network using the output of the teacher network, the loss function from the teacher network to the first student network is:

[0023]

[0024] Where n is the number of layers; λ j To balance the hyperparameters of the loss terms in different layers; D is the Euclidean distance function used to calculate the distance between two features; The features output by the j-th layer teacher network; This is the output of the first student network at layer j;

[0025] The output of the teacher network includes

[0026] Optionally, in the step of supervising the training of the second student network using the output of the teacher network, the loss function from the teacher network to the second student network is:

[0027]

[0028] Where n is the number of layers; λ j To balance the hyperparameters of the loss terms in different layers; D is the Euclidean distance function used to calculate the distance between two features; The features output by the j-th layer teacher network; This is the output of the second student network at layer j;

[0029] The output of the teacher network includes

[0030] Optionally, the step of fusing the outputs of the first student network and the second student network to generate a semantic segmentation result includes:

[0031] The first student network and the second student network are each divided into n layers, forming n first student network layers and n second student network layers;

[0032] The output of the first student network layer and the corresponding output of the second student network layer are reweighted by a fusion module, and a weighted feature is output. The number of fusion modules is n, and the weighted feature is used as the input to the next first student network layer and the next second student network layer.

[0033] The average value of the weighted features output by the fusion module that combines the final first student network layer and the second student network layer is input into the fully connected layer.

[0034] The fully connected layer generates semantic segmentation results.

[0035] Optionally, the step of reweighting the output of the first student network layer and the corresponding output of the second student network layer includes:

[0036] Attention values ​​are obtained based on the outputs of the first student network layer and the corresponding outputs of the second student network layer; wherein, the attention values ​​include event feature attention values ​​e along the height direction. h Event feature attention value e along the width direction w Attention value i of pseudo-image features along the height direction h And the pseudo-image feature attention value i along the width direction w ;

[0037] The outputs of the first student network layer and the corresponding outputs of the second student network layer are reweighted according to a weighting formula, which is:

[0038] f′ e =f e +f i ×i h ×i w

[0039] f′ i =f i +f e ×e h ×e w

[0040] Where, f′ e The weighted event characteristics; f e The event features output by the first student network layer; f′ i The weighted pseudo-image features; f i The event characteristics output by the second student network layer.

[0041] Optionally, the step of obtaining attention values ​​along the width and height directions based on the output of the first student network layer and the corresponding output of the second student network layer includes:

[0042] Perform a concatenation operation along the channel dimension to obtain the concatenated feature f, f = [f e f i ];

[0043] The feature f is encoded using average pooling kernels in both the width and height directions to obtain the feature f after pooling along the width direction. w (c, w) and the feature f after pooling along the height direction h (c, h):

[0044]

[0045]

[0046] For the feature f w(c, w) and the feature f h Perform the first convolution transformation operation on (c, h) to obtain the transformed feature z:

[0047] z = ReLU(BN(Conv([f h f w ])))

[0048] Segment feature z into height feature z h and width feature z w Then, a second convolution operation is performed to obtain the event feature attention value e along the height direction. h Attention value e of event features along the width direction w Attention value i of pseudo-image features along the height direction h And the pseudo-image feature attention value i along the width direction w :

[0049]

[0050]

[0051]

[0052]

[0053] Where c is the channel direction index, h is the height direction index, w is the width direction index, H is the length of feature f in the height direction, and W is the length of feature f in the width direction; Conv, and The functions are: 1×1 convolution function; Tanh is the activation function; BN is the batch normalization function; ReLU is the first activation function; z∈R 2C / r×1×(H+W) r is the reduction ratio controlling the size of the intermediate characteristic channel; R is the real number field; C is f h and f w The number of channels.

[0054] Optionally, the backbone network of the teacher network is ResNet, and the 7x7 convolutional kernel in the first layer of the ResNet is replaced with three 3x3 convolutional kernels.

[0055] An event semantic segmentation network includes: a first student network, wherein the input of the first student network is event data;

[0056] The second student network takes pseudo-image data as its input; the pseudo-image data is obtained by transforming the event data.

[0057] A fusion network is used to fuse the outputs of the first student module and the second student module to generate a semantic segmentation result;

[0058] The first student network and the second student network are obtained through supervised training using the output of the teacher network. The teacher network is trained using grayscale images, which correspond to the event data.

[0059] Optionally, the first student network is divided into multiple first student network layers, and the second student network is divided into multiple second student network layers, with a first student network layer and a second student network layer forming a student network layer pair;

[0060] The fusion network includes a fully connected layer and multiple fusion modules, which are configured one-to-one with the student network layer pairs. The fusion modules are used to reweight the features output by the student network layer pairs to obtain weighted features, and the weighted features are used as the input of the next student network layer pair.

[0061] The fully connected layer is used to generate semantic segmentation results based on the output of the last fusion module.

[0062] The beneficial effects of the event semantic segmentation method and event semantic segmentation network provided by the embodiments of the present invention include:

[0063] Embodiments of the present invention provide an event semantic segmentation method, which includes training a teacher network using a first dataset containing multiple image-label pairs; training a first student network and a second student network using a second dataset and supervised training with the output of the teacher network; the second dataset containing multiple grayscale image-event data pairs, where the input to the teacher network is grayscale images, the input to the first student network is event data, and the input to the second student network is pseudo-image data converted from event data; and fusing the outputs of the first and second student networks to generate a semantic segmentation result. Because the inputs to the first and second student networks are different, they learn knowledge in different ways, making the knowledge of the first and second student networks diverse and complementary. Therefore, the semantic segmentation result obtained by fusing the first and second student networks is more effective.

[0064] Embodiments of the present invention also provide an event semantic segmentation network, comprising a first student network, a second student network, and a fusion network. Both the first and second student networks are obtained through supervised training using the output of a teacher network. The input to the first student network is event data, and the input to the second student network is pseudo-image data, which is obtained through event data transformation. The fusion network is used to fuse the outputs of the first and second student modules to generate a semantic segmentation result. Because the inputs to the first and second student networks are different, they learn knowledge in different ways, resulting in diverse and complementary knowledge between the first and second student networks. Therefore, the semantic segmentation result generated by this event semantic segmentation network is more effective. Attached Figure Description

[0065] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0066] Figure 1 A flowchart of an event semantic segmentation method according to one aspect of the present invention is shown;

[0067] Figure 2 A flowchart illustrating the steps of an event semantic segmentation method according to one aspect of the present invention is shown.

[0068] Figure 3 A framework diagram of an event semantic segmentation network according to one aspect of the present invention is shown;

[0069] Figure 4 A training framework diagram of a student network according to one aspect of the present invention is shown;

[0070] Figure 5 A schematic diagram of the structure of a fusion module according to one aspect of the present invention is shown;

[0071] Figure 6 A schematic diagram of event data used in a first example provided according to an aspect of the present invention is shown;

[0072] Figure 7 It shows that according to Figure 6 The semantic segmentation result diagram generated by semantic segmentation of the event data shown.

[0073] Figure 8 A schematic diagram of event data used in a second example provided according to one aspect of the present invention is shown;

[0074] Figure 9 It shows that according to Figure 8The semantic segmentation result diagram generated by semantic segmentation of the event data shown.

[0075] Figure 10 A schematic diagram showing the comparative results of a comparative experiment provided according to one aspect of the present invention is shown. Detailed Implementation

[0076] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0077] In the description of this invention, it should be noted that the terms "first" and "second" are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0078] The inventors discovered that current cross-modal knowledge distillation methods neglect the influence of different input modes between teacher and student networks. Specifically, while directly aligning their intermediate features and outputs can train the student network, it can only transfer limited knowledge because the feature encoding differs when using different modalities as input. Therefore, only a portion of the knowledge in the teacher network is useful in another form of training. This limited knowledge cannot train a high-performance student network. Therefore, to improve upon these issues, this invention provides an event semantic segmentation method and network that enhances semantic segmentation performance by introducing more useful cross-modal knowledge. Furthermore, embodiments of this invention provide semantic segmentation for fundamental computer vision tasks, which plays a crucial role in fields such as autonomous driving.

[0079] Please refer to the reference. Figures 1-3 Embodiments of the present invention provide an event semantic segmentation method and an event semantic segmentation network. It should be noted that... Figure 1 In the diagram, dashed lines represent the calculation process of the loss function, while solid lines represent the calculation process of the input image or event.

[0080] The event semantic segmentation method provided in the embodiments of the present invention includes the following steps:

[0081] S01: Train the teacher network using the first dataset.

[0082] In this embodiment, 2975 image-label pairs from the Cityscapes dataset are selected to form the first dataset for training the teacher network. Simultaneously, for the color images in the original data pairs, they are converted to grayscale images by averaging the RGB channels. It is understood that in other embodiments, other datasets containing hand-labeled data, such as the Mapillary Vistas dataset, can also be used to construct the dataset required for this invention.

[0083] A grayscale image-based teacher network is trained using the dataset constructed above. Since the dataset selected in this embodiment is the Cityscapes dataset, the trained teacher network can map grayscale images to probability values ​​of 19 semantic categories.

[0084] In this embodiment, DeepLabv3+ is selected as the teacher network, and ResNet (forming the residual module) is used as the backbone network. The 7x7 convolutional kernel in the first layer of ResNet is replaced with three 3x3 convolutional kernels, thereby improving the semantic segmentation results. Optionally, the number of convolutional layers in ResNet is 50, that is, the backbone network is ResNet-50. In other embodiments, other numbers of layers can also be used.

[0085] Loss function of the teacher network: The output of the teacher network is constrained using the cross-entropy loss function.

[0086] Data augmentation strategies for teacher networks include at least one of the following: random horizontal flipping, random scaling, Gaussian blur, color dithering, and random cropping.

[0087] S02: Using the second dataset, the first and second student networks are trained in a supervised manner using the output of the teacher network.

[0088] In this embodiment, the DDD17 basic dataset is selected as the second dataset, which has multiple pairs of grayscale image-event data, with corresponding data in each pair of grayscale image-event data.

[0089] During the training of the first and second student networks, the Multi-level Cross-modality Knowledge Distillation (MCKD) method was employed. The input to the first student network was event data, while the input to the second student network was pseudo-image data obtained by transforming event data from the second dataset. Accordingly, the first student network can also be called an event-based student network, and the second student network can also be called an image-based student network.

[0090] Since the pseudo-image data used as input to the second student network is obtained through event data transformation, the process also includes a step of converting the event data into pseudo-image data. Specifically, the event data is converted into pseudo-image data using a modality transformation network. Before performing this transformation, a step of constructing a modality transformation network (EFIT network, Event to fake image translation) is included. The modality transformation network and the second student network can be trained synchronously; therefore, the training process of the modality transformation network will be described later in conjunction with the training of the second student network.

[0091] Figure 3 A framework diagram for training the student network is shown. The student network includes a first student network and a second student network. Specifically, step S02 includes the following steps:

[0092] S21: First Student Network Training.

[0093] Given a pair of data {(x e x i )},use Represents all outputs of the teacher network T, where This represents the intermediate features of the teacher's network output. The segmentation features representing the teacher's network output, i.e. Where, x e This represents event data, which serves as the input to the first student network, x. i This represents grayscale image data, which serves as input to the teacher network.

[0094] Accordingly, the first student network S e The feature representation of the corresponding scale output of the teacher network T is as follows: in This represents the intermediate features of the first student's network output. This represents the final output of the first student's network, i.e.

[0095] Therefore, the loss of MCKD can be expressed as:

[0096]

[0097] Where n is the number of layers; λ j To balance the hyperparameters of the loss terms in different layers; D is the Euclidean distance function used to calculate the distance between two features; The features output by the j-th layer teacher network; This represents the output of the first student network at layer j. Optionally, the specific value of n can also be set to 1-3, that is, in a specific embodiment, the value of n can be set to 1, 2, or 3. In this embodiment, n = 3, and λ1 = 10, λ2 = 10, and λ3 = 1. Compared to training the first student network solely through the segmentation features output by the teacher network, the first student network can resemble the teacher network more closely with the supervision of the intermediate features output by the teacher network. It is understood that in other embodiments, the number of layers can also be specifically set according to requirements.

[0098] Specifically, features from different layers are extracted through the following modules: residual module, dilated spatial pyramid pooling module, and decoder module. In other words, both the teacher network and the first student network framework include residual module, dilated spatial pyramid pooling module, and decoder module.

[0099] S22: Second student network training.

[0100] To transfer more knowledge from teacher networks, this invention proposes a mode transition network G. e→i The event data x is converted through a modality conversion network. e (Also known as event modality) is converted into pseudo-image data x fi (Also known as pseudo-image modality), thereby training a second student network S with pseudo-images as input. fi .

[0101] Second Student Network S fi With mode transition network G e→i Training is conducted synchronously, meaning that during training, the second student network S... fi With mode transition network G e→i The training process involves synchronous iterative updates. Specifically, the second student network S... fi The training process and the first student network S e The training process is roughly the same. Similarly, with The features representing the corresponding scale outputs of the second student network and the teacher network T, and The final output of the second student network, i.e. The loss of MCKD can be expressed as:

[0102]

[0103] Where n is the number of layers; λ j To balance the hyperparameters of the loss terms in different layers; D is the Euclidean distance function used to calculate the distance between two features; The features output by the j-th layer teacher network; This is the output of the second student network at layer j. In this embodiment, n = 3, and λ1 = 10, λ2 = 10, and λ3 = 1.

[0104] Accordingly, the framework of the second student network includes a residual module, a hollow spatial pyramid pooling module, and a decoder module. Features of different layers in the second student network are also extracted by the residual module, the hollow spatial pyramid pooling module, and the decoder module, respectively.

[0105] To train the mode transfer network G e→i A simple pixel-by-pixel loss Defined as:

[0106]

[0107] In addition, MCKD is also used to train G in order to better extract knowledge. e→i MCKD is composed of multiple mean squared error loss functions, therefore the mode transfer network is trained using the mean squared error loss function. The final loss L... EFIT for:

[0108]

[0109] Where, λ pw It is a hyperparameter of the loss term, and in this embodiment, λ pw =100. Add L mckd-fi G e→i It can benefit S fi The distillation process, that is, using L mckd-fi Simultaneously train G e→i and S fi .

[0110] Although the pseudo-image data is generated from event data, and we already have an event-based student network, namely the first student network S e However, the introduction of pseudo-image data helps to supplement knowledge distillation. Specifically, with pseudo-image data as input, the second student network S fi It extracts features of scene grayscale intensity, rather than events reflecting scene edges. Therefore, the second student network S fi With the first student network S e In contrast, they learn knowledge in different ways (i.e., their inputs are different), the first student network S e Second Student Network S fi The knowledge will be diverse and complementary. Furthermore, because the generated pseudo-image data looks like real images, the second student network S... fi Knowledge can be transferred more easily because of the second student network S. fiThe feature extraction process is similar to that of the teacher network T. In general, to train the second student network S... fi We used a mode transition network to generate other modes, which laid the foundation for the subsequent first student network S. e Second Student Network S fi This laid the foundation for the fusion of outputs.

[0111] S03: Combine the outputs of the first student network and the second student network to generate semantic segmentation results.

[0112] Since the input modes of the first student network and the second student network are different, the output modes of the first student network are also different. In order to achieve the fusion of the outputs of the first student network and the second student network, the embodiments of the present invention propose a residual-based coordinate attention fusion module (RCAF, hereinafter referred to as the fusion module) based on coordinate self-attention.

[0113] The event semantic segmentation network provided in the embodiments of the present invention is formed by a trained first student network, a second student network, a mode conversion network, and the aforementioned fusion module. Therefore, the event semantic segmentation network provided in the embodiments of the present invention can be used to execute step S03. From another perspective, steps S01-S02 in the event semantic segmentation method provided in the embodiments of the present invention can be regarded as at least part of the steps for constructing the event semantic segmentation network. This event semantic segmentation network can also be called a dual-branch fusion network.

[0114] Step S03 includes:

[0115] S31: Divide the first student network and the second student network into n layers respectively.

[0116] The first student network and the second student network are divided separately, resulting in n first student network layers and n second student network layers. The n first student network layers are configured using Si. e -block-1, ..., S e -block-n is used for representation, and n second student network layers adopt S fi -block-1, ..., S fi The term is represented as -block-n. Here, n corresponds to the number of layers n during the training of the first and second student networks. In other words, in this embodiment, the first and second student networks are each divided into 3 layers, forming three first student network layers and three second student network layers.

[0117] S32: The output of the first student network layer and the output of the corresponding second student network layer are reweighted by the fusion module, and the weighted features are output.

[0118] A first student network layer and a second student network layer of the same scale constitute a student network layer pair, and each student network layer pair contains a first student network layer and a second student network layer. The number of fusion modules is adapted to the number of student network layer pairs; that is, in this embodiment, the number of fusion modules is three. The output of each student network layer pair is reweighted through a fusion module to obtain weighted features.

[0119] If the fusion module is not the last fusion module, then the weighted features output by the fusion module are used as the input to the next student network layer pair.

[0120] If this fusion module is the last fusion module, that is, this fusion module is used to fuse the final first student network layer S. e -block-n and the final second student network layer S fi If `-block-n` is selected, the average value of the weighted features output by the fusion module is input into the fully connected layer. Accordingly, the fusion network of the event semantic segmentation network includes both a fusion module and a fully connected layer.

[0121] The process of weighting the output features of the fusion module is as follows:

[0122] S321: Obtain attention values ​​based on the outputs of the first student network layer and the corresponding outputs of the second student network layer.

[0123] The second student network layer corresponding to the first student network layer is the second student network layer that forms a student network layer pair with the first student network layer. The output of the first student network layer is f. e The output of the second student network layer is f i . Figure 5 The structure of the fusion module is shown, where M is the stitching operation, X is the average pooling operation along the width direction, Y is the binning average pooling operation along the height direction, T1 is the first convolution transformation operation, S is the channel splitting operation, T2 is the second convolution transformation operation, × is the multiplication operation, and + is the addition operation. Figure 5 As shown, according to f e and f i The process of obtaining attention values ​​includes:

[0124] Perform a concatenation operation along the channel dimension to obtain the concatenated feature f, f = [f e f i ].

[0125] Encode the feature f using average pooling kernels in the width direction (or horizontal direction) and the height direction (or vertical direction) to obtain the feature f after pooling along the width direction. w (c, w) and the feature f after pooling along the height direction h (c, h). Obtain feature f w (c, w) and feature f h The process of (c, h) can be represented as:

[0126]

[0127]

[0128] Where c is the channel direction index, h is the height direction index, w is the width direction index, H is the length of feature f in the height direction, and W is the length of feature f in the width direction.

[0129] For feature f w (c, w) and feature f h (c, h) (f is used in the following formula) w and f h (For representation) Perform the first convolution transformation operation to obtain the transformed feature z:

[0130] z = ReLU(BN(Conv([f h f w ])))

[0131] Where Conv is a 1×1 convolution function; BN is a batch normalization function; ReLU is the first activation function; z∈R 2C / r×1×(H+W) r is the reduction ratio controlling the size of the intermediate characteristic channel; R is the real number field; C is f h and f w The number of channels.

[0132] Segment feature z into height feature z h and width feature z w This allows for the extraction of coordinate attention in both the height and width directions.

[0133] A second convolutional transformation is performed to obtain the event feature attention value e along the height direction. h Event feature attention value e along the width direction w Attention value i of pseudo-image features along the height direction h And the pseudo-image feature attention value i along the width direction w The second convolution operation can be represented as:

[0134]

[0135]

[0136]

[0137]

[0138] Where Tanh is the activation function; and Both are 1×1 convolution functions, but and The inputs and outputs are different, specifically, The input is z h The output is e h ; The input is z w The output is e w ; The input is z h The output is i h ; The input is z w The output is i w .

[0139] S322: Reweight the outputs of the first student network layer and the corresponding outputs of the second student network according to the weighting formula. The weighting formula is:

[0140] f e ′=f e +f i ×i h ×i w

[0141] f i ′=f i +f e ×e h ×e w

[0142] Among them, f e ′ represents the weighted event feature, f i ' represents the weighted pseudo-image feature. The weighted feature mentioned above is f. e ′ and f i A general term for ′.

[0143] That is, the fusion module provided in the embodiments of the present invention is similar to a residual form. In this way, features can be fully fused and different modules can benefit from each other.

[0144] If the fusion module is not the last fusion module (along the data processing direction, i.e.) Figure 3 (From left to right in the middle), then the output f of the fusion modulee ′ and f i As the input to the next student network layer pair, specifically, f e f is the input of the first student network layer to the next student network layer. i ′ is used as the input to the second student network layer in the next student network layer; if this fusion module is the last fusion module (along the data processing direction, i.e. Figure 3 (From left to right in the middle), then the f output by the fusion module e ′ and f i The average value of f' e ′+f i ′) / 2 is input into the fully connected layer.

[0145] S33: The fully connected layer generates semantic segmentation results.

[0146] The fully connected layer generates semantic segmentation results by averaging the obtained values. Specifically, Figure 6 A schematic diagram of the event data used in the first example is shown. Figure 7 It shows according to Figure 6 The semantic segmentation result image generated by semantic segmentation; Figure 8 A schematic diagram of the event data used in the second example is shown. Figure 9 It shows according to Figure 8 The semantic segmentation result image generated by semantic segmentation.

[0147] Experimental verification

[0148] The event semantic segmentation method provided by the embodiments of the present invention was tested using the DDD17 basic dataset. In the experiment, events of 50ms, 10ms and 250ms were selected for testing, and 50ms events were used for training.

[0149] The Davis Driving Dataset 2017 (DDD17) is available at sensors.ini.uzh.ch / databases. This data was collected from highway driving in Switzerland and Germany under various conditions. It includes both Davis data and vehicle data.

[0150] Table 1 shows the experimental results of the event semantic segmentation method provided by the embodiments of the present invention and the results of comparative experiments using other methods. According to the data shown, the event semantic segmentation method and event semantic segmentation network provided by the present invention have good performance.

[0151] Table 1

[0152] Network type MIoU (50ms) MIoU (10ms) MIoU (250ms) MIoU (Average) EvSegNet 54.81％ 45.85％ 47.56％ 49.41％ V2E 56.01％ 45.82％ 51.61％ 51.15％ EvDistill 58.02％ 49.21％ 52.01％ 53.08％ DTL 58.80％ 50.01％ 52.96％ 53.95％ MTF 60.28％ 47.96％ 57.02％ 55.09％

[0153] Among them, MIoU represents the average ratio of the intersection and union of the true and predicted values ​​in each segmentation category; EvSegNet (Event Segmentation Network) was proposed by Alonso and is trained in a supervised manner; V2E (Video to Event Transformation Network) was proposed by Gehrig and is trained in a supervised manner by sharing image labels through video-to-event conversion; EvDistill (Event Distillation Network) was proposed by Wang and introduces a teacher network for cross-modal distillation; DTL (Dual Branch Transfer Network) was proposed by Wang and introduces a cross-task learning module based on image reconstruction.

[0154] at the same time, Figure 10 The diagram shows a comparison of semantic segmentation results between an embodiment of the present invention and a DTL network. Specifically, Figure 10 From left to right in the diagram, the data represents the image data output by the event camera, the event data corresponding to that image data, the semantic segmentation result output by the DTL network, the semantic segmentation result output by the network of this invention, and the evaluation label corresponding to the image data. Based on the parts circled in each box in the comparison diagram, it can be seen that the network of this invention performs better than DTL.

[0155] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An event semantic segmentation method, characterized in that, include: The teacher network is trained using a first dataset; wherein the first dataset contains multiple image-label pairs; The first and second student networks are trained using a second dataset and supervised by the output of the teacher network; wherein the second dataset has multiple pairs of grayscale image-event data; the input of the teacher network is the grayscale image, the input of the first student network is the event data, and the input of the second student network is pseudo-image data converted from the event data; The outputs of the first student network and the second student network are combined to generate a semantic segmentation result; The step of fusing the outputs of the first student network and the second student network to generate a semantic segmentation result includes: The first student network and the second student network are each divided into n layers, forming n first student network layers and n second student network layers; The output of the first student network layer and the corresponding output of the second student network layer are reweighted by a fusion module, and a weighted feature is output. The number of fusion modules is n, and the weighted feature is used as the input to the next first student network layer and the next second student network layer. The average value of the weighted features output by the fusion module that combines the final first student network layer and the second student network layer is input into the fully connected layer. The fully connected layer generates semantic segmentation results; The step of reweighting the output of the first student network layer and the corresponding output of the second student network layer includes: Attention values ​​are obtained based on the outputs of the first student network layer and the corresponding outputs of the second student network layer; wherein, the attention values ​​include event feature attention values ​​e along the height direction. h Event feature attention value e along the width direction w Attention value i of pseudo-image features along the height direction h And the pseudo-image feature attention value i along the width direction w ; The outputs of the first student network layer and the corresponding outputs of the second student network layer are reweighted according to a weighting formula, which is: f e ′=f e +f i ×i h ×i w f i ′=f i +f e ×e h ×e w Among them, f e ′ represents the weighted event feature; f e The event characteristics output by the first student network layer; f i ′ represents the weighted pseudo-image features; f i The event characteristics output by the second student network layer; The steps of obtaining attention values ​​along the width and height directions based on the output of the first student network layer and the corresponding output of the second student network layer include: Perform a concatenation operation along the channel dimension to obtain the concatenated feature f, f = [f e ,f i ]; The feature f is encoded using average pooling kernels in both the width and height directions to obtain the feature f after pooling along the width direction. w (c,w) and the feature f after pooling along the height direction h (c,h): For the feature f w (c,w) and the feature f h Perform the first convolution transformation operation at (c,h) to obtain the transformed feature z: z=ReLU(BN(Conv([f h ,f w ]))) Segment feature z into height feature z h and width feature z w Then, a second convolution operation is performed to obtain the event feature attention value e along the height direction. h Event feature attention value e along the width direction w Attention value i of pseudo-image features along the height direction h And the pseudo-image feature attention value i along the width direction w : Where c is the channel direction index, h is the height direction index, w is the width direction index, H is the length of feature f in the height direction, and W is the length of feature f in the width direction; Conv, and The functions are: 1×1 convolution function; Tanh is the activation function; BN is the batch normalization function; ReLU is the first activation function; z∈R 2C / r×1×(H+W) r is the reduction ratio controlling the size of the intermediate characteristic channel; R is the real number field; C is f h and f w The number of channels.

2. The event semantic segmentation method according to claim 1, characterized in that, It also includes the step of converting the event data into pseudo-image data, and the step includes: Construct a mode transition network; The event data is converted into pseudo-image data using the modality conversion network. The mode conversion network is trained synchronously with the second student network.

3. The event semantic segmentation method according to claim 2, characterized in that, The conversion loss function of the mode conversion network is: Among them, L EFIT λ is the conversion loss of the mode conversion network. pw The hyperparameter for balancing the loss term; x i This is grayscale image data; x e For event data; D is the Euclidean distance function used to calculate the distance between two features; L mckd-fi The loss from the teacher network to the second student network.

4. The event semantic segmentation method according to claim 1, characterized in that, In the step of supervising the training of the first student network using the output of the teacher network, the loss function from the teacher network to the first student network is: Where n is the number of layers; λ j To balance the hyperparameters of the loss terms in different layers; D is the Euclidean distance function used to calculate the distance between two features; The features output by the j-th layer teacher network; This is the output of the first student network at layer j; The output of the teacher network includes 5. The event semantic segmentation method according to claim 1, characterized in that, In the step of supervising the training of the second student network using the output of the teacher network, the loss function from the teacher network to the second student network is: Where n is the number of layers; λ j To balance the hyperparameters of the loss terms in different layers; D is the Euclidean distance function used to calculate the distance between two features; The features output by the j-th layer teacher network; This is the output of the second student network at layer j; The output of the teacher network includes 6. The event semantic segmentation method according to any one of claims 1-5, characterized in that, The backbone network of the teacher network is ResNet, and the 7x7 convolutional kernel in the first layer of the ResNet is replaced with three 3x3 convolutional kernels.

7. An event semantic segmentation module, characterized in that, The event semantic segmentation module is used to execute the event semantic segmentation method as described in any one of claims 1-6, and the event semantic segmentation module includes: The first student network, whose input is event data; The second student network takes pseudo-image data as its input; the pseudo-image data is obtained by transforming the event data. A fusion network is used to fuse the outputs of the first student network and the second student network to generate a semantic segmentation result; The first student network and the second student network are obtained through supervised training using the output of the teacher network. The teacher network is trained using grayscale images, which correspond to the event data.

8. The event semantic segmentation module according to claim 7, characterized in that, The first student network is divided into multiple first student network layers, and the second student network is divided into multiple second student network layers. A first student network layer and a second student network layer constitute a student network layer pair. The fusion network includes a fully connected layer and multiple fusion modules, which are configured one-to-one with the student network layer pairs. The fusion modules are used to reweight the features output by the student network layer pairs to obtain weighted features, and the weighted features are used as the input of the next student network layer pair. The fully connected layer is used to generate semantic segmentation results based on the output of the last fusion module.

Images