Unsupervised Event Camera Semantic Segmentation Method and Device Based on Hybrid Pseudo Labels

Abstract

The present invention relates to an unsupervised event camera semantic segmentation method and device based on hybrid pseudo-labels. First, an unsupervised domain adaptation framework is used as a benchmark, and a set of pseudo-labels is generated through self-training. Then, Offline Event-to-Image Reconstruction (Offline E2VID) is incorporated into the framework, and another set of pseudo-labels is obtained by predicting the segmentation map on the reconstructed image. Subsequently, a noise label learning strategy is introduced to mix the two sets of pseudo-labels and improve the quality, thus realizing the training of the model. In addition, a soft alignment algorithm for the center points of the target domain is adopted to further improve the consistency of the target domain features. Extensive experiments show that the present invention effectively improves the negative impact of pseudo-label noise on semantic segmentation by mixing pseudo-labels.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an unsupervised event camera semantic segmentation method and device based on hybrid pseudo-labels. Background Art

[0002] Semantic segmentation based on event cameras has become increasingly popular due to its ability to handle scenarios under high-speed motion and extreme lighting conditions, which cannot be solved by traditional RGB cameras. Since it is difficult to annotate event camera data, previous methods relied on event-to-image reconstruction to obtain pseudo-labels for training. However, this will inevitably introduce noise and learning from noisy pseudo-labels, especially when generated from a single source, which may exacerbate errors. This drawback is also known as confirmation bias in pseudo-labels.

[0003] Traditional event cameras lack texture information and cannot perform semantic segmentation tasks.

[0004] Chinese Patent Application Publication No. CN117097960A discloses a panoramic simulation event stream generation method based on unsupervised domain adaptation. By simulating the event stream and the real event stream and further using some interpolation models, a panoramic event stream close to the real one is obtained. In this application, although the learning framework based on unsupervised domain transfer supplements the texture information, due to the huge difference between the event camera domain and the traditional image domain, the performance of the semantic segmentation task is affected.

[0005] The solution to solve the above problems by introducing a simulation tool to convert the image and event domains depends on the simulation tool and introduces additional pseudo-label noise, affecting the final performance.

[0006] In summary, how to improve the effect of camera semantic segmentation is an urgent problem to be solved. Summary of the Invention

[0007] The purpose of the present invention is to overcome the above-mentioned defects existing in the prior art and provide an unsupervised event camera semantic segmentation method and device based on hybrid pseudo-labels. During the training process of the semantic segmentation model, the negative impact of pseudo-label noise on semantic segmentation is improved by mixing pseudo-labels.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] In one aspect of the present invention, an unsupervised event camera semantic segmentation method based on hybrid pseudo-labels is provided. The data collected by the target event camera is semantically segmented by using a pre-trained semantic segmentation model based on knowledge distillation. Among them, the semantic segmentation model includes a student network and a teacher network, and the training process of the semantic segmentation model includes:

[0010] Obtain event camera data and its corresponding source image information;

[0011] Based on the event camera data, generate a first set of pseudo-labels through self-training;

[0012] Based on the event camera data, obtain reconstructed image information through event-to-image reconstruction;

[0013] Based on the reconstructed image information, generate a second set of pseudo-labels by predicting a segmentation map;

[0014] Based on the first set of pseudo-labels and the second set of pseudo-labels, calculate the value of the loss function and update the parameters of the student network to realize the training of the semantic segmentation model.

[0015] As a preferred technical solution, it further includes:

[0016] During the training process, based on each output of the student network in knowledge distillation, calculate the center point of the source domain feature distribution as an anchor point, calculate the distances from the features of the event camera data and the reconstructed image information to the anchor point respectively, and use Jensen-Shannon divergence to reduce the two distances, and perform subsequent loss function value calculation based on the output after reducing the distances.

[0017] As a preferred technical solution, in the semantic segmentation model, the training process of the student network in knowledge distillation includes:

[0018] Input the pre-obtained source image information corresponding to the event camera data into the student network to obtain a source image representation, and calculate the value of the first loss function;

[0019] Input the event camera data into the student network and the teacher network in knowledge distillation respectively, use the output of the teacher network as pseudo-labels, and calculate the value of the second loss function;

[0020] Based on the first set of pseudo-labels and the second set of pseudo-labels, calculate the value of the third loss function;

[0021] Based on the value of the first loss function, the second loss function, and the third loss function, update the parameters of the student network to realize the training of the student network.

[0022] As a preferred technical solution, the values of the first loss function, the second loss function, and the third loss function are all cross-entropy loss function values.

[0023] As a preferred technical solution, the calculation of the three loss function values includes the following steps:

[0024] Perform weighted aggregation on the first set of pseudo-labels and the second set of pseudo-labels, and calculate the third loss function value based on the representation of the event camera data output by the student network.

[0025] As a preferred technical solution, the calculation of the first loss function value includes the following steps:

[0026] Calculate the first loss function value based on the source image representation and the ground truth label corresponding to the pre-acquired source image information.

[0027] As a preferred technical solution, the process of obtaining the reconstructed image information includes:

[0028] Screen the data to be reconstructed from the event camera data, and obtain the reconstructed image information based on offline event-to-image reconstruction.

[0029] As a preferred technical solution, during the training process of the semantic segmentation model, update the parameters of the teacher network by exponential moving average based on the parameters of the student network.

[0030] Another aspect of the present invention provides an electronic device, which is characterized in that it includes: one or more processors and a memory, and the memory stores one or more programs, and the one or more programs include instructions for executing the above-mentioned unsupervised event camera semantic segmentation method based on hybrid pseudo-labels.

[0031] Another aspect of the present invention provides a computer-readable storage medium, which is characterized in that it includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the above-mentioned unsupervised event camera semantic segmentation method based on hybrid pseudo-labels.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) Mitigate the negative impact of pseudo-label noise on semantic segmentation: In the training process of the semantic segmentation model of the present application, event camera data is used as input, and a set of pseudo-labels is generated through teacher-student network self-training in knowledge distillation. Then, taking the reconstructed image information obtained through event-to-image reconstruction as input, another set of pseudo-labels is obtained by predicting the segmentation map on the reconstructed image. Subsequently, a noise label learning strategy is introduced, and the two sets of pseudo-labels are mixed and trained, effectively reducing the challenges brought by noise pseudo-labels by introducing pseudo-label loss.

[0034] (2) Improve segmentation performance: By narrowing the distribution between the event camera data and the reconstructed image in the target domain, overcome the problem of divergence in the target domain in unsupervised and adaptation tasks, thereby improving segmentation performance, and significantly improving the consistency and reliability of target features. Brief Description of the Drawings

[0035] Figure 1 It is a schematic diagram of the unsupervised event camera semantic segmentation process of the hybrid pseudo-label in the embodiment;

[0036] Figure 2 It is a schematic diagram of the target domain soft alignment process in the embodiment;

[0037] Figure 3 It is a schematic diagram of the visualization result of semantic segmentation in the embodiment. Detailed Embodiment

[0038] Next, in combination with the drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0039] Terms such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first", "second" may explicitly or implicitly include one or more of such features.

[0040] It should be noted that, without conflict, the features in the following embodiments and implementation manners can be combined with each other.

[0041] Embodiment 1

[0042] Aiming at the problems existing in the foregoing prior art, this embodiment provides an unsupervised event camera semantic segmentation method based on hybrid pseudo-labels, which adopts a new hybrid pseudo-label framework for semantic segmentation based on event cameras to reduce the influence of noisy pseudo-labels. Specifically, referring to Figure 1 , first, an unsupervised domain adaptation framework is used as a benchmark, and a first set of pseudo-labels can be generated through self-training corresponding to Figure 1 ③ in . Then, offline event-to-image reconstruction (Offline E2VID) is incorporated into the framework, and a second set of pseudo-labels is obtained by predicting the segmentation map on the reconstructed image Figure 1 corresponding to

[0043] ② in Figure 1Schematic diagram of the semantic segmentation model in this method, which is constructed based on the student network and teacher network of knowledge distillation. Figure 1 On the left in is the four types of input information involved in the training process of the model. From bottom to top, they are the full amount of event camera data Event camera data to be reconstructed Reconstructed image information (Recon.Image), source image information (Source Image). Among them, the full amount of event camera data Is obtained by the event camera; the event camera data to be reconstructed Is obtained by screening from the full amount of event camera data The reconstructed image information Is obtained by offline event-to-image reconstruction from the event camera data to be reconstructed; the source image information is obtained by the image capture device, which includes the sample O And the corresponding true label U i And the corresponding true label U i . It should be noted that the superscript l represents the data to be reconstructed, and the superscript u represents the data that does not need to be reconstructed.

[0044] During model training, by calculating the values of three types of loss functions, the student network F ψ Is trained, which will be described separately below.

[0045] The value of the first loss function corresponds to Figure 1 Path ① in. In this path, using the sample I in the source image information i As the input of the student network F ψ To obtain the representation F ψ (I i ), and using the true label Y i Calculate the value of the first loss function.

[0046] The value of the second loss function corresponds to Figure 1 Path ④ in. In this path, using the full amount of event camera data Respectively as the input of the student network F ψ And the teacher network F φ To obtain the representations And Using the representation of the teacher network As the pseudo label to calculate the value of the second loss function.

[0047] The value of the third loss function is the focus of this application, corresponding to Figure 1 Paths ② and ③ in. First, the event camera data to be reconstructed Respectively as the input of the student network F ψ And the teacher network F φThe input is used to obtain the representations and of the teacher network as the first group of pseudo-labels. Additionally, using the reconstructed image information as the input of the student network, the representation is obtained as the second group of pseudo-labels. The two groups of pseudo-labels are weighted and aggregated to complete the representation mixing, and the output of the student network is used to calculate the value of the third loss function.

[0048] Specifically, the above loss function value can be the cross-entropy loss function value, or other loss function value calculation methods can be adopted, which are not limited here.

[0049] See Figure 1 , during the training process, the teacher network is updated through the exponential moving average (EMA) of the student network parameters. The stop gradient in paths ③ and ④ means that the teacher network will not be trained through the backpropagation of the loss function.

[0050] During the prediction process, the event camera data collected is input into the trained student network F ψ to obtain the output representation Semantic segmentation is implemented based on the output representation, as shown by the dashed line in Figure 1 . Only event camera data is required during the prediction process, without the need for source image information, which can improve the task performance while greatly reducing the computational overhead during the model implementation phase.

[0051] This embodiment provides a novel hybrid pseudo-labeling framework for event-based semantic segmentation. This framework effectively alleviates the challenges brought by noisy pseudo-labels, which is a common problem in this field. This method uniquely incorporates unsupervised domain adaptation of self-training and offline event-to-image reconstruction to generate high-quality hybrid pseudo-labels. The pseudo-labels are further improved by introducing a noisy label learning strategy.

[0052] Embodiment 2

[0053] Based on Embodiment 1, see Figure 1 , the unsupervised event camera semantic segmentation method based on hybrid pseudo-labels in this embodiment combines the SoftPrototypical Alignment (SPA) algorithm during the event camera semantic segmentation process to further improve the consistency of the target domain features.

[0054] Specifically, see Figure 2 , the SPA algorithm is based on all the output representations F ψ of the student network F ψ (Ii ), and By calculating the center point of the source domain feature distribution as the anchor point, the distance from the event camera data (TargetEvents) and its reconstructed image (Reconstructed Image) to the anchor point is calculated respectively. And using Jensen-Shannon divergence Close the two distances. By closing the distribution of event camera data and reconstructed images in the target domain, the problem of target domain divergence in unsupervised and adaptation tasks can be overcome, thereby improving the segmentation performance and significantly improving the consistency and reliability of target features.

[0055] To illustrate the effectiveness of this method, quantitative experiments are conducted on two event camera semantic segmentation datasets. The experimental results are shown in Tables 1 and Figure 3 As mentioned above, the results show that the model achieves better results than other current related technical methods, and this method does not rely on an online event camera simulator and can improve the performance in most mainstream event camera semantic segmentation tasks.

[0056] Table 1 Comparative experimental results

[0057]

[0058] The framework of unsupervised domain transfer learning based on hybrid pseudo-labels provided in this application realizes the unsupervised event camera semantic segmentation task. The noise pseudo-label learning and target domain soft alignment modules provided can further improve the performance of semantic segmentation. In extensive experiments, this method outperforms existing unsupervised domain transfer methods and is also very good compared with several supervised methods. This achievement marks a significant progress in the field of event-based semantic segmentation.

[0059] Example 3

[0060] This embodiment 1 provides an electronic device, characterized in that it includes: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the unsupervised event camera semantic segmentation method based on hybrid pseudo-labels as described in embodiment 1 or embodiment 2.

[0061] Example 4

[0062] This embodiment provides a computer-readable storage medium, characterized in that it includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the unsupervised event camera semantic segmentation method based on hybrid pseudo-labels as described in Example 1 or Example 2.

[0063] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An unsupervised event camera semantic segmentation method based on hybrid pseudo-labels, characterized in that Use the pre-trained semantic segmentation model based on knowledge distillation to perform semantic segmentation on the data collected by the target event camera. The semantic segmentation model includes a student network and a teacher network. Among them, the training process of the semantic segmentation model includes: Obtain event camera data; Based on the event camera data, generate a first set of pseudo-labels through self-training; Based on the event camera data, obtain reconstructed image information through event-to-image reconstruction; Based on the reconstructed image information, generate a second set of pseudo-labels through predicted segmentation maps; Based on the first set of pseudo-labels and the second set of pseudo-labels, calculate the loss function value and update the parameters of the student network to implement the training of the semantic segmentation model, wherein, the loss function value includes: The first loss function value, using the samples in the source image information as the input of the student network to obtain a representation , and using the true label to calculate the first loss function value; The second loss function value, using all event camera data as the input of the student network and the teacher network respectively, and obtaining the representations and respectively. Using the representation of the teacher network as the pseudo label to calculate the second loss function value; The third loss function value. First, the event camera data to be reconstructed is used as the input of the student network and the teacher network respectively, and the representations and are obtained respectively. The representation of the teacher network is used as the first group of pseudo-labels. Additionally, the reconstructed image information is used as the input of the student network, and the representation is obtained as the second group of pseudo-labels. The two groups of pseudo-labels are weighted and aggregated to complete the representation mixing, and the third loss function value is calculated using the output of the student network. During the semantic segmentation of the event camera, it includes a soft alignment process of the center point of the target domain: Based on student network All output representations , , and , by calculating the center point of the source domain feature distribution as the anchor point, and calculating the distance from the event camera data and its reconstructed image features to the anchor point , , and using the Jensen-Shannon divergence Close the distance between the two.

2. The unsupervised event camera semantic segmentation method based on hybrid pseudo-labels according to claim 1, characterized in that The first loss function value, the second loss function value, and the third loss function value are all cross-entropy loss function values.

3. A method for unsupervised event camera semantic segmentation based on hybrid pseudo-labels according to claim 1, characterized in that, The process of obtaining the reconstructed image information includes: Screen the data that needs to be reconstructed from the event camera data, and obtain the reconstructed image information based on offline event-to-image reconstruction.

4. A semi-supervised event camera semantic segmentation method based on hybrid pseudo-labels according to claim 1, characterized in that, During the training process of the semantic segmentation model, based on the parameters of the student network, update the parameters of the teacher network through exponential moving average.

5. An electronic device, characterized in that, Includes: One or more processors and a memory. The memory stores one or more programs, and the one or more programs include instructions for executing the unsupervised event camera semantic segmentation method based on hybrid pseudo-labels as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, Includes one or more programs for execution by one or more processors of an electronic device. The one or more programs include instructions for executing the unsupervised event camera semantic segmentation method based on hybrid pseudo-labels as described in any one of claims 1-4.

Images