Method for training a domain-adaptive neural network and associated device

Abstract

To provide a computer-implemented method for training a domain-adapted neural network configured to generate a semantic segmentation of an image.SOLUTION: A method comprises: inputting at least one labeled source image belonging to a source domain, at least one unlabeled target image belonging to a target domain different from the source domain, at least one stylized source image, and at least one stylized target image into the neural network; and training the neural network by minimizing a loss function penalizing differences between features generated from a given image and features generated from the at least one stylized version of the given image.SELECTED DRAWING: Figure 6

Description

[Background technology]

[0001] 1. Field of the Invention The present disclosure relates to the field of computer vision, and more precisely, to neural networks that may be capable of generating semantic segmentations of images belonging to different domains.

[0002] 2. Description of Related Technology Machine learning techniques typically require a huge amount of labeled data associated with a given domain to achieve reliable performance. However, collecting and annotating large datasets corresponding to new domains is very costly, time-consuming, and error-prone. Furthermore, in some domains, sufficient training data is not available. And models trained on datasets from one domain provide poor results when tested on data from another domain, due to "domain shift."

[0003] However, it often happens that a large amount of data is available for another domain related to the one under consideration, and it is therefore advantageous to adapt a learning model trained on examples from one domain to apply the task to examples from another domain.

[0004] Domain adaptation (DA) is a learning task that addresses this problem by considering the case of a model that is trained using examples from one or more domains (termed the "source domain") and then configured to apply the task to examples from a different but related domain (termed the "target domain"). The goal of domain adaptation is to address shifts between source and target domains that typically degrade the performance of a learning model by providing a model that is first trained on examples from a given domain but then adapted to process samples from another domain related to the given domain. Domain adaptation is typically considered a specific type of transfer learning, where the data distributions of the source and target domains are different, but the label spaces of the source and target domains are the same.

[0005] Adapting a model to process samples in a target domain by considering only samples from the source domain leads to poor performance, even when the domain shift is small. To address this, unsupervised domain adaptation (UDA) techniques process labeled samples from the source domain and unlabeled samples from the target domain. UDA techniques are particularly useful for semantic segmentation, where learning models require a huge amount of manually labeled data, which is expensive to acquire because pixel-by-pixel labeling is required. Semantic segmentation is defined as the task of assigning a label representing its semantic content to each pixel in an input image. Typically, initial supervised training on the source domain is followed by adaptation to the target domain using unsupervised learning methods, which aim to achieve reliable performance even on the target domain where labels are not available.

[0006] The technique is particularly adapted to bridge the gap between these synthetic and real-world data. Indeed, the computer graphics field generates a large amount of annotated synthetic data, which can ultimately be used to train models dedicated to processing real-world data. UDA techniques then make it possible to avoid the time-consuming and very expensive manual labeling required for real-world training data.

[0007] However, most feature-level adaptation methods validated in synthetic-to-real adaptation provide small gains in normal-to-adverse level adaptation.

[0008] Thus, there is room for improvement in the field of semantic segmentation models that are trained using data representing normal conditions but are intended to handle data representing adverse conditions. Summary of the Invention

[0009] To this end, the present invention first provides a computer-implemented method for training a domain-adaptive neural network to generate a semantic segmentation of an image, the method comprising: inputting at least one labeled source image belonging to a source domain, at least one unlabeled target image belonging to a target domain different from the source domain, at least one stylized source image, and at least one stylized target image into a neural network (F); training the neural network by minimizing a loss function that penalizes differences between features generated from at least one labeled source image and features generated from at least one stylized source image, and / or a loss function that penalizes differences between features generated from at least one target image and features generated from at least one stylized target image.

[0010] Therefore, the above method proposes to train a neural network using labeled images belonging to a given domain (termed the "source domain") and unlabeled images belonging to another domain (termed the "target domain"), where the source images and at least one stylized source image are labeled, while the target image and at least one stylized target image are unlabeled.

[0011] However, the neural network is trained under the condition that the source image and at least one stylized source image input to the neural network show the same scene from the same viewpoint, and the at least one target image and at least one stylized target image show another single scene from the same viewpoint.

[0012] As used herein, a "viewpoint" corresponds to the viewpoint relative to a camera used to capture an image of a scene.

[0013] The source and target images differ in their content and style, i.e., they represent scenes captured under different lighting and / or weather conditions. In an example, the source images belong to a source domain consisting of scene images captured under normal conditions (e.g., daytime, approximately 20,000 lux with bright blue sky), and at least one target image belongs to a target domain consisting of a batch of scene images captured under adverse conditions (e.g., nighttime with illumination below 1 lux or foggy weather).

[0014] "Normal conditions" are defined herein as a combination of daylight and bright weather without precipitation or snow covering the ground, i.e., good visibility, while "bad conditions" include "bad weather," possibly with precipitation and / or snow covering the ground and / or unfavorable light intensity, i.e., poor visibility.

[0015] As described in further detail below, at least one stylized source image is generated by mapping the style of the target image to the source image. In this way, the source image and the at least one stylized source image have similar structural content, and the at least one stylized source image has the style of images belonging to the target domain. In other words, in the above method, only style information is exploited from the target image to generate at least one stylized source image. Training then attempts to reduce the distance between features extracted from the source image and the at least one stylized source image; because both images have similar structural content, the neural network is encouraged to generate feature representations that are invariant to the styles of these input images.

[0016] Similarly, at least one stylized target image is generated by mapping the style of the source image to the target image. In this way, the target image and the at least one stylized target image have similar structural content, and the at least one stylized target image has the style of the image belonging to the source domain. In other words, in the above method, only style information is utilized from the source image to generate at least one stylized target image. Training also attempts to reduce the distance between features extracted from the target image and features related to the at least one stylized target image, and because both images have similar structural content, the neural network is encouraged to generate feature representations that are invariant to the styles of these input images.

[0017] Neural networks are trained to generate semantic segmentations of input images. The goal of semantic segmentation is to assign to each pixel of the input image a label that represents its semantic content. Semantic segmentation differs from image classification in that in image classification, a single label is assigned to the entire scene. This label indicates the dominant object in the scene. Semantic segmentation also differs from object detection in that in object detection, objects are identified by bounding boxes and a label is assigned to each box.

[0018] Semantic segmentation of an input image results in the image being clustered into regions corresponding to different objects and structures labeled with categories that represent its semantic content.

[0019] A "category" of an object may be referred to as a type or class of object, as is commonly done by those skilled in the art of object detection or semantic segmentation. As an example, a category may be selected from a list of object categories typically present in scenes observed on roads. For example, a category may be selected within a list comprising cars, pedestrians, cyclists, buses, trucks, etc. The categories may also be more precisely defined (e.g., for cars, it is possible to consider sedans, compact cars, etc.).

[0020] The training of the neural network can be performed iteratively, where multiple source images and multiple target images represent different scenes, each with different structural content. Multiple stylized source images and stylized target images are then generated, and the training step is applied iteratively.

[0021] The neural network is trained by minimizing a loss function that penalizes differences between features extracted from images. The differences can be expressed as distances, and after multiple distances are calculated (in the case of batch training), stochastic gradient descent or any other suitable training algorithm or variant of stochastic gradient descent can be used. Stochastic gradient descent can be used to adapt the weights of the neural network. The training method according to the present invention involves minimizing the loss function, e.g., backpropagating this loss.

[0022] In certain implementations, the labeled source image and at least one unlabeled target image are captured under different lighting and / or weather conditions.

[0023] In certain implementations, at least one stylized source image is generated by applying a stylization method to a source image, the stylization method maintaining the structural content of the source image, and the at least one stylized source image having the style of an image belonging to a target domain.

[0024] In certain implementations, at least one stylized target image is generated by applying a stylization method to the target image, the stylization method maintaining the structural content of the target image, and the at least one stylized target image having the style of an image belonging to the source domain.

[0025] In certain implementations, the neural network includes an encoder configured to generate intermediate features, and minimizing the loss function prompts the encoder to generate intermediate features that are invariant to lighting and / or weather conditions.

[0026] In certain implementations, the labels of the source images are used as ground truth for supervising the semantic segmentation prediction of the source images.

[0027] In certain implementations, the neural network further comprises a decoder configured to parse the generated features into "semantic classes" or "categories."

[0028] As mentioned above, the concept of "category" may be referred to as a type or class of object, as is commonly done by those skilled in the art of object detection or semantic segmentation.

[0029] In a particular implementation, the loss function is: a first feature invariant loss that penalizes differences between features generated from at least one labeled source image and features generated from at least one stylized source image, the at least one stylized source image having the same structural content as the labeled source image; a second feature invariant loss that penalizes differences between features generated from at least one target image and features generated from at least one stylized target image, the at least one stylized target image having the same structural content as the target image; Source Image I S Set of labels Y S The cross-entropy error L at the output of the trained neural network for the labeled source images, CE (F, I S , Y S ), and -Target Image I T A set of pseudo labels

number

number

[0030] In a specific implementation, the first and second feature-invariant losses are: I, an image among the source and target images; I′, a stylized version of I; and φ(I), φ(I′)∈R. D×M×N The features generated by the bottleneck layer are φ(I) and φ(I') and the Frobenius norm |||| F and D is the number of channels, and M and N are the spatial dimensions of the feature maps.

number

[0031] In certain implementations, the method further includes generating at least one stylized source image by applying the stylization method to the source image, and generating at least one stylized target image by applying the stylization method to the target image.

[0032] In certain implementations, applying the stylization method to the source image includes: Applying a Fourier transform to the source image gives the source image phase F P (I S ) and the source image amplitude F A (I S ) and Applying a Fourier transform to at least one target image to obtain the target image amplitude F A (I T ) and Source image amplitude F A (I S ) the low frequency part of the target image amplitude F A (I T ) and replacing it with the low frequency part of M is a mask β have

number

[0033] The invention also relates to a neural network trained by the method according to the invention.

[0034] According to a third aspect, the invention relates to an electronic device comprising at least one processor and a memory on which a program implementing the method described above is stored.

[0035] According to a fourth aspect, the present invention relates to a computer-implemented method for generating a semantic segmentation of an input image, the method comprising: Obtaining a neural network to be trained by performing the method for training a neural network described above; and generating a semantic segmentation of the input image using the obtained neural network.

[0036] As mentioned above, generating a semantic segmentation of an input image aims to assign to each pixel of the input image a label that represents its semantic content.

[0037] According to a fifth aspect, the invention relates to an electronic device comprising at least one processor and a memory on which a program implementing the method for generating a semantic segmentation as described above is stored.

[0038] According to a sixth aspect, the present invention relates to a system comprising an electronic device configured to implement a method for training a neural network and an electronic device configured to implement a method for generating a semantic segmentation of an input image.

[0039] Embodiments of the present invention also extend to a program which, when executed on a computer or processor, causes the computer or processor to perform the above-mentioned method, or which, when loaded into a programmable device, causes the device to become the above-mentioned device. The program may be provided by itself or may be carried on a carrier medium. The carrier medium may be a memory or recording medium, or a transmission medium such as a signal. A program embodying the present invention may be transitory or non-transitory. [Brief explanation of the drawings]

[0040] The features, advantages, and technical and industrial importance of preferred embodiments of the present invention will be described below with reference to the accompanying drawings, in which like reference numerals refer to like elements.

[0041] [Figure 1] 1 illustrates a particular implementation of an electronic device (10A) configured to implement a method for training a neural network. [Figure 2] FIG. 10B illustrates a particular implementation of an electronic device (10B) configured to generate a semantic segmentation of an input image. [Figure 3] FIG. 2 illustrates an example of the hardware architecture of the electronic device (10A) illustrated in FIG. [Figure 4] FIG. 3 illustrates an example of the hardware architecture of the electronic device (10B) illustrated in FIG. 2. [Figure 5] FIG. 2 shows a functional representation of the electronic device (10A) shown in FIG. [Figure 6] 1 is a flowchart of a method for training a neural network according to a specific implementation of the present invention. [Figure 7] 1 is a flowchart of a method for generating a semantic segmentation of an input image according to a particular implementation of the present invention; [Figure 8]10A shows qualitative results obtained with the electronic device (10A) of FIG. 1 for a "normal vs. wicked domain adaptation" scenario using the Cityscapes dataset and the Awkward Conditions Dataset with Correspondences (ACDC). DETAILED DESCRIPTION OF THE INVENTION

[0042] For purposes of the present invention, it is to be understood that the specific devices and processes illustrated in the accompanying drawings, and described in the following specification, are merely preferred embodiments or aspects of the subject matter of the present disclosure. Accordingly, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein are not to be considered as limiting, unless specifically indicated.

[0043] As used herein, aspects, components, elements, modules, entities, steps, functions, instructions, and / or the like should not be construed as critical or essential unless expressly stated otherwise. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more" and "at least one." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, the terms "has," "have," "having," or the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise.

[0044] FIG. 1 illustrates a particular implementation of an electronic device 10A configured to implement a method for training a neural network.

[0045] FIG. 2 illustrates a particular implementation of an electronic device 10B configured to generate a semantic segmentation of an input image.

[0046] The remaining description is more specifically aimed at training a neural network to provide a semantic segmentation of an input image. Whatever the nature of the neural network considered (convolutional, perceptron, autoencoder, recurrent, etc.), the invention remains applicable, in particular for any deep neural network.

[0047] Additionally, no limitations are placed on the type or content of images (also termed "input images") that may be input to a semantic segmentation model, nor on the type of semantic segmentation that may be output based on the input images (i.e., the nature of the classes or "categories" is not a limiting factor of the present invention). In particular, such images may include, for example, images of urban scenes with pixel-level semantic annotations captured by an autonomous vehicle.

[0048] Incidentally, semantic segmentation remains a difficult and complex task in automatic visual understanding when compared to simpler problems such as image classification or object detection: in fact, in image classification, a single label is assigned to the entire input image referring to the dominant object in the scene captured by the input image, while in object detection, objects are identified by 2D bounding boxes and a label is assigned to each box. On the other hand, in semantic segmentation, each pixel of the input image is assigned a label corresponding to its semantic content.

[0049] The electronic device 10A is configured to train a neural network (F) so as to improve the accuracy of the semantic segmentation of an image. The trained neural network is then used as input to the electronic device 10B, which implements a method for generating a semantic segmentation of an input image. The method is hereinafter referred to as the "semantic segmentation method." Here, reference is made by "improving the accuracy of the semantic segmentation" to obtaining more accurate semantic segmentation results than would be possible if the neural network had not been improved (or "adapted") in accordance with the present invention.

[0050] FIG. 3 illustrates an example of the hardware architecture of the electronic device 10A illustrated by FIG.

[0051] The electronic device 10A has a computer hardware architecture. As shown in FIG. 3, the electronic device 10A includes a processor 1. Although a single processor 1 is shown, two or more processors may be used according to the particular needs, desires, or implementation of the electronic device 10A. Generally, the processor 1 executes instructions and manipulates data to perform the operation of the electronic device 10A and any algorithms, methods, functions, processes, flows, and procedures described in this disclosure.

[0052] The electronic device 10A also comprises a communication means 5, which is used by the electronic device 10A to communicate with another electronic device communicatively connected to the electronic device 10A, for example to connect to a database storing a plurality of labeled source images and a plurality of target images, where each source image of the plurality is associated with at least one target image. Generally, the communication means 5 is operable to communicate with a wireless network and comprises logic encoded in software, hardware, or a combination of software and hardware. More specifically, the communication means 5 may comprise software supporting one or more communication protocols associated with the communication, such that the wireless network or interface hardware is operable to communicate physical signals in and out of the depicted electronic device.

[0053] Although shown as a single communication means 5 in Figure 3, two or more communication means may be used according to the particular needs, desires, or implementation of the system. The electronic device 10A also comprises a random access memory 2, a read only memory 3, and a non-volatile memory 4.

[0054] The non-volatile memory 4 of the electronic device 10A stores a computer program PROG_IMP according to the invention, which contains instructions for carrying out the method for training a neural network according to the invention.

[0055] The program PROG_IMP defines functional modules of the electronic device 10A, which are based on or control the aforementioned elements 1 to 5 of the electronic device 10A, in particular: At least one labeled source image (I) belonging to the source domain S ), at least one unlabeled target image (I T ), at least one stylized source image (I S→T ), and at least one stylized target image (I T→Sa module MOD_OB_IMG configured to input the OB_IMG data into a neural network (F); At least one labeled source image (I S ) generated from the feature (φ(I S )) and at least one stylized source image (I S→T ) generated from the feature (φ(I S→T )) and the loss function (L CISS ), and / or at least one target image (I T ) generated from the feature (φ(I T )) and at least one stylized target image (I T→S ) generated from the feature (φ(I T→S )) and the loss function (L CISS and a module MOD_IMP configured to train the neural network (F) by minimizing

[0056] The different functional modules mentioned above can be part of a single electronic device (10A) or can be distributed among several electronic devices belonging to a system.

[0057] FIG. 4 illustrates an example of the hardware architecture of the electronic device 10B illustrated by FIG.

[0058] The electronic device 10B has a computer hardware architecture. As shown in FIG. 4, the electronic device 10B includes a processor 1. Although a single processor 1 is shown, two or more processors may be used according to the particular needs, desires, or implementation of the electronic device 10B. Generally, the processor 1 executes instructions and manipulates data to perform the operation of the electronic device 10B, as well as any algorithms, methods, functions, processes, flows, and procedures described in this disclosure.

[0059] The electronic device 10B also comprises communication means 5, which are used by the electronic device 10B to communicate with another electronic device communicatively connected to the electronic device 10B, for example to connect to the electronic device 10A to receive a neural network once trained according to the method for training a neural network described above.

[0060] Generally, the communication means 5 comprises logic operable to communicate with a wireless network and encoded in software, hardware, or a combination of software and hardware. More specifically, the communication means 5 may comprise software supporting one or more communication protocols associated with the communication, such that the wireless network or interface hardware is operable to communicate physical signals in and out of the illustrated system.

[0061] 4 as a single communication means 5, two or more communication means may be used according to the particular needs, desires, or particular implementation of the electronic device 10B. The electronic device 10B also comprises a random access memory 2, a read-only memory 3, and a non-volatile memory 4.

[0062] The non-volatile memory 4 of the electronic device 10B stores a computer program PROG_GEN according to the invention, which includes instructions for executing the method for generating a semantic segmentation of an input image according to the invention.

[0063] The program PROG_GEN defines functional modules of the electronic device 10B, which are based on or control the aforementioned elements 1 to 5 of the electronic device 10A, in particular: a module MOD_OB_F adapted to obtain a neural network (F) generated by carrying out the method for training a neural network (F) according to the invention; and a module MOD_GEN configured to generate a semantic segmentation of the input image using the obtained neural network (F).

[0064] The different functional modules mentioned above can be part of a single electronic device (10B) or can be distributed among several electronic devices belonging to a system.

[0065] FIG. 5 shows a functional representation of the electronic device 10A shown by FIG.

[0066] As previously described, electronic device 10A implements a method for training a neural network to process images belonging to several related domains. Thus, electronic device 10A learns representations by penalizing differences between encoded features of pairs of images of the same scene, where one image of the pair is a "stylized version" of the other.

[0067] More precisely, the electronic device 10A is configured to align features extracted from the two images using a feature-invariant loss that penalizes differences between the two feature maps. Intuitively, the encoder of a semantic segmentation network should output features that are invariant to the style of the scene, so that the decoder can then generate identical outputs for different versions of the same scene, since the ground truth semantics of these versions are also identical.

[0068] To this end, as shown in Figure 5, we use a labeled source image I S and an unlabeled target image I belonging to the target domain T is input to the neural network F. The labeled source image I S and target image I Trepresent different scenes, each captured under different lighting and / or weather conditions. A "labeled source image" refers to an image having pixels associated with labels that represent the semantic content of the pixels. In certain cases, a labeled source image may refer to an image in which each pixel of the image is associated with a label.

[0069] The stylization function g is the stylized source image I S→T The source image I S is applied to the target image I T The "stylized function" g is also taken as input. T→S The source image I S By further taking as input the target image I T is then applied to the stylized source image I S→T and Stylized Target Image I T→S is also input to the neural network F.

[0070] As further described below, the source image I S The stylized function g applied to the source image I S and stylized source image I S→T have similar structural content, and at least one stylized source image has the style of an image belonging to the target domain. S→T In other words, the stylization method allows the generation of a source image I S Similarly, the target image I T The stylized function g applied to the target image I T and Stylized Target Image I T→S have similar structural content, and the stylized target image I T→S A stylized target image I such that I has the style of an image belonging to the source domain. T→S Allows the generation of

[0071] Then, the representation backbone, also called the "encoder" φ, is the image I S , I T , I S→T , and I T→S From the "feature representation" φ(I S ), φ(I T ), φ(I S→T ), and φ(I T→S ) which generates a representation φ(.)∈R corresponding to the output of the bottleneck layer. D*M*N where D corresponds to the channel dimension of the feature map (i.e., the number of channels), and M and N are the two spatial dimensions of the feature map (i.e., the number of rows and columns of the feature map, respectively).

[0072] The neural network is based on the source image I S and stylized source image I S→T It is trained to improve the accuracy of semantic segmentation of images belonging to different domains by minimizing a feature-invariant loss that penalizes the difference between the features φ(I S ) and φ(I S→T ) is the feature invariant loss module L INV is fed to the feature invariant loss module L INV L INV (F, I S , I S→T ) is generated by the feature-invariant loss. S and stylized source image I S→T have similar structural content but differ only in terms of their style, and the functions L INV Minimizing φ encourages the encoder φ to generate a feature representation that is independent of the modality of the input image. In a specific way, this minimizes the S ) and φ(I S→T ) can be implemented by minimizing the distance between

[0073] The neural network also computes the target image I Tand stylized target image I T→S Feature invariant loss L that penalizes the difference between INV To do this, the feature φ(I T ) and φ(I T→S ) is the feature invariant loss module L INV is fed to the feature invariant loss module L INV L INV (F, I T , I T→S ) to generate a feature-invariant loss. T and Stylized Target Image I T→S have similar structural content but differ only in terms of their style, and the functions L INV Minimizing φ encourages the encoder φ to generate a feature representation that is independent of the modality of the input image. In a specific way, this minimizes the T ) and φ(I T→S ) can be implemented by minimizing the distance between

[0074] Then, the source image I S The feature φ(I S ), and the source image I T The feature φ(I T ) are fed to a shared decoder ω, which is configured to compute softmax predictions and respective cross-entropy errors.

[0075] More precisely, the decoder ω is a cross-entropy error module L CE The source image I is fed to S The softmax output F(I S ) is generated. This cross-entropy error module L CE Here is the source image I S Set of labels Y S Cross-entropy error L with CE (F, I S , Y S) Similarly, the decoder ω is configured to generate a cross-entropy error module L CE The target image I is fed to T The softmax output F(I T ) is generated. This cross-entropy error module L CE is the target image I T A set of pseudo labels

number

number

[0076] Feature invariant loss L INV (.) and the cross-entropy error L CE Further details regarding (.) are provided below.

[0077] FIG. 6 is a flowchart of a method for training a neural network according to a specific implementation of the present invention.

[0078] The method to train a neural network is to provide at least one labeled source image (I) belonging to the source domain. S ) and at least one unlabeled target image (I T ) and at least one stylized source image (I S→T ) and obtain at least one stylized target image (I T→S ) which may be implemented by the module MOD_OB_IMG of the electronic device 10A.

[0079] This step S100 involves generating at least one source image (I S ) and at least one unlabeled target image (I T) is acquired. Typically, a plurality of source images (I S ) and the target image (I T ) is obtained, but for simplicity, we then use one source image (I S ) and one target image (I T ) is obtained.

[0080] Source image (I S ) and the target image (I T ) can be obtained by accessing a database D′ that stores a dataset comprising a plurality of sets of images, each set being composed of one or several source images I S and one or several target images I T and a source image (I S ) and the target image (I T ) represent different scenes captured under different lighting and / or weather conditions. The electronic device 10A and the database D' may be connected using a wired or wireless connection over a telecommunications network.

[0081] Step S100 is to extract the source image (I S ) and the target image (I T ) in a stylized version.

[0082] More precisely, the purpose of this substep S120 is to generate a stylized version of a given image belonging to a given domain, such that the given image and the stylized image have similar structural content, and the stylized image has the style of an image belonging to a different domain from the given domain. In this way, the two domains are "aligned" at the pixel level, and the semantics of the given image are maintained when generating the stylized image. Therefore, the semantic annotations of the given image can be used as ground truth for supervising the prediction of the semantic segmentation of the stylized image.

[0083] In certain implementations, lightweight shallow mapping is applied to generate a stylized version of the image. "Lightweight shallow mapping" provides the benefits of on-the-fly stylization (e.g., step S120) without introducing unnecessary additional complexity into the overall architecture.

[0084] More formally, stylized source image I S→T I S→T =g(I S , I T ) for each stylized target image, I T→S =g(I T , I S ) can be calculated as

[0085] I S→T and I T→S The stylized mapping function g used to generate

[0086] While the method of the present invention is agnostic to the exact mapping g used for stylization, in certain implementations, a Fourier Domain Adaptation (FDA)-based method is applied. FDA-based methods reduce the mismatch between the source and target distributions by exchanging the low-frequency spectrum of one with that of the other. Using the FDA-based method, the stylized source image I S→T To generate the signal, the electronic device 10A

number

[0087] More precisely, the electronic device 10A implements the following steps: Source image (I S ) to obtain the source image phase F P (I S ) and the source image amplitude F A (I S ) Target image (I T ) to obtain the target image amplitude F A (I T ) Source image amplitude F A (I S ) the low frequency part of the target image amplitude F A (I T ) by substituting the low frequency part of M β is the mask corresponding to the ideal low-pass filter.

number

[0088] Next, I S The modified spectral representation of (whose phase component is unchanged) is the image I S→T and its structural content is mapped back to I S , but in the style of an image belonging to the target domain.

[0089] Those skilled in the art may refer to the paper "Fourier Domain Adaptation for Semantic Segmentation" by Yanchao Yang and Stefano Soatto at the IEEE / CVF Conference on Computer Vision and Pattern Recognition in June 2020 for further details regarding FDA-based methods.

[0090] The main difference with FDA-based methods in the art is that in our method of training a neural network, both the images belonging to the source domain and the images belonging to the target domain are processed in a way that promotes constancy of the features generated by the neural network (F) across the two domains, thus better aligning the two domains at the feature level, thereby improving domain adaptation.

[0091] Step S100 is to extract the labeled source image (I S ), target image (I T ), stylized source image (I S→T ), and stylized target images (I T→S ) into a trained neural network (F).

[0092] The way to train a neural network is to use labeled source images (I S ) generated from the feature (φ(I S )) and at least one stylized source image (I S→T ) generated from the feature (φ(I S→T )) and the loss function (L CISS ), and / or at least one target image (I T ) generated from the feature (φ(I T )) and at least one stylized target image (I T→S ) generated from the feature (φ(I T→S )) and the loss function (L CISS ) which may be implemented by a module MOD_IMP of the electronic device 10A.

[0093] More precisely, step S200 includes sub-steps S220, S230, S240, and S250. As mentioned above, the neural network (F) is composed of an encoder and a decoder. In sub-step S220, the neural network (F) generates an image IS , I T , I S→T , and I T→S "Representation" φ(I S ), φ(I T ), φ(I S→T ), and φ(I T→S ) to generate a feature map.

[0094] The encoder should be a fully convolutional encoder. In that case, the representation φ(.)∈R corresponds to the output of the bottleneck layer. D*M*N where D corresponds to the channel dimension of the feature map (i.e., the number of channels), and M and N are the two spatial dimensions of the feature map (i.e., the number of rows and columns of the feature map, respectively).

[0095] In substep S230, the feature invariant loss L INV (F, I S , I S→T ) and L INV (F, I T , I T→S ) is calculated.

[0096] By using a specific feature-invariant loss rather than the typical cross-entropy error, I S and I S→t Between and I t and I t→S This provides the advantage of imposing stronger constraints on the neural network (F) regarding the shared semantic content between and.

[0097] The feature representations generated in sub-step S220 should ideally be invariant to the particular modality or visual conditions of the input image, allowing the decoder to focus on parsing the features and outputting semantic classes without the need to further abstract from the particular modality of the input image.

[0098] Therefore, the neural network (F) is computed using a feature-invariant loss function L that penalizes the difference between features generated from a given image and features generated from a stylized version of the given image. INV It is trained by minimizing (.).

[0099] More formally, let F be the composition of the encoder φ and the decoder ω, then

number

[0100] The feature invariant loss can be expressed as:

number

[0101] Returning to step S230, the feature-invariant loss function is S , I S→T ) and (I T , I T→S ), which is applied to align features from each pair, and the feature invariant loss L INV (F, I S , I S→T ) and L INV (F, I T , I T→S ) is calculated.

[0102] Then, the method for training the neural network (F) is to use the source image I S The softmax output F(I S ) and target image IT The softmax output F(I T ) is generated.

[0103] In step S250, the softmax output F(I S ) is the cross-entropy error L CE (F, I S , Y S ) is generated by the cross-entropy error module L CE and the softmax output F(I T ) also, target image I T A set of pseudo labels

number

number

[0104] In certain implementations, the target image I T The pseudo-labels of are determined by applying the teacher-student self-training framework of DAFormer

[12] with confidence-weighted pseudo-labels.

[0105] More formally, if the semantic labels Y are one-hot-encoded in a C×H×W tensor, then the cross-entropy error L associated with the network's softmax output F(I) for image I is CE (.) is defined as follows:

number

[0106] The resulting two feature invariant losses L INV (F, I S , I S→T ) and L INV (F, IT , I T→S ) is the cross-entropy error L CE (F, I S , Y S ) and

number

number

[0107] In step S260, the neural network (F) calculates the resulting loss function L CISS The neural network is trained by minimizing the loss function L. Stochastic gradient descent or any other suitable training algorithm or variant of stochastic gradient descent may be used. Stochastic gradient descent may be used to adapt the weights of the neural network. The training method of the present invention is CISS This involves minimizing , for example, backpropagating this loss.

[0108] 7 is a flowchart of a method for generating a semantic segmentation of an input image according to a specific implementation of the present invention, the method being implemented by the electronic device 10B shown in FIG.

[0109] 7, the method for generating a semantic segmentation of an input image includes a first step S300 of obtaining an input image, which may belong to a target domain, i.e., the input image may represent a scene captured under adverse conditions.

[0110] The method for generating a semantic segmentation of an input image further includes a step S400 of obtaining a trained neural network (F), which is generated by performing the method for training a neural network described above. This step may be implemented by module MOD_OB_F of electronic device 10B.

[0111] In a particular implementation, "obtaining a trained neural network" includes applying, by this electronic device 10B, the method for training a neural network described above.

[0112] In a variant, electronic device 10A implements a method for training a neural network to generate a domain-adaptive neural network, e.g., a neural network that achieves reliable performance even for images captured under adverse conditions. Then, the step of "obtaining the trained neural network" includes "receiving by electronic device 10B the trained neural network generated by electronic device 10A." In that case, electronic device 10A and electronic device 10B may be connected using a wired or wireless connection via a telecommunications network.

[0113] The method for generating a semantic segmentation of the input image further includes a step S500 of generating a semantic segmentation of the input image using the neural network (F) obtained in step S400, which may be implemented by module MOD_GEN of electronic device 10B.

[0114] FIG. 8 shows qualitative results obtained with the electronic device (10A) of FIG. 1 for the “normal vs. ill-conditioned domain adaptation” scenario using the Cityscapes dataset and the Adverse Conditions Dataset with Correspondences (ACDC).

[0115] Implementation details

[0116] The default implementation of the method for training the neural network is based on HRDA

[13] . The semantic segmentation network comprises a MiT-B5 encoder by SegFormer

[42] and a context-aware feature fusion decoder

[12] .

[0117] We also implemented our method using the DeepLabv2 [3] architecture with a ResNet-101 backbone [9] for direct comparison with several previous UDA methods that use this architecture. For the default HRDA-based implementation, we followed the teacher-student self-training framework of DAFormer

[12] with confidence-weighted pseudo-labels, rare class sampling, and target data augmentation according to DACS

[34] , achieving a 6 × 10 -5 and 6×10 for the decoder. -4 We used the AdamW optimizer

[21] with a learning rate of , a linear learning rate warmup, and mini-batches of size B = 2, and trained on two Titan RTX GPUs, each processing one of the two samples in each mini-batch.

[0118] The default configuration and parameters of HRDA were applied for its multi-resolution setup. When applying the FDA-based method of stylizing the images, β = 0.01 was used as the bandwidth parameter for the low-frequency band of the Fourier spectrum. The default value for the weight of the feature invariant loss was set to λ for the default HRDA-based implementation of CISS. S = 200 and λ t = 100 and for the alternative DeepLabv2-based implementation, S =λ t =10.

[0119] Dataset

[0120] The experiments focus on domain adaptation and generalization settings from normal vs. poor visual conditions, since the method for training the neural network is tuned for conditional level domain shifts that affect the style and texture of objects in the scene, rather than structural level shifts as in typical synthetic vs. real scenarios.

[0121] In our experiments, we used the Cityscapes dataset [6] as the labeled source domain set. Cityscapes is a large dataset of urban driving scenes captured in several Central European cities under normal conditions and contains high-quality pixel-level semantic annotations for a set of 19 common classes in driving scenes. The dataset consists of a training set with 2,975 images, a validation set with 500 images, and a test set with 1,525 images. When training the UDA method in our experiments, we sampled source images only from the Cityscapes training set. In addition, we used the ACDC dataset

[31] as the unlabeled target domain set to model the ill-conditioned domain for normal-versus-ill UDA. ACDC consists of 4,006 images of driving scenes evenly distributed across four common ill-conditions: night, fog, rain, and snow. The images feature high-quality semantic annotations for the same set of 19 classes as Cityscapes. The training, validation, and test sets contain 1,600, 406, and 2,000 images, respectively.

[0122] We then used BDD100K

[44] as the target set for domain generalization.,In particular, we first performed zero-shot testing on a validation,set of a 10,000-image semantic segmentation subset of BDD100K,containing 1,000 images captured under varying visual,conditions.,We then used BDD100K-night

[30] , a split of BDD100K with,87 nighttime images with accurate segmentation labels.

[0123] Comparison of state-of-the-art unsupervised domain adaptation and generalization methods in Cityscape → ACDC.

[0124] The cityscape serves as the source domain, and the entire ACDC, including all four adverse conditions, serves as the target domain.

[0125] Our method, CISS, significantly outperforms all competing methods, both domain adaptation and domain generalization, with a 4% difference in the main mIoU metric from the next-best method. Furthermore, our method achieves the best IoU in 11 of 19 individual classes, outperforming classes critical to driving recognition, such as roads, sidewalks, and cars. Focusing on methods using the DeepLabv2 architecture, CISS-DeepLabv2 also has the best performance among them, demonstrating that the advantages of our method are general across different UDA architectures.

[0126] Figure 8 shows qualitative results for Cityscapes → ACDC, e.g., the qualitative results provided by a neural network trained with the method of the present invention using the Cityscapes dataset as the source domain and the ACDC dataset as the target domain, and then tested / validated on the ACDC dataset.

[0127] Column A contains four different images belonging to the ACDC dataset, column B corresponds to their ground truth annotations, column C corresponds to the semantic segmentation results when applying the HRDA method, and column D corresponds to the results when applying the method for training a neural network of the present invention.

[0128] In the top snow image, our method accurately captures the sky, road, and sidewalk, while HRDA misclassifies both the sidewalk and sky as a road. In the nighttime images in the second row, our method successfully segments one of two very dark pedestrians on the left sidewalk and detects two motorcycles nearby, while HRDA completely mistakes the pedestrian for a plant and the motorcycle for a car. In the bottom rain image, HRDA incorrectly segments the green reflection of two traffic lights on the road as traffic lights, while our method correctly assigns the reflection to the road and segments the sidewalk on the right much more accurately. The invention disclosed in this specification includes the following aspects. [Aspect 1] 1. A computer-implemented method for training a neural network, the method comprising: At least one labeled source image (I) belonging to the source domain S ), at least one unlabeled target image (I T ), at least one stylized source image (I S→T ), and at least one stylized target image (I T→S ) into the neural network; The labeled source image (I S ) generated from the feature (φ(I S )) and said at least one stylized source image (I S→T ) generated from the feature (φ(I S→T )) and / or penalizing the difference between said at least one target image (I T ) generated from the feature (φ(I T )) and said at least one stylized target image (I T→S ) generated from the feature (φ(I T→S )) and the loss function (L CISS training the neural network by minimizing 11. A computer-implemented method comprising: [Aspect 2] The at least one labeled source image (I S ) and the at least one unlabeled target image (I T 2. The method of embodiment 1, wherein the images are captured under different lighting and / or weather conditions. Aspect 3 said at least one stylized source image (I S→T ) is the source image (I S ) by applying a stylization method to the source image (IS ) and maintaining the structural content of said at least one stylized source image (I S→T 3. The method of claim 1 or 2, wherein the image data has a format that belongs to the target domain. Aspect 4 The at least one stylized target image (I T→S ) is the target image (I T ) by applying a stylization method to the target image (I T ) and maintaining the structural content of said at least one stylized target image (I T→S 3. The method of claim 1 or 2, wherein the image data has a format belonging to the source domain. Aspect 5 The neural network comprises an encoder (φ) configured to generate intermediate features, and the loss function (L CISS 3. The method of claim 1 or 2, wherein minimizing φ(I), φ(I′) prompts the encoder to generate intermediate features (φ(I), φ(I′)) that are invariant to lighting and / or weather conditions. Aspect 6 The loss function (L CISS )teeth, The at least one labeled source image (I S ) generated from the feature (φ(I S )) and said at least one stylized source image (I S→T ) generated from the feature (φ(I S→T a first feature-invariant loss that penalizes the difference between The at least one target image (I T ) generated from the feature (φ(I T )) and said at least one stylized target image (I T→S ) generated from the feature (φ(I T→S a second feature-invariant loss that penalizes the difference between The source image I S Set of labels Y S the labeled source image (I S ) the cross-entropy error L at the output of the trained neural network CE (F, I S 、Y S ), and The target image I T A set of pseudo labels [Number 1] JPEG0007791225000016.jpg14170 The target image (I T ) the cross-entropy error at the output of the trained neural network [Number 2] JPEG0007791225000017.jpg15170 3. The method of claim 1 or 2, wherein Aspect 7 The first and second feature invariant losses are S ) and the target image (I T ), I is an image in the image tree, I' is a stylized version of I, and φ(I), φ(I')∈R D×M×N The features φ(I) and φ(I') generated by the bottleneck layer are calculated using the Frobenius norm |||| F and D is the number of channels, and M and N are the spatial dimensions of the feature maps. [Number 3] JPEG0007791225000018.jpg17170 7. The method of embodiment 6, wherein Aspect 8 The source image (I S ) by applying a stylization method to said at least one stylized source image (I S→T ) and generating the target image (I T ) by applying the stylization method to the at least one stylized target image (I T→S 3. The method of any one of embodiments 1 to 2, further comprising generating a Aspect 9 The source image (I S ) applying the stylization method to The source image (I S ) to obtain the source image phase F P (I S ) and the source image amplitude F A (I S ) and The at least one target image (I T ) to obtain the target image amplitude F A (I T ) and - the source image amplitude F A (I S ) the low frequency portion of the target image amplitude F A (I T ) and replacing it with the low frequency part of M is a mask β have [Number 4] JPEG0007791225000019.jpg17170 The inverse Fourier transform F -1 by applying a stylized source image (I S→T ) and 9. The method of embodiment 8, comprising: Aspect 10 A neural network trained by the method of embodiment 1 or 2. Aspect 11 3. An electronic device comprising: at least one processor; and a memory having stored thereon a program that implements the method for training a neural network according to aspect 1 or 2. Aspect 12 3. A computer program product that, when executed by an electronic device, causes the electronic device to perform the method for training a neural network according to aspect 1 or 2. Aspect 13 3. A recording medium readable by a computer and having recorded thereon a computer program comprising instructions for executing the method for training a neural network according to claim 1 or 2. Aspect 14 1. A computer-implemented method for generating a semantic segmentation of an input image, the method comprising: Obtaining a neural network trained by performing the method according to aspect 1 or 2; generating a semantic segmentation of the input image using the obtained neural network; 11. A computer-implemented method comprising: Aspect 15 An electronic device comprising at least one processor and a memory having stored thereon a program that implements the method for generating a semantic segmentation of an input image described in aspect 14. Aspect 16 A computer program that, when executed by an electronic device, causes the electronic device to perform the method for generating a semantic segmentation of an input image according to aspect 14. Aspect 17 A recording medium readable by a computer and having a computer program recorded thereon, the computer program including instructions for executing the method for generating a semantic segmentation of an input image according to aspect 14.

[0129] Reference

[0130] [3] Chen&Al.'s DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-848, 2018.

[0131] [6] Cordts et al., The Cityscapes dataset for semantic urban scene understanding. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[0132] [9] He et al., Deep residual learning for image recognition. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.

[0133]

[12] Hoyer et al., DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

[0134]

[13] Hoyer et al., HRDA: Context-aware high-resolution domain-adaptive semantic segmentation. The European Conference on Computer Vision (ECCV), 2022

[0135]

[21] Loshchilov et al., Decoupled weight decay regularization. ICLR, 2018.

[0136]

[30] Sakaridis&Al.'s Map-guided curriculum domain adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

[0137]

[34] Tranheden&Al.'s DACS: Domain adaptation via crossdomain mixed sampling. Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision (WACV), 2021.

[0138]

[42] Xie&Al.'s SegFormer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 2021.

[0139]

[44] Yu&Al.'s BDD100K: A diverse driving dataset for heterogeneous multitask learning. IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

Claims

1. 1. A computer-implemented method for training a neural network, the method comprising: At least one labeled source image (I) belonging to the source domain Ｓ ), at least one unlabeled target image (I) belonging to a target domain different from the source domain. Ｔ ), at least one stylized source image (I Ｓ→Ｔ ), and at least one stylized target image (I Ｔ→Ｓ ) into the neural network; The labeled source image (I Ｓ ) generated from the feature (φ(I Ｓ )) and said at least one stylized source image (I Ｓ→Ｔ ) generated from the feature (φ(I Ｓ→Ｔ )) and / or by penalizing the difference between said at least one target image (I Ｔ ) generated from the feature (φ(I Ｔ )) and said at least one stylized target image (I Ｔ→Ｓ ) generated from the feature (φ(I Ｔ→Ｓ )) and the loss function (L ＣＩＳＳ training the neural network by minimizing 12. A computer-implemented method comprising:

2. The at least one labeled source image (I Ｓ ) and the at least one unlabeled target image (I Ｔ 10. The method of claim 1, wherein the images are captured under different lighting and / or weather conditions.

3. said at least one stylized source image (I Ｓ→Ｔ ) is the source image (I Ｓ ) by applying a stylization method to the source image (I Ｓ ) and maintaining the structural content of said at least one stylized source image (I Ｓ→Ｔ 3. The method of claim 1, wherein the image data includes a style of the image belonging to the target domain.

4. The at least one stylized target image (I Ｔ→Ｓ ) is the target image (I Ｔ ) by applying a stylization method to the target image (I Ｔ ) and maintaining the structural content of said at least one stylized target image (I Ｔ→Ｓ 3. The method according to claim 1, wherein the image data has a format belonging to the source domain.

5. The neural network comprises an encoder (φ) configured to generate intermediate features, and the loss function (L ＣＩＳＳ 3. The method of claim 1, wherein minimizing φ(I), φ(I′) encourages the encoder to generate intermediate features (φ(I), φ(I′)) that are invariant to lighting and / or weather conditions.

6. The loss function (L ＣＩＳＳ )teeth, The at least one labeled source image (I Ｓ ) generated from the feature (φ(I Ｓ )) and said at least one stylized source image (I Ｓ→Ｔ ) generated from the feature (φ(I Ｓ→Ｔ a first feature-invariant loss that penalizes the difference between The at least one target image (I Ｔ ) generated from the feature (φ(I Ｔ )) and said at least one stylized target image (I Ｔ→Ｓ ) generated from the feature (φ(I Ｔ→Ｓ a second feature-invariant loss that penalizes the difference between The source image I Ｓ Set of labels Y Ｓ the labeled source image (I Ｓ ) the cross-entropy error L at the output of the trained neural network ＣＥ (F, I Ｓ , Y Ｓ ), and The target image I Ｔ A set of pseudo labels [Equation 1] The target image (I Ｔ ) the cross-entropy error at the output of the trained neural network for [Equation 2] 3. The method of claim 1, wherein the sum of

7. The first and second feature invariant losses are Ｓ ) and the target image (I Ｔ ), I is an image in the image tree, I' is a stylized version of I, and φ(I), φ(I')∈R Ｄ×Ｍ×Ｎ The features φ(I) and φ(I') generated by the bottleneck layer using ||·|| Ｆ and D is the number of channels, and M and N are the spatial dimensions of the feature maps. [Equation 3] The method of claim 6, wherein:

8. The source image (I Ｓ ) by applying a stylization method to said at least one stylized source image (I Ｓ→Ｔ ), and generating the target image (I Ｔ ) by applying the stylization method to the at least one stylized target image (I Ｔ→Ｓ 3. The method of claim 1 or 2, further comprising:

9. The source image (I Ｓ ) applying the stylization method to The source image (I Ｓ ) to obtain the source image phase F Ｐ (I Ｓ ) and the source image amplitude F Ａ (I Ｓ ) and The at least one target image (I Ｔ ) to obtain the target image amplitude F Ａ (I Ｔ ) and the source image amplitude F Ａ (I Ｓ ) by dividing the low frequency portion of the target image amplitude F Ａ (I Ｔ ) and replacing it with the low frequency part of M is a mask β have [Equation 4] The inverse Fourier transform F －１ by applying Ｓ→Ｔ ) and The method of claim 8, comprising:

10. 1. A neural network for causing a computer to generate a semantic segmentation of an input image, the neural network comprising an encoder (φ) configured to generate intermediate features, the encoder (φ) comprising: The neural network is configured such that, when at least one labeled source image (I S ) belonging to a source domain, at least one unlabeled target image (I T ) belonging to a target domain different from the source domain, at least one stylized source image (I S → T ), and at least one stylized target image (I T → S ) are input to the neural network, a loss function (L CISS ) that penalizes the difference between features (φ(I S )) generated from the labeled source image (I S ) and features (φ(I S → T )) generated from the at least one stylized source image (I S → T ) and / or the difference between features (φ(I T )) generated from the at least one target image (I T ) and features (φ(I T → S )) generated from the at least one stylized target image (I T → S ) is minimized.

11. 3. An electronic device comprising at least one processor and a memory having stored thereon a program implementing the method for training a neural network according to claim 1 or 2.

12. A computer program which, when executed by an electronic device, causes the electronic device to carry out the method for training a neural network according to claim 1 or 2.

13. A recording medium readable by a computer and having recorded thereon a computer program comprising instructions for carrying out the method for training a neural network according to claim 1 or 2.

14. 1. A computer-implemented method for generating a semantic segmentation of an input image, the method comprising: Obtaining a neural network trained by carrying out the method according to claim 1 or 2; generating a semantic segmentation of the input image using the obtained neural network; 12. A computer-implemented method comprising:

15. An electronic device comprising at least one processor and a memory having stored thereon a program implementing the method for generating a semantic segmentation of an input image according to claim 14.

16. A computer program product that, when executed by an electronic device, causes the electronic device to perform the method for generating a semantic segmentation of an input image according to claim 14.

17. A computer-readable recording medium having recorded thereon a computer program comprising instructions for performing the method for generating a semantic segmentation of an input image according to claim 14.

Images