Method and system for training a model to perform semantic segmentation on foggy images

Abstract

A system and method for training a model to be used for semantic segmentation of images, comprising: a‑obtaining (S01) a first plurality of foggy images (101), b‑training (S02) a classification model for estimating fog density, c‑classifying (S03) a second plurality of images (101) having light fog, d‑obtaining (S04) a third plurality of foggy images (103) having light fog, e‑training (S05) a semantic segmentation model using the third plurality of foggy images, f‑applying (S06) the semantic segmentation model to the second plurality of foggy images (102) to obtain semantic segmentation (102′), g‑obtaining (S07) a fourth plurality of foggy images (104) having thick fog, and h‑training (S08) using previously obtained foggy images.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more particularly to semantic segmentation of images in which fog may be present. Background Art

[0002] Semantic segmentation is a method for determining the types of objects that are visible (or partially visible) in an image. For example, an image may be acquired by a camera installed in a vehicle. Semantic segmentation of this image allows distinguishing other vehicles, pedestrians, lanes of traffic, etc. Therefore, semantic segmentation is particularly useful for self-driving vehicles and other types of automated systems.

[0003] Semantic segmentation methods typically use models such as neural networks or convolutional neural networks to perform segmentation. These models must be trained.

[0004] Training a model typically involves feeding known images into the model. For these images, the predetermined semantic segmentation is known (the operator may have prepared the predetermined semantic segmentation for each image by annotating the images). The output of the model is then evaluated taking into account the predetermined semantic segmentation, and if the output of the model differs from the predetermined semantic segmentation of the image, the parameters of the model are adjusted.

[0005] Therefore, in order to train a semantic segmentation model, a large number of images and predetermined semantic segmentations are required.

[0006] It has been observed that adverse weather conditions can cause visibility problems for both drivers and automated systems. Despite the continuous improvement of sensors and computer vision algorithms, they often fail to work in many outdoor conditions. This prevents the practical use of these automated systems: it is unthinkable that a vehicle will avoid bad weather, and the vehicle must be able to distinguish between different objects in every possible weather condition.

[0007] Fog is a classic example of adverse weather that can severely reduce the visibility of a scene depending on the density of the fog.

[0008] Therefore, it is desirable to train a semantic segmentation model with foggy images (images on which fog appears).

[0009] However, obtaining semantic segmentation data on foggy images (e.g., foggy pictures taken by a camera) is particularly difficult and time-consuming, especially when operators have to manually annotate the foggy images before feeding them to the model.

[0010] To overcome this problem, generating synthetic foggy images has been proposed.

[0011] In fact, obtaining semantic segmentation is easier to perform on clean images, and semantic segmentation of clean images can be used to train a model that has been fed with images with synthetic fog.

[0012] The prior art document "Semantic foggy scene understanding with synthetic data" (International Journal of Computer Vision, Sakaridis, C., Dai, D., Van Gool, L., 2018) discloses a method for adding synthetic fog to clear weather images. These images can then be used to train a neural network. The solution disclosed in this document is unsatisfactory for semantic segmentation of images presenting dense fog.

[0013] A primary object of the present invention is to provide methods and systems that overcome the deficiencies of currently available systems and methods. Summary of the invention

[0014] The present invention overcomes one or more deficiencies of the prior art by proposing a method for training a model to be used for semantic segmentation of an image, the method comprising:

[0015] a- obtaining a first plurality of foggy images with synthetic fog of varying density (e.g., random fog density on each image),

[0016] b- using the first plurality of foggy images and the associated different densities to train a classification model for estimating fog density,

[0017] c-classifying the second plurality of images according to the fog density in the images using the classification model of step b to obtain a second plurality of foggy images, wherein each foggy image has a fog density included within the first fog density threshold and the second fog density threshold,

[0018] d-obtaining a third plurality of foggy images, wherein each foggy image has synthetic fog including a fog density within a first fog density threshold and a second fog density threshold,

[0019] e- training a semantic segmentation model using the third plurality of foggy images and the predefined semantic segmentation of the third plurality of images,

[0020] f- applying the semantic segmentation model of step e to the second plurality of foggy images to obtain semantic segmentation of the second plurality of foggy images,

[0021] g-obtaining a fourth plurality of foggy images, wherein each foggy image has synthetic fog having a fog density included within a third fog density threshold and a fourth fog density threshold, both of which are greater than the first fog density threshold and the second fog density threshold,

[0022] h - Train the semantic segmentation model from step e using:

[0023] - a fourth plurality of foggy images and a predefined semantic segmentation of the fourth plurality of foggy images,

[0024] - Semantic segmentation of the second plurality of foggy images and the plurality of foggy images obtained in step f.

[0025] In this application, a predefined semantic segmentation is a semantic segmentation that can be obtained through operator annotations. Therefore, when referring to a predefined semantic segmentation, it means that the corresponding (foggy) image can be used to train a model or neural network. Because the operator may have prepared annotations that guide the semantic segmentation, these annotations may be of high quality.

[0026] Therefore, using the predefined semantic segmentation training model in step e can provide good training for the semantic segmentation model.

[0027] Because both the third and fourth fog density thresholds are greater than the first and second fog density thresholds, the first and second density thresholds may correspond to light fog or at least lighter fog than the third and fourth fog density thresholds corresponding to dense fog or at least denser fog.

[0028] Those skilled in the art may select a fog density threshold value to distinguish between light fog and dense fog according to the application.

[0029] It should be noted that obtaining an image of synthetic fog with a selected fog density can be performed according to the method disclosed in the document "Semantic Fog Scene Understanding Using Synthetic Data" (International Journal of Computer Vision, Sakaridis, C., Dal, D., Van Gool, L., 2018). This method allows the use of parameters to select the density of the fog: thus, the model of step b can use the parameters for adding synthetic fog.

[0030] The images of the second plurality of images do not contain synthetic fog, they may be images of scenes that may contain real fog. Using the trained model of step b, only images containing real light fog may be selected. Since the model of step e has been trained using the predefined semantic segmentation of images with synthetic light fog, and since the second plurality of foggy images only contains images with light fog, the semantic segmentation obtained in step f is also of good quality.

[0031] Therefore, the data input to the model in step h is of good quality and allows the model to adapt to perform semantic segmentation on thicker fog.

[0032] Additionally, it should be noted that synthetic fog (and especially dense synthetic fog) may contain artifacts. Therefore, it is interesting to adjust the training using real fog (the second plurality of foggy images) in step h.

[0033] According to a specific embodiment, step h comprises inputting a mixture of images from the second plurality of foggy images and images from the fourth plurality of foggy images into the semantic segmentation model of step e so as to obtain an image stream to be fed to the model, wherein the image stream comprises a larger proportion of images from the fourth plurality of foggy images than images from the second plurality of foggy images.

[0034] The inventors have observed that, since the predefined semantic segmentations of the fourth plurality of foggy images may be of good quality (eg, because they may be obtained by operator annotations prior to performing the method), it is preferred to use a larger number of images from the plurality of images.

[0035] According to a particular embodiment, step h comprises minimizing the following value:

[0036]

[0037] in:

[0038] l is the number of images in the fourth plurality of foggy images,

[0039] x i ″ is the image with index i in the fourth plurality of foggy images,

[0040] y i is the image x i ″Predefined semantic segmentation,

[0041] L(x, y) is the cross entropy loss function of x and y,

[0042] Φ″(x i ″) is the output of the semantic segmentation model,

[0043] μ is the number of images in the second multiple of foggy images,

[0044] Where w is a predefined weight,

[0045] is an image in the second plurality of foggy images, and

[0046] is the image x obtained in step f i "Semantic segmentation of .

[0047] In the above equation, the loss L(x, y) is calculated after each processing of the image, and the parameters of the model are adjusted to minimize this loss.

[0048] The training eventually results in minimizing the above equation by mixing images from the fourth plurality of foggy images and the second plurality of foggy images in a ratio that may be 1:w (a predefined weight, for example equal to 1 / 3):

[0049]

[0050] According to certain embodiments, in steps a, d, and g, obtaining a foggy image of the first, third, or fourth plurality of foggy images comprises:

[0051] - obtain the depth map of the initial image,

[0052] - obtain a predefined semantic segmentation of the initial image,

[0053] - make a transmittance map of the initial image, where for each pixel of the initial image, the transmittance decreases as the depth of the pixel in the depth map increases,

[0054] - filtering the transmittance map by considering a predefined semantic segmentation of the image to obtain a filtered transmittance map,

[0055] - Use the filtered transmittance map to simulate the appearance of fog on the initial image to obtain a foggy image.

[0056] The inventors of the present invention have observed that by using a predefined semantic segmentation of the image, the transmittance map better reflects the transmittance of a real foggy image (obtained by acquiring images of a foggy scene using a camera).

[0057] It should be noted that in this application, the (filtered) transmittance map determines the amount of scene radiance that can reach the camera (viewpoint of the image). Transmittance maps are notably disclosed in the document "Semantic Fog Understanding Using Synthetic Data". The transmittance map can have the same resolution as the image: the pixels of the image have associated transmittance values. Similarly, the depth map can also have the same resolution as the image.

[0058] The refinement of the transmittance map with decreasing transmittance as depth increases provides a first approximation of the fog. This first approximation can be improved by subsequent filtering steps.

[0059] It should be noted that transmittance map and radiance are terms known to those skilled in the art and are further described by way of example in "Single image haze removal using dark channel prior" (He, K., Sun, J., Tang, X., IEEE Transactions on Pattern Analysis and Machine Intelligence 33(12) (2011) 2341-2353) and "Bayesian defogging" (Nishlno, K., Kratz, L., Lombardi, S., International Journal of Computer Vision 98(3) (2012) 263-278).

[0060] According to certain embodiments, filtering the transmittance map comprises applying a bilateral filter.

[0061] Therefore, the initial transmittance of the pixel is denoised and enhanced.

[0062] Bilateral filtering allows smoothing of transmittance while preserving edges and, in the present invention, edges between separate semantic objects.

[0063] According to a specific embodiment, the bilateral filter is defined as:

[0064]

[0065] in:

[0066] p and q are the pixels of the original image,

[0067] N(p) is the neighbor of p,

[0068] t(p) is the transmittance of pixel p in the filtered transmittance map,

[0069] is the transmittance of pixel q in the transmittance map,

[0070] is the spatial Gaussian kernel,

[0071] is the color gamut Gaussian kernel,

[0072] δ is the Kronecker delta,

[0073] μ is the predetermined weight,

[0074] h(p) and h(q) are the semantic labels of pixels p and q, respectively, and

[0075] J(q) and J(p) are the color values ​​in the CIELAB domain, respectively.

[0076] The inventors of the present invention have observed that the above equations can be easily implemented, thus facilitating the implementation of the present invention. More precisely, a bilateral grid as disclosed in the document "A fast approximation of the bilateral filter using a signal processing approach" (International Journal of Computer Vision, Paris, S., Durand, F., 2009) can be used.

[0077] For example, it is possible to generate two separate bilateral grids, one for each semantic and color domain, and then operate on each grid separately to perform filtering by combining the final results. Thus, a 3-dimensional grid (pixel position and label) and a 5-dimensional (pixel position and three color components) for the semantic domain are used instead of a single joint 6-dimensional grid (which would increase computation time).

[0078] According to a specific embodiment, the transmittance map is defined as:

[0079]

[0080] in:

[0081] is the transmittance of pixel q in the transmittance map,

[0082] β is the attenuation coefficient, and

[0083] l(q) is the depth of pixel q in the depth map.

[0084] This transmittance map allows to simulate homogeneous fog in a first approximation. The attenuation coefficient β can be chosen to reduce visibility and increase the density of the fog as a function of depth.

[0085] According to a specific embodiment, wherein using the filtered transmittance map to simulate the appearance of fog on the initial image comprises: calculating the color value I(x) of each pixel as:

[0086] I(x)=R(x)+L(1-t(x))

[0087] in:

[0088] x is the pixel position,

[0089] R(x) is the color value of the image,

[0090] L(1-t(x)) is the atmospheric light, and

[0091] t(x) is the transmittance of pixel x in the filtered transmittance map.

[0092] According to a specific embodiment, a depth-by-stereo method and an optional outlier suppression method are used to obtain the depth map.

[0093] Outlier suppression methods may include suppression of wells.

[0094] Stereo depth methods may include acquiring two images using two cameras that are horizontally spaced a certain distance apart.

[0095] The outlier suppression method may include suppression of missing values ​​in the depth map obtained from the stereo depth method. These holes are caused by failures of the initial stereo depth method due to occlusions: areas that are visible on one camera but occluded in another camera used for stereo depth (which results in missing depth values). The present invention also proposes a system for training a model to be used for semantic segmentation of an image, comprising:

[0096] Module A, for obtaining a first plurality of foggy images with synthetic fog of different densities,

[0097] Module B, using the first plurality of foggy images and the associated different densities for training a classification model for estimating fog density,

[0098] module C, for classifying the second plurality of images using the classification model of module B according to the fog density in the images, so as to obtain a second plurality of foggy images, each of the second plurality of foggy images having a fog density included within a first fog density threshold and a second fog density threshold,

[0099] Module D, for obtaining a third plurality of foggy images, each of the third plurality of foggy images having synthetic fog having a fog density within a first fog density threshold and a second fog density threshold,

[0100] module E for training a semantic segmentation model using a third plurality of foggy images and the predefined semantic segmentations of the third plurality of images,

[0101] Module F, for applying the semantic segmentation model of step e to the second plurality of foggy images to obtain semantic segmentation of the second plurality of foggy images,

[0102] Module G, for obtaining a fourth plurality of foggy images, each foggy image having synthetic fog with a fog density included within a third fog density threshold and a fourth fog density threshold, both of which are greater than the first fog density threshold and the second fog density threshold,

[0103] Module H is used to train the semantic segmentation model of step e using the following:

[0104] - a fourth plurality of foggy images and a predefined semantic segmentation of the fourth plurality of foggy images,

[0105] - Semantic segmentation of the second plurality of foggy images and the plurality of foggy images obtained by module F.

[0106] The system may be configured to perform all embodiments of the method as described above.

[0107] In a specific embodiment, the steps of the above method are determined by computer program instructions.

[0108] Therefore, the present invention also relates to a computer program for performing the steps of the method as described above, when the computer program is executed by a computer.

[0109] The program may use any programming language and take the form of source code, object code, or an intermediate code between source code and object code, such as a partially compiled form or any other desired form.

[0110] The invention is also directed to a computer-readable information medium comprising instructions of a computer program as described above.

[0111] The information medium may be any entity or device capable of storing a program. For example, the medium may include a storage component such as a ROM (e.g., a CD ROM or a microelectronic circuit ROM), or a magnetic storage component such as a floppy disk (floppy disk) or a hard disk.

[0112] Alternatively, the information medium may be an integrated circuit in which the program is incorporated, the circuit being suitable for executing the method in question or for use in its execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] How the invention may be implemented will now be described by way of example with reference to the accompanying drawings, in which:

[0114] - Figure 1 is a block diagram of an exemplary method; and

[0115] - Figure 2 is a schematic diagram of an exemplary system. DETAILED DESCRIPTION

[0116] exist Figure 1 The method used to train the model is shown in .

[0117] Initially, the model may be a neural network or a convolutional neural network, which may be conceived to perform semantic segmentation on images. However, initially, the model has not been trained to perform semantic segmentation on foggy images.

[0118] The images that can be processed by the model (after being fully trained) can be pictures taken by an image sensor. Multiple objects (preferably of different types that may or may not overlap) can be visible on these images.

[0119] For example, the image shows a scene that might be visible from a vehicle on a road (eg, in a street).

[0120] In a first step S01 , a first plurality of foggy images having different fog densities are synthesized.

[0121] This step may be performed using multiple images that have no fog on them.

[0122] In order to have realistic synthetic fog on these images, synthetic fog may be added to each of the multiple initial images as follows.

[0123] The initial image is in RGB image. To perform the method, a depth map is obtained for the image. Preferably, a stereo depth method and an outlier suppression method are used to obtain the depth map. In other words, two cameras can be used to acquire the image, typically a left camera and a right camera (the actual image to be processed can be a left image or a right image). On the depth map, known methods (for example, the method disclosed in the document "Semantic Fog Understanding Using Synthetic Data" (International Journal of Computer Vision, Sakaridis, C., Dai, D., Van Gool, L., 2018)) can be used to remove outliers.

[0124] In addition, for processing the initial image, it is proposed to use a predefined semantic segmentation of the image. This semantic segmentation of the image may have been prepared by an operator annotating each part (group of pixels) of the image in order to indicate their type. For example, this results in a segmentation of the image indicating whether an object is a vehicle, a pedestrian, a road, a lane, etc.

[0125] The segmentation can be expressed mathematically as a function h:

[0126] h:P→{1,...,C}

[0127] in:

[0128] p is the discrete domain of pixel locations,

[0129] C is the total number of semantic classes (or types) in the scene (or image).

[0130] First, elaborate the transmittance map. This step can include adding "synthetic fog" using the following equation:

[0131]

[0132] in:

[0133] is the transmittance of pixel q in the transmittance map,

[0134] β is the attenuation coefficient, and

[0135] l(q) is the depth of pixel q in the depth map.

[0136] The coefficient β is known from the field of meteorology. In practice, the visibility of an object (also referred to by the acronym "MOR: Meteorological Optical Range") is defined as the maximum distance from the camera (or viewpoint) at which the transmittance is greater than or equal to 0.05. This means that:

[0137]

[0138] Therefore, the attenuation coefficient can be selected to be greater than or equal to 2.996×10 -3 m -1 , which corresponds to light fog.

[0139] The inventors observed that the transmittance graph was not satisfactory at this stage.A further filtering step was performed.

[0140] In this step, semantic segmentation of the image is considered. For example, filtering the transmittance map may include using a bilateral filter.

[0141] This can be achieved using the following equation:

[0142]

[0143] in:

[0144] p and q are the pixels of the image,

[0145] N(p) is the neighbor of p,

[0146] t(p) is the transmittance of pixel p in the filtered transmittance map,

[0147] is the transmittance of pixel q in the transmittance map,

[0148] is the spatial Gaussian kernel,

[0149] is the color gamut Gaussian kernel,

[0150] δ is the Kronecker function,

[0151] μ is the predetermined weight,

[0152] h(p) and h(q) are the semantic labels of pixels p and q, respectively, and

[0153] J(q) and J(p) are the color values ​​in the CIELAB domain, respectively.

[0154] Typical values ​​for the parameters might be:

[0155] σ s =20

[0156] σ c =10

[0157] μ=5

[0158] Then, fog is simulated on the image. Using the filtered transmittance map to simulate the appearance of fog on the image involves calculating the color value I(x) of each pixel as:

[0159] J(x)=R(x)+L(1-t(x))

[0160] in:

[0161] x is the pixel position,

[0162] R(x) is the color value of the image,

[0163] L(1-t(x)) is the atmospheric light, and

[0164] t(x) is the transmittance of pixel x in the filtered transmittance map.

[0165] It should be noted that R(x) is a value having three components in the red-green-blue space, each of which is an integer between 0 and 255.

[0166] Preferably, the atmospheric light is considered to be constant, for example the atmospheric light is a constant value estimated from the image. For example, the pixel with the highest intensity value may be considered to be the atmospheric light.

[0167] The image I(x) is also represented in red-green-blue space and is called a foggy image.

[0168] Thus, a first plurality of foggy images 100 can be obtained. By using various values ​​for the coefficient β, for example, greater than 2.996×10 -3 m -1 Random values ​​of , synthetic fog of various densities can be seen on the image.

[0169] In step S02, a model for estimating fog density is trained using the first plurality of foggy images 100. The model is configured to receive an image as input and output a value (eg, coefficient β) associated with the image.

[0170] Because the values ​​of the coefficients β for the images in the first plurality of foggy images are known, the network can be trained by minimizing the difference between the output of the model and the values ​​of the coefficients β used to generate synthetic fog.

[0171] In step S03 , the second plurality of images 101 are classified using the model trained in step S02 .

[0172] Preferably, the second plurality of images 101 include images containing real fog. In other words, these images are photographs taken outdoors on which real fog with different fog densities may exist.

[0173] Before performing the classification, a first fog density threshold and a second fog density threshold are defined such that images showing a light fog density are included within the first fog density threshold and the second fog density threshold.

[0174] A person skilled in the art will be able to determine these thresholds on the output of the model of step S02.

[0175] Using the model and the two thresholds, only images having fog densities included within the first fog density threshold and the second fog density threshold in the second plurality of images 101 may be selected to obtain a second plurality of foggy images 102 .

[0176] Then, step S04 may be performed, in which a third plurality of foggy images 103 are obtained. The third plurality of foggy images may be obtained using the same method used in step S01. The fog density of an image in the third plurality of foggy images is selected so as to be contained within the first fog density threshold and the second fog density threshold. Thus, in the third plurality of foggy images, the image presents light fog.

[0177] In the figure, a semantic segmentation 103' of each image of the third plurality of foggy images is also represented. This segmentation is predefined and may have been obtained before the annotation step by an operator.

[0178] The third plurality of foggy images 103 and the corresponding predefined semantic segmentation are then used in step S05 where a semantic segmentation model is trained.

[0179] The training is performed by inputting each image of the third plurality of foggy images 103 to the model and comparing the output of the model to a predefined semantic segmentation of the image in order to adjust the parameters of the model and train the model.

[0180] Then, in step S06, the second plurality of foggy images 102 may be input to the training model of step S05. For each image, a semantic segmentation 102' is obtained.

[0181] In step S07 , a fourth plurality of foggy images 104 is obtained.

[0182] The fourth plurality of foggy images may be obtained using the same method used in step S01. The fog density of the images in the fourth plurality of foggy images is selected so as to be contained within a third fog density threshold and a fourth fog density threshold, both of which are greater than the first fog density threshold and the second fog density threshold.

[0183] The third and fourth fog density thresholds may be selected to illustrate dense fog.

[0184] In the figure, semantic segmentations 104' of a fourth plurality of foggy images are shown. These semantic segmentations may have been used for the generation of synthetic fog.

[0185] In the final step S08, the model trained (or pre-trained) in step S05 is trained (or further trained) using the following:

[0186] - a fourth plurality of foggy images 104 and a predefined semantic segmentation 104' of the fourth plurality of foggy images,

[0187] - A semantic segmentation 102' of the second plurality of foggy images 102 and the plurality of foggy images obtained in step S06.

[0188] Step S08 comprises inputting a mixture of images from the second plurality of foggy images 102 and images from the fourth plurality of foggy images 104 into the semantic segmentation model of step S05 so as to obtain an image stream to be fed to the model, wherein the image stream comprises a larger proportion of images from the fourth plurality of foggy images than images from the second plurality of foggy images.

[0189] According to a particular embodiment, step h comprises minimizing the following value:

[0190]

[0191] in:

[0192] l is the number of images in the fourth plurality of foggy images,

[0193] x i ″ is the image with index i in the fourth plurality of foggy images,

[0194] y i is the image x i ″Predefined semantic segmentation,

[0195] L(x, y) is the cross entropy loss function of x and y,

[0196] Φ″(x i ″) is the output of the semantic segmentation model,

[0197] μ is the number of images in the second multiple of foggy images,

[0198] where w is a predefined weight (e.g., equal to 1 / 3),

[0199] is an image in the second plurality of foggy images, and

[0200] is the image x obtained in step f i "Semantic segmentation of .

[0201] It should be noted that semantic segmentation is easier to perform on images with light fog than on images with dense fog. Therefore, a model trained on synthetic data (e.g., synthetic fog) can be generalized to real data if both data sources contain light fog rather than dense fog.

[0202] Likewise, dense synthetic fog and light real fog reflect different and complementary characteristics: on the one hand, dense synthetic fog has similar visibility impairments as real dense fog, but may include artifacts; on the other hand, light real fog captures the real non-uniform and spatially varying structure of fog, but with a different density.

[0203] Therefore, it is recommended to use both for training models.

[0204] refer to Figure 1 The steps of the described method can be determined by computer program instructions. These instructions can be determined by, for example, Figure 2 The processor of the system shown executes.

[0205] In the figure, a system 200 for training a model is shown. The system 200 (which may be a computer) includes a processor 201 and a non-volatile memory 202.

[0206] In the non-volatile memory 202, a set of instructions is stored, and the set of instructions includes instructions to execute the method for training a model, and in more detail:

[0207] - instructions 203A, to obtain a first plurality of foggy images having synthetic fog of different densities,

[0208] - instructions 203B to train a classification model for estimating fog density using the first plurality of foggy images and the associated different densities,

[0209] - instructions 203C to classify the second plurality of images using the classification model of instructions 203B according to the fog density in the images so as to obtain a second plurality of foggy images, each foggy image in the second plurality of foggy images having a fog density included within the first fog density threshold and the second fog density threshold,

[0210] - instruction 203D, to obtain a third plurality of foggy images, each foggy image of the third plurality of foggy images having a synthetic fog having a fog density included within a first fog density threshold and a second fog density threshold,

[0211] - instructions 203E, to train a semantic segmentation model using a third plurality of foggy images and the predefined semantic segmentation of the third plurality of images,

[0212] - instruction 203F to apply the semantic segmentation model of instruction 203E to the second plurality of foggy images to obtain semantic segmentation of the second plurality of foggy images,

[0213] - instruction 203G, to obtain a fourth plurality of foggy images, each foggy image in the fourth plurality of foggy images having a synthetic fog having a fog density included within a third fog density threshold and a fourth fog density threshold, the third fog density threshold and the fourth fog density threshold both being greater than the first fog density threshold and the second fog density threshold,

[0214] - Instruction 203H to train the semantic segmentation model of instruction 203E using the following:

[0215] - a fourth plurality of foggy images and a predefined semantic segmentation of the fourth plurality of foggy images,

[0216] - A second plurality of foggy images and semantic segmentation of the plurality of foggy images obtained in instruction 203F.

[0217] Instructions 203A to 203H and processor 201 form eight modules respectively:

[0218] - a module A for obtaining a first plurality of foggy images with synthetic fog of different densities,

[0219] - a module B for training a classification model for estimating fog density using the first plurality of foggy images and the associated different densities,

[0220] - a module C for classifying a second plurality of images using the classification model of module B according to the fog density in the images so as to obtain a second plurality of foggy images, each foggy image in the second plurality of foggy images having a fog density included within a first fog density threshold and a second fog density threshold,

[0221] - a module D for obtaining a third plurality of foggy images, each foggy image of the third plurality of foggy images having a synthetic fog having a fog density within a first fog density threshold and a second fog density threshold,

[0222] - a module E for training a semantic segmentation model using a third plurality of foggy images and the predefined semantic segmentation of the third plurality of images,

[0223] - a module F for applying the semantic segmentation model of step e to a second plurality of foggy images to obtain semantic segmentation of the second plurality of foggy images,

[0224] - a module G, for obtaining a fourth plurality of foggy images, each of the fourth plurality of foggy images having a synthetic fog having a fog density within a third fog density threshold and a fourth fog density threshold, wherein the third fog density threshold and the fourth fog density threshold are both greater than the first fog density threshold and the second fog density threshold,

[0225] - Module H, which is used to train the semantic segmentation model of step e using:

[0226] - a fourth plurality of foggy images and a predefined semantic segmentation of the fourth plurality of foggy images,

[0227] - Semantic segmentation of the second plurality of foggy images and the plurality of foggy images obtained by module F.

[0228] It should be noted that obtaining an image with synthetic fog can be performed by using the method disclosed in an international patent application filed by the same applicant as the present application on the same day and entitled "Method and system for processing an image to obtain a foggy image", which is incorporated into the present application in its entirety.

[0229] Although the invention has been described above with reference to certain specific embodiments, it is to be understood that the invention is not limited to the particularities of the specific embodiments. Many variations, modifications and developments may be made in the above-described embodiments within the scope of the appended claims.

Claims

1. A method for training a model for semantic segmentation of an image, comprising: a- obtaining a first plurality of foggy images (100) having synthetic fog of different densities, b- using said first plurality of foggy images and the associated different densities to train a classification model for estimating fog density, c- classifying a second plurality of images (101) using the classification model of step b according to the density in the images to obtain a second plurality of foggy images (102), each foggy image in the second plurality of foggy images having a density included within a first fog density threshold and a second fog density threshold, d- obtaining a third plurality of foggy images (103), each foggy image in the third plurality of foggy images having synthetic fog having a fog density within the first fog density threshold and the second fog density threshold, e- training a semantic segmentation model using the third plurality of foggy images and the predefined semantic segmentation (103') of the third plurality of foggy images, f- applying the semantic segmentation model of step e to the second plurality of foggy images (102) to obtain a semantic segmentation of the second plurality of foggy images (102'), g-obtaining a fourth plurality of foggy images (104), each of the fourth plurality of foggy images having synthetic fog with a fog density included within a third fog density threshold and a fourth fog density threshold, wherein the third fog density threshold and the fourth fog density threshold are both greater than the first fog density threshold and the second fog density threshold, h- Train the semantic segmentation model from step e using: - the fourth plurality of foggy images (104) and a predefined semantic segmentation (104') of the fourth plurality of foggy images, - the second plurality of foggy images (102) and the semantic segmentation (102') of the second plurality of foggy images obtained in step f.

2. The method according to claim 1, wherein: Step h comprises inputting a mixture of images from the second plurality of foggy images and images from the fourth plurality of foggy images into the semantic segmentation model of step e to obtain an image stream to be fed to the model, wherein the image stream comprises a larger proportion of images from the fourth plurality of foggy images than images from the second plurality of foggy images.

3. The method according to claim 1 or 2, wherein: Step h consists in minimizing the following values: in: l is the number of images in the fourth plurality of foggy images, x i ″ is an image with index i in the fourth plurality of foggy images, y i is the image x i ″The predefined semantic segmentation, L(x, y) is the cross entropy loss function of x and y, Φ″(x i ″) is the output of the semantic segmentation model, μ is the number of images in the second plurality of foggy images, Where w is a predefined weight, is an image of the second plurality of foggy images, and is the image x obtained in step f i ”The semantic segmentation of 4. The method according to claim 1 or 2, wherein: In steps a, d and g, obtaining a foggy image for the first, third or fourth plurality of foggy images comprises: - obtain the depth map of the initial image, - obtaining a predefined semantic segmentation of said initial image, - making a transmittance map of the initial image, wherein for each pixel of the initial image, the transmittance decreases when the depth of the pixel in the depth map increases, - filtering said transmittance map by taking into account said predefined semantic segmentation of said image to obtain a filtered transmittance map, - simulating the appearance of fog on said initial image using said filtered transmittance map to obtain said foggy image.

5. The method according to claim 4, wherein: Filtering the transmittance map includes applying a bilateral filter.

6. The method according to claim 5, wherein: The bilateral filter is defined as: in: p and q are the pixels of the original image, N(p) is the neighborhood of p, t(p) is the transmittance of pixel p in the filtered transmittance map, is the transmittance of pixel q in the transmittance map, is the spatial Gaussian kernel, is the color gamut Gaussian kernel, δ is the Kronecker function, μ is the predetermined weight, h(p) and h(q) are the semantic labels of pixels p and q, respectively, and J(q) and J(p) are the color values ​​in the CIELAB domain, respectively.

7. The method according to claim 4, wherein: The transmittance map is defined as: in: is the transmittance of pixel q in the transmittance map, β is the attenuation coefficient, and l(q) is the depth of pixel q in the depth map.

8. The method according to claim 4, wherein: Using the filtered transmittance map to simulate the appearance of fog on the initial image includes calculating the color value I(x) of each pixel as: I(x)=R(x)+L(1-t(x)) in: x is the pixel position, R(x) is the color value of the image, L(1-t(x)) is the atmospheric light, and t(x) is the transmittance of pixel x in the filtered transmittance map.

9. The method according to claim 4, wherein: The depth map is obtained using a stereo depth method and an outlier suppression method.

10. The method according to claim 5 or 6, wherein: The transmittance map is defined as: in: is the transmittance of pixel q in the transmittance map, β is the attenuation coefficient, and l(q) is the depth of pixel q in the depth map.

11. The method according to any one of claims 5 to 7, wherein: Using the filtered transmittance map to simulate the appearance of fog on the initial image includes calculating the color value I(x) of each pixel as: I(x)=R(x)+L(1-t(x)) in: x is the pixel position, R(x) is the color value of the image, L(1-t(x)) is the atmospheric light, and t(x) is the transmittance of pixel x in the filtered transmittance map.

12. The method according to any one of claims 5 to 8, wherein: The depth map is obtained using a stereo depth method and an outlier suppression method.

13. A system for training a model for semantic segmentation of an image, comprising: Module A, for obtaining a first plurality of foggy images (100) having synthetic fog of different densities, Module B, for training a classification model for estimating density using the first plurality of foggy images and the associated different densities, Module C, for classifying a second plurality of images (101) using the classification model of module B according to the density in the images to obtain a second plurality of foggy images (102), each of the second plurality of foggy images having a density included within a first fog density threshold and a second fog density threshold, Module D, for obtaining a third plurality of foggy images (103), each of the third plurality of foggy images having synthetic fog of a density within the first fog density threshold and the second fog density threshold, module E, for training a semantic segmentation model using the third plurality of foggy images and the predefined semantic segmentation (103') of the third plurality of foggy images, Module F, for applying the semantic segmentation model of module E to the second plurality of foggy images to obtain semantic segmentation of the second plurality of foggy images (102'), Module G, for obtaining a fourth plurality of foggy images (104), each of the fourth plurality of foggy images having synthetic fog of a density included within a third fog density threshold and a fourth fog density threshold, wherein the third fog density threshold and the fourth fog density threshold are both greater than the first fog density threshold and the second fog density threshold, Module H is used to train the semantic segmentation model of module E using the following: - the fourth plurality of foggy images (104) and a predefined semantic segmentation (104') of the fourth plurality of foggy images, - said second plurality of foggy images (102) and a semantic segmentation (102') of said second plurality of foggy images obtained by module F.

14. A recording medium readable by a computer and having a computer program recorded thereon, the computer program including instructions for executing the steps of the method according to any one of claims 1 to 12.

Images