Vehicle environment scene classification device

Abstract

A device for classifying a scene in the environment of a vehicle is proposed, comprising a classification unit using a neural network to classify a scene in response to a received image representing the scene, and a learning unit to train the neural network from a set of images stored in a database (3). The device includes: - an image preprocessing module (120) to extract a set of thumbnails from a received image based on a position criterion within the image and at least one criterion relating to an image parameter, - a classification module (122) to determine, for each thumbnail, a set of membership probabilities, each associated with a class from a set of predefined classes, and - an image annotation module (124) to determine an annotation corresponding to the image from the sets of probabilities determined for all the thumbnails. Abbreviated: Figure 2

Description

Title of the invention: Vehicle environment scene classification device technical field

[0001] The invention relates generally to image processing systems and in particular to a device and a method for scene classification of the environment of an automated vehicle.

[0002] Automation technologies for driving systems have experienced significant growth in recent years, particularly in the field of automated vehicles, such as autonomous vehicles, whether land-based, marine, or underwater. Autonomous navigation is made possible by real-time recognition of the vehicle's environment, which has the advantage of requiring very little hardware. In fact, generally, only an image acquisition device combined with a computing unit, whether onboard or external, is necessary. In existing solutions, the computing unit is configured to perform semantic classification of environmental scenes. To do this, the computing unit uses a semantic segmentation process that provides information on the class of each pixel in an image.This is a complex computer vision task that classically relies on deep neural networks trained for the semantic classification of scenes in the vehicle's environment.

[0003] Thus, when the vehicle moves in its environment, it is able to finely analyze the environment to detect obstacles, avoid them and optimize its movement in accordance with the environment in which it is evolving.

[0004] In current systems, neural network training is performed using a database containing thousands of semantically annotated images. The neural networks thus learn to automatically recognize the pixel class of an image based on these annotations.

[0005] However, a significant obstacle to the deployment of such systems is the high human cost associated with implementing these approaches, known as semantic segmentation. Indeed, such systems require manual annotation at the pixel level of each image, across the thousands of images in the database, to enable robust classification. These semantic annotations are necessary to implement the artificial intelligence training phase. Thus, deploying autonomous vehicles in unexplored environments requires substantial manual re-annotation work, leading to significant delays in the exploitation of these new environments. Furthermore, the precise annotation of these Images are sometimes subjective due to the environment being studied where the boundary between two classes is not clearly defined (for example, the boundary between sand and the water column in an underwater environment).

[0006] There is therefore a need for an improved device and method for classifying scenes in the environment of an automated vehicle.

[0007] General definition of the invention

[0008] The invention improves the situation by providing a device for classifying a scene in the environment of a vehicle, comprising a classification unit using a neural network configured to classify a scene in response to a received image representing the scene, and a learning unit configured to train the neural network from a set of images stored in a database. Advantageously, the classification device includes an image annotation unit comprising: - an image preprocessing module configured to extract a set of thumbnails from a received image based on a position criterion within the image and at least one criterion relating to an image parameter, - a classification module configured to determine, for each image, a set of membership probabilities, each associated with a class from a set of predefined classes, and - an image annotation module configured to determine an annotation corresponding to the image from the sets of probabilities determined for all the thumbnails, the annotation being passed to the database for storage in association with the image in the database.

[0009] According to one aspect of the invention, the image annotation module can be configured to merge the membership probabilities determined for all classes and for all imagelets and to reconstruct an annotation, pixel by pixel, for the received image, which provides the annotation.

[0010] In one embodiment, each set of probabilities determined for an image can be represented by a probability vector having a set of components corresponding to the membership probabilities, each associated with one of the classes.

[0011] In one embodiment, an image parameter can be an image frequency information, the criterion relating to the image parameter being a frequency level.

[0012] The image criterion used for extraction can be relative to ranges of values ​​of the image parameter.

[0013] In one embodiment, the preprocessing unit uses a fixed or variable step to implement image extraction by sliding window.

[0014] In one embodiment, the image annotation module can be configured to select, for each image pixel, the class having a maximum membership probability among the membership probabilities obtained for all the imagelets.

[0015] In one embodiment, the pre-processing module can be configured to implement, prior to determining the probability set for an image, an operation to resize the image based on the level of its image parameter.

[0016] In one embodiment, the pre-processing module can be configured to add, for each KxK-sized imagelet among the P imagelets extracted from the MxN-sized input image, the C membership probabilities obtained for the imagelet in a KxK-sized patch, corresponding to the imagelet, onto an MxNxC-sized matrix corresponding to the input image.

[0017] In one embodiment, the probability matrix Mp can be defined according to the equation:

[0018] Mp(i, j, n) = Mp(i, jn) + P(n)*!!^ '

[0019] where P denotes the probability vector P at the output of the classification unit and Æ the vector of the different dimensions of image extraction.

[0020] In one embodiment, the probability matrix Mp can be defined according to the equation:

[0021] j +

[0022] P denotes the probability vector P at the output of the classification unit, G(i, j) denotes a Gaussian window and # denotes the vector of the different dimensions of image extraction.

[0023] According to one aspect of the invention, the Gaussian window G(i, / ) can be defined by:

[0024] G(i, / ) = ^W(i, / ) [0, / ]

[0025] The annotation of the received image can then be determined by selecting, at each point of the probability matrix Mp, the index having the maximum value in the probability vector associated with each pixel.

[0026] A method for classifying a scene in the environment of a vehicle is further proposed, comprising a classification step using a neural network configured to classify a scene in response to a received image representing the scene, and a training step configured to train the neural network from a set of images stored in a database. Advantageously, the method comprises: - image pre-processing consisting of extracting a set of image fragments from a received image based on a position criterion within the image and at least one criterion relating to an image parameter, - a classification step consisting of determining, for each image, a set of membership probabilities, each associated with a class from among a set of predefined classes, and - an image annotation step consisting of determining an annotation corresponding to the image from the sets of probabilities determined for all the imagelets, the annotation being transmitted to the database for storage in association with the image in the database.

[0027] The embodiments of the invention thus make it possible to annotate the pixels of the images in the training database, with reduced complexity and processing time. Brief Description of the Figures

[0028] Other features, details and advantages of the invention will become apparent from the description given with reference to the accompanying drawings provided by way of example, which represent, respectively:

[0029] [Fig-1] - Fig. 1 schematically represents a vehicle implementing a control system, according to the embodiments of the invention.

[0030] [Fig.2] - The [Fig.2] schematically represents the structure of the annotation unit according to embodiments of the invention.

[0031] [Fig.3] Fig.3 describes the image annotation process implemented by the image annotation unit according to embodiments of the invention.

[0032] [Fig.4] Fig.4 represents an example of a probability matrix and patches, according to an example embodiment.

[0033] [Fig.5] The [Fig.5] illustrates a change in the size of the sliding window depending on the class.

[0034] [Fig.6] Fig.6 illustrates the method of determining image annotations, in an embodiment where a single-scale approach is used.

[0035] [Fig.7] Fig.7 illustrates the process of determining image annotations, in an embodiment where a multi-scale approach is used.

[0036] [Fig.8] Fig.8 illustrates the image annotation determination process, applied to a sequence of images, using the pre-trained classifier.

[0037] Detailed description of the application

[0038] Fig. 1 schematically represents a vehicle 100 implementing a vehicle control system 10, according to embodiments of the invention.

[0039] The control system 10 is configured to recognize in real time the environment of the vehicle 100 and to adapt the driving of the vehicle 100 dynamically according to the recognition carried out in order to allow automated navigation, such as autonomous navigation.

[0040] The control system 10 includes a vehicle environment scene classification device 1 (also more simply called a "classification system") and a perception system 2.

[0041] The classification device 1 is configured to perform a semantic classification of the scenes in the environment of vehicle 100 from the images detected by the perception system 2 of vehicle 100.

[0042] The vehicle 100 can be an automated or semi-automated vehicle capable of moving in an environment, such as a land, sea, or underwater vehicle. The environment can therefore be a land, sea, or underwater environment.

[0043] The environment of vehicle 100 is associated with a set of predefined classes, depending on the application of the invention, such as, for example, the following set of classes: [roadway, sidewalk, vegetation, vehicle]. In embodiments, the classes associated with the environment can be modified (addition, deletion of classes, for example) by implementing a system relearning.

[0044] The control system 10 can be a driver assistance system configured to optimize vehicle driving and ensure its safety (obstacle detection for example) based on information determined by the perception system 2.

[0045] The perception system 2 comprises a set of detection devices arranged on the vehicle 100 and configured to provide signals representative of an image of the environment in which the vehicle 100 is moving. The detection devices 20 may include one or more cameras. A camera may be equipped with a photosensitive sensor sensitive to radiation from a light spectrum including infrared and / or visible light. The cameras may further include one or more acoustic and / or laser cameras (in applications of the invention in underwater environments, for example).

[0046] The detection devices 20 can be arranged on the vehicle so as to acquire images of the vehicle's environment located in front of the vehicle 100, to the sides, above, and / or below the vehicle 100, depending on the application of the invention. For example, in a terrestrial environment, the detection devices can be mounted on the front of the vehicle and / or inside the vehicle's passenger compartment and / or on the vehicle's roof. The detection devices may further include a lidar (Laser Detection and Ranging) system and / or a radar system.

[0047] Other types of sensors can be used by the perception system 2 such as an ultrasonic sensor, a steering wheel angle sensor, a wheel speed sensor, a brake pressure sensor, a yaw rate and lateral acceleration sensor, or a combination thereof.

[0048] The perception system 2 is configured to determine images of the environment from the information detected by the set of detection devices. In some embodiments, the perception system 2 may further include a processing device 22 configured to implement multi-sensor fusion algorithms capable of combining information from the different detection devices 20 to determine the images of the environment.

[0049] The classification device 1 includes a learning unit 11 for pre-training the neural network used on real datasets stored in an image database 3. The database contains for each data an image / annotation pair.

[0050] The classification device 1 further includes an image annotation unit 12 configured to automatically annotate one or more images from the image database 3 used for the training phase. The image annotation unit 12 is thus configured to generate the annotations stored in the image database 3. An image annotation is represented by a matrix of the same size as the image where each element has a value between 0 and Nl, with N denoting the number of classes (the number of pixels belonging to a class).

[0051] Such annotations are used by the learning unit 11 to train the neural network(s) used.

[0052] The classification device 1 includes a classification unit 14 configured to semantically segment each image detected by the perception system 2 in real time, using the previously trained neural network.

[0053] To facilitate understanding of the embodiments of the invention, definitions or concepts relating to neural networks are detailed below in relation to the embodiments of the invention.

[0054] A neural network comprises neurons interconnected by synapses, which can be implemented in the form of digital memories. A neural network can comprise a series of successive layers, including an input layer carrying the input signal and an output layer carrying the result of the prediction made by the neural network (network output), and one or more intermediate layers. Each layer of a neural network takes its inputs from the outputs of the preceding layer. The number of neurons in each layer is equal to the number of inputs to the neurons in the following layer. A layer The neural network data thus includes a set of neurons taking their inputs from the neurons of the previous layer.

[0055] The signals propagated into and out of the network layers can be numerical values ​​(information coded in the value of the signals), or electrical pulses in the case of pulse coding (information coded temporally according to the order of arrival of the pulses or according to the frequency of the pulses).

[0056] A neural network comprises a set of input data (or 'input coefficients') and output data (or 'output coefficients').

[0057] The output values ​​are calculated from the inputs and synaptic weights by applying an activation function to the input coefficients.

[0058] Each neuron in the neural network is configured to calculate a weighted sum of its input coefficients using a combination function and synaptic weights, then applying an activation function to the weighted sum obtained to produce its output:

[0059] The synaptic weights of a neural network are determined by learning in a learning phase implemented by the learning unit. Random values ​​are initially assigned to the weights of the neural network, and then a set of real data from database 3 is used to perform the learning. Learning a neural network consists of determining the optimal values ​​of the synaptic weights, for each neuron in the neural network, from the last layer of the network to the first, using a learning function.

[0060] The learning phase can implement a plurality of iterations of the learning function, each iteration comprising a forward propagation step and a backpropagation step to correct the errors between the outputs obtained in the forward propagation phase and the expected outputs for the input sample under consideration.

[0061] The learning phase thus makes it possible to compare the output obtained with the expected output (in the case of a supervised method), and based on this comparison, to update the links between the neurons represented by the synaptic weights to improve the final result.

[0062] In the forward propagation phase, input datasets from database 3 are used to implement the learning. Each dataset forms a sample x associated with desired (or expected) values. The signal corresponding to the input sample is propagated forward through the layers of the neural network, starting from the first layer, from layer (k-1) to the next layer (k) up to the last layer. In the phase of forward propagation, the activation function q> and synaptic weights connecting neurons from a previous layer (k-1) and a next layer (k) are used.

[0063] When forward propagation is complete, a result is obtained at the output.

[0064] In the backpropagation phase, any errors obtained by a Backpropagation occurs at the neuron's synapses and at the neurons connected to it. Backpropagation can, for example, be gradient backpropagation to modify synaptic weights based on their impact on the generated errors. Thus, synaptic weights that contribute to a large error can be modified more significantly than weights that have generated a smaller error.

[0065] The duration of the training phase may depend on the size of the database 3 storing the samples used for training and the size of the network.

[0066] After the training phase, a faster generalization phase is implemented by the classification unit 14. In the generalization phase, the weights learned from the training phase are used (static neural network in which the weights are fixed). Input data corresponding to the scene images detected in real time by the perception system 2 are presented to the neural network, which provides a response representing the probability of each pixel belonging to the predefined classes.

[0067] The control system 2 can implement a set of driving applications using the classification determined by the classification device 1 such as a lane change application to help the driver change lane or a collision warning application.

[0068] Fig. 2 schematically represents the structure of the annotation unit 12 according to embodiments of the invention.

[0069] The image annotation unit 12 is configured to determine an annotation to be assigned to a representative image of a scene of the environment to be analyzed, in response to the reception of this image determined by the perception system 2. The classification device 1 can then store the image in association with the determined annotation in the database 3.

[0070] The scene classification device 1 can thus perform a semantic annotation of the images allowing to produce a semantic segmentation of the environment of the vehicle 100 in a minimal time, even in the case of scenes still unexplored where no annotation is available.

[0071] In one embodiment, as illustrated in [Fig.2], the image annotation unit 12 may comprise: - an image preprocessing module 120 configured to preprocess images and extract sub-images (or imagelets) from the received image ('input image'); - a classification module 122 (also called "classifier") configured to perform a classification of the imagelets extracted by the preprocessing unit 102, using the neural network, to determine the probabilities of each imagelet ('thumbnail image' in English) belonging to each of the classes of the vehicle environment; - An annotation module 124 configured to merge the responses from the classification module 122 obtained for each image and to reconstruct a semantic annotation, pixel by pixel, for the complete image considered, which provides the annotation to be assigned to the received image.

[0072] The image preprocessing module 120 can be configured to collect local information from the image under consideration, according to several levels, and to extract a set of thumbnails based on an image position criterion and at least one image criterion related to an image parameter. An thumbnail can thus correspond to position information within the image and to image criteria defined from the scale levels of the image parameter under consideration (i.e., ranges of values ​​of the image parameter).

[0073] The image criterion used for image extraction can be related to different types of image parameters, such as, for example, image frequency, image texture, image contours.

[0074] The image parameter scale levels considered for extraction can be, for example, a scale pyramid of a given factor, such as, for example, a factor of 2. In an example of application of the invention where the image criterion used for extraction relates to the image parameter corresponding to the maximum image resolution, for a maximum resolution of 224x224 pixels, and a minimum resolution of 14x14 pixels, the factor 2 scale pyramid used includes the levels: 224x224; 112x112; 56x56; 28x28; and 14x14.

[0075] The maximum resolution may depend on the resolution of the perception system (2). The imagelets are extracted for each scale level.

[0076] The preprocessing unit 120 can take into account the low and high frequency information inherent to each of the scene classes of the environment under consideration. The preprocessing unit 120 can use a fixed step size to implement multi-scale sliding window extraction. The step size can be predefined. The step size can be applied to the pixel grid of the image (vertically and horizontally). The step size and scale levels allow the fineness of the semantic annotations that will be automatically generated to be configured.

[0077] The classifier 122 is configured to determine, for each extracted image, a set of membership probabilities, each associated with a class from among a set of predefined classes.

[0078] In one embodiment, the classifier 122 is configured to assign, to each image extracted by the image preprocessing module 120, a characteristic vector (also called a "probability vector") comprising components, each component being associated with one of the classes of the scene of the environment under consideration and having a value corresponding to the probability of the image belonging to the associated class. Thus, the probability vector assigned to an image represents the probability of the image belonging to each of the classes of the scene of the environment under consideration. The classifier 122 uses the neural network, which may be, in particular, a deep neural network, pre-trained beforehand on a suitable dataset, during a training phase, to enable robust classification.

[0079] The annotation module 124 is configured to merge the responses of the classifier 122 (represented by the probability vectors obtained for each image) so as to aggregate the membership probabilities obtained for each image, corresponding to different image positions and image criteria. The annotation module 124 is specifically configured to assign to each pixel of the initial image a probability vector indicating its degree of membership in the classes present in the scene under consideration. The annotation module 124 thus associates a semantic annotation with the image under consideration by selecting, for each image pixel, the class with the highest probability of membership. As used here, the term "annotation" is associated with an image and refers to a set of class information, each piece of class information being associated with a pixel of the image and corresponding to the determined highest probability class.

[0080] The image annotation unit 12 can then store the annotations determined in association with the corresponding image in the image database 3.

[0081] The learning unit 11 can then implement a learning of the semantic segmentation neural network not using annotations of the same type.

[0082] To carry out the learning, the learning unit 11 can use the annotations thus generated.

[0083] It should be noted that the annotations determined by the reconstruction unit 124 may not be perfect and may form a set of so-called degraded annotations. However, since the neural network construction of modules 11 and 14 is of the encoder / decoder type, this type of network will naturally smooth the input annotations and thus limit the impact of the imperfection of such degraded annotations.

[0084] The classifier 122 can be trained to recognize classes on imagelets of a given size, such as, for example, 224x224 images. The classifier 122 can be configured to predict the probabilities of belonging to predefined classes on each imagelet extracted from the image, at several scale levels. In a mode In particular, the 122 classifier can be configured to predict the majority class (i.e. having a maximum probability of belonging) on ​​each image extracted from the image, at several scale levels.

[0085] Such predictions can be obtained by applying an underlying multi-scale sliding window principle. The sliding window is a technique that allows an image to be analyzed by dividing it into small regions called windows and progressively moving this window across the entire image to perform a local analysis. The sliding window process comprises three steps: - A window initialization step is performed to determine its position and size. The window can be square, and its initial position can be located at the origin of the image. The window size depends on the scale set in the multi-scale approach. - a step of analyzing the pixels of the window by performing a classification of the window to extract its probabilities of belonging to the classes of the system. - a window movement step, after analyzing the current region. In this step, the window is moved by a certain number of pixels, called the "step". This step is repeated until the window has covered the entire image. The step can be fixed and equal to the resolution of the minimum scale (for example, for an image size of 14x14, the step is equal to 14).

[0086] A multi-scale approach allows for as many sliding windows as desired sizes (scale pyramid).

[0087] The fusion of the class predictions available for each pixel, performed by the reconstruction unit 124, makes it possible to obtain a semantic annotation of the image under consideration, which can be degraded. Each annotation thus obtained enriches the semantic annotation database 3 and can be used for the implementation of learning by the learning unit 11, which makes it possible to use a strong semantic segmentation neural network.

[0088] Figure 3 describes the image annotation process implemented by the image annotation unit according to embodiments of the invention.

[0089] At step 300, an input image of size MxN is received.

[0090] In step 302, the input image is pre-processed. This step includes the extraction sub-images (or thumbnails) of the input image. The image can be a color image. Each thumbnail can be associated with an image resolution.

[0091] In step 302, the features of the input image received in step 300 can be stored in an image memory (for example, an image catalog), in which the different imagelets extracted from the image are associated with fixed positions and image resolutions, which may be different. The imagelets can advantageously be defined according to several scale levels. Taking into account the multi-scale information of the imagelets allows the description of The image is analyzed both in terms of the textures present within it, characterizing each class, and in terms of the structures and contours visible in the image. In one embodiment, the process can, for example, implement a multi-frequency approach allowing the simultaneous extraction of low and high-frequency information.

[0092] In step 304, the probabilities of belonging of each extracted image to each of the vehicle environment classes are determined by implementing a classification (i.e. a class prediction) of the image based on the neural network, providing a probability vector for each class (this step can be implemented by the classifier 122).

[0093] Step 304 (the classification step) consists of determining, from the neural network previously trained on annotated images of a fixed size (e.g., 224x224), the probabilities of belonging to the different classes locally present in each image extracted in step 302, and the probability vector grouping these membership probabilities for the different classes. The neural network can be a deep neural network, such as, for example, an AlexNet or SqueezeNet. The neural network is used to determine the membership probabilities of each extracted image in each of the C classes, where C is the number of classes present in the considered scene of the environment. These classes can be predefined during the training (i.e., learning) of the image classification network.In some embodiments, step 304 may include, before determining the probability vector of an image, an operation of resizing an image based on its scale level, in order to make the input of classifier 122 compatible with the size of the image.

[0094] Step 304 thus provides, for each image, C membership probabilities which locally describe the content of the original image at a given position and scale of the image corresponding to an image (the C membership probabilities can constitute the C components of the probability vector which is therefore then of size C).

[0095] In step 306, a semantic annotation is determined for each pixel of the image using the probability vectors obtained for the different imagelets derived from the input image. Step 306 can be implemented by the image annotation reconstruction unit 124.

[0096] In step 306, a semantic annotation is determined for each image based on the class membership probabilities determined in step 304, for all the imagelets derived from the input image. Thus, considering P imagelets extracted in step 302, step 306 uses the P probability vectors determined for the P imagelets.

[0097] Step 306 comprises, for each KxK image among the P imagelets extracted from the MxN input image, adding the C membership probabilities obtained for the imagelet in a KxK patch (a patch is a square centered on an image pixel), corresponding to the imagelet, onto an MxNxC matrix corresponding to the input image. Such a matrix thus represents the probabilities that each of the classes appears at every point of the original image. Step 302, which extracts the imagelets according to position and scale level criteria, allows the entire probability matrix to be constructed.

[0098] Figure 4 shows an example of a 40 probability matrix and 400 patches, according to an exemplary embodiment. Figure 4 illustrates a change in the step size of the sliding window that extracts the images, the offset between two images being fixed at the beginning of the process and remaining unchanged throughout its application. It should be noted that it is possible to go down to a step size of 1 pixel. However, the smaller the step size, the greater the number of operations. For example, for a step size of 224 / x pixels, the number of operations is greater than for a step size of 224 pixels.

[0099] In one embodiment, a class can be constrained to a given image resolution to account for the variety of environments in which autonomous vehicles operate. Indeed, in such environments, it is common for certain classes to represent texture information rather than structural or contour information, and vice versa. Depending on the application of the invention, it is possible to observe the results at different extraction dimensions (i.e., scales) and to choose the most consistent dimension for each class.

[0100] Figure 5 illustrates a change in the size of the sliding window depending on the class. As shown in Figure 5, a different factor can be applied for each class to account for the varying size of the patches added to the probability matrix.

[0101] On the same image, small images of different sizes are extracted and allow the probabilities of belonging to specific classes to be established according to the information they contain (for example, frequency level information, low or high frequencies).

[0102] Considering a probability matrix Mp, a probability vector P at the output of the neural network-based classifier 122 and the vector D of the different extraction dimensions (i.e., scale levels), the probability matrix can be defined according to the following equation (1):

[0103] j,n) -Mp(i,j,n) +P(n):i'-^

[0104] For a given image dimension, only the probabilities on the classes concerned by this dimension are added in step 302.

[0105] In one embodiment, the step size set in step 302 of image extraction (the sliding window step size) can be used to control the level of detail in the semantic annotation of the final image. In other embodiments, Gaussian filtering-type smoothing can be implemented to improve the readability of information at class boundaries, which are defined by the extraction step size, according to equation (3). Indeed, with a given step size, it is generally not possible to obtain different predicted classes on an area smaller than such a pixel step size. In other embodiments, each probability matrix from step 304 can be weighted by a two-dimensional Gaussian window for each scale level used.Thus, instead of directly adding the probability of the class under consideration over a surface ZxZ, the results of a two-dimensional Gaussian window of size ZxZ multiplied by the probability of the class are added according to the following equation (2): .

[0106] Mpd, j, n) jn} +G(i, (2)

[0107] In equation (2), the Gaussian window G(i, j) is defined by:

[0108] Gdj) =^W4Wourie [0,Z],je [0, Z] (3)

[0109] The semantic annotation of the initial image can then be determined by selecting, at each point of the probability matrix Mp, the index between 0 and C of the class with the highest probability. Thus, each pixel of the input image is assigned the class to which its membership is most probable.

[0110] Fig. 6 illustrates the method of determining image annotations, in an embodiment where a single-scale approach is used.

[0111] In the example of Figure 6, the method according to embodiments of the invention determines degraded annotations for C classes. The original image is divided, for example, into 224x224 pixels, and the classifier 122 is applied to each pixel to form a probability vector for belonging to each of the classes. The vector thus obtained is added to the portion corresponding to the extracted pixel in a probability matrix of the same size as that of the input image, but having a depth equal to C, instead of the depth of the input image (which can be an RGB image for 'Red Green Blue') equal to 3. In one embodiment, at step 308, a color can be associated with each pixel of the probability matrix Mp to determine annotations suitable for visualization in a human-machine interface. Furthermore, the index exhibiting the maximum probability (i.e.index having the maximum value in the vector. (probabilities associated with each pixel) can be extracted to automatically generate the semantic annotation of the initial image.

[0112] Figure 7 illustrates the method for determining image annotations in an embodiment where a multi-scale approach is used. In such an embodiment, small images of different resolutions are extracted from the initial image to encapsulate scale-level information (e.g., high and low frequency information) and thus extract not only the image contours but also the image texture. The different small images can then be resized for classification in step 304 before being merged in step 306 to determine the final probability matrix for automatically constructing the semantic annotation of the input image.

[0113] Figure 8 illustrates the method for determining image annotations, applied to a sequence of images, using the pre-trained classifier 122 (block 81) from a classifier model 82 and annotated images 80A for each class composing the scene under study, and a sequence of input images 80B for which a semantic annotation is to be determined. The method implements a weighted sliding window multi-scale classification approach (block 83) to establish a probability matrix (block 85) of each pixel belonging to a given class in order to determine (block 84) the semantic annotation of the initial image, thus obtaining annotated images 85.

[0114] The embodiments of the invention are thus particularly well-suited to any application using a semantic segmentation neural network, and more specifically its retraining from new data, such as, for example, a seabed segmentation application. They enable the deployment of AUVs in such environments with a reduced semantic annotation cost. In such an application example, the embodiments of the invention can be used for risk detection. Indeed, they allow for the automatic or dynamic determination of semantically annotated images based on a predetermined number of classes representative of the seabed, highlighting the proportion of each class in an image, and analyzing whether a part of the image does not belong to these classes, thus indicating the probability of encountering an unknown element.According to another example, the system and method according to embodiments of the invention can be used to calculate the proportion of the representation of the class "water" in the image (i.e. the class representing water in the images) in order to determine if an AUV (acronym for "Autonomous Underwater System") is at an acceptable depth to navigate in its environment using SLAM-type technologies (Acronym . for "simultaneous localization and mapping" meaning 'mapping and simultaneous localization') for example.

[0115] Those skilled in the art will understand that the system or subsystems according to embodiments of the invention can be implemented in various ways by hardware, software, or a combination of hardware and software, in particular in the form of program code that can be distributed as a program product in various forms. In particular, the program code can be distributed using computer-readable media, which may include computer-readable storage media and communication media. The methods described herein can, in particular, be implemented in the form of computer program instructions executable by one or more processors in a computer system. These computer program instructions can also be stored in computer-readable media.

[0116] The invention is not limited to the embodiments described above by way of non-limiting example. It encompasses all the alternative embodiments that could be considered by a person skilled in the art.

Claims

1. Demands A device for classifying a scene in the environment of a vehicle, comprising a classification unit (14) using a neural network configured to classify a scene in response to a received image representing the scene, and a learning unit (11) configured to train said neural network from a set of images stored in a database (3), the scene being associated with a set of predefined classes, characterized in that it comprises an image annotation unit comprising: - an image preprocessing module (120) configured to extract a set of thumbnails from a received image based on a position criterion in the image and at least one criterion relating to an image parameter, - a classification module (122) configured to determine, for each thumbnail, a probability vector comprising components,each component being associated with one of the scene classes and having a value corresponding to the probability of the image belonging to the associated class, the probability vector determined for an image representing the probability of the image belonging to each of the scene classes, said probability vector being determined using said neural network and, - an image annotation module (124) configured to fuse the responses from the classification module (122) obtained for all the imagelets, represented by the probability vectors obtained for the imagelets, and to determine an annotation corresponding to said image, by selecting for each pixel of the image the class having a maximum probability of membership from said probability vectors determined for all said imagelets, said annotation being transmitted to said database (3) for storage of the annotation in association with said image in the database, and in that the classification device further comprises a classification unit (14) configured to semantically segment the received image using the previously trained neural network

2. Classification device according to claim 1, wherein an image parameter is an image frequency information, the criterion relating to said image parameter being a frequency level among a low frequency and a high frequency.

3. Classification device according to any one of the preceding claims, wherein the preprocessing unit uses a fixed or variable step to implement image extraction by sliding window.

4. Classification device according to any one of the preceding claims, wherein the pre-processing module is configured to implement, prior to determining the probability vector for an image, an image resizing operation based on the level of its image parameter.

5. Classification device according to any one of the preceding claims, wherein the pre-processing module is configured to add, for each KxK-sized imagelet among the P imagelets extracted from the MxN-sized input image, the C membership probabilities obtained for the imagelet in a KxK-sized patch, corresponding to the imagelet, onto an MxNxC-sized probability matrix corresponding to the input image.

6. Classification device according to claim 5, wherein the probability matrix Mp is defined according to the equation: jy n) = Mp(i, j, n) +P(n)' where P denotes the probability vector P at the output of the classification unit and D the vector of the different dimensions of image extraction.

7. Classification device according to claim 5, wherein the probability matrix Mp is defined according to the equation: =Mp(i,j,n) + G(i, j)-''P(n)^~^— where P denotes the probability vector P at the output of the classification unit, G(i, j) a Gaussian window and D the vector of the different dimensions of image extraction.

8. Classification device according to claim 7, wherein the Gaussian window G(L j) is defined by: G(i, j) = for ig [0, Z], jG [0, Z], the annotation of the received image being then determined by selecting, at each point of the probability matrix Mp, the index having the maximum value in the probability vector associated with each pixel.

9. A method for classifying a scene in the environment of a vehicle, comprising a classification step using a neural network configured to classify a scene in response to a received image representing the scene, the scene being associated with a set of predefined classes, and a training step configured to train said neural network, from a set of images stored in a database (3), characterized in that the method comprises: - an image preprocessing (120) consisting of extracting a set of thumbnails from a received image according to a position criterion in the image and at least one criterion relating to an image parameter, - a classification step (122) consisting of determining, for each thumbnail, a probability vector comprising components,each component being associated with one of the classes of the scene and having a value corresponding to the probability of the image belongs to the associated class, the probability vector determined for an image representing the probability of the image belonging to each of the classes of the scene, said probability vector being determined using said neural network, and - an image annotation step (124) consisting of merging the responses of the classification module (122) obtained for all the images, represented by the probability vectors obtained for the images, and determining an annotation corresponding to said image, by selecting for each pixel of the image the class having a maximum probability of belonging from said probability vectors determined for all said images,said annotation being transmitted to said database (3) for storage in association with said image in the database, and in that the method comprises semantic segmentation of the received image using the previously trained neural network.

Images