Deep neural network classification method of an off-distribution road situation image captured by a sensor on a moving vehicle.
The neural network with a reconstruction error-regularized loss function effectively detects out-of-distribution road situation images, enhancing safety and compliance with SOTIF norms by modeling the training distribution, thus improving ADAS and AD system performance.
Patent Information
- Application Number
- FR2023005113
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Current deep neural networks used in ADAS and AD systems for classifying road situations suffer from OOD generalization, leading to low precision in classifying out-of-distribution images, which can compromise road safety and require multiple networks or training data distributions.
A neural network with an encoder and decoder is trained using a loss function regularized by reconstruction error to model the distribution of training images, allowing detection of OOD images during inference by comparing input images to the learned distribution.
The method provides reliable detection of OOD images without requiring multiple networks or continuous training, ensuring safer road situation classification and compliance with SOTIF norms by avoiding misclassifications.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Method for classifying by deep neural network an image of an out-of-distribution road situation captured by a sensor of a vehicle in circulation. Technical field
[0001] The present invention relates to methods for classifying road situation images captured by a sensor of a vehicle in circulation. In particular, the present invention relates to a training phase of a neural network and an inference phase which uses the trained neural network to classify road situation images captured by a sensor of a vehicle in circulation. Technological background
[0002] Contemporary vehicles incorporate an increasing number of on-board systems that contribute to vehicle control by assisting the driver and / or ensuring the comfort of the driver and passengers of the vehicle. Some of these systems are known as AD AS systems (from the English “Advanced Driver-Assistance System” or in French “Advanced Driving Assistance System”).
[0003] Some current AD AS systems use neural networks to classify images of road situations captured by an on-board imaging device of a vehicle. These images of road situations can be used, for example, by DMS systems (from the English "Driver Monitoring System", in French vehicle driver behavior monitoring system also known as vehicle driver attention monitoring system, driver attention monitor or driver fatigue warning system), or by autonomous driving systems called AD systems (from the English Automatic Driving).
[0004] These images of road situations are images of the immediate environment outside the vehicle which can be classified to determine, for example, the danger of road situations and to warn, for example, an automatic driving system which can then activate at least one component of the vehicle to ensure the safety of the vehicle and the occupants of this vehicle.
[0005] In modern AD AS and AD systems, so-called deep neural networks are usually used and trained on tens or even hundreds of millions of images, for example taken from relevant road situations encountered in real life and for which the ground truth is known. Examples include the deep neural networks EfficientNet-v2 (M.Tan, Q. Le, “EfficientNetV2: Smaller Models and Faster Training”, Proceedings of the 38th International Conference on Machine Learning, PMLR 139:10096-10106, 2021) and MobileNet-v2 [M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and LC Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” Proc. IEEE Comput. Log. Conf. Comput. Screw. Pattern Recognit., pp. 4510-4520, 2018] for image classification of road situations.
[0006] The performance of neural networks depends heavily on how an input image relates to the distribution of the images used to train these neural networks. When the statistics of an input image are within the distribution resulting from the images used to train neural networks, i.e. the input image is identical to an image used for training or to an image interpolated from an image used for training (this is referred to as the generalization capacity of neural networks), neural networks generally display good classification performance. On the other hand, when the input image is OOD (Out Of Distribution), these neural networks extrapolate a classification response with low accuracy.
[0007] This problem is known as ODD generalization.
[0008] One of the problems to be solved in using deep neural networks is to detect such OOD input images and to find workarounds to improve the performance of the trained deep neural networks when an ODD image is presented at its input.
[0009] Current solutions to the ODD generalization problem use either multiple neural networks that assume multiple training data distributions, or a single neural network trained from multiple training data distributions relevant to real-life unusual road situations.
[0010] These solutions are not satisfactory because they require the training and use of several neural networks and / or several training data distributions. In addition, real road situations may always occur which do not correspond to the road situations corresponding to the training data distributions.
[0011] Summary of the present invention
[0012] An object of the present invention is to solve at least one of the problems of the technological background described above.
[0013] Another object of the present invention is to improve the performance of a deep neural network used to classify images of road situations.
[0014] Another object of the present invention is to improve the detection of road situations which do not correspond to road situations envisaged during training the neural network.
[0015] Another object of the present invention is to improve road safety.
[0016] According to a first aspect, the present invention relates to a method for classifying a road situation image captured by a sensor of a vehicle in circulation, said image, called input image, being presented as input to a neural network trained to provide a road situation class identifier, the neural network comprising an encoder and a decoder, the encoder providing a feature vector formed of features used to associate a road situation class identifier with the input image, the feature vector being defined in a feature space whose dimension is smaller than that of the input image, the decoder providing an image reconstructed from the feature vector, the neural network being trained during a learning phase by minimizing a loss function,the learning phase comprising the following steps for each current image of a set of images of road situations presented as input to the neural network: , - obtaining, at the output of the encoder and for the current image, a vector of characteristics defined in the characteristic space; - obtaining, from the neural network and for the current image, a road situation class identifier; - obtaining, at the output of the decoder and for the current image, a reconstructed image - obtaining, from the neural network and for the reconstructed image, a road situation class identifier; - the loss function per current image depending on a first data item and a second data item, the first data item being representative of a non-correspondence between the road situation class identifier obtained for the current image and the road situation class identifier obtained for the reconstructed image and the second data item being representative of a reconstruction error between the current image and the reconstructed image; the method being implemented by at least one computer of the vehicle comprises the following steps: - obtaining, from the trained neural network, a feature vector defined in the feature space for the input image; - verification that the input image is not an image outside a distribution of images in the set of road situation images; - if the input image is not an image outside the distribution of images in the set of road situation images, obtaining, from the trained neural network, a road situation class identifier for the input image.
[0017] The neural network is special because it uses a loss function regularized by a reconstruction error. The determination of the feature vectors is then constrained during the learning phase so that the feature vectors model the distribution of the road situation images used for training the neural network. This modeling is used during deployment on board the vehicle (inference phase) to detect whether a road situation image captured by a sensor of the vehicle in circulation and presented as input to the trained neural network is an OOD image, i.e. an image outside the distribution of the images in the set of road situation images.The method is therefore advantageous because it provides reliable ODD input image detection and because it only provides road situation class identifiers for input images of the trained neural network that are not ODD images.
[0018] The classification of ODD images presented as input to a trained neural network is a major cause of accidents involving autonomous vehicles in recent years and the on-board vehicle systems based on this classification must comply with SOTIF (Safety Of The Intended Functionality) normative constraints as formalized in the ISO 26262 standard.
[0019] The method is also advantageous because it does not use several neural networks or several distributions of images of road situations during the learning phase, hence an easier implementation of the learning and inference phases.
[0020] The method is also advantageous because it does not require any knowledge of the images presented as input to the trained neural network and does not require continuous training of the neural network as ODD images are presented as input to the neural network.
[0021] According to a variant, the step of verifying that the input image is not an image outside the distribution of the images of the set of images of road situations comprises the following steps: - obtaining, at the output of the decoder and for the input image, an image reconstructed from the characteristic vector obtained for the input image; - calculation of a reconstruction error between the input image and the reconstructed image; - the input image being considered as not being outside the distribution of images in the set of road situation images if the reconstruction error is less than a threshold value.
[0022] This variant is advantageous because it makes it possible to evaluate that an input image presented as input to the trained neural network is not an image outside a distribution of images from the set of images of road situations in comparing a reconstruction error with a threshold value. This calculation is simple to implement.
[0023] According to a variant, the step of verifying that the input image is not an image outside the distribution of the images of the set of images of road situations comprises the following steps: - memorization of a sum vector obtained by summing the characteristic vectors obtained for the current images of the set of road situation images during the learning phase; - calculation of distances between elements of the characteristic vector obtained for the input image and corresponding elements of the stored sum vector; - the input image is considered not to be outside the distribution of images in the road situation image set if at least one distance is less than a threshold value.
[0024] This variant is advantageous because the storage of a sum vector makes it possible to store the distribution of each element (characteristic) of characteristic vectors and to compare the distribution of each element of the sum vector with a corresponding element (characteristic) of the characteristic vector obtained for the input image.
[0025] According to a variant, the loss function is a weighted linear combination of the first data and the second data.
[0026] According to a variant, the loss function is regularized by a third data representative of curvature at the level of minima of a curve defined in the space of characteristics by the characteristic vectors obtained for the images of the set of images of road situations.
[0027] The regularization of the loss function by the third data ensures that the internal parameters of the neural network determined during the learning phase allow similar classification performances for images presented as input to the trained neural network which are identical to those of the set of images of road situations used during the learning phase and for images which are visually close to these images of the set of images of road situations, that is to say images which are not identical in terms of semantic contents but which are close to them.
[0028] According to a variant, the loss function is a weighted linear combination of the first data, the second data and the third data.
[0029] According to a second aspect, the present invention relates to a road situation image classification device, the device comprising a memory associated with a processor configured for implementing the steps of the method according to the first aspect of the present invention.
[0030] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.
[0031] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.
[0032] Such a computer program may use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.
[0033] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.
[0034] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording means or a hard disk.
[0035] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.
[0036] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures
[0037] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 6, in which:
[0038] [Fig-1] schematically illustrates a neural network according to an example of rea particular and non-limiting use of the present invention.
[0039] [Fig.2] illustrates a flowchart of the different stages of a learning phase of the method for classifying an image of a road situation, according to a particular and non-limiting exemplary embodiment of the present invention;
[0040] [Fig.3] illustrates a flowchart of the different stages of an inference phase of the method for classifying an image of a road situation, according to a particular and non-limiting exemplary embodiment of the present invention;
[0041] [Fig.4] illustrates a flowchart of the different sub-steps of a stage of the phase inference of the method for classifying an image of a road situation, according to a particular and non-limiting exemplary embodiment of the present invention;
[0042] [Fig.5] illustrates a flowchart of the different sub-steps of a stage of the phase inference of the method for classifying an image of a road situation, according to a particular and non-limiting exemplary embodiment of the present invention;
[0043] [Fig.6] schematically illustrates a device configured to classify an image of road situation, according to a particular and non-limiting exemplary embodiment of the present invention;
[0044] Description of the examples of embodiment
[0045] A method and a device for classifying an image of a road situation will now be described in the following with joint reference to Figures 1 to 6. The same elements are identified with the same reference signs throughout the description which follows.
[0046] [Fig.l] schematically illustrates a neural network according to a particular and non-limiting exemplary embodiment of the present invention.
[0047] The neural network comprises a set of layers of artificial neurons.
[0048] For example, each artificial neuron is a perceptron, that is to say a linear classifier generally comprising several inputs and a single output and characterized by an activation function, weights (or synaptic coefficients) and a bias (or threshold).
[0049] For example, a perceptron with n inputs .... x„) and a single output o is defined by n weights ( , h'h ) and a bias (or threshold) 0: 100501 fl, It o = f(z) = ' 1 0 otherwise
[0051] The output o then results from the application of the Heaviside function to the post-synaptic potential z given by:
[0052]
[0053] with a non-linear activation function H(x) given for example by:
[0054] w .. uz \ (0 six<0 VxeR. HLx) = , v 7 11 if x > 1
[0055] The internal parameters of the neural network are then these weights and biases for all the perceptrons.
[0056] The present invention is not limited to this definition of perceptron nor to the use of other basic elements forming a layer of the neural network. It is also not limited to the number of perceptrons (or other building blocks) used per layer or by the number of layers. The internal parameters of the neural network are usually weights and biases regardless of the building blocks of the neural network layers.
[0057] The primary function of the neural network is to calculate the post-synaptic potential z which is usually defined by a vector in a feature space (called latent space in English). The vector z is formed of neural activations which we will subsequently call features. The feature vector z is then used to associate a road situation class identifier with an input image presented as input to the trained neural network. The feature space has a dimension lower than that of the input image space. For this reason, the neural network can be seen as comprising an encoder 10 and a decoder 11. The input image x presented as input to the neural network is then the input image presented as input to the encoder 10. The output of the encoder 10 is then the feature vector z ([Fig.l]) obtained from the input image x by applying operations defined by the perceptrons (or other basic elements) of the different layers of the neural network.
[0058] The neural network is trained from a set of images of road situations with the aim of classifying an input image of a road situation into a road situation class, i.e. of associating a road situation class identifier with the image presented as input to the trained neural network.
[0059] For example, if several classes of road situations correspond to road situations of different danger levels, the neural network can provide a road situation class identifier y of one of these classes of road situations to indicate that the input image corresponds to a road situation whose danger level corresponds to that of this class.
[0060] The decoder 11 makes it possible to obtain a reconstructed image x from the characteristic vector z. The decoder 11 applies to the characteristic vector z operations dual to those of the encoder 10 which are implemented to obtain the characteristic vector z from the input image x.
[0061] Feedback connection weights and other biases may be determined during training of the neural network and be part of the internal parameters of the neural network.
[0062] The neural network is also configured to provide a road situation class identifier y to a reconstructed image at the output of the decoder 11.
[0063] Preferably, the neural network is a convolutional neural network of the MobileNet-v2 type adapted, according to the present invention, to allow the reconstruction of an image x from a vector of characteristics z and the obtaining of an identifier of road situation class y to a reconstructed image at the output of the decoder 11. The loss function F used during the learning phase of the neural network is also modified according to the present invention as will be seen later.
[0064] A vehicle, according to the present invention, corresponds for example to a vehicle with a thermal engine, with electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle thus corresponds for example to a land vehicle, for example a car, a truck, a bus.
[0065] The vehicle advantageously carries an imaging device which provides a set of images to a device 2 on board the vehicle which is configured for implementing at least one step of the method for classifying a road situation image described in relation to FIGS. 2 to 5.
[0066] The imaging device comprises a camera 10 associated with an optical system for acquiring images outside the vehicle.
[0067] The image device also comprises processing means such as a computer and storage means such as non-volatile memories of the EEPROM or flash type for storing the acquired images.
[0068] The processing means may be configured to adapt the resolution and / or the dimensions and / or the format of an image acquired by the imaging device to a resolution, dimensions and / or an image format at the input of the neural network.
[0069] The camera 101 corresponds for example to a monocular optical camera with a CCD sensor (from the English Char-Coupled Device, in French charge transfer device) or with a CMOS sensor (from the English Complementary Metal-Oxide Semi-conductor, in French complementary metal oxide semiconductor).
[0070] The imaging device provides high resolution images (e.g. >1 megabits / pixel) comprising several components.
[0071] For example, the images comprise three components: one of luminance and two of chrominance. According to this example, the captured images are of dimension WxHx3 if W indicates the number of columns of the images and H the number of rows of these images.
[0072] The input images of the neural network can have a resolution identical to that of the captured images, i.e. a number of pixels equal to WxH and a format identical to that of the captured images, i.e. a luminance component and two chrominance components. In this case, the dimension of the space of the input images is equal to WxHx3 and the dimension of the space of the characteristics is less than WxHx3
[0073] The vehicle also comprises a road situation image classification device 2 described in relation to [Fig. 6] which is configured to implement a road situation image classification method described in relation to FIGS. 2 to 5. The steps of the method are divided into a learning phase ([Fig.2]) and an inference phase ([Fig.3]).
[0074] During the learning phase, the above neural network is trained from the set of road situation images. For each current image x of this set of current situation images, the method comprises steps 31 to 34.
[0075] In step 31, a characteristic vector z defined in the characteristic space is obtained at the output of the encoder 10 for the current image x.
[0076] According to a variant, in a step 311, the characteristic vector z is stored in a memory.
[0077] In step 32, a road situation class identifier y is obtained for the current image x from the neural network.
[0078] In step 33, a reconstructed image v is obtained at the output of the decoder 11 and for the current image x.
[0079] In step 34, a road situation class identifier y is obtained for the reconstructed image $ from the neural network.
[0080] The loss function F per current image x, which is minimized to determine the internal parameters of the neural network, depends on a first CE data and a second RL data
[0081] The first CE data is representative of a non-correspondence between the road situation class identifier y obtained for the current image and the road situation class identifier y obtained for the reconstructed image:
[0082] For example, the first CE data is a signed difference between the class identifier y and the class identifier y.
[0083] The second RI data is representative of a reconstruction error between the current image x and the reconstructed image x.
[0084] For example, the reconstruction error is a mean square error between the current image x and the reconstructed image .x.
[0085] The loss function F is then a function of the first data CE and the second data RL
[0086] According to a variant, the loss function F is then given by a weighted linear combination of the first and second data obtained by equation (1) below:
[0087] [Math.l] F = CE (y, $) + aRl
[0088] with a weighting coefficient.
[0089] The inference phase comprises steps 35 to 37.
[0090] In step 35, a feature vector z defined in the feature space is obtained from the trained neural network for an input image x presented as input to the trained neural network.
[0091] In step 36, it is verified that the input image x is not an image outside a distribution of the images of the set of images of road situations.
[0092] If the input image x is not an image outside the distribution of images in the set of road situation images, in step 37, a road situation class identifier y is obtained from the neural network trained for the input image x.
[0093] If the input image x is an image outside the distribution of images in the set of road situation images, the method stops and no road situation class identifier is provided by the trained neural network.
[0094] The threshold value can be set during a validation phase of the method for classifying images of road situations, i.e. the threshold value can be determined as a function of constraints, for example regulatory constraints, which define, for example, the proximity of the semantic contents of the interpolated images with those of the images interpolated from images in the set of images of road situations.These constraints can for example be expressed in terms of false negatives or false positives: a high threshold value produces more false negatives (ODD images but which are considered by the trained neural network as being within the distribution of road situation images in the set of road situation images; a low threshold value produces fewer false positives (images which are considered as being within the distribution of road situation images in the set of road situation images but which are considered as ODD by the trained neural network).
[0095] According to a variant, illustrated in [Fig.4], step 36 comprises sub-steps 361 and 362.
[0096] In sub-step 361, a reconstructed image r is obtained, at the output of the decoder 11 and for the input image x presented at the input of the trained neural network, from the characteristic vector z obtained for the input image x.
[0097] In sub-step 362, a reconstruction error E is calculated between the input image x and the reconstructed image x.
[0098] The input image x is then considered not to be outside the distribution of the images of the set of images of road situations if the reconstruction error E is less than a threshold value T.
[0099] According to a variant, illustrated in [Fig.5], step 36 comprises step 311 ([Fig.2]) and a sub-step 364.
[0100] In step 311, a sum vector is stored. The sum vector is obtained by summing the feature vectors zi obtained for the current images xi of the set of road situation images during the learning phase.
[0101] In sub-step 364, distances Di are calculated between the elements of the vector of features z obtained for the input image x and the corresponding elements of the stored sum vector.
[0102] The input image is considered not to be outside the distribution of the images of the set of images of road situations if at least one distance Di is less than the threshold value T.
[0103] According to a variant, the loss function F is regularized by a third data R2 representative of curvature at the level of minima of a curve defined in the space of characteristics by the characteristic vectors y obtained for the images of the set of images of road situations (learning phase).
[0104] According to a variant, the third data R2 is obtained from a Hessian matrix of the curve in the characteristic space.
[0105] According to a variant, the third data R2 is the sum of the squares of the eigenvalues of the Hessian matrix given by the equation below:
[0106] [Math.2]
[0107] The loss function F then depends on the first data CE and the third data R2.
[0108] According to a variant, the loss function F is given by a weighted linear combination of the first, second and third data obtained by equation (2) below:
[0109] [Math.3] F = CE(y, y) + aRl + @R2
[0110] with P a weighting coefficient.
[0111] According to a variant, the loss function F is given by a weighted linear combination of the first and second data obtained by equation (3) below:
[0112] [Math.4] F^CE(y,y)+pR2
[0113] with P a weighting coefficient.
[0114] According to a variant, during the learning phase, for each current image, a first characteristic vector zl is obtained at the output of the encoder 10 as a function of the loss function F of equation (1) (step 31) and a second characteristic vector z2 is obtained at the output of the encoder 10 as a function of the loss function F of equation (3) (step 31).
[0115] This variant allows the determination of the internal parameters of the neural network to be separated, which can thus be optimized separately to obtain optimized modeling of the images of all the road situation images (equation 1) and on the other hand to improve the generalization of the neural network (equation 3).
[0116] [Fig.6] schematically illustrates a device 2 configured to classify road situation images, according to a particular and non-limiting exemplary embodiment of the present invention.
[0117] The device 2 corresponds for example to a device on board the vehicle, for example a computer.
[0118] The device 2 is for example configured for the implementation of at least one step of the method described with regard to figures 2 to 5. Examples of such a device 2 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 2, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 2 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.
[0119] The device 2 comprises one (or more) processor(s) 20 configured to execute instructions for carrying out at least one step of the method and / or for executing the instructions of the software(s) embedded in the device 2. The processor 20 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 2 further comprises at least one memory 21 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.
[0120] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the memory 21.
[0121] According to various particular and non-limiting embodiments, the device 2 is coupled in communication with other similar devices or systems and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.
[0122] The neural network may be trained by a remote device, such as a server or a series of servers, in communication with the device 2 via a communication network. The internal parameters of the neural network may then be transmitted by this remote device to the device 2 which receives them. The device 2 is then able to implement the classification method when an acquired input image by the vehicle's imaging device is presented as input to the trained neural network.
[0123] According to a particular and non-limiting exemplary embodiment, the device 2 comprises a block 22 of interface elements for communicating with external devices, for example a remote server or the “cloud”, other nodes of the ad hoc network. The interface elements of the block 22 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); - HDMI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).
[0124] Data, for example internal parameters of the trained neural network, are for example loaded to the device 2 via the interface of the block 22 using a Wi-Fi® network such as according to IEEE 802.11, an ITS G5 network based on IEEE 802.1 Ip or a mobile network such as a 4G (or 5G) network based on the LTE (Long Term Evolution) standard defined by the 3GPP consortium, in particular an LTE-V2X network.
[0125] According to another particular and non-limiting exemplary embodiment, the device 2 comprises a communication interface 23 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 24. The communication interface 23 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 24. The communication interface 23 corresponds for example to a wired network of the CAN (from the English “Controller Area Network” or in French “Réseau de contrôles”) type, CAN FD (from the English “Controller Area Network Flexible Data-Rate” or in French “Réseau de contrôles à débit de données flexible”), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).
[0126] Internal parameters of the trained neural network may be received via the communication interface 23.
[0127] According to a particular and non-limiting exemplary embodiment, the device 2 can provide output signals to one or more external devices, such as a display screen 25, touch-sensitive or not, one or more speakers 26 and / or other peripherals 27 (projection system) via output interfaces 28, 29 and 30 respectively. According to a variant, one or other of the external devices is integrated into the device 2.
[0128] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for classifying a road situation image which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.
[0129] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 2 of [Fig.6].
Claims
1. Claims Method for classifying a road situation image captured by a sensor of a vehicle in circulation, said image, called input image, being presented as input to a neural network trained to provide a road situation class identifier, the neural network comprising an encoder and a decoder, the encoder providing a feature vector formed of features used to associate a road situation class identifier with the input image, the feature vector being defined in a feature space whose dimension is smaller than that of the input image, the decoder providing an image reconstructed from the feature vector, the neural network being trained during a learning phase by minimizing a loss function,the learning phase comprising the following steps for each current image of a set of images of road situations presented as input to the neural network:, - obtaining (31), at the output of the encoder and for the current image, a characteristic vector (z) defined in the characteristic space; - obtaining (32), from the neural network and for the current image, a road situation class identifier (y); - obtaining (33), at the output of the decoder and for the current image, a reconstructed image (x); - obtaining (34), from the neural network and for the reconstructed image, a road situation class identifier (y); - the loss function per current image depending on a first data item (CE) and a second data item (RI), the first data item (CE) being representative of a non-correspondence between the road situation class identifier (y) obtained for the current image and the road situation class identifier (^) obtained for the reconstructed image and the second data item being representative of a reconstruction error between the current image and the reconstructed image; the method being implemented by at least one computer of the vehicle comprises the following steps: - obtaining (35), from the trained neural network, a feature vector (z) defined in the feature space for the input image; - verification (36) that the input image is not an image outside of a distribution of the images of the set of road situation images; - if the input image is not an image outside the distribution of the images of the set of road situation images, obtaining (37), from the trained neural network, a road situation class identifier (y) from the neural network for the input image.
2. Method according to claim 1, in which the step of verifying (36) that the input image is not an image outside the distribution of the images of the set of road situation images comprises the following steps: - obtaining (361), at the output of the decoder and for the input image, an image reconstructed from the characteristic vector (z) obtained for the input image; - calculating (362) a reconstruction error between the input image and the reconstructed image; - the input image being considered as not being outside the distribution of the images of the set of road situation images if the reconstruction error is less than a threshold value.
3. Method according to claim 1, for which the step of verifying (36) that the input image is not an image outside the distribution of the images of the set of road situation images comprises the following steps: - storing (363) a sum vector obtained by summing feature vectors obtained for the current images of the set of road situation images during the learning phase; - calculating (364) distances between elements of the feature vector obtained for the input image and corresponding elements of the stored sum vector; - the input image is considered not to be outside the distribution of the images of the set of road situation images if at least one distance is less than a threshold value.
4. Method according to one of the preceding claims, for which the loss function is a weighted linear combination of the first data and the second data.
5. Method according to one of claims 1 to 4, for which the loss function is regularized by a third data item (R2) representative of curvature at the minima of a curve defined in the feature space by the feature vectors obtained for the images in the road situation image set.
6. The method of claim 5, wherein the loss function is a weighted linear combination of the first data, the second data and the third data.
7. Computer program comprising instructions for implementing the method according to any one of the preceding claims, when these instructions are executed by a processor.
8. A computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to one of claims 1 to 6.
9. Device (2) for classifying road situation images comprising a memory (21) associated with at least one processor (20) configured for implementing at least one step of the method according to any one of claims 1 to 6.
10. Vehicle (10) comprising the device (2) according to claim 9.
Citation Information
Patent Citations
Techniques for identification of out-of-distribution input data in neural networks
WO2022216506A1
