Method and device for processing a road situation image using an energy-optimized neural network for a vehicle

Abstract

The present invention relates to a method and a device (10) for processing a road situation image (IG1) captured by a sensor (16) of a vehicle (2). The method, implemented by an artificial neural network (NN1), referred to as the target network, comprises at least one training phase of the target network (NN1) including: selection of a subgroup of candidate parameters (w*) from a group of parameters (w) of the target network; obtaining an output result from the target network (NN1), configured by pruning the subgroup of candidate parameters (w*), when examples of road situation images are presented as input to the target network; estimation of an energy cost representative of the energy consumption caused by obtaining said result; and configuration of the target network (NN1) according to parameters (w) selected from the energy cost to minimize the energy consumption caused by the target network (NN1). Figure for the abstract: Figure 1

Description

Title of the invention: Method and device for processing a road situation image using an energy-optimized neural network for a vehicle Technical field

[0001] The present invention relates to a device and a method for processing a road situation image captured by a vehicle sensor and aims in particular at training an energy-efficient artificial neural network intended to carry out the processing of said road situation image. Technological background

[0002] Contemporary vehicles incorporate an increasing number of on-board systems that contribute to vehicle control by assisting the driver and / or ensuring the comfort of the driver and passengers of the vehicle. Some of these systems are known as AD AS systems (from the English “Advanced Driver-Assistance System” or in French “Advanced Driving Assistance System”).

[0003] Some current AD AS systems use neural networks to classify or process images of road situations captured by an on-board imaging device of a vehicle. These images of road situations can be used, for example, by DMS systems (from the English "Driver Monitoring System", in French vehicle driver behavior monitoring system also known as vehicle driver attention monitoring system, driver attention monitor or driver fatigue warning system), or by autonomous driving systems called AD systems (from the English Autonomous Driving).

[0004] These images of road situations are images of the immediate environment outside the vehicle which can be classified to determine, for example, the danger of road situations and to warn, for example, an automatic driving system which can then activate at least one component of the vehicle to ensure the safety of the vehicle and the occupants of this vehicle.

[0005] In modern AD AS and AD systems, so-called deep neural networks are usually used and trained on tens or even hundreds of millions of images, for example taken from relevant road situations encountered in real life and for which the ground truth is known. Examples include the deep neural networks EfficientNet-v2 (M.Tan, Q. Le, “EfficientNetV2: Smaller Models and Faster Training”, Proceedings of the 38th International Conference on Machine Learning, PMLR 139:10096-10106, 2021) and MobileNet-v2 [Mr. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and LC Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” Proc. IEEE Comput. Soc. Conf. Comput. Screw. Pattern Recognition, pp. 4510-4520, 2018] for the classification of images of road situations.

[0006] One of the problems observed is that contemporary neural networks require significant processing and memory resources, which can pose difficulties for implementation in vehicles. This is explained in particular by the fact that neural networks are increasingly large and involve an increasing number of parameters that must be taken into account and trained to obtain satisfactory performance.

[0007] A known technique, called elimination or pruning, consists of eliminating parameters from the neural network, which allows it to be lightened or reduced in size, while minimizing the loss of performance. Certain parameters of the network can thus be set to 0 (i.e. "pruned"), which amounts to removing synaptic connections or neurons. Parameter pruning constitutes a specific area of ​​machine learning, which has for example been addressed in the document entitled “What is the State of neural network pruning?” (D. Blalock, JJ Gonzalez Ortiz, J. Frankle, and J. Guttag,, Proceedings of machine learning and Systems, vol. 2, pp. 129-146, 2020). Up to 90% or more of an artificial neural network's parameters can be pruned or removed depending on the pruning technique used during the training phase.

[0008] The parameter pruning technique can be used to reduce the resource requirements of a neural network embedded in a vehicle, but this is not always sufficient to obtain a high-performance neural network. Summary of the present invention

[0009] In practice, an artificial neural network intended to be implemented in a vehicle device to process road situation images is generally not trained in execution situation in the on-board device itself. Training such a neural network directly in the on-board device is generally impractical, in particular due to the fact that the on-board device does not have the hardware and / or software resources (memory, software layer, etc.) necessary to carry out the training phase of the neural network. Also, the parameter pruning must be carried out outside the vehicle (i.e. outside the execution environment of the neural network), typically in a dedicated graphics card (or unit).

[0010] Training a neural network outside of its execution environment can pose certain constraints, in particular due to the fact that it can be difficult to evaluate how the entire embedded device would react when training the model. of the neural network.

[0011] In particular, the applicant has considered taking into account the energy consumption of the embedded device in the training phase of the neural network, which poses difficulties, in particular when this phase is carried out outside the execution environment, namely the embedded device.

[0012] The applicant has in fact found that the energy performance of a neural network that has been subjected to training with parameter pruning does not necessarily have lower energy consumption than for non-pruned neural networks. It has been observed that, even when applying such a pruning technique, the neural network is likely to have excessive energy consumption, which can pose a problem when running on board a vehicle.

[0013] One of the objects of the present invention is to solve at least one of the problems or deficiencies described above.

[0014] Another object of the present invention is to configure an artificial neural network capable of efficiently processing a road situation image while being on board any vehicle, including a vehicle with limited resources.

[0015] Another object of the present invention is to limit the energy resources necessary for processing a road situation image by an artificial neural network embedded in a vehicle, while minimizing the loss of performance.

[0016] According to a first aspect, the present invention relates to a method for processing a road situation image captured by a sensor of a vehicle, said method being implemented by an artificial neural network, called the target network, said method comprising at least one phase of training said target network comprising: - selection of a subgroup of candidate parameters from a group of parameters of the target network; - obtaining a result at the output of the target network, configured by pruning the subgroup of candidate parameters, when examples of road situation images are presented at the input of the target network; - estimation of an energy cost representative of an energy consumption caused by obtaining said result; and - configuration of the target network according to parameters selected from the energy cost to minimize the energy consumption caused by the target network.

[0017] The present invention advantageously makes it possible to configure the target network so that it is capable of efficiently processing (during an inference phase) one or more road situation images while being on board any vehicle 2, including in a vehicle with limited resources. The invention makes it possible in particular to limit the energy resources required for processing at least one road situation image by the target network embedded in a vehicle, while minimizing the loss of performance.

[0018] The method according to the invention may include other characteristics which may be taken separately or in combination, in particular among the embodiments which follow.

[0019] According to a particular embodiment, the parameters comprise connection weights of the target network.

[0020] According to a particular embodiment, the estimation of the energy cost is carried out by prediction of a second artificial neural network, called the estimator network, distinct from the target network, said estimator network being trained to predict the energy cost based on the candidate parameters of the target network.

[0021] According to a particular embodiment, the method comprises, prior to the training phase of the target network, a preliminary training of the estimator network comprising: - training a predictive model of the estimator network from training data comprising examples of pruned parameters and associated energy costs for a plurality of examples of neural networks,, said estimation of the energy cost during said training phase of the target network being determined by applying the predictive model trained from the subgroup of candidate parameters presented as input to the estimator network.

[0022] According to a particular embodiment, the method comprises, for each iteration of the training phase of the target network: - determination of an error between the result of the target network and a reference value; - determination, from the error and the estimated energy cost, of a value of a cost function representative of a performance of the target network when the candidate parameters are pruned; and - adaptation of the candidate parameters selected during a next iteration of the training phase according to the value of the cost function to minimize the energy consumption of the target network.

[0023] According to a particular embodiment, the adaptation of the candidate parameters during the training phase comprises: - calculation of a partial derivative of the cost function for each parameter of the target network parameter group; - said adaptation comprising an increase in a candidate parameter if the associated partial derivative is negative and a reduction in said candidate parameter if the associated partial derivative is positive.

[0024] According to a particular embodiment, a plurality of said training phases are carried out iteratively, the subgroup of candidate parameters being increased at each iteration of the training phase.

[0025] According to a second aspect, the present invention relates to a device configured to implement the processing method according to the first aspect of the invention. The invention relates in particular to a processing method configured to process a road situation image captured by a vehicle sensor, the device comprising a memory associated with a processor configured to implement the steps of the processing method according to the first aspect of the present invention.

[0026] It should be noted that the various embodiments mentioned above in relation to the training method according to the first aspect of the invention as well as the associated advantages apply in a similar manner to the training device according to the second aspect of the invention.

[0027] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type or of the land motor vehicle type, comprising a drive device according to the second aspect of the present invention. This vehicle may for example be an autonomous or semi-autonomous vehicle.

[0028] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the control method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor. In other words, the different steps of the training method are determined by computer program instructions. This computer program is configured to be implemented in a training device of the second aspect of the invention, or more generally in a computer.

[0029] Such a computer program may use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0030] According to a fifth aspect, the present invention relates to a recording medium (or information medium), readable by the control device according to the second aspect or more generally by a computer (or a processor), on which is recorded a computer program comprising instructions for executing the steps of the control method according to the first aspect of the present invention.

[0031] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a means of storage, such as ROM, CD-ROM, or microelectronic circuit-type ROM, or magnetic recording media or hard disk.

[0032] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.

[0033] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0034] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 6, in which:

[0035] [Fig-1] schematically illustrates a vehicle comprising an on-board device, according to at least one particular and non-limiting embodiment of the present invention;

[0036] [Fig.2] illustrates a diagram of different stages of a method of treating a artificial neural network implemented in the embedded device of [Fig.l], according to at least one particular and non-limiting exemplary embodiment of the present invention;

[0037] [Fig.3] schematically illustrates an external device configured to drive a artificial neural network, such as the network implemented in the embedded device of [Fig.l], according to at least one particular and non-limiting exemplary embodiment of the present invention;

[0038] [Fig.4] illustrates a diagram of different stages of a method of training a artificial neural network, such as the network implemented in the embedded device of [Fig.l], according to at least one particular and non-limiting exemplary embodiment of the present invention;

[0039] [Fig.5] schematically illustrates the training of an artificial neural network, such as the network implemented in the embedded device of [Fig.l], according to at least one particular and non-limiting exemplary embodiment of the present invention; and

[0040] [Fig.6] illustrates a diagram of different stages of a method of training a artificial neural network, such as the network implemented in the embedded device of [Fig.l], according to at least one particular and non-limiting exemplary embodiment of the present invention.

[0041] Description of the examples of embodiment

[0042] A method and a device for controlling a vehicle will now be described in the following with reference to Figures 1-6. Unless otherwise indicated, elements common or similar to several figures bear the same reference signs and have identical or similar characteristics, so that these common elements are generally not described again for the sake of simplicity.

[0043] The terms “first(s)”, “second(s)”, etc.) are used in this document by arbitrary convention to enable different elements (such as operations, means, etc.) implemented in the embodiments described below to be identified and distinguished.

[0044] As previously indicated, the invention relates in particular to a method for processing a road situation image captured by a sensor of a vehicle, the method being implemented by an artificial neural network, called the target network, or more generally by a device comprising at least the target network. This target network is in particular intended to be implemented in an on-board device of the vehicle to process the road situation image.

[0045] The invention aims in particular, but not exclusively, at the processing of a road situation image representative of a road situation in which a vehicle is located, for example of the automobile or other type, or more generally a vehicle of the motorized land vehicle type.

[0046] According to a particular and non-limiting example of embodiment of the present invention, this processing method comprises at least one phase of training said target network comprising: - selection of a subgroup of candidate parameters from a group of parameters of the target network; - obtaining a result at the output of the target network, configured by pruning (or eliminating) the subgroup of candidate parameters, when examples of road situation images are presented at the input of the target network; - estimation of an energy cost representative of an energy consumption caused by obtaining said result; and - configuration of the target network from parameters selected from the energy cost to minimize the energy consumption caused by the target network.

[0047] This method advantageously makes it possible to configure the target network so that it is capable of efficiently processing a road situation image while being embedded in any vehicle, including in a vehicle with limited resources. The target network embedded in the vehicle can thus efficiently process a road situation image while minimizing the consumption of associated energy.

[0048] Other aspects and advantages of the present invention will emerge from the exemplary embodiments described below with reference to the drawings mentioned above.

[0049] [Fig.l] schematically illustrates a vehicle carrying a sensor 16 and an artificial neural network NN1. In this example, the artificial neural network NN1, also called the target network, is implemented in an on-board device 10. The vehicle 2 may optionally further comprise a control unit 18.

[0050] More precisely, the target network NN1 implemented by the on-board device 10 is configured to carry out a processing 11 of one or more road images IG1 captured by means of the sensor 16. The processing 11 carried out may be any processing (for example in real time), for example recognition, classification, risk prediction, etc., carried out by the target network NN1 as a function of the vehicle 2 (or more precisely of the on-board device 10). This function (or system) may for example be configured to carry out any control operation of the vehicle 2, such as for example the control of at least one function (or system) of the vehicle, such as an AD AS and / or AD function of the vehicle, other examples being however conceivable.

[0051] According to one example, the device 10 is configured to execute an AD AS, and / or DMS, and / or AD function (or system), for example to classify or process at least one road situation image IG1 captured by the on-board sensor 16.

[0052] For this purpose, the sensor 16 may comprise any image sensor 16a (or video sensor) configured to capture images IG1 of the road situation in connection with the vehicle 2. This sensor 16 may for example comprise at least one on-board camera or any other equivalent equipment, for example an image sensor of the CCD (for “Charge-Coupled Device”) or CMOS (for “Complementary Metal-Oxide Semiconductor”) type. Alternatively or cumulatively, the sensor 16 may comprise any optical sensor 16b (for example of the LIDAR type) configured to capture images IG1 of the road situation in connection with the vehicle 2.

[0053] It is assumed hereinafter that a single sensor 16 is used although it is possible to use a plurality of such sensors to capture one or more images IG1 of the road situation.

[0054] As illustrated, the sensor 16 can be configured to generate DTI image data representative of at least one road situation image IG1 (or more generally data representative of a road situation), these data being transmitted as input to the target network NN1 implemented by the device 10. To do this, the sensor 16 and the on-board device 10 are coupled together via an appropriate communication link.

[0055] According to a particular example, the on-board device 10 is configured to perform a any pre-processing of the DTI image data (compression, cropping, reshaping, etc.) to adapt the format of the DTI data provided as input to the target network NN1.

[0056] The image(s) IG1 of the road situation captured by the sensor 16 are representative of an environment in which the vehicle 2 is located. This environment is for example a close environment outside the vehicle 2 (for example a road environment), the extent of this environment being variable depending in particular on the acquisition range of the sensor 16.

[0057] The processing 11 carried out by the target network NN1 when it is on board the vehicle 2 may comprise, for example, a classification of at least one road situation image IG1 and a determination of a road situation danger from a result of the classification. The device 10 may then be configured, upon detection of a given level of road situation danger (for example beyond a predefined threshold), to carry out a predefined safety action to secure the vehicle 2, such as, for example, the activation of a component of the vehicle 2 (for example, the control unit 18) to secure the vehicle and its occupants.

[0058] The type and characteristics of the vehicle 2 may be adapted as appropriate. The vehicle 2 may, for example, be of the automobile type or equivalent. Alternatively, this vehicle may be a coach, a bus, a truck, a utility vehicle or a motorcycle, or more generally a vehicle of the motorized land vehicle type.

[0059] According to a particular example, the vehicle 2 travels under the total supervision of a driver.

[0060] According to a particular example, the vehicle 2 travels in an autonomous or semi-autonomous mode. In other words, the first vehicle 4 may be an autonomous or semi-autonomous vehicle.

[0061] More particularly, the vehicle 2 can travel according to a level of autonomy equal to 0 or according to a level of autonomy ranging from 1 to 5 for example, according to the scale defined by the American federal agency which has established 5 levels of autonomy ranging from 1 to 5. Level 0 corresponds to a vehicle having no autonomy, the driving of which is under the total supervision of the driver; level 1 corresponding to a vehicle with a minimal level of autonomy, the driving of which is under the supervision of the driver with minimal assistance from an AD AS system (from the English “Advanced Driver-Assistance System” or in French “Système avance d’aide à la conduite”); and level 5 corresponding to a completely autonomous vehicle.

[0062] The 5 levels of autonomy of the classification of the federal agency responsible for road safety are:

[0063] level 0: no automation, the vehicle driver has full control of the main vehicle functions (engine, accelerator, steering, brakes);

[0064] level 1: driver assistance, automation is active for certain vehicle functions, the driver retaining overall control over the driving of the vehicle; cruise control is part of this level, as are other aids such as ABS (anti-lock braking system) or ESP (programmed electro-stabilizer);

[0065] level 2: automation of combined functions, the control of at least two main functions is combined in the automation to replace the driver in certain situations; for example, adaptive cruise control combined with lane centering allows a vehicle to be classified level 2, as does automatic parking assistance (from the English “Park assist”);

[0066] level 3: limited autonomous driving, the driver can cede complete control of the vehicle to the automated system which will then be in charge of critical safety functions; autonomous driving can however only take place in certain determined environmental and traffic conditions (only on motorways for example);

[0067] level 4: complete autonomous driving under conditions, the vehicle is designed to ensure all critical safety functions alone over a complete journey; the driver provides a destination or navigation instructions but is not required to make himself available to take back control of the vehicle;

[0068] level 5: completely autonomous driving without driver assistance in all circumstances.

[0069] The vehicle 2 can be configured according to any of the autonomy levels mentioned above. According to a particular embodiment, the vehicle 2 is configured according to a semi-autonomous or autonomous mode, that is to say with an autonomy level greater than or equal to 2 according to the classification above.

[0070] As illustrated in [Fig.l] in a particular example, the on-board device 10 comprises for example at least one processor 12 and a non-volatile memory 14. The on-board device 10 may for example be all or part of a computer, or a combination of computers.

[0071] Examples of the on-board device 10 include, but are not limited to, on-board electronic equipment such as a vehicle on-board computer, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the control device 10, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. The on-board device 10 may be implemented in the form of electronic circuits or software (or computer) modules or a combination of electronic circuits and software modules.

[0072] The device 10 is configured to implement the processing 11 from the IGL image(s). For this purpose, the device 10 may comprise a computer program PG1 stored in the non-volatile memory 14 accessible by the processor 12, this computer program PG1 comprising instructions for the processing 10. The processor 12 is thus configured to execute in particular the instructions defined by the computer program PG1.

[0073] The non-volatile memory 14 may constitute an information medium according to a particular embodiment in that it comprises the computer program PG1 comprising instructions for carrying out all or part of the steps of a processing method of the invention.

[0074] In particular, the computer program PG1 can define the target network NN1 to carry out the processing 11. The target network NN1 is thus configured to obtain an output result from the input data DTI comprising the image(s) IG1 captured by the sensor 16.

[0075] The target network NN1 comprises (or is characterized by) a group of parameters denoted w forming part of (or defined by) a model ML1 (or predictive model) according to which said network is configured ([Fig.l]). These parameters w (or variables of the model ML1) can comprise any trainable parameters of the target network NN1. In other words, these parameters w are modular (configurable, adjustable) during a PHI training phase described below ([Fig.2]).

[0076] By way of example, the parameter group w comprises connection weights (or synaptic weights) and / or biases of the target network NN1. According to a particular example, this parameter group w comprises all the trainable parameters of the target network NN1 or, alternatively, any sub-part of the trainable parameters.

[0077] By definition, a connection weight is respectively applied to the input of each neuron to calculate an output data of said neuron. This weight represents the importance (or preponderance, or weight) given to an input of the neuron of the target network NN1. By definition, the biases are additional parameters, according to which the neurons of the target network NN1 are configured, to adjust the input values ​​to which the weights are applied. These weights influence the behavior of the activation function of the neuron considered.

[0078] As illustrated in [Fig.2], before carrying out the processing 11 during a so-called inference phase PH2, the target network NN1 must first be trained during at least one training phase PHI. This training phase PHI makes it possible to train (or configure) the predictive model ML1 of the target network NN1, and more particularly its parameters w, so that said target network NN1 is capable of processing ul subsequently at least one IG1 image of road situation during the PH2 inference phase (processing 11) which follows.

[0079] Examples of implementation of the PHI training phase of the target network NN1 are described below. This PHI training phase notably comprises an elimination (or pruning) step for eliminating certain parameters from the parameter group w of the target network NN1.

[0080] In the following, consider by way of example that at least one PHI training phase of the target network NN1 is carried out outside the execution environment of said target network NN1, that is to say outside the on-board device 10, or even outside the vehicle 2. Note however that variants are possible where the training of the target network NN1 is carried out in an on-board manner in the vehicle 2, for example by the on-board device 10.

[0081] Training the target neural network NN1 directly in the embedded device 10 may in fact be impractical, for example due to the fact that the embedded device 10 does not necessarily have the hardware and / or software resources (memory, software layer, etc.) required for carrying out the PHI training phase of said target network NN1.

[0082] Training a neural network outside of its execution environment can pose certain technical difficulties, in particular due to the fact that it can be difficult to evaluate how the entire embedded device 10 would react when training the ML1 model of the target network NN1.

[0083] [Fig. 3] schematically represents a device 20 external to the vehicle 2, this external device 20 (also called training device 20) comprising the target network NN1 (or a copy of this target network NN1) and a second artificial neural network NN2, called estimator network, distinct from the target network NN1. The external device 20 is configured to train the target network NN1 (and more particularly its model ML1) during at least one training phase PHI, prior to the inference phase PH2 carried out subsequently by the on-board device 10 on board the vehicle 2 (figures 2-3).

[0084] In other words, the external device 20 is configured to carry out a training method comprising at least one PHI training phase for training the target network NN1. This PHI training aims to train the ML1 model of the target network NN 1 to carry out a given task, namely the processing 11 in the examples which follow.

[0085] As illustrated in [Fig.l] in a particular example, the external device 20 comprises for example at least one processor 22 and a non-volatile memory 24. The external device 20 may for example comprise all or part of a server, or a combination of servers. By way of example, the external device 20 comprises a graphics card (or unit) configured to perform at least one PHI training phase of the target network NN1.

[0086] For this purpose, the external device 20 may comprise a computer program PG2 stored in the non-volatile memory 24 accessible by the processor 22, this computer program PG2 comprising instructions for carrying out at least one PHI training phase of the target network NN1. The processor 12 is thus configured to execute in particular the instructions defined by the computer program PG2.

[0087] The non-volatile memory 24 may constitute an information medium according to a particular embodiment in that it comprises the computer program PG2 comprising instructions for carrying out the steps of a training method of the invention.

[0088] The computer programs PG1 and PG2 collectively form a computer program executable by the devices 10 and 20 to carry out the processing method illustrated in [Fig.2] causing the processing 11 of a road situation image IG1.

[0089] In particular, the computer program PG2 can define the target network NN1 and the estimator network 24. The external device 20 is thus configured to train the target network NN1, by means of the estimator network NN2, to carry out the processing 11.

[0090] According to a particular example, the external device 20 does not have access to the vehicle 2 (or to any information characterizing the, or originating from the, on-board device 10), and is therefore not able to determine the behavior of the device 10 when the target network NN1 executes the processing 11 of at least one road situation image IG1.

[0091] According to a particular example, the estimator network NN2 is configured according to a model (or predictive model) ML2. In other words, this model ML2 comprises parameters according to which the estimator network NN2 is configured. The parameters of the model ML2 may comprise any trainable parameters of the target network NN2. In other words, these parameters are modular (configurable, adjustable) during a preliminary training phase PH3 described below ([Fig.6]).

[0092] As indicated above, the external device 20 is configured to implement a training method, the latter comprising at least one PHI training phase of the target neural network NN1. To do this, the external device 20 executes for example the computer program PG2.

[0093] This training method is now described in conjunction with Figures 3-5 according to particular embodiments. In particular, a said PHI training phase comprising steps 31-34 is now described in particular examples, it being understood that this PHI training phase can be carried out once or a plurality of times iteratively to train the ML1 model of the target network NN1. In other words, the method of training the target network NN1 may comprise one or multiple iterations of the PHI training phase.

[0094] During a selection step 31, a subgroup of candidate parameters w* is selected from (or among) the group of parameters w of the target network NN1. The candidate parameters w* therefore belong to the group of parameters w. As indicated below, the parameters w* of this subgroup are candidates for being “pruned”, i.e. eliminated from the target network NN1 (by parameter pruning).

[0095] Generally, pruning a parameter of an artificial neural network is performed by setting the value of that parameter to zero.

[0096] According to a particular example, this subgroup of candidate parameters w* excludes at least one (unselected) parameter w from said group. In other words, at least one parameter w is not selected as a candidate parameter w*.

[0097] During a step 32 of obtaining (or executing), a result T(x, w) (or more simply noted T) is obtained at the output of the target network NN1, configured by pruning (or eliminating) the subgroup of candidate parameters w*, when examples - noted x - of road situation image are presented at the input of the target network NN1. In other words, the target network NN1 receives as input examples x of road situation image and produces a result T(x, w) at the output by carrying out the processing 11 from the examples x. This result T(x, w) represents a prediction generated by the target network NN1 from the inputs x when the subgroup of parameters w* is eliminated (or pruned). This step 32 of obtaining makes it possible to test the behavior of the target network NN1 in the hypothesis where the parameters w* are pruned.

[0098] According to a particular example, the examples x presented at the input of the target network x are previously extracted and received from a training base noted DB1 ([Fig.5])•

[0099] During an estimation step 33, an energy cost E(w, w*) (or more simply denoted E) is estimated, this energy cost E being representative of an energy consumption caused by obtaining said result T. In other words, the energy cost represents an energy consumption caused by the processing 11 carried out by the target network NN1 during the obtaining step 32.

[0100] According to a particular example, the estimation 33 of the energy cost E is carried out by prediction of the estimator network NN2, the latter being trained to predict the energy cost E as a function of the subgroup of candidate parameters w* selected from the target network NN1 (and possibly also from the group of parameters w).

[0101] In particular, the estimator network NN2 can be trained to predict the energy cost E that would be caused if the target network NN1, configured by pruning the candidate parameters w*, were implemented in the device 10 (embedded in the vehicle 2). In other words, the estimator network NN2 estimates the gain (or loss) in energy consumption that would be obtained if the candidate parameters w* were eliminated (or pruned) from the target network NN1 and if said target network NN1 thus configured were implemented in the device 10 to obtain the result T as output from the examples x received as input. It is thus possible to estimate with precision and reliability the genetic consumption occurring in the target hardware, namely the embedded device 10 in this example.

[0102] The use of such an estimator neural network NN2 is advantageous in that it makes it possible to accurately and reliably predict the energy consumption E that the target network NN1 would cause if it were implemented on board the vehicle 2, without it being necessary to carry out the training of the target network NN1 on board the vehicle 2. The training of the target network NN1 can advantageously be carried out outside the vehicle 2, for example by means of the external device 20 which can comprise software and / or hardware means more suitable for carrying out this type of operation. The external device 20 can for example comprise one or more servers, which makes it possible to have greater computing power and memory resources than in a system embedded in the vehicle 2.

[0103] During a configuration step 34, the target network NN1 is configured according to parameters w selected from the energy cost E obtained in the estimation step 33 to minimize the energy consumption caused by the target network NN1. In other words, the energy consumption caused by the target network NN1 is minimized by eliminating by pruning parameters w which are selected according to the energy cost E. It is thus possible to take into account the energy cost estimated at each iteration of the PHI training phase to determine the parameters w of the target network NN1 which are eliminated (or pruned) during the configuration step 34.

[0104] According to one example, all or part of the candidate parameters w* selected in the selection step 31 are eliminated (or pruned) during the configuration step 34. This configuration step 24 can be carried out at each iteration of the PHI training phase or, alternatively, only at certain iterations, or even only during a final iteration.

[0105] As previously indicated, the estimator network NN2 (or more particularly its predictive model ML2) can be trained during a preliminary training phase PH3 to predict at output (step 33, [Fig.4]) the energy consumption E caused by the obtaining (step 32) by the target network of the result T. The predictive model ML2 of the estimator network NN2 is thus itself trained before being used for training the target network NN1.

[0106] An example of the implementation of such a preliminary training phase PH3 implemented by the external device 20 is now described with reference to [Fig.5].

[0107] During the preliminary training phase PH3, the predictive model ML2 of the estimator network NN2 is trained from training data received as input from the estimator network NN2. This training data comprises examples e(w*) of pruned parameters and examples e(E) of associated energy costs, i.e. pairs (e(w*), e(E)), and this for a plurality of examples of neural networks, for example for a plurality of neural networks.

[0108] The training data used to train the NN2 estimator network are for example extracted and received from a DB2 training base ([Fig.5]).

[0109] According to a particular example, the training data provided as input to the estimator network NN2 during the preliminary training phase PH3 comprise, in addition to e(w*) and e(E), examples e(w) of groups of parameters w, i.e. groups of parameters w from which the associated parameters w* are pruned.

[0110] According to a particular example, the training data used to train the estimator network NN2 characterize the behavior of the energy consumption of the target hardware, namely the embedded device 10 in this example. This training data can thus define the evolution of an average energy consumption e(E) as a function of the parameters w* and w for a plurality of artificial neural networks implemented by the target device 10. Different parameters w* have been eliminated by pruning in the examples of neural networks considered in order to allow the predictive model ML2 once trained to generalize the energy consumption E caused on the target device 10 for parameters w and w* received as input from the estimator network NN2.

[0111] The preliminary training PH3 of the estimator network NN2 includes for example the determination of a correlation between the data e(w*) and e(w) on the one hand, and the associated results e(E) on the other hand. This correlation is for example determined by a calculation of a regression. To this end, the estimator network NN2 can thus carry out this regression: its output attempts to reproduce the target value present in the database for this input configuration. Since the energy consumption is a real value that must be approached as closely as possible, we speak of regression.The NN2 estimator network can for example learn to do this regression task in two steps: an error is calculated between the network output and the target value (e.g. by "Mean Squared Error"), then this error is back-propagated into the NN2 estimator network and used by its learning algorithm to change (reconfigure) the NN2 estimator network so that the next prediction of energy consumption by the NN2 estimator network is better.

[0112] The estimation (step 33, [Fig.4]) of the energy cost E during the PHI training phase of the target network NN1 is thus obtained at the output of the network es NN2 estimator, from the subgroup of candidate parameters w* presented as input to the NN2 estimator network, and this by applying the ML2 predictive model trained by the NN2 estimator network. According to a particular example, the NN2 estimator network performs this estimation from the subgroup of candidate parameters w* and the group of parameters w presented as input to the NN2 estimator network. A technical advantage of this particular example lies in particular in the input format of this network, which can use the 2 types of information: what are all the parameters (i.e. w), and what are the pruned parameters (i.e. w*).

[0113] According to a particular example, the NN2 estimator network is a transformer network, also called a “transformer” or “self-attentive model” (for “Transformer network” in English). The NN2 estimator network can in particular be a “vision transformer” (also called ViT for “Vision Transformer network” in English). This type of artificial neural network is particularly suitable for processing image data. The NN2 estimator network can thus be configured to receive as input the weights e(w) and e(w*) of the parameters w and the pruned parameters w*, respectively.

[0114] According to a particular example, the estimator network NN2 is configured to receive as input an image in which each pixel corresponds to a parameter of the target network NN1. Thus, if we consider for example a target network NN1 comprising approximately 3 million parameters w, we can restructure these parameters as an image of 3x1024x1024 pixels. We can then indicate the pruned parameters by setting the corresponding pixels to 0 (black pixels) and setting all the others to 1 (white pixels). The Vision Transformer is then used without any other modification than changing its input layer to allow it to process images of 3x1000x1000. It is also possible to distribute the pixels not on 3 channels but on 300, giving an input “image” of format 300x100x100. Other variant embodiments are conceivable.

[0115] An example of a ViT network is described in the paper “An image is worth 16x16 words: Transformers for image recognition at scale” (A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.; International Conference on Learning Representations, 2021).

[0116] Furthermore, the external device 20 may iteratively repeat the PHI training phase as previously described (steps 31-34; [Fig. 4]). Thus, candidate parameters w* may be selected (step 31) during a current iteration of the PHI training phase to test their impact on the energy consumption of the target network NN2 in the device 10 when these candidate parameters w* are eliminated by pruning. As illustrated in [Fig. 4] according to a particular example, the candidate parameters w* to be selected during a next phase iteration PHI training devices are adapted during an adaptation process 35 ([Fig.4]).

[0117] An exemplary embodiment of this adaptation process 35, comprising steps 36-38, is now described with reference to FIGS. 5-6. To do this, the external device 20 executes in this example the computer program PG2. The execution of the adaptation process 35 in a current iteration of the PHI training phase of the target network NN1 is considered below, it being understood that this adaptation process 35 can be executed at each iteration of the PHI training phase.

[0118] More precisely, during a determination step 36, an error is determined between the result T(x, w) obtained at the output of the target network NN2 at the current iteration on the one hand and a reference value y on the other hand. This error can be defined as follows:

[0119] [Math.l] Crit(T(x, w), j)

[0120] where Crit is a criterion function which takes as input the result T(x, y) and the reference value y. This value y corresponds for example to a real result, to the expected theoretical result, for an image x received as input of the target network NN1.

[0121] The error Crit can be calculated for a plurality of images x received at the input of the target network NN1. The error thus determined is the sum of the errors obtained for each input image x, which can be translated as:

[0122] [Math.2] ^( x ,y) eD Crit ^ y)

[0123] where D is the set of target inputs and outputs (x,y) of the batch of copies from the learning base.

[0124] During a determination step 37, a value of a cost function L(w) (also called loss function) is determined from the error Crit and the energy cost E estimated by the estimator network NN2 for the current iteration. This cost function L(w) is representative of a performance of the target network NN1 (or capacity of the network NN1 to carry out the processing 11) when the candidate parameters w* are eliminated by pruning. The higher the value of the cost function L(w), the worse the performance of the target network.

[0125] According to a particular example, the cost function is defined as follows:

[0126] [Math.3] = E Uv) eD Crit(T(x, w), y) + f i || w || [Math.3] + E(w, w*)afi [Math.3] w* II

[0127] where p is a weight (or weighting coefficient) of a weight penalty (or "weight penalty" in English) which limits the weight of the target network NN1 to the extent that the more parameters w there are in the network, the higher the value of the cost function L(w), which in particular makes it possible to avoid over-learning of the target network NN1; where a is a weight (or weighting coefficient) which weights the order of magnitude of the weight penalty, this coefficient being incremented (or increased) at each complete cycle, called “Epoch” (or epoch), of the input data set x used to train the ML1 predictive model of the target network NN1; where E(w, w*) is the energy cost determined by the NN2 estimator network, this energy cost serving as weight (weighting coefficient) of the candidate parameters w* which are pruned.

[0128] As indicated above, the energy cost E(w, w*) estimated by the estimator network NN2 is advantageously introduced into the definition of the cost function L(w) in order to weight the weight of the candidate parameters w* which are pruned (or tested for pruning) in the target network NN1. The higher the energy cost E(w, w*) determined at the current iteration, the greater the contribution (or weight) of the candidate parameters w* in the value of the cost function L(w). Thus, the cost function L(w) weights the weight penalty according to the energy cost E(w, w*), which makes it possible to minimize the energy consumption of the target network NN1 by training the latter during a plurality of iterations of the PHI training phase.

[0129] During an adaptation step 38, the candidate parameters w* selected during a next iteration of the PHI training phase are adapted according to the value of the cost function L(w) obtained in step 37 for the current iteration, so as to minimize the energy consumption of the target network NN1. In other words, the process of pruning the parameters w of the target network is regularized (or controlled) during the training of the target network NN1 from the estimation E of the energy consumption caused by the target network NN1, that is to say caused by the performance of the task (processing 11) of the target network NN1.

[0130] According to a particular example, during the adaptation process 35, a partial derivative of the cost function L(w) is calculated for each parameter of the parameter group w of the target network. A candidate parameter w* is then increased if the associated partial derivative is negative and decreased if the associated partial derivative is positive. The value of the partial derivative indicates in which direction to modify each parameter w to decrease its contribution to the error defined by the cost function L(w).

[0131] According to a particular example, a plurality of PHI training phases of the target network NN1 are carried out iteratively, the group of candidate parameters w being increased at each iteration of the PHI training phase in order to compress (or lighten) the target network NN1. It is thus advantageous to converge towards a target network NN1 with fewer and fewer parameters. In this way, the energy cost of the target network NN1 can be progressively reduced.

[0132] In particular, it is possible to carry out energy cost estimates E continuously, i.e. throughout the training method of the target network NN1 (at each iteration of the training phase PHI), which advantageously makes it possible to limit the disturbances that may affect the target network NN1, in particular in a case where a single parameter pruning is carried out at the end of training of the target network NN1.

[0133] According to a particular example, once the ML1 model has been trained according to the PHI training method previously described, model data (not shown) are transmitted from the external device 20 to the embedded device 10, this model data being representative of the trained ML1 model. The embedded device 10 can thus advantageously execute the processing 11 from the trained target network NN1.

[0134] The present invention advantageously makes it possible to configure the target network NN 1 so that it is capable of efficiently processing (during an inference phase PH2) a road situation image IG1 while being embedded in any vehicle 2, including in a vehicle 2 with limited resources. The invention makes it possible in particular to limit the energy resources necessary for the processing 11 of a road situation image IG1 by the target network NN1 embedded in a vehicle, while minimizing the loss of performance. In other words, it is advantageously possible to compress the target network so as to minimize its energy consumption in the target hardware 10 while maintaining good performance during the inference phase, for example when carrying out the processing 11.

[0135] To do this, parameters w of the artificial neural network NN1 are eliminated using a technique called pruning. The parameters w that are pruned are selected based on their contribution to the energy cost of the target network NN1. The greater their contribution, the greater the probability that they will be eliminated. It is thus possible to minimize the energy consumption of the target network in the vehicle 10.

[0136] Furthermore, it is possible to efficiently train the target network model NN1 at distance of the vehicle 2, using a second separate network, namely the estimator network NN2, which participates in the training of the predictive model ML1 of the target network NN1. As previously described, this prior training of the target network NN1 is carried out for example by the external device 20. The use of such an estimator network NN2 makes it possible to efficiently train the target network NN1 by taking into account the energy cost associated with examples x of image received as input to the target network NN1.

[0137] In the examples described above, the target network NN1 and the estimator network NN2 implemented by the external device 20 are two distinct artificial neural networks. Alternatively, the target network NN1 and the estimator network NN2 form a single artificial neural network. In other words, the target network NN1 is configured to carry out the operations or steps of the estimator network NN2.

[0138] According to a particular example, the method of training the target network NN1 is carried out by the device 10 on board the vehicle 2. The on-board device 10 can then carry out one or more PHI training phases as already described above.

[0139] According to a particular example, the on-board device 10 in the vehicle 2 is configured to comprise (or implement) the target network NN1 and the estimator network NN2 used to train the target network NN1. The on-board device 10 carries out all the operations or steps of the processing method, namely the phases PHI, PH2 and PH3.

[0140] As understood by a person skilled in the art, all the embodiments and variants described above, some of which have been deliberately simplified to facilitate explanations, constitute only non-limiting examples of implementation of the present disclosure. In particular, a person skilled in the art may envisage any adaptation or combination of the embodiments and variants described above, in order to meet a particular need.

[0141] The present invention is therefore not limited to the embodiments described above but extends in particular to the methods previously described which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to the respective devices configured for the implementation of such methods.

Claims

Claims

1. Method for processing a road situation image (IG1) captured by a sensor (16) of a vehicle (2), said method being implemented by an artificial neural network (NN1), called target network, said method comprising at least one training phase (PHI) of said target network comprising: - selection (31) of a subgroup of candidate parameters (w*) from a group of parameters (w) of the target network (NN1); - obtaining (32) a result (T) at the output of the target network, configured by pruning the subgroup of candidate parameters (w*), when examples (x) of road situation image are presented at the input of the target network (NN1); - estimation (33) of an energy cost (E) representative of an energy consumption caused by obtaining said result; and - configuration (34) of the target network (NN1) according to parameters (w) selected from the energy cost € to minimize the energy consumption caused by the target network.

2. The method of claim 1, wherein the parameter group (w) comprises connection weights of the target network (NN1).

3. Method according to claim 1 or 2, in which the estimation of the energy cost (E) is carried out by prediction of a second artificial neural network (NN2), called the estimator network, distinct from the target network, said estimator network being trained to predict the energy cost (E) as a function of the candidate parameters (w*) of the target network.

4. Method according to claim 3, wherein the method comprises, prior to the training phase (PHI) of the target network, a preliminary training of the estimator network (NN2) comprising: - training a predictive model (ML2) of the estimator network from training data comprising examples (e(w), e(w*), e(E)) of pruned parameters and associated energy costs for a plurality of examples of neural networks, said estimation of the energy cost (E) during said training phase (PHI) of the target network (NN1) being determined by applying the predictive model (ML2) trained from the subgroup of candidate parameters (w*) presented as input to the estimator network (NN2).

5. Method according to any one of the preceding claims, in which the method comprises, for each iteration of the training phase (PHI) of the target network (NN1): - determination (36) of an error between the result of the target network and a reference value; - determination (37), from the error and the estimated energy cost, of a value of a cost function representative of a performance of the target network when the candidate parameters are pruned; and - adaptation (38) of the candidate parameters selected during a next iteration of the training phase as a function of the value of the cost function to minimize the energy consumption of the target network.

6. Method according to claim 5, wherein the adaptation (38) of the candidate parameters (w*) during the training phase (PHI) comprises: - calculation of a partial derivative of the cost function (L(w)) for each parameter of the group of parameters (w) of the target network (NN1); - said adaptation comprising an increase of a candidate parameter if the associated partial derivative is negative and a reduction of said candidate parameter if the associated partial derivative is positive.

7. A method according to any preceding claim, wherein a plurality of said training phases are performed iteratively, the subgroup of candidate parameters being increased at each training phase iteration.

8. Computer program (PG1; PG2) comprising instructions for implementing the method according to any one of the preceding claims, when these instructions are executed by a processor (12, 22).

9. Processing device (10; 20) configured to process a road situation image captured by a vehicle sensor, said device comprising a memory (14; 24) associated with at least one processor (12; 22) configured to implement the steps of the method according to any one of claims 1 to 7.

10. Vehicle (2) comprising the device (10) according to claim 9.

Images