Method and device for radio reception comprising reinforcement learning based on user feedback

Abstract

The present disclosure relates to a method (40) of radio reception, implemented by a radio reception device comprising a selection model configured to select a control action of processing carried out to extract audio content from radio signals received by the radio reception device, and a reward allocation model, said method comprising, during at least one phase of use of the radio reception device: at the start of the at least one phase of use: a modification (S40) of the reward allocation model, during the duration of the at least one phase of use: reinforcement learning (S41) of the selection model using the modified reward allocation model, and a sound broadcast (S42) of an extracted audio content, after the end of the at least one phase of use: a transmission (S43),to a user of the radio reception device, a request for evaluation of the quality of the broadcast audio content, when the quality of the broadcast audio content is validated by the user: a storage (S44) of the modified selection model and of the reward allocation model for a following phase of use. Figure accompanying the abstract: Figure 4,

Description

Title of the invention: Method and device for radio reception comprising reinforcement learning based on user feedback Technical field

[0001] The present invention belongs to the field of radiophony, and more particularly relates to a method and a device for radiophonic reception comprising reinforcement learning based on user feedback. State of the art

[0002] A radio reception device is equipment, for example installed in a motor vehicle, which allows a user to access audio content from different radio stations.

[0003] By "radio station" is meant here a source generating audio content. Audio content generated by radio stations is broadcast to users via a network of geographically distributed transmitting antennas which emit this audio content in the form of radio signals.

[0004] Nowadays, many radio signal processing algorithms are implemented by a radio receiving device, in order to improve the listening experience of users of said radio receiving device. Generally, different processing operations can be applied to the received radio signals, in order to extract audio content to be broadcast to the user, and the control of the applied processing operations is done according to a measurement of a radio state of the radio receiving device and predetermined processing parameters of said radio receiving device.

[0005] The values ​​of these processing parameters are generally set during the development of the radio reception device, and are then no longer modified once the production of the radio reception device is launched. During development, the adjustment of the values ​​of the processing parameters is generally carried out on the basis of reference scenarios (a reference scenario corresponding to a predefined radio state or a predefined sequence of radio states), and said values ​​of the processing parameters are adjusted to ensure that the radio reception device applies predefined reference processes associated with these reference scenarios.

[0006] Generally speaking, it is not possible to foresee all the possible scenarios to which the radio reception device may be exposed during its use, so that such an approach does not make it possible to guarantee that the values ​​of the processing parameters thus set will be optimal, from a user experience point of view, when using said radio reception device, after it has been produced.

[0007] Furthermore, a radio reception device, for example installed in a motor vehicle, will most often be exposed to the same scenarios repeating themselves recurrently, given that it is not uncommon for a motor vehicle to regularly make the same journeys, and that the user of this motor vehicle also makes the same choices in terms of radio station(s) listened to. If these recurring scenarios are poorly covered by the reference scenarios, then the user experience will most often be suboptimal.

[0008] International application PCT WO 2012 / 059782 A1 proposes providing several sets of processing parameter values, and selecting a set of values ​​to be used based on a measurement of a radiophonic state of the radiophonic reception device. International application PCT WO 2012 / 059782 A1 also proposes requesting real-time feedback from the user on the listening quality obtained using the selected set of values.

[0009] International application PCT WO 2012 / 059782 A1 therefore makes it possible to cover a greater number of different scenarios in an improved manner. However, such an approach also does not allow for correct adaptation to all possible scenarios. Increasing the number of different scenarios that can be supported by the radio reception device requires increasing the number of different sets of processing parameter values, which prohibitively increases the requirements in terms of memory capacity. Furthermore, when the user is the driver of the motor vehicle, it is not possible for the user, from a road safety point of view, to provide real-time feedback on each set of processing parameter values ​​applied. Statement of the invention

[0010] The present invention aims to remedy all or part of the limitations of the solutions of the prior art, in particular those set out above, by proposing a solution which makes it possible to develop the processing applied by a radio reception device in a manner which improves the user experience, in particular for scenarios of use of the radio reception device which occur repeatedly.

[0011] To this end, it proposes according to a first aspect a radio reception method, implemented by a radio reception device comprising a selection model configured to select, as a function of a radio state of the radio reception device, a processing control action. performed to extract audio content from radio signals received by the radio reception device, the selection model being a machine learning model, the radio reception device further comprising a reward allocation model based on a radio state and a selected control action, said method comprising, during at least one phase of use of the radio reception device: - at the start of at least one phase of use: a modification of the reward allocation model, so as to obtain a modified reward allocation model, - during the duration of the at least one phase of use: reinforcement learning of the selection model using the modified reward allocation model, so as to modify the selection model, and sound broadcasting of audio content extracted from radio signals processed according to control actions selected by the modified selection model, - after the end of the at least one phase of use: a transmission, to a user of the radio reception device, of a request for quality evaluation of the broadcast audio content, - when the quality of the broadcast audio content is validated by the user: a storage of the modified selection model as a selection model and of the modified reward allocation model as a reward allocation model, for a subsequent phase of use.

[0012] Thus, it is proposed to develop the processing applied by the radio reception device by automatic learning (“machine learning” in the English literature), and more particularly by reinforcement learning based on user feedback (“reinforcement learning via human feedback”).

[0013] Conventionally, reinforcement learning consists, for an autonomous agent (here the radio reception device), in learning the best action or the best sequence of actions to perform by experimentation. More precisely, the autonomous agent observes an environment (here the radio state in which the radio reception device evolves), and decides on a control action or a sequence of control actions to perform according to its current radio state. A control action here corresponds to an action controlling the processing carried out on the radio signals, for example an action modifying values ​​of processing parameters.In response to a control action, the autonomous agent receives a reward that can be positive if the control action had a positive impact on the extraction of audio content, or negative if the control action had a negative impact on the extraction of audio content, or zero. These steps of selecting a control action, transitioning to a new radio state and . Receiving a reward is iterated until a stopping criterion is verified, and allows the autonomous agent to learn or determine a behavior, also referred to as "strategy" or "policy" ("policy" in Anglo-Saxon literature), which is optimal in that it tends to maximize not the reward received in the short term but the sum (possibly weighted) of the cumulative rewards in the medium or long term.

[0014] Such reinforcement learning is therefore based on a selection model, which implements the selection policy to select a control action based in particular on the radio state of the radio reception device, and a reward allocation model, which allocates a reward to a selected control action. The selection model is a machine learning model such as an (artificial) neural network.

[0015] It is proposed to evolve the selection model (and therefore to evolve the processing applied by the radio reception device) during at least one phase of use of the radio reception device. A phase of use corresponds to a phase beginning with the activation of said radio reception device (for example when the engine of the motor vehicle is started) and ending with the deactivation of said radio reception device (for example when the engine of the motor vehicle is stopped). At the start of the at least one phase of use, the reward allocation model is modified, in order to modify the implemented reward function, then a reinforcement learning algorithm is implemented to evolve the selection model over the duration of the at least one phase of use as a function of the modified reward allocation model.After the end of the at least one usage phase (i.e., in delayed time), the user is asked to evaluate the quality of the streamed audio content. If the user indicates that he is satisfied with the quality of the streamed audio content, then the changes made to the reward assignment model (at the beginning of the at least one usage phase) and to the selection model (by reinforcement learning) are considered to improve the user experience, and are retained for the next usage phase.

[0016] In this way, the user experience can be improved as successive phases of use progress, in particular for scenarios of use of the radio reception device that occur repeatedly (such as those encountered on daily journeys in a motor vehicle), on the basis of user feedback that is not in real time (only after the end of a phase of use) and is therefore not a source of inattention for the user / driver during the journey of the motor vehicle. Such an evolution of the applied processing is also done without having to increase the required memory capacity, since it is not no need to increase the number of different sets of processing parameter values ​​to be stored.

[0017] In particular embodiments, the radio reception method may further comprise one or more of the following optional features, taken individually or in all technically possible combinations.

[0018] In particular embodiments, when the quality of the broadcast audio content is not validated by the user, the modified reward allocation model is not stored as the reward allocation model for the next usage phase. In other words, the modifications made to the reward allocation model are discarded, and the initial reward allocation model, before its modification at the start of the at least one usage phase, will be reused (and possibly re-modified) during the next usage phase.

[0019] In particular embodiments, when the quality of the broadcast audio content is not validated by the user, the modified selection model is not stored as a selection model for the next phase of use. In other words, the modifications made to the selection model are discarded, and the initial selection model, before its modification by reinforcement learning during the at least one phase of use, will be reused (and possibly re-modified) during the next phase of use.

[0020] In particular embodiments, the modified reward allocation model is obtained by randomly modifying parameter values ​​of the reward allocation model.

[0021] In particular modes of implementation, the random modification of a value of a parameter of the reward allocation model follows a Normal law.

[0022] In particular embodiments, the selection model is an (artificial) neural network, for example a deep neural network. For example, the selection model is a feedforward neural network (FNN).

[0023] In particular modes of implementation, the selection model is previously trained, for example by supervised learning.

[0024] In particular embodiments, the reward assignment model is a machine learning model, for example, a neural network such as a deep neural network. For example, the reward assignment model is a feedforward neural network (FNN).

[0025] In particular modes of implementation, the reward allocation model is previously trained by supervised learning.

[0026] In particular modes of implementation, the control action selected by the selection module controls at least one of: - a filter to be applied to a radio signal, - a filter to be applied to audio content extracted from a radio signal phonic, - a decision to switch between a first reception frequency of a radio station and a second reception frequency of the same radio station, - a decision to switch between an analog radio broadcasting system and a digital radio broadcasting system.

[0027] In particular embodiments, the radio reception device being on board a motor vehicle, a phase of use corresponds to an activation of the radio reception device on a journey of the motor vehicle.

[0028] According to a second aspect, there is provided a computer program product comprising instructions which, when executed by a radio reception device comprising at least one tuner, a processing circuit, a sound broadcasting module and a human-machine interface, configure said radio reception device to implement a radio reception method according to any one of the embodiments of the present disclosure.

[0029] According to a third aspect, there is provided a computer-readable recording medium on which is recorded a set of instructions which, when executed by a radio reception device comprising at least one tuner, a processing circuit, a sound broadcasting module and a human-machine interface, configure said radio reception device to implement a radio reception method according to any of the embodiments of the present disclosure.

[0030] According to a fourth aspect, a radio reception device is proposed comprising at least one tuner, a processing circuit, a sound broadcasting module and a human-machine interface, configured to implement a radio reception method according to any one of the embodiments of the present disclosure.

[0031] According to a fifth aspect, there is provided a motor vehicle comprising a radio reception device according to any one of the embodiments of the present disclosure. Presentation of figures

[0032] The invention will be better understood upon reading the following description, given as a non-limiting example, and made with reference to the figures which represent: - [Fig. 1][Fig. 1]: a schematic representation of an exemplary embodiment of a radio broadcasting system, - [Fig.2][Fig.2]: a schematic representation of an example of an embodiment of a radio reception device, - [Fig.3][Fig.3]: a schematic representation of a reinforcement learning system based on user feedback from the radio reception device, - [Fig.4][Fig.4]: a diagram illustrating the main steps of an example implementation of a radio reception process.

[0033] In these figures, identical references from one figure to another designate identical or similar elements. For reasons of clarity, the elements represented are not to scale, unless otherwise indicated.

[0034] Furthermore, the order of steps shown in these figures is given solely as a non-limiting example of the present disclosure which may be applied with the same steps performed in a different order and / or with steps performed in parallel and / or jointly. Description of the embodiments

[0035] [Fig.l] schematically represents an example of a radio broadcasting system comprising a transmitting antenna 11, which transmits radio signals in a coverage area 12.

[0036] It should be noted that the present disclosure relates to radio reception in general, and is applicable to all types of radio broadcasting systems, in particular to all types of analog ("Amplitude Modulation", AM, "Frequency Modulation, FM, etc.) and / or digital ("Digital Audio Broadcasting", DAB, etc.) radio broadcasting systems, including in the case of a switch from an analog radio broadcasting system to a digital radio broadcasting system, and vice versa, for example to perform radio station tracking.

[0037] [Fig.l] also represents a motor vehicle 13 in which a radio reception device 20 is mounted. The radio reception device 20 carries out the sound broadcasting of the audio content originating from a radio station selected by a user of the motor vehicle 13. This audio content is extracted from radio signals received from the transmission antenna 11.

[0038] [Fig.2] schematically represents an exemplary embodiment of a radio reception device 20.

[0039] As illustrated by [Fig.2], the radio reception device 20 comprises at least one tuner 21 ("tuner" in English literature). In the example illustrated by [Fig.2], the radio reception device 20 comprises a single tuner. In other examples not illustrated by figures, the radio reception device 20 may comprise two or more tuners.

[0040] In a known manner, a tuner 21 makes it possible to bring a radio signal, received in a channel associated with a frequency, back into baseband, the frequency being for example selected by a user. For example, a tuner 21 provides at output, in analog or digital form, a baseband signal which corresponds to the selected radio signal brought back to a zero or intermediate frequency. Such a tuner 21 comprises equipment (amplifier(s), local oscillator(s), mixer(s), analog and / or digital filter(s), analog / digital converter(s), etc.) considered to be known to the person skilled in the art. Such a tuner 21 may also comprise and / or be connected to other equipment (antenna(s), etc.) considered to be known to the person skilled in the art.

[0041] As illustrated by [Fig.2], the radio reception device 20 also comprises a processing circuit 22. The processing circuit 22 comprises, for example, one or more processors 220 (CPU, DSP, GPU, FPGA, ASIC, etc.). In the case of several processors 220, these may be integrated into the same equipment and / or integrated into hardware-distinct equipment communicating with each other. The processing circuit 22 also comprises one or more memories 221 (magnetic hard disk, electronic memory, optical disk, etc.) in which, for example, a computer program product 222 is stored, in the form of a set of program code instructions to be executed by the processor(s) to implement all or part of the steps of a radio reception method 40 which will be described below.

[0042] As illustrated by [Fig.2], the radio reception device 20 also comprises a sound broadcasting module 23, making it possible to deliver audio content to users. The sound broadcasting module 23 may for example comprise one or more loudspeakers and / or one or more connection modules for headphones. A connection module may for example allow a wired connection (jack plug, USB port, etc.) or a wireless connection (Bluetooth communication module, WiFi, etc.) with headphones.

[0043] As illustrated in [Fig. 2], the radio reception device 20 also includes a human-machine interface 24 (HMI), allowing a user of the radio reception device 20 to interact therewith. LTHM 24 may take any suitable form allowing the radio reception device 20 to transmit information to a user and allowing a user to transmit information to the radio reception device 20. For example, LTHM 24 includes a touch screen and, optionally, a microphone. It should also be noted that, in certain exemplary embodiments, LTHM 24 may use the sound diffusion module 23 to transmit information to the user.

[0044] In certain exemplary embodiments, and as illustrated in the non-limiting example of [Fig. 2], the radio reception device 20 may optionally comprise a location module 25, making it possible to determine a geographical position of the radio reception device 20. Any type of location module 25 may be considered within the scope of the present disclosure and the choice of a particular type of location module 25 constitutes only a variant implementation of the present disclosure. In preferred embodiments, the location module 25 corresponds to a reception module of a global navigation satellite system (GNSS), for example a GPS reception module (Global Positioning System).

[0045] As indicated previously, the present disclosure is based on a reinforcement learning system. [Fig. 3] schematically represents the main components implemented in such a reinforcement learning system. As illustrated by [Fig. 3], the radio reception device 20 comprises a selection model 223, which implements a selection policy for selecting a control action based in particular on the radio state of the radio reception device 20, and a reward allocation model 224, which allocates a reward to a selected control action. As illustrated by [Fig. 3], the selection model 223 and the reward allocation model 224 are for example implemented by the processing circuit 22.The radiophonic state of the radiophonic reception device 20 is for example measured and / or determined from measurements carried out by means of the tuner 21 (and possibly by means of the location module 25), and the control action selected by the selection model 223 is for example executed by said tuner 21 and / or by the processing circuit 22.

[0046] In a non-limiting manner, O denotes a set of radio states of the radio reception device 20, A a set of control actions that can be implemented to control the processing of the radio signals by the radio reception device 20, and T a set of rewards that can be allocated by the reward allocation module 224.The selection model 223 receives at each instant1 an observation ot GO of the radiophonic state in which the radiophonic reception device 20 is located (for example provided by the tuner 21), and determines a control action at £ A to be carried out on the basis of a selection policy TT ■ A, that is to say at = JT(of) • Following the carrying out of this control action at, the radiophonic reception device 20 passes into a radiophonic state °t+i and the reward allocation model 224 generates a reward rt e Y on the basis of a reward function . r : O x A -» T, i.e. rt = r( ot, at)- The selection model 223 is a machine learning model, which can therefore be adjusted by reinforcement learning to select a control action which tends to optimize a predicted (possibly weighted) sum of rewards which can be accumulated over time following the implementation of this control action.

[0047] Generally, any reinforcement learning algorithm can be implemented to optimize the selection policy implemented by the selection module 223. According to a non-limiting example, the reinforcement learning algorithm is a “Q-learning” type algorithm (see for example the document “Reinforcement Learning, An Introduction, Second Edition”, Richard S. Sutton and Andrew G. Barto, The MIT Press, Cambridge, Massachusetts, 2018), preferably a deep “Q-learning” type algorithm. In such a case, the selection model 223 includes a deep neural network (usually referred to as a "deep Q-network" in the English literature) to model a Q-function that predicts a future gain (or Q-value) resulting from a control action at selected at a time \ For example, for a selection policy that must select a sequence of control actions ( ay a2, ■ ■ ■, ), the Q-function is by example defined according to the following expression:

[0048] [Math.l] 2 O st, at) =

[0049] expression in which Et corresponds to the expectation with respect to the decision policy 77.

[0050] According to another example, the function Q is defined according to the following expression:

[0051] [Math.2] ( J \

[0052] expression in which 0 < y < 1 is a weighting factor used to give more importance to rewards obtained in the long term than to rewards obtained in the short term.

[0053] The reinforcement learning algorithm aims to automatically learn the Q function, so that the optimal selection policy 77 can be constructed by selecting the control action which, from a given radio state, maximizes the Q function. Any reinforcement learning method can be implemented to perform the automatic learning of the Q function, and the choice of a particular reinforcement learning method is only one implementation variant of the present disclosure.

[0054] As indicated above, the selection model 223 determines an action of control as a function of a radiophonic state of the radiophonic reception device 20. The radiophonic state is for example defined by one or more state parameters such as: - a temporal or frequency representation of a radio signal received by the radio reception device 20, or parameters determined from an analysis of such a temporal or frequency representation (for example an evaluation of variability of the amplitude of the radio signal, in particular in the case of a modulation assumed to be at constant amplitude, or even an evaluation of a width of a frequency band occupied by the radio signal, etc.), - an estimate of a quantity representative of a reception quality of a radio signal (for example an estimate of a reception power of the radio signal, of a frequency drift affecting the radio signal, of a signal-to-noise ratio of the radio signal, of a propagation channel between the transmission antenna 11 and the radio reception device 20, etc.), - one or more reception frequencies on which audio content from the same given radio station is received in the geographical area in which the radio reception device 20 is located, - one or more identifiers of radio stations whose audio content is received in the geographical area in which the radio reception device 20 is located, and optionally the associated reception frequencies, - a parameter determined by analysis (for example temporal or frequency) of audio content extracted from a radio signal (for example an estimation of the type of audio content - for example among speech, music, etc. -, or an estimation of the type of music corresponding to the audio content - for example among classical music, rock, pop, etc. -, etc.), - a geographical position of the radio reception device 20, for example provided by the location module 25, - etc.

[0055] The control action makes it possible to control all or part of the processing applied to the radio signals by the radio reception device 20. For example, the control action can modify the values ​​of processing parameters applied by the radio reception device 20. For example, a control action selected by the selection module 223 can for example provide one or more of the following processing parameters: - the coefficients of a filter to be applied to a radio signal (for example by calculating the filter coefficients, or by selecting a filter from among several predefined filters, etc.), - the coefficients of a filter to be applied to audio content extracted from a radio signal (for example by calculating the filter coefficients, or by selecting a filter from several predefined filters, etc.), - a decision to switch from a first reception frequency to a second reception frequency in the context of monitoring a radio station whose audio content is received, in the geographical area in which the radio reception device 20 is located, both on the first reception frequency and the second reception frequency (such a decision corresponds for example to an indication of the reception frequency to be considered, or even to a determination of threshold value(s) to be used to make the decision whether or not to switch from one reception frequency to another, etc.), - a decision to switch between an analogue radio broadcasting system and a digital radio broadcasting system (such a decision corresponds, for example, to an indication of the radio broadcasting system to be considered, or to a determination of threshold value(s) to be used to make the decision whether or not to switch between an analogue radio broadcasting system and a digital radio broadcasting system, etc.), - etc.

[0056] For example, a filter to be applied to a radio signal (for example brought back to an intermediate or zero frequency), which can be determined by the control action, may correspond to a dynamic propagation channel equalization filter (for example to obtain a substantially constant amplitude, in the case of a modulation assumed to be at constant amplitude, or to obtain a substantially constant phase, in the case of a modulation assumed to be at constant amplitude, or more generally to reduce the negative effects of multiple paths of the propagation channel, etc.). In such a case, the control action may for example adapt the coefficients of the dynamic propagation channel equalization filter as a function of the observed radio state, for example as a function of an evaluation of amplitude or phase variability of the radio signal, and / or as a function of an estimation of the propagation channel, etc.

[0057] Another non-limiting example of a filter to be applied to the radio signal, which can be determined by the control action, corresponds for example to a band-pass filter, the control action being able for example to adapt the bandwidth and / or cut-off frequencies of said band-pass filter as a function of the observed radiophonic state, for example as a function of a frequency representation of the radiophonic signal or of parameters determined from such a frequency representation (such as for example an evaluation of the width of the frequency band occupied by the radiophonic signal), etc.

[0058] For example, a filter to be applied to audio content extracted from a radio signal, which can be determined by the control action, may correspond to an audio equalization filter (for example to reinforce or attenuate the low frequencies or the high frequencies of the audio spectrum, etc.). In such a case, the control action may for example adapt the coefficients of the dynamic audio equalization filter according to the observed radio state, for example according to an identifier of the radio station whose audio content is broadcast to the user, and / or according to a parameter determined by temporal and / or frequency analysis of the extracted audio content (for example the type of music), etc.

[0059] For example, the decision to switch between a first reception frequency and a second reception frequency of the same radio station can be taken by the selection module 223 as a function of state parameters such as the estimated reception qualities (for example the estimated reception power, the estimated signal-to-noise ratio, etc.) on the first reception frequency and the second reception frequency respectively, and / or the geographical position of the radio reception device 20, etc.

[0060] For example, the decision to switch between an analog radio broadcasting system and a digital radio broadcasting system may be made by the selection module 223 based on state parameters such as the estimated reception qualities (e.g., the estimated reception power, the estimated signal-to-noise ratio, etc.) for the analog radio broadcasting system and the digital radio broadcasting system, respectively, and / or the geographical position of the radio reception device 20, etc.

[0061] It should be noted that the reinforcement learning algorithm is implemented, to evolve the selection model 223, during the use of the radio reception device 20, that is to say after the latter has been produced and installed in a motor vehicle purchased by a user.

[0062] Several strategies are possible for initializing the selection model 223, i.e. for defining the selection model 223 (for example a deep neural network such as an FNN) which will be implemented during the very first use of the radio reception device 20 by a user. For example, the selection model 223 can be initialized randomly. However, such random initialization could lead to a degraded user experience during first uses of the radio reception device 20, although the user experience will be gradually improved thanks to the reinforcement learning algorithm.

[0063] In preferred embodiments, the selection model 223 is previously trained, during the development of the radio reception device 20, by supervised learning. Conventionally, such supervised learning is based on a set of training data comprising reference data (reference scenarios) to be provided as input to the selection model 223, as well as the outputs (control actions) expected in response to these reference data, and the supervised learning aims to obtain a selection model 223 which tends to select the expected control actions for the reference scenarios considered. Such a set of training data is for example determined by experts and / or by calibration or simulation.The training data set may also be supplemented over time with information received from radio reception devices 20 in use, after their selection models 223 have been optimized by reinforcement learning according to the present disclosure. Such supervised learning therefore makes it possible to initialize the selection model 223 in such a way that the initial user experience provided by the radio reception device 20 is comparable to that provided by the prior art solutions. This user experience will, however, in the present case, be progressively improved thanks to the reinforcement learning algorithm.

[0064] As indicated above, the reinforcement learning algorithm further relies on user feedback. For this purpose, the reward allocation model 224 may also be required to evolve during use of the radio reception device 20, in order to attempt to progressively obtain an allocated reward (or gain / Q-value) that is more representative of the listening quality as perceived by the user of the radio reception device 20. The reward allocation model 224 must therefore be able to be modified by the radio reception device 20. The reward function implemented by the reward allocation model 224 is for example defined by different parameters, at least some of which have values ​​that can be modified.In preferred embodiments, the reward allocation model 224 is a machine learning model, such as a neural network, preferably a deep neural network (e.g., a FNN). Where applicable, everything said above for the initialization of the selection model 223 is also applicable for the initialization of the reward allocation model 224. In particular, in preferred embodiments, the reward allocation model 224 is prea . lably trained, during the development of the radio reception device 20, by supervised learning. Conventionally, such supervised learning is based on a training data set comprising reference data (reference scenarios) to be provided as input to the reward allocation model 224, as well as the outputs (rewards) expected in response to these reference data, and the supervised learning aims to obtain a reward allocation model 224 which tends to allocate the expected rewards for the reference scenarios considered. Such a training data set is for example determined by experts and / or by calibration or simulation.The training data set may also be supplemented over time with information received from radio reception devices 20 in use, after their reward allocation models 224 have been optimized by reinforcement learning based on user feedback according to the present disclosure.

[0065] Reinforcement learning is carried out after the production of the radio reception device 20, the latter being embedded in a motor vehicle, during phases of use of the radio reception device 20.

[0066] A phase of use of the radio reception device 20 corresponds to a phase beginning with the activation of said radio reception device 20 (such activation being able to be simultaneous with the starting of the engine of the motor vehicle 13) and ending with the deactivation of said radio reception device 20 (such deactivation being able to be simultaneous with the stopping of the engine of the motor vehicle 13).

[0067] To be able to carry out reinforcement learning by user feedback, the reward allocation model 224 is modified at the beginning of a usage phase, so as to obtain a modified reward allocation model 224, and this modified reward allocation model 224 is used for reinforcement learning of the selection model 223 over the duration of this usage phase (during which the reward allocation model 224 was modified), and possibly also over the duration of one or more subsequent usage phases. Generally, reinforcement learning can be carried out (before requiring user feedback as described below) over a single usage phase.However, if this is considered too short (for example, of a duration less than a first predetermined threshold value), then it is possible to continue reinforcement learning over one or more subsequent phases of use, for example until a cumulative duration considered sufficient is obtained (for example, greater than a second predetermined threshold value). In certain examples, it is also possible to carry out learning by ren . force over a predetermined number of usage phases. In summary, depending on the case, reinforcement learning, using the modified reward allocation model 224, can be carried out (before requiring user feedback as described below) over the duration of one or more usage phases.

[0068] [Fig.4] schematically represents the main steps of an example of implementation of a radio reception method 40.

[0069] As illustrated by [Fig.4], and as indicated previously, the radio reception method 40 comprises, at the start of a use phase, a step S40 of modifying the reward allocation model 224, so as to obtain a modified reward allocation model.

[0070] Such a modification of the reward allocation model 224 is for example carried out by modifying the parameter values ​​of the reward allocation model 224, for example randomly. For example, the random modification of a value of a parameter of the reward allocation model 224 follows a Normal distribution. For example, the Normal distribution used has a zero mean, and its variance is for example determined from the starting value (at the start of step S40) of the modified parameter of the reward allocation model 224. For example, for a parameter X whose starting value is %o, the random modification introduced follows for example a Normal distribution N ( 0, e • x0 ), expression in which 0 < s < 1 (and preferably 0 < s < 0.1, for example £ = 0.05), so that X~N ( Xq, s • x0) •

[0071] As illustrated by [Fig.4], the radio reception method 40 comprises, during the duration of one or more phases of use of the radio reception device 20, a step S41 of reinforcement learning of the selection model 223, which uses the modified reward allocation model 224 obtained during the step S40. As indicated above, the reinforcement learning step S41 can implement any type of suitable reinforcement learning algorithm, and the choice of a particular algorithm constitutes only a variant implementation of the present disclosure.By this reinforcement learning, the selection model 223 is modified iteratively, and the selection model 223, thus modified iteratively, is used to select the control actions of the processing applied to continuously extract audio content from the radio signals received by the radio reception device 20. The audio content thus extracted is broadcast to the user of the motor vehicle 13, by means of the sound broadcasting module 23, during a step S42.

[0072] As illustrated by [Fig.4], after the end of one or more phases of use implementing reinforcement learning using the modified reward allocation model 224, the radio reception method 40 comprises a step S43 of transmitting, by means of the HMI 24 and intended for the user of the radio reception device 20, a request for evaluation of the quality of the audio content broadcast during the duration of the usage phase which has just ended. It should be noted that the execution of the transmission step S43 may also, optionally, be subject to other conditions in addition to that relating to the completion of a usage phase (i.e. the fact that the radio reception device 20 is deactivated, i.e. no longer broadcasts audio content). In particular, in certain examples, the transmission step S43 may be executed only if the motor vehicle 13 is stopped (stationary), or even only if the engine of said motor vehicle 13 is stopped (if the radio reception device 20 is deactivated before stopping the engine of said motor vehicle 13).Such provisions make it possible to ensure that the user's return is requested when the motor vehicle 13 is stationary, that is to say in good conditions from a road safety point of view.

[0073] As illustrated in [Fig. 4], when the quality of the broadcast audio content is validated by the user (reference S43a in [Fig. 4]), the radio reception method 40 comprises a step S44 of storing the modified selection model 223 as selection model 223 and the modified reward allocation model 224 as reward allocation model 224, for a subsequent use phase. In other words, a positive feedback from the user amounts to validating the modifications made to the selection model 223 and the reward allocation model 224, and these modifications are consequently retained for subsequent use phases.

[0074] When the quality of the broadcast audio content is not validated by the user (reference S43b in [Fig.4]), the modified reward allocation model 224 is not stored as the reward allocation model 224 for the next use phase, and the radio reception method 40 comprises a step S45 during which the modifications made to the reward allocation model 224 are discarded. In this way, it is the reward allocation model 224 in force before the modification step S40 which is used for the next use phase.

[0075] In some cases, it is also possible, when the quality of the broadcast audio content is not validated by the user, not to store the modified selection model 223 as the selection model 223 for the next usage phase. Where applicable, the modifications made to the selection model 223 (by reinforcement learning using the modified reward allocation model 224) are also discarded during step S45. Where applicable, it is the selection model 223 in force before the modification step S40 which is used. for the following phase of use. However, it is also possible, in certain examples not illustrated by figures, to retain the modifications made to the selection model 223 even when the quality of the broadcast audio content is not validated by the user, because the reinforcement learning of said selection model 223 will continue during the following phases of use.

[0076] It should be noted that the quality of the audio content is considered not to be validated if the radio reception device 20 receives, via 1THM 24, negative feedback from the user (i.e., if the user indicates that he is not satisfied with the quality of the broadcast audio content). However, the quality of the audio content may also be considered not to be validated if the radio reception device 20 does not receive feedback from the user. For example, the quality of the audio content may be considered not to be validated if the user does not respond within a predetermined response time.

[0077] More generally, it should be noted that the implementation and embodiment methods considered above have been described as non-limiting examples, and that other variants are consequently conceivable.

[0078] In particular, the invention has been described by considering mainly the case of a radio reception device 20 on board a motor vehicle 13. It should however be noted that it is also possible, according to other examples, to have a radio reception device 20 which is not on board a motor vehicle 13, but which is for example carried by a user.

[0079] Furthermore, the invention has been described by considering mainly a selection module 223 and a reward allocation module 224. It should however be noted that it is also possible, according to other examples, to consider several pairs of selection module 223 / reward allocation module 224 to control different treatments. For example, it is possible to provide: - a first selection module 223, associated with a first reward allocation module 224, for selecting a filter to be applied to a radio signal or to audio content extracted from this radio signal, - a second selection module 223, associated with a second reward allocation module 224, to decide whether to switch between different reception frequencies or between different radio broadcasting systems, etc.

Claims

Claims

1. Method (40) for radio reception, implemented by a radio reception device (20) comprising a selection model (223) configured to select, as a function of a radio state of the radio reception device, a control action of processing carried out to extract audio content from radio signals received by the radio reception device, the selection model being a machine learning model, the radio reception device (20) further comprising a reward allocation model as a function of a radio state and a selected control action, said method comprising, during at least one phase of use of the radio reception device: - at the start of the at least one phase of use: a modification (S40) of the reward allocation model, so as to obtain a modified reward allocation model, - during the duration of the at least one phase of use: reinforcement learning (S41) of the selection model using the modified reward allocation model, so as to modify the selection model, and sound broadcasting (S42) of audio content extracted from radio signals processed according to control actions selected by the modified selection model, - after the end of the at least one phase of use: a transmission (S43), to a user of the radio reception device, of a request for evaluation of the quality of the audio content broadcast during the duration of the at least one phase of use, - when the quality of the broadcast audio content is validated by the user: a storage (S44) of the modified selection model as a selection model and of the modified reward allocation model as a reward allocation model, for a following use phase.

2. The method (40) of claim 1, wherein, when the quality of the If the streamed audio content is not validated by the user, the modified reward attribution model is not remembered as the reward attribution model for the next usage phase.

3. Method (40) according to any one of the preceding claims, wherein, when the quality of the broadcast audio content is not validated by the user, the modified selection model is not stored as a selection model for the next phase of use.

4. A method (40) according to any preceding claim, wherein the modified reward allocation model is obtained by randomly modifying parameter values ​​of the reward allocation model.

5. Method (40) according to claim 4, wherein the random modification of a value of a parameter of the reward allocation model follows a Normal distribution.

6. A method (40) according to any preceding claim, wherein the selection model is a neural network.

7. A method (40) according to any preceding claim, wherein the selection model is pre-trained by supervised learning.

8. A method (40) according to any preceding claim, wherein the reward allocation model is a machine learning model, for example a neural network.

9. The method (40) of claim 8, wherein the reward allocation model is pre-trained by supervised learning.

10. Method (40) according to any one of the preceding claims, wherein the control action selected by the selection module controls at least one of: - a filter to be applied to a radio signal, - a filter to be applied to audio content extracted from a radio signal, - a decision to switch between a first reception frequency of a radio station and a second reception frequency of the same radio station, - a decision to switch between an analog radio broadcasting system and a digital radio broadcasting system.

11. Method (40) according to any one of the preceding claims, in which, the radio reception device being on board a motor vehicle, a phase of use corresponds to an activation of the radio reception device on a journey of the motor vehicle.

12. Computer program product (222) comprising instructions which, when executed by a radio reception device (20) comprising at least one tuner (21), a processing circuit (22), a sound broadcasting module (23) and a human-machine interface (24), configure said radio reception device to implement a radio reception method (40) according to any one of the preceding claims.

13. A computer-readable recording medium on which is recorded a set of instructions which, when executed by a radio reception device (20) comprising at least one tuner (21), a processing circuit (22), a sound broadcasting module (23) and a human-machine interface (24), configure said radio reception device to implement a radio reception method (40) according to any one of claims 1 to 11.

14. 1 1. Radio reception device (20) comprising at least one tuner (21), a processing circuit (22), a sound broadcasting module (23) and a human-machine interface (24), configured to implement a radio reception method (40) according to any one of claims 1 to 11.

15. Motor vehicle (13) comprising a radio reception device (20) according to claim 14.

Images