Method for speech audiometry testing using speech recognition and associated electronic device

Abstract

Disclosed is a method for speech audiometry testing for a first patient (200) comprising: -a first step of acoustically transmitting a recording of a linguistic utterance including at least one transmission phoneme; -a first step of acoustically receiving a first voice response from the first patient to the first acoustic transmission comprising at least one response phoneme; -a step of determining, using an artificial neural network comprising an input and an output, on the basis of an item of input data obtained from the first response, a first output character string that is representative of the at least one response phoneme; -a step of comparing the first character string with a second character string that is representative of the at least one input phoneme.

Description

Speech audiometry testing method implementing voice recognition and associated electronic device

[0001] The invention relates to a method of speech audiometry.

[0002] Speech audiometry testing procedures conducted by an audiologist allow determining a patient's audio perception of linguistic expressions, particularly words.

[0003] These methods include:Acoustically emitting a recording of a linguistic expression,Acoustically receiving and recognizing a first patient's response, andComparing the linguistic expression with the patient's response.

[0004] Response recognition is implemented by the audiologist or more generally a person (in other words: human), which requires the mobilization of a person for the entire duration of the test.

[0005] To overcome this drawback, the invention relates to a method for testing the vocal audiometry of a first patient comprising the following steps: First acoustic emission of a recording of a linguistic expression comprising at least one emission phoneme (The first patient then reproduces by speech what he has heard.In other words, it emits a first vocal response),First acoustic reception of a first vocal response from the first patient to the first acoustic emission comprising at least one response phoneme,Determination, by an artificial neural network comprising an input and an output, from an input data item obtained from the first response, of a first output character string representative of the at least one response phoneme,Comparison of the first character string with a second character string, representative of said at least one input phoneme (in other words: representative of said at least one emission phoneme),.

[0006] The artificial neural network being trained, prior to the determination step, by implementing (or repeating) the following training steps (and the test method may include these steps): Second acoustic emission of the expression (The second patient mentioned below then reproduces by speech what he has heard.In other words, it emits a vocal response),Second acoustic reception of a second vocal response from a second patient to the second acoustic emission, the second response comprising at least one training phoneme (the second response is then, for example, recorded in memory, for example, in an audio file),Listening to the second response by a human (therefore, for example, the audio file),Reception of a third character string representative of the at least one training phoneme,Supervised training of the artificial neural network from the second input response labeled by the third character string (i.e.: the training of the neural network tends to ensure that the neural network produces the third character string as output when the second response is received as input).

[0007] Thus, the neural network automates the acquisition of the patient's response. Training the neural network from the expression allows:To avoid overinterpretation as in classic speech recognition neural networks (i.e.: classic neural networks look for the closest word in the language, even if the patient did not respond with this word)The network is trained to recognize a patient's response, even when the word responded by the patient is not a word in the language. Since the network is trained during speech audiometric tests, it receives words that are not words in the language (because the words emitted are meaningless, or because the patient responds with an error).

[0008] The steps of the above method can be repeated to assess the hearing of the first patient, for example for different input words (in the expression) for which the neural network was trained. The intensity of the acoustic emission can be varied during this repetition to estimate the patient's speech intelligibility thresholds.

[0009] According to one embodiment, the expression consists of a word (or several isolated words) or a word (or several words) preceded by an isolated article. Alternatively, the expression consists of more than one word (or more than two words) or one or more sentences.

[0010] The training step is preferably repeated with at least several different second patients. According to one embodiment, the training step is repeated (for example, at least 10,000 times) for, as input, less than 300 different expressions (and for example, more than 50 words) constituting lists, for different second patients. For example, these lists are the cochlear lists of Lafon or the disyllabic lists of Fournier;

[0011] Alternatively, the training step can be repeated for a larger number of words.

[0012] Preferably, the training steps (or the method steps) are repeated (e.g. at least 10,000 times) for a series of different expressions and / or for different second patients.

[0013] The series of different expressions consists of (or includes) fewer than 2000 different expressions, for example, fewer than 300 different expressions, (the series of different expressions including, for example, at least 10).

[0014] For example, the steps of the audiometric testing process are repeated for part of the series of expressions.

[0015] According to one embodiment, all or some of the different second patients are hard of hearing (or the second patient is hard of hearing). A patient is hard of hearing if, for example, at least one of his ears has an average hearing loss of more than 20 (or 41 or 71) decibels on at least one audiometric frequency, for example, equal to 500 hertz, 1000 hertz, 2000 hertz or 4000 hertz.

[0016] The input to the neural network can consist of multiple expressions. It is indeed more efficient to allow the neural network to work on multiple expressions at the same time.

[0017] For example, the neural network comprises a first part, comprising a first series of layers of the neural network, capable of producing a vector encoding the input data, and a second part, comprising at least one layer of the neural network, capable of producing the first output character string from the vector, the first part being pre-trained (and the method may comprise this pre-training step), prior to the training steps, from more than 10,000, at least, different expressions (or words) (and at most 1,000,000 different expressions or different words) as input.

[0018] According to one embodiment, at least a portion of the first part (for example, connected with the input) has fixed weights (i.e., connections between the layers) during the training step. The remainder of the neural network, apart from said at least one portion, is modified during the training.

[0019] Alternatively, the entire first part can be trained.

[0020] The first part is for example the neural network described in the article:

[0021] Alexei Baevski, Yuhao Zhou, Abdelrahman-Mohamed, Michael Auli: “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations” NeurIPS 2020.

[0022] According to one embodiment, in this first part, the weights of the convolutional encoder (denoted f) of the article which produces a latent representation of the input are fixed during the training step. The rest of the first part, in particular the transformer (denoted g) is modified during training.

[0023] Other neural networks than the one presented in this article are of course possible.

[0024] The data is for example an audio file (or part of an audio file) including the audio recording of the first response (for example in a format called "wav", in English for "Waveform Audio File Format" translated into French by the expression audio waveform format).

[0025] According to one embodiment, the first part is pre-trained from a truncated input (e.g., some parts of the audio recording have been removed).

[0026] Alternatively, the first part is trained from labeled data.

[0027] Thus, following pre-training, the first part is able to encode an input audio file (or audio data) into a vector containing a context representation.

[0028] Alternatively, the data is an image file, for example a spectrogram obtained from the first voice response. Such an approach is for example presented for general speech recognition in the following article:

[0029] Zhang, Wei, et al. “Towards end-to-end speech recognition with deep multipath convolutional neural networks. » International Conference on Intelligent Robotics and Applications. Springer, Cham, 2019.

[0030] For example, the training steps include the following step: Input, using a keyboard, of the third string of characters by the human, following the listening step.

[0031] Input can be done by a keyboard or any other type of human-machine interface.

[0032] The electronic device stores, for example, the recording of the vocal expression in memory.

[0033] The first acoustic emission of the recording of a linguistic expression can, for example, be made by headphones or a loudspeaker.

[0034] The second acoustic emission of the recording of a linguistic expression can, for example, also be carried out by headphones or a loudspeaker.

[0035] The first reception of a first response and / or the second reception of the second response is for example carried out by a microphone.

[0036] The artificial neural network is, for example, a convolutional network. The first part may include a transformer. In the case where the first part is the neural network presented in the article Alexei Baevski et al. above, the second part may consist of a single output layer comprising all the phonemes contained in the lists (the output vector indicates which phonemes were input).

[0037] The artificial neural network can be implemented by the central processing unit which can have the architecture of a computer, a microprocessor and a microcontroller.

[0038] By comparing the first string with the second string, the hearing abilities of the first patient can be determined conventionally.

[0039] The second answer can be listened to using headphones.

[0040] Each of the above headsets can be replaced with a speaker.

[0041] The process is implemented, for example, by an electronic device.

[0042] The invention therefore also relates to an electronic audiometric testing device configured to implement the steps of the method according to the invention.

[0043] The invention further relates to a computer program comprising instructions, executable by a microprocessor or a microcontroller, for implementing the method according to the invention, when the computer program is executed by a microprocessor or a microcontroller.

[0044] The characteristics and advantages of the electronic device and the computer program are identical to those of the method, which is why they are not repeated here.

[0045] According to a variant, the invention relates to a method for vocal audiometry of a first patient comprising the following steps: First acoustic emission of a recording of a linguistic expression comprising at least one emission phoneme (The first patient then reproduces by speech what he has heard. In other words he emits a vocal response), First acoustic reception of a first vocal response from the first patient to the first acoustic emission comprising at least one response phoneme, Determination, by an artificial neural network comprising an input and an output, from an input data item obtained from the first response, of a first output character string representative of a comparison of said at least one response phoneme with the at least one emission phoneme,

[0046] The artificial neural network being trained, prior to the determination step, by implementing (or repeating) the following training steps (and the test method may include these steps): Second acoustic emission of the expression (The second patient mentioned below then reproduces by speech what he has heard.In other words, it emits a second vocal response),Second acoustic reception of a second vocal response from a second patient to the second acoustic emission, the second response comprising at least one training phoneme (the second response is then, for example, recorded in memory, for example, in an audio file),Reception of a third character string representative of a comparison of said at least one emission phoneme with the at least one training phoneme,Supervised training of the artificial neural network from the second input response labeled by the third character string (i.e.: the training of the neural network tends to ensure that the neural network produces the third character string as output when the second response is received as input).

[0047] The hearing abilities of the first patient can be determined conventionally from the first character string.

[0048] Compared to the speech audiometry testing method, the speech audiometry method may have the advantage of not requiring reception of the training phoneme, which may avoid, for example, input during training following the second response.

[0049] The advantages and characteristics of the speech audiometry method are identical to those of the speech audiometry test method above (without the need to repeat them here), except for the listening and comparison step.

[0050] The invention therefore also relates to an audiometric electronic device configured to implement the steps of the audiometry method according to the invention.

[0051] The invention further relates to a computer program comprising instructions, executable by a microprocessor or a microcontroller, for implementing the speech audiometry method according to the invention, when the computer program is executed by a microprocessor or a microcontroller.

[0052] The characteristics and advantages of the electronic audiometric device and the computer program are identical to those of the audiometry procedure, therefore they are not repeated here.

[0053] It is understood that an element such as the electronic device, the central unit, or another element is “configured to” carry out a step or an operation, by the fact that the element comprises means for (in other words “is shaped to” or “is adapted to”) carrying out the step or the operation. These are preferably electronic means, for example a computer program, data in memory and / or specialized electronic circuits.

[0054] When a step or operation is performed or implemented by such an element, this generally implies that the element includes means for (in other words "is shaped to" or "is adapted to") performing the step or operation. This also includes, for example, electronic means, for example, a computer program, data in memory and / or specialized electronic circuits.

[0055] Other characteristics and advantages of the present invention will appear more clearly on reading the detailed description which follows, comprising embodiments of the invention given as non-limiting examples and illustrated by the appended drawings, in which:

[0056] represents an electronic device according to one embodiment of the invention.

[0057] represents an artificial neural network according to the invention

[0058] represents the method according to the invention, in a first exemplary embodiment, implemented by the electronic device of the.

[0059] represents the method according to the invention, in a second exemplary embodiment, implemented by the electronic device of the.

[0060] Detailed description of an exemplary embodiment of the invention

[0061] With reference to the, the neural network 400 comprises a first part 430, comprising a first series of layers of the neural network, capable of producing a vector 450 encoding an input data item 410, and a second part 440, comprising at least one layer of neurons, capable of producing the first output character string 420 from the vector 450.

[0062] The input data 410 is for example an audio file comprising the expression (for example in a format called “wav”, in English for “Waveform Audio File Format” translated into French by the expression waveform audio format).

[0063] For example, the neural network 400 is implemented by the central unit 110.

[0064] With reference to Figures 1, 2, 3, and 4, in step S10, the first part 430 is pre-trained, prior to the training steps below, from hundreds of hours of input audio files 410, comprising more than 10,000 different linguistic expressions (and at most 1,000,000 different first words).

[0065] Preferably, these are any expressions of a natural language.

[0066] This is self-supervised training.

[0067] For example, entry 410 is truncated (e.g., parts of the audio file have been removed).

[0068] The first part 430 is for example as described in the article:

[0069] Alexei Baevski, Yuhao Zhou, Abdelrahman-Mohamed, Michael Auli: “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations” NeurIPS 2020.

[0070] During pre-training, for example, this first part 430 attempts to predict the truncated parts of the input 410 and / or uses a contractive loss to evaluate the performance and modify the weights of the neural network (A cost function quantifies the error of the neural network in comparison to the label and represents the cost as a function of the combinations of parameters of the neural network).

[0071] Thus, following the pre-training, the first part 430 is able to encode an audio file (or audio data) as input 410 into a vector 450 comprising a representation of the linguistic expressions contained in the audio file.

[0072] According to one embodiment, in this first part 430, the weights of the convolutional encoder 431 (denoted f) in the article Alexei Baevski et al. above which produces a latent representation of the input, are fixed during the training step. The remainder 432 of the first part 430, in particular in the transformer (denoted g), in the article Alexei Baevski et al. above, is modified during training.

[0073] The second part 440 can be a single output layer comprising all the phonemes contained in the lists (the output vector indicates which phonemes are known). A "softmax" and a "log" can be applied to the output. This layer can be linear, i.e. comprising a linear activation function.

[0074] In step S20, the electronic device 100 controls the emission of a linguistic expression by a headset 170.

[0075] The linguistic expression is for example the word "raw".

[0076] The expression can consist of a single word or a single word and an article preceding that word (such as "le rondin") or a sentence.

[0077] The expression may include a logatome (i.e., a meaningless word) or consist of logatomes.

[0078] At step S30, patient 210 responds by reproducing in speech what he heard.

[0079] In step S40, the central unit 110 (which has for example the architecture of a computer, a microprocessor or a microcontroller) receives the patient's vocal response, via the microphone 130 and records it in memory 111 in the form of an audio file.

[0080] For example, patient 210 may respond “dru” when saying this word.

[0081] In step S50, the recorded response is then listened to by the human operator 220 using the headset 180.

[0082] In step S60, the human operator 220 enters “dru” via the keyboard 160 a character string by the operator 220.

[0083] In step S70, the character string “dru” is received from the keyboard 160 by the central unit 110.

[0084] In step S80, the neural network is trained from the recorded response (in the form of an audio file) as input 410 labeled by the character string “dru”.

[0085] According to one embodiment, the training steps S20, S30, S40, S50, S60, S70 and S80 are repeated (for example, at least ten thousand times), for different patients, for a series of different expressions instead of the expression "raw".

[0086] The set of different expressions can consist of less than 300 different expressions or less than 2000 different expressions.

[0087] The series of different expressions can be made up, for example, of Lafon's cochlear lists comprising 20 lists of 17 words, or 340 expressions, numbered from 1 to 20. Table 1 contains an extract from these lists (more precisely lists 1 to 6).

[0088] For example, steps S20, S30, S40, S50, S60, S70 and S80 can be performed for 6500 patients: 3500 patients for each word in lists 1 to 5, then 1500 (other) patients for each word in lists 6 to 10, then 2000 (other) patients for each word in lists 11 to 15, then 2500 (other) patients for each word in lists 16 to 20.

[0089] List 1List 2 List 3) List 4 List 5 List 6DustRideJibActWaveFangLobeBetterMatColStrongSoupMowingCalfSwimStumpGrogneBileDros sageSheathThreadRawBallWedgeGoodBankGroundTempleFawnPhaseMuleCatReignProwlsCleftStemGrainCellarBubbleSumMaineWorthyEdgeRustDareSiteBuoySaveChanceWinAbbotSouthFalseJoistDaggerAcquiredTownMareWeddingBaitRoadEyelashPartyWeakChairTailDungBallSoudWallShipChangePledgeHoleEvilTonneFearCrawlsFleaCornQuicklyRanceFlyGirlbilleddoubtBillDouteBeechLoinGaveAloneFriendCupOakMeadowOnHairFlightForeheadCunningLadlePrison

[0090] Of the 10,500 patients, 3,000 may be hard of hearing, for example.

[0091] For training, the cost function used is, for example, of the connectionist temporal classification type.

[0092] In step S90, an audiometric test on a patient 200 is initiated by the electronic device 100. To simplify the description of this embodiment, the pre-training, the training on the patient 210, and the audiometric test on the patient 200 are carried out by the same electronic device 100, but generally, most often, these three steps are carried out by different devices.

[0093] In step S90, the expression “raw” is output by the central unit 110 using the headset 120. The expression can be stored in memory 111.

[0094] Patient 200 may respond, for example, "dru" by saying this word.

[0095] In step S100, the central unit 110 receives the patient's voice response "dru" via the microphone 130 and records it in the form of an audio file.

[0096] In step S110, the neural network 400, implemented by the central unit 110, determines at output 420 the character string “dru” from the response of the patient 200 in the form of an audio file at input 410 of the neural network 400.

[0097] At step S120, “raw” is compared with “dru” to evaluate the hearing of patient 200.

[0098] Steps S90, S100, S110 and S120 may be repeated for different expressions for which the neural network 400 has been trained. Thus, the hearing of the patient 200 is evaluated.

[0099] For example, steps S90, S100, S110 and S120 can be repeated for the other words in list 2 table 1.

[0100] The artificial neural network 400 is for example a convolutional network. The first part 430 may comprise a transformer.

[0101] La represents a second example of embodiment of the method according to the invention. In this second example, the steps S10', S20', S30', S40', S90' and S100' respectively are identical to the steps S10, S20, S30, S40, S90 and S100 respectively,

[0102] In step S70', the character string "F" is received from the keyboard 160 by the central unit 110. "F" indicates that the patient 210 issued an incorrect response, because "cru" is different from "dru". The character string "V" would indicate that the patient 210 issued an accurate response. In another example, the character string "FVV" may be received from the keyboard 160 by the central unit 110. "F" indicates that the patient 210 issued an incorrect response and "V" indicates that the following phonemes are correct, because "c" is different from "d".

[0103] In step S80', the neural network is trained from the recorded response (in the form of an audio file) at input 410 labeled by the character string "F" or "FVV"

[0104] The training steps can be repeated in the same way as for the first example in the.

[0105] In step S110', to evaluate the hearing of the patient 200, the neural network 400, implemented by the central unit 110, determines at output 420 the character string "F" or "FVV" from the response of the patient 200 in the form of an audio file at input 410 of the neural network 400.

[0106] Steps S90', S100', S110' can be repeated for different expressions for which the neural network 400 has been trained. Thus, the hearing of the patient 200 is evaluated.

Claims

A speech audiometry test procedure for a first patient (200) comprising the following steps: First acoustic emission (S90) of a recording of a linguistic expression comprising at least one emitted phoneme; First acoustic reception (S100) of a first speech response from the first patient to the first acoustic emission comprising at least one response phoneme; Determination (S110), by an artificial neural network (400) comprising an input (410) and an output (420), from input data (410) obtained from the first response, of a first output string (420) representing at least one response phoneme; Comparison (S120) of the first string with a second string, representative of said at least one input phoneme; The artificial neural network (400) being trained, prior to the determination step,by implementing the following training steps: Second acoustic emission (S20) of the expression, Second acoustic reception (S30) of a second vocal response from a second patient (210) to the second acoustic emission, the second response including at least one training phoneme, Reception of a third string of characters (S70) representative of the at least one training phoneme, Supervised training (S80) of the artificial neural network (400) from the second input response (410) labeled by the third string of characters. A speech audiometry test method according to claim 1, wherein the expression consists of a single word or a single word and an article preceding that word. A speech audiometry test method according to claim 1 wherein the expression consists of more than one word. A speech audiometry test method according to any one of the preceding claims, wherein the neural network (400) comprises a first part (430), comprising a first series of neural network layers, capable of producing a vector (450) encoding the input data (410), and a second part (440), comprising at least one layer of neurons, capable of producing the first output character string (420) from the vector (450). A speech audiometry test method according to the preceding claim, wherein the first part (430) is pre-trained, prior to the training steps, from more than 10,000 different words. A speech audiometry test method according to claim 4 or 5 wherein the first part (430) is pre-trained from a truncated input (410). A speech audiometry test method according to any one of claims 4 to 6 wherein a portion of the first part (430) is at fixed weights during the supervised training step. A speech audiometry test method according to any one of the preceding claims, wherein the data is an audio file comprising the expression. A speech audiometry test method according to any one of the preceding claims, wherein the training steps comprise the following step: Input, via a keyboard, of the third string of characters by a human. A speech audiometry test method according to any one of the preceding claims, wherein the training steps are repeated for different second patients. A speech audiometry test method according to the preceding claim, wherein all or some of the different second patients are hearing impaired. A speech audiometry test method according to any one of the preceding claims, wherein the training steps are repeated for a series of different expressions. A speech audiometry test method according to the preceding claim, wherein the series of different expressions consists of fewer than 2000 different expressions. electronic audiometric test device (100) configured to implement the steps of the process according to any one of claims 1 to 13. Computer program comprising instructions, executable by a microprocessor or microcontroller, for implementing the method according to any one of claims 1 to 13, when the computer program is executed by the microprocessor or microcontroller.

Images