Very low bitrate audio compression

Abstract

Very low bitrate audio compression. The invention relates to a method and device for generating textual data representative of an audio signal comprising at least one voice segment. It comprises, on a terminal (DE), the following steps: - identifying in the audio signal (S1) at least one voice segment containing at least one voice information; - extracting (S4) from the voice segment at least one voice context data point (LEM), representative of an emotion associated with said voice information; - preparing (S6) a textual description of said voice segment, comprising at least said voice information and said voice context data point. Fig. 4

Description

Title of the invention: Very low bitrate audio compression Scope of the invention

[0001] The invention relates generally to telecommunications, and more specifically to communications involving the transmission of audio streams containing voice information. It applies more particularly to terminals equipped with physical and software resources including a microprocessor and a speech recognition and synthesis module. Previous Art

[0002] Voice compression generally uses conventional audio compression techniques, for example MPEG3, AAC, etc. These techniques are capable of efficiently compressing audio signals, including speech. However, they take little account of the signal's content, particularly the presence of speech information and the emotions associated with such speech, which enable a faithful reconstruction of the speech signal.

[0003] Recently, new techniques have emerged in this field for associating emotions with a speech signal. These are known as "emotional speech synthesis" (hereafter abbreviated as "ESS"). For example, US patent application 2022 / 0122580 A1 describes a system capable of associating an emotion with a human voice and synthesizing it in such a way as to reflect that emotion.

[0004] However, none of these techniques are concerned with the encoding of such voice and emotion information, and with the transmission of such information in a very compact manner in order to transmit it over a very low-speed network between a transmitter and a receiver.

[0005] An additional difficulty arises when such a speech signal is accompanied by sound signals such as music, background noise, etc.

[0006] There are therefore needs to encode and transmit an audio signal containing speech at very low bitrates. Description of the invention

[0007] The invention improves upon the state of the art.

[0008] To this end, it proposes a method for generating textual data representative of an audio signal comprising at least one voice segment, the method being characterized in that it comprises, on a terminal called the transmitting terminal, the following steps: - identify in the audio signal said at least one voice segment containing at least one voice information; - extract from the vocal segment at least one piece of vocal context data, representative of an emotion associated with the vocal information; - prepare a textual description of said vocal segment, including at least said vocal information and said vocal context data;

[0009] Advantageously, according to the invention, a textual description of the audio signal containing speech is performed by taking into account emotional information associated with the vocal context. The speech and the emotional information are represented and optionally encoded in text form. It is thus possible to store or transmit this textual description at very low data rates, since the representation of a text, or sequence of ASCII characters, occupies, as is known, very little space compared to audio data, even compressed, on a storage medium or on a transmission channel.

[0010] By "audio signal" we mean the representation of any sound, in analog or digital form, for a certain duration.

[0011] By "vocal segment" is meant a portion of the audio signal that contains speech information. Such a segment may, for example, correspond to a fixed duration, or to a speech segment ending in silence, etc. It may comprise a syllable, a word, a sentence, or a set of sentences spoken by one or more speakers. It may also contain, in addition, other sound information that is not considered speech (music, song, ambient noise, etc.).

[0012] By “emotion” we mean a state of the speaker which is reflected in a variation of the characteristics of the audio signal which carries his voice, for example anger, joy, etc.

[0013] By “textual description”, we mean a description of the audio signal in the form of text, that is to say a sequence of characters (letters, numbers, special characters, symbols, etc.) which can be represented for example by ASCII codes.

[0014] Other non-specifically emotional contextual data (such as voice intensity) may also be part of the textual description.

[0015] According to a particular embodiment of the invention, the process as described above further comprises the steps of extracting from said at least one vocal segment at least one piece of sound context data, representative of the sound content of the vocal segment, and in which said textual description further comprises said sound context data.

[0016] Advantageously, according to this method, the voice segment further includes a sound context, to which sound context data is associated. This sound context data is also represented in textual form. It may be a description of the sound content (for example, a song title, a piece of music) or an address of the audio content, the address being represented as text (for example, an http address). Advantageously, this also means that the audio content takes up very little space on the transmission channel.

[0017] According to a particular embodiment of the invention, the audio signal further comprises at least one non-vocal segment, and the method as described above further comprises the following steps: - identify said at least one non-vocal segment containing at least one piece of sound context data, representative of the non-vocal content; - prepare a textual description of said non-vocal segment, including at least said sound context data.

[0018] Advantageously, according to this method, the signal includes non-vocal segments in addition to vocal segments. By "non-vocal segment" is meant a portion of the audio signal that does not contain speech information as defined herein. Such a segment may, for example, correspond to a fixed duration, or to a section of music ending in silence, etc. It is also prepared before transmission in the form of text data. As explained previously, this may be a description of the audio content (such as a song title) or an address of the audio content, said address being represented in text form (for example, an HTTP address). Thus, advantageously, the audio signal is represented as a succession of text representations of vocal and non-vocal segments, which take up very little space on the disk or transmission channel.

[0019] According to a particular embodiment of the invention, the textual description is a set of data in XML or JSON format.

[0020] Advantageously according to this mode, a standardized text format is used to describe the voice and non-voice segments.

[0021] The XML language (from the English "eXtended Markup Language") comprises a list of data in the form of fragments classically described between an opening tag (< >) and a closing tag<!-- --> It offers many advantages, including being easy to read for both humans and machines. XML is a standard, and it is structured (the structure of an XML document is defined and can be validated by a schema), hierarchical, etc.

[0022] JSON (JavaScript Object Notation) is a lightweight language for exchanging textual data. It also offers many advantages. It is also more compact than XML.

[0023] According to a particular embodiment of the invention, the voice context data is generated by a classification module.

[0024] Advantageously, according to this method, a classification module such as a neural network can be trained to recognize the emotions conveyed by the voice. of a speaker, and subsequently perform automatic recognition. Thus, when a vocal segment is presented to the classifier, it automatically detects an associated emotion data.

[0025] According to a particular embodiment of the invention, the textual description is transmitted over a network.

[0026] Advantageously, according to this method, the textual description consisting solely of text is very well suited to a transmission network since, occupying very little space, it limits the network load. This is particularly important in the case of a very low-bandwidth network, for example a LoRa technology network, allowing for the structuring of a low-power and low-cost wide area network.

[0027] According to one variant, the transmission is carried out in the form of an SMS message.

[0028] Advantageously, according to this method, the voice and non-voice segments, in other words, the entire text stream, can be transmitted directly over a radio channel in the form of SMS messages. This offers numerous advantages, including relieving the mobile network between two devices, since the voices of the interlocutors use the SMS channel with a very low data rate and no longer the traditional voice channel.

[0029] SMS (Short Message Service) refers to a text message sent from one device, such as a mobile phone, to another device. The terms micro-messages or text messages are also used. The SMS messaging service allows the transmission of short text messages. It is one of the services of mobile telephony. It was introduced by the GSM standard.

[0030] According to a particular embodiment of the invention, the method further comprises a learning phase, including the steps of: - receive training audio segments including at least one voice prompt; - receive labeling data including at least one vocal context data representative of an emotion associated with the vocal information of the segment; - adapt a voice context detection system based on training audio segments and received labeling data.

[0031] Advantageously, according to this method, a classifier is trained beforehand to extract an emotion from an audio segment. The classifier, which can be, for example, a neural network, is thus well trained on the speaker's voice.

[0032] Correspondingly, the invention also proposes a method for generating an audio signal from a textual description comprising at least one voice segment, comprising the steps of: - extract from the textual description of the vocal segment at least one vocal piece of information and one piece of vocal context data representative of an emotion associated with the vocal piece of information; - synthesize the voice segment from the aforementioned voice information and voice context data at least.

[0033] According to one embodiment, the method of generating an audio signal further comprises a step of extracting from the textual description of the voice segment at least one sound context data representative of the sound content of the voice segment, and in which synthesizing the voice segment further uses the sound context data.

[0034] According to a material aspect, the invention also relates to a device for generating textual data representative of an audio signal comprising at least one voice segment, the device comprising a memory and a processor configured to: - identify in the audio signal at least one voice segment containing at least one voice information; - extract from the vocal segment at least one piece of vocal context data, representative of an emotion associated with the vocal information; - prepare a textual description of said voice segment, including at least said voice information and said voice context data.

[0035] According to a material aspect, the invention further relates to a device for generating an audio signal from a textual description comprising at least one speech segment, comprising a memory and a processor configured to: - extract from the textual description of the vocal segment at least one vocal piece of information and one piece of vocal context data representative of an emotion associated with the vocal piece of information; - synthesize the voice segment from the aforementioned voice information and voice context data at least.

[0036] The invention also relates to a system comprising a device for generating textual data representative of an audio signal and a device for generating an audio signal from a textual description as described above, the system being characterized in that the textual data representative of the audio signal are transmitted over a network between the textual data generation device and the audio signal generation device.

[0037] The invention also relates to a computer program comprising instructions for implementing one of the above methods according to any one of the particular embodiments described above, when said program is executed by a processor. The method can be implemented in various ways, including in hardwired or software form. This The program can use any programming language, and be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0038] The invention also relates to a computer-readable recording or information medium containing instructions for a computer program as described above. The aforementioned recording media can be any entity or device capable of storing the program. For example, the medium may include a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a hard drive. Furthermore, the recording media may be a transmissible medium such as an electrical or optical signal, which can be transmitted via an electrical or optical cable, by radio, or by other means. The programs according to the invention can, in particular, be downloaded from a network such as the Internet.

[0039] Alternatively, the recording media may correspond to an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the process in question.

[0040] These devices and computer programs have characteristics and advantages similar to those described above in relation to the corresponding methods. List of figures

[0041] Other features and advantages of the invention will become more apparent upon reading the following description of particular embodiments, given by way of simple illustrative and non-limiting examples, and the accompanying drawings, among which:

[0042] [Fig-1] Fig. 1 illustrates an example of an implementation environment for the invention according to a particular embodiment;

[0043] [Fig.2] Fig.2 illustrates the architecture of a data generator DE device textual (or "coder") according to embodiments of the invention;

[0044] [Fig. 3] [Fig. 3] illustrates the architecture of a DR signal-generating device audio (or "decoder") according to embodiments of the invention;

[0045] [Fig.4] Figure 4 illustrates steps in the text data generation process. according to one embodiment of the invention;

[0046] [Fig. 5] Fig. 5 illustrates steps in the audio signal generation process according to an embodiment of the invention;

[0047] [Fig. 6] Fig. 6 schematically represents an example of a device emotion prediction for the implementation of the process of [Fig.1] according to one embodiment;

[0048] [Fig.7] Fig.7 represents, in the form of a flowchart, the main steps of a process for training an artificial neural network for such an emotion prediction device;

[0049] [Fig.8] Fig.8 represents schematically a database of emotions according to one embodiment of the invention.

[0050] Description of an embodiment of the invention General principle of the invention

[0051] The general principle of the invention consists of transmitting, in place of a portion of audio stream comprising a speech signal, which would be compressed by a conventional technique, a textual representation of the content of this signal, for example in the form of ASCII characters, including in particular the voice of the user, as well as the associated emotions, and optionally information relating to a portion of non-vocal audio signal. Specific embodiments of the invention.

[0052] Fig. 1 illustrates an example of an implementation environment for the invention according to a particular embodiment.

[0053] According to this embodiment, a transmitting device, denoted DE, captures, analyzes and transmits the voice of a user to a receiving device, denoted DR, through a RES network. Optionally, the transmitted video stream may include, in addition to the user's voice, audio data, or sounds, such as background noise (traffic, birdsong, etc.) or music.

[0054] According to the embodiment presented, the transmitting device is a mobile terminal (smartphone). According to other embodiments, it may take the form of another terminal, for example a PC, a tablet, etc.

[0055] According to the embodiment presented, the receiving device is a laptop computer (PC). According to other embodiments, it may take the form of another terminal, for example a smartphone, a tablet, etc.

[0056] The RES network can be conventionally a mobile network (e.g., UMTS, 4G, 5G, etc.), the Internet, or other. It can be local or wide area network.

[0057] At the transmitting terminal, responsible for generating the stream, the user's voice is captured. This involves identifying the user's voice, but also their emotions, timbre, etc. The incoming stream is segmented into voice and non-voice segments. A speech-to-text conversion engine (STT) is used, along with an emotion analysis engine. The terminal then transmits the audio stream, which may contain only speech, or optionally a mixture of voice and traditional audio. The voice segments are encoded as text. The emotion information Associated emotions are encoded as specific data, known as vocal context data, reflecting the emotions associated with the speech segment. This context data is also represented as text. Optionally, traditional data corresponding to non-vocal segments (such as music, ambient noise, etc.), juxtaposed or superimposed on the speech, are also encoded as text.

[0058] At the receiving terminal, the signal is received and analyzed, and then a voice segment is synthesized from the received text and a model capable of reconstructing the speaker's voice with the emotions of the original signal. Such methods exist and are known as "emotional speech synthesis" (hereafter abbreviated as "ESS"). Non-vocal sound information can be added to the synthesized signal.

[0059] Figure 2 illustrates an architecture of a text data generator device (or "coder") according to embodiments of the invention.

[0060] The DE device has the classic architecture of a computer and includes, in particular, a memory memory (MEM), a processing unit equipped, for example, with a processor (PROC), and controlled by the computer program (PGM) stored in memory. At initialization, the code instructions of the computer program (PGM) are, for example, loaded into memory before being executed by the processor (PROC). The processor (PROC) of the processing unit implements, in particular, the steps of the text data generation process according to any one of the specific embodiments described in relation to Figures 4 and 6-8, according to the instructions of the computer program (PGM).

[0061] The DE device also includes: - An audio signal acquisition module (AUDIO IN) capable of separating an audio input into voice segments (containing speech) and non-voice segments (ambient sound, music, etc.). - A voice segment processing module (VOICE) working in association with an STT speech-to-text conversion module; - An EMOT module for processing user emotions on a voice segment, capable of generating one or more emotion labels from the voice segment; - An ENTR module for learning emotions, for example a neural network, capable of learning to associate an emotion with a vocal segment; the EMOT module corresponds to the implementation of this module, once the learning is complete. - A text output module for audio signals containing speech data and vocal context data. (representative of an emotion), and optionally sound context data. - A SOUND module for processing sound information; - A database containing, for example, an index or label associated with each known emotion (e.g., "1" for joy, "2" for anger, etc.). Optionally, it can also include prosodic characteristics of the signals associated with a particular emotion (variations in frequency, intensity, pronunciation, etc. related to that emotion).

[0062] Figure 3 illustrates an architecture of a DR device generating an audio signal from textual data (or "decoder") according to embodiments of the invention.

[0063] The DR device has the classic architecture of a computer and includes, in particular, a memory MEM', a processing unit UT', equipped, for example, with a processor PROC', and controlled by the computer program PGM' stored in memory MEM'. At initialization, the code instructions of the computer program PGM' are, for example, loaded into memory before being executed by the processor PROC'. The processor PROC' of the processing unit UT' implements, in particular, the steps of the process for generating an audio signal from a textual description according to any one of the specific embodiments described in relation to Figures 5 or 8, according to the instructions of the computer program PGM'.

[0064] The DR device also includes: - A text audio signal acquisition module (TEXT IN) capable of separating a text audio input into voice segments (containing speech and optionally non-voice information) and non-voice segments (ambient sound, music, etc.). - A voice segment processing module (VOICE') working in association with a TTS module for text-to-speech conversion; - An EMOT' module for processing user emotions on a voice segment, capable of modifying the audio playback of a voice segment by taking into account the emotion labels associated with the voice segment; - An emotion database BD', including data for the EMOT' module, for example prosodic parameters characteristic of signals associated with a particular emotion (variations in frequency, intensity, pronunciation, etc.), or a pointer to a specific neural network. - A SOUND' module for processing sound information; - An audio signal output module (AUDIO OUT) capable of reproducing speech audio segments associated with emotions and optionally non-vocal audio segments.

[0065] Figure 4 illustrates steps in the process of generating textual data according to one embodiment of the invention.

[0066] The method according to this embodiment receives a voice input in the form of an audio stream; it separates it into voice audio segments and non-voice audio segments. The voice segments are processed to provide a textual representation composed of character strings corresponding to the user's words and associated emotion data. It thus encodes a voice adapted to the emotions of the labeling data. The voice segments may also include sound (non-voice) information. The non-voice information (music, background noise, ambient noise, etc.) may also be associated with a textual description.

[0067] During an initial S0 step, the initialization of all the modules necessary for the implementation of the process is carried out, in particular the training of the EMOT module if it is carried out in the form of a neural network, or the appropriate populating of a database of emotions associated with recordings, etc.

[0068] In step SI, an audio stream is acquired by the audio acquisition module (AUDIO IN of [Fig. 2]). This audio stream is decomposed into different segments corresponding to speech-containing speech segments (SVC) or non-speech segments not containing speech to be processed within the meaning of the invention (SNVC). It should be noted that a song can be considered non-speech information within the meaning of the invention. There are various methods well known to those skilled in the art for extracting speech segments from an audio signal while ignoring noise, music, silence, etc. For example, it is possible to use, within a given time window (of a given duration, for example, 10 seconds), thresholding based on signal energy or other acoustic parameters, segmentation using hidden Markov models, or neural networks, etc. According to another example, the segment stops as soon as 3 seconds of silence are exceeded and a new segment begins.

[0069] Generally speaking, a speech segment corresponds to acoustic properties related to the speaker's vocal production mechanism. In particular, speech is characterized by a formant and non-stationary structure that reflects the resonance of the vocal tract. The alternation of voiced, unvoiced, and silent sounds also allows it to be distinguished from the properties of other sounds. A non-vocal segment, on the other hand, includes sounds possessing their own characteristics, such as silence, noise, music, etc. Music, for example, is characterized by a harmonic and stationary structure, a repetitive rhythm, and an absence of silence.

[0070] These notions can naturally be expanded or reduced without loss of generality: one can retain or not the non-vocal segments of silence, or of music, etc. Similarly, a vocal segment can contain music superimposed on speech, or an ambient noise (birdsong, lawnmower noises, etc.), which one can choose to treat (i.e. to include in the textual description) or not.

[0071] In an optional step S2, non-vocal information is extracted from a vocal or non-vocal segment. In one embodiment, a segment may be considered vocal (since it contains speech) but may also contain sound. In this case, step S2 can be applied to that segment. In one embodiment, a vocal segment has no sound, and step S2 is ignored. In another embodiment, only non-vocal sound segments are processed in step S2. In yet another embodiment, sound is not processed, meaning that step S2 is not implemented. When step S2 is implemented, it results in the extraction of information, or data, of the sound context associated with the segment, for example, the description of background noise (birdsong) or the address of a sound recording. Step S3 processes its textual representation.Steps S4 and S5, on the other hand, concern speech information (VOICE) from vocal segments, with or without associated sound.

[0072] In a step S4, the encoder submits the speech segment to an emotion detection module (EMOT module of [Fig. 2]). For this purpose, it uses, for example, an artificial neural network, as shown in [Fig. 6], previously trained on a corpus of speech segments, as shown in [Fig. 7]. In this case, the data is presented to the input of the previously trained neural network in an expected format, for example, a normalized vector of input data for the neural network, which is called the "input data" DEi. This data is of the same nature as that used for training.

[0073] According to another embodiment, a simple signal analysis technique, for example the analysis of the fundamental frequency (pitch), intensity, rate, articulation, etc. of speech, in other words prosody, can be envisaged: a given vocal segment is compared to the characteristics which are recorded in the database BD, and a decision is made on the nature of the emotion conveyed.

[0074] According to another embodiment, the vocal segment includes a keyword allowing the emotion label to be deduced (the speaker says: "I am sad and annoyed"; two emotion labels corresponding to "sadness" and "annoyance" are deduced).

[0075] At the exit of this step, one or more LEMi emotion labels are assigned to the vocal segment, such as 1 for "joy", N for "anger", etc. The emotion label corresponds to a vocal context data.

[0076] It should be noted that "emotion" is to be taken here in a broad sense. It can be characteristic of the intensity of the speech segment, the speed of elocution, etc. More generally, it is a contextual element representative of the vocal content.

[0077] In a step S5, the speech segment is converted into text using a speech-to-text conversion module (STT module of [Fig. 2]). Such a module is well known to those skilled in the art and will not be described further. For example, DeepSpeech software, a speech recognition engine from Mozilla©, can be used for this purpose.

[0078] Steps S4 and S5 can be parallel, as shown in [Fig. 4], or successive. In particular, one could consider "removing" the emotional component from the speech segment before converting the speech into text, for improved speech recognition.

[0079] Following steps S4 and S5, the speech segment is prepared for coding in a step S6. In one embodiment, the preparation consists of generating a text in the form of an XML language. In one example, a known markup language is used which provides a standard way of annotating text for the generation of synthetic speech, proposed by the W3C, called SSML (Speech Synthesis Markup Language (SSML) Version 1.1 - W3C Recommendation 7 September 2010), available at https: / / www.w3.org / TR / speech-synthesisll / .

[0080] An example of the use of such a language, applied to the Alexa© voice synthesizer from Amazon, is reproduced below:

[0081] <speak>

[0082] <amazon:émotion name="tristesse" intensity="medium">

[0083] I want to tell you a secret.

[0084]

[0085] cvoice name="Kendra"> I am not a real human

[0086] <amazon:émotion name="colère" intensity="high"> Impossible! < / Amazon: emotion

[0087] < / speak>

[0088] In the example above, the emotion label "sadness," or vocal context data, is associated with the following text ("I want to tell you a secret") and allows this emotion to be reflected during the audio playback of the segment by the decoder. The label "medium" is another vocal context data item, not specifically associated with an emotion, which allows the sound level to be adjusted during the audio playback of the segment.

[0089] During an optional S3 step, a textual representation of a (non-vocal) sound is generated. An example of a sound information representation is provided below. The "src" tag gives an address in the form of a URI (Uniform URI). A Resource Identifier (RID), in HTTP format, for an audio stream that, in this example, includes a musical recording and a recording of birdsong. The "descr" tag provides a description that can correspond to alternative content in case the source cannot be played (synthesized birdsong, or a statement such as "the birds are singing"). The "clipBegin" tag indicates the point at which the non-vocal sound is inserted into the segment. The non-vocal sound can be inserted over the voice at the beginning of the segment, or later. It can also have an end time (clipEnd), a volume level (soundLevel), etc. Reference can be made to the aforementioned W3C specification, which proposes a number of possible parameters. Naturally, the list specified in this specification is not exhaustive.

[0090] <audio>

[0091] <descr="Neuvième symphonie">

[0092] <src="https: / / www.mybd.com / fr / 9th" clipBegin="10s" clipEnd="20s" soundLevel="-6dB"

[0093] < / audio >

[0094] <audio> <descr="birds"><src="https: / / www.mybd.com / birds7th">< / audio >

[0095] In another example, the address may consist of an index pointing to a database known to both the sender and the receiver. For example, it may be a private music library belonging to the user of the sending device and known to the receiving device. In another example, the address provided may be intended to be recognized by a music recognition tool (e.g., an entry recognizable by Shazam©):

[0096] < audio > < src ="seven nation army" >< / audio >

[0097] <audio> <src="4567345">< / audio >

[0098]

[0099] Naturally, any other textual description within the grasp of a person skilled in the art may be considered. According to another embodiment, a JSON-type format is used.

[0100] During an S7 step, the segments are prepared for encoding and optionally concatenated into a single text file or stream. If, during the optional S3 step, non-speech (sound) information was extracted from a speech or non-speech segment, it is also inserted into the text representation. During an S7 step, speech and non-speech (sound) context data can be concatenated into the same text segment. Several segments can also be concatenated into the same XML description (the same "file," intended for storage or transmission). For example, this would result in a markup file describing three segments, as shown below: the first contains only speech; the second contains speech and sound (the Ninth Symphony, superimposed (to speech); the third contains only sound (birdsong). Each segment is introduced by the tag "segment" as in this example.

[0101] <segment=l>

[0102] <speak>

[0103] <amazon:émotion name="triste" intensity="medium">

[0104] I want to tell you a secret.

[0105]

[0106] < / speak>

[0107]

[0108] <segment=2>

[0109] cvoice name="Kendra">

[0110] <amazon:émotion name="colère" > I am not a real human. [YES]

[0112] <audio>

[0113] <descr="Neuvième symphonie">

[0114] <src="https: / / www.mybd.com / fr / 9th" clipBegin="2s">

[0115] < / audio>

[0116]

[0117] <segment=3>

[0118] <audio> <descr="birds"><src="https: / / www.mybd.com / birds7th">< / audio >

[0119]

[0120] Finally, during step S8, the data is encoded and transmitted over the communication channel. In one embodiment, the data is in XML format and encoded in ASCII. In another embodiment, the data is compressed using a standard tool suitable for compressing such data, such as the W3C's EXI (Efficient Extensible Interchange) encoding.

[0121] According to a preferred embodiment, the data is transmitted over a mobile network in the form of SMS-type messages, adapted to the text format.

[0122] According to one embodiment, the data is transmitted over a LORA type network, at very low speed and long range.

[0123] According to one embodiment, the data is stored in a file.

[0124] Figure 5 illustrates steps in the process of generating an audio signal from textual data (or decoding) according to one embodiment of the invention.

[0125] The method according to this embodiment receives input in the form of text and labeling data comprising emotional information associated with a voice segment, as well as optionally text inputs associated with non-vocal sounds. On this basis, it generates an audio stream comprising a voice adapted to the emotions of the labeling data and optionally sound data such as ambient noises, music, etc. Other non-specifically emotional data (such as voice intensity) can also be part of the textual description.

[0126] During an initial S10 step, all modules necessary for the implementation of the process are initialized.

[0127] In step SI 1, a text data stream is acquired by the text acquisition module (TEXT IN of [Fig. 3]). This text stream is decomposed into different segments corresponding to speech-containing (SVC) or non-speech-containing (SNVC) segments. Any implementation within the grasp of a person skilled in the art, allowing the beginning and end of a text segment to be identified, is possible. According to the example given in support of [Fig. 4], the stream is in XML format and each segment beginning and end is identified by an XML tag. According to the example provided in support of step S7, the extraction process receives a stream of three segments identified by appropriate tags. <segment>: the first segment contains only speech; the second contains speech and sound; the third contains only sound.

[0128] In an optional step S12, non-vocal information is extracted from a vocal or non-vocal segment. In one embodiment, a segment may be considered vocal (since it contains speech) but may also contain sound. In this case, step S12 can be applied to that segment. This is the case for segment number 2 in the example. In one embodiment, if a vocal segment has no sound, step S12 is ignored. In another embodiment, the sound is not processed, meaning that step S12 is not implemented. When step S12 is implemented, it results in the extraction of sound context information associated with the segment, for example, the description of background noise (birdsong) or the sound reproduction of a recording. Step S3 handles its audio synthesis. Steps S14 and S15, on the other hand, concern speech information (VOICE) from vocal segments.

[0129] In step S15, the extracted speech segment (“I want to tell you a secret”) is converted into sound (or synthesized) using a text-to-speech conversion module (TTS module of [Fig. 3]). Such a module is well known to those skilled in the art and will not be described further. For example, DeepSpeech, a speech synthesis engine from Mozilla©, can be used for this purpose.

[0130] In step S14, if it detects an emotion label in the segment, the decoder submits the speech segment to an emotion synthesis module (EMOT' module of [Fig. 3]). According to the embodiment proposed in support of [Fig. 4], the emotion label is extracted from the segment by reading the "emotion" tag (e.g., "sad"). Alternatively, an index ("1") pointing to parameters associated with the emotion can be read from a database.

[0131] The decoder uses, in one embodiment, a prosody-based signal synthesis technique for emotion representation, as presented in the article by Shroeder: “Emotional Speech Synthesis: A Review, EUROSPEECH 2001 Scandinavia, 7th European Conference on Speech Communication and Technology, 2nd INTERSPEECH Event, Aalborg, Denmark, September 3-7, 2001.” Such a model of emotion in speech relies on a number of prosodic parameters such as fundamental frequency (pitch), intensity, rate, articulation, etc.: a given speech segment is modified (or synthesized) by taking into account the characteristic parameters of emotion that are recorded in the database BD' of [Fig. 3] (or BD of [Fig. 2], or BD_1, BD_2 of [Fig. 8]). Such a set of parameters is illustrated in support of [Fig. 8].

[0132] According to another embodiment, the decoder uses an artificial neural network, which has been previously trained to modify a speech segment by adding an emotion to it. In this case, the emotion label and the synthesized speech data from step S15 are presented to the input of the previously trained neural network in an expected format, for example, a normalized vector of input data for the neural network. This data is of the same nature as that used for training. An example of such a neural network is detailed in the article "Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention," 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4784-4788, by H. Tachibana et al. This network can be adapted to take into account the emotion label(s) as input. [Fig.[8] presents an example of the use of neural networks specialized by emotion, or class of emotions.

[0133] Steps S14 and S15 can be parallel, as shown in [Fig. 5], or successive. In one embodiment, step S15 is performed first, then the speech is presented with the emotion at the input of step S14, in order to synthesize the speech with the emotion.

[0134] Following steps S14 and S15, the speech segment is prepared for audio playback in step S16. According to one embodiment, the preparation consists of generating an audio segment in a standardized format (WAV, MP3, AAC, etc.).

[0135] During an optional step S13, non-vocal sound information (which may belong to a vocal or non-vocal segment) is processed. In the example given above, the "audio" tag allows such sound information to be detected in segments 2 and 3. As discussed previously, it can take the value of a URI of a sound file, or an index pointing to a database known to both the sender and receiver, such as a private database or music recognition engine, etc. Optionally, a tag ("descr") provides a description that may correspond to alternative content in the event that the source could not be played; there may then be a synthesis of birdsong, or a statement of the type "the birds are singing".

[0136] During step S17, the segments are prepared for sound playback and optionally concatenated into a single audio file or stream. If, during the optional step S13, non-vocal (sound) information was extracted from a vocal or non-vocal segment, it is also inserted into the audio stream. During step S17, audio and vocal data (possibly including emotion) can be superimposed within the same audio segment. According to the example given above, this would result in an audio file comprising three segments: the first contains only speech; the second contains speech and sound (the Ninth Symphony, superimposed on the speech); the third contains only sound (birdsong).

[0137] Finally, during an S18 step, the data is output to a sound device (AUDIO OUT on [Fig.3]), for example a loudspeaker.

[0138] Fig. 6 schematically represents an emotion prediction device for implementing the user emotion processing module on a voice segment (EMOT module of Fig. 2) according to one embodiment.

[0139] It includes an artificial neural network, for example of the convolutional type.

[0140] The artificial neural network shown has been previously trained to identify an emotion label from among a set of emotion labels, according to a process detailed in support of [Fig.7].

[0141] Once the network is trained, it is capable of outputting an index, or label, LEMi, or voice context data, which allows access to an emotion (for example: "joy"), or a set of emotion labels (for example: "joy" and "rushed"), in the database containing the emotion labels. If the network has been trained correctly, this index corresponds to that of the emotion (or set of emotions) closest to the emotion conveyed by the user's voice in that segment.

[0142] According to the example shown: - The neural network comprises an input layer to which the DEi training data corresponding to the speech segment to be classified is presented, an output layer, and one or more hidden layers, each layer comprising a plurality of nodes, also called neurons. Each hidden layer, as well as the output layer, is associated with a mathematical operation, also called an activation function, performed at each node of said layer. In addition, each node of a hidden layer and of the output layer is "linked" to each node of the preceding layer by a weight, each node thus taking as input the result (or the value, for the input layer) of each node of the previous layer multiplied by the associated weight, i.e., the weight linking said node to said node of the previous layer. In such a case, step SO of [Fig.4] / S20 of [Fig.7] allows defining the number of layers, as well as the activation functions and the number of nodes of the hidden and output layers. - Training data can correspond to sampled raw audio data or to a set of audio signal parameters (prosody parameters such as variations in pitch, intensity, duration, etc. of the speech signal) or to a spectral image of the signal (for example a logarithmic spectral image, or MEL type spectrogram, whose frequencies are adapted to a human listener). - The neural network delivers in each node of the output layer a prediction score (SCi) corresponding to the score of belonging to emotion i, identified by its LEMi index (LEM1, LEM2 ... LEMN). - The neural network also includes a layer called "softmax" one of whose objectives, as is known, is to normalize the scores obtained to make them correspond to probabilities of belonging to label i, the sum of the probabilities of belonging being for example equal to 1. - During training, the index of the output node associated with the training emotion (which is known) must be as close as possible to the highest probability. Training consists of refining the different weights associated with the different neurons in the different layers to obtain such an output probability; a feedback loop (also called backpropagation), denoted here as RETRO, connected to a "softmax" type function, is used for this purpose. - Finally, the neural network according to this example includes an OHE (for One Hot Encoding) module which allows choosing a unique LEMi index as output (typically, the one with the highest probability).

[0143] Figure 7 shows, in flowchart form, the main steps of a method for training an artificial neural network for such an emotion prediction device, which can be part of the SO initialization step of Figure 4. This method is implemented by an ENTR training module (Figure 2) which is used to configure the neural network as shown as an example in Figure 6.

[0144] The artificial neural network is trained to identify one (or more) emotion label from a set of possible emotions; more specifically, the label is an emotion index from a set of emotion indices. The emotion label corresponds to a piece of vocal context data.

[0145] In a step S20, the structure of the artificial neural network is defined. Following the example given in [Fig. 6], this is a classification network based on the use of a CNN, a softmax function (SM), a backpropagation module (CE), and a one-hot encoding module (OHE). Alternatively, the structure used to define the neural network could be, for example, that of a forward-propagating neural network, a multilayer perceptron, etc. Any other example within the grasp of those skilled in the art may be used.

[0146] In a step S21, the method obtains training data DEj for an audio segment from a set of training audio segments SEGj. For example, the training set comprises N training segments, where N takes the value 10000, numbered from 1 to 10000, and corresponding to the emotions EMi of label LEMi with index j ranging from 1 to 10000 and index i ranging from 1 to 10 (i.e., there are 10 emotion labels). Audio segments may or may not be stored in association with the LEMi labels. Indeed, once the network is trained, according to this embodiment, it is no longer necessary to have the training audio segments. However, it may be useful to be able to access them for calculating distortions, etc. According to one variant, they are removed from the database. According to another variant, they are removed after training. According to another variant, they are stored in the database.These training audio segments can be obtained, without loss of generality, from a database, a hard drive, network access, the output of a decoder, or live from a speaker's microphone, etc. In one embodiment, a smartphone-type terminal can train the network on the fly using the voice of its user and those of their interlocutors to extract emotions and synthesize them simultaneously with the voice. As mentioned previously, the training segments can correspond to audio sequences of varying lengths that include speech. Each segment comprises a set of audio samples.

[0147] For example, an audio segment SEGi comprises 20,000 samples. The training audio segments may or may not contain the same number of samples. In this step S21, the audio segment SEGi is processed to obtain the first training dataset DEi. For example, the audio segment SEGi is converted into a vector where each index represents an audio sample value encoded on a predefined number of bits. In another example, the audio segment SEGi is converted into a vector of (normalized) characteristic values ​​such as its prosodic features (pitch, pitch, speech rate, etc.). In yet another example, the input is a MEL-type spectrogram.

[0148] During an S22 step, the DEi training dataset is applied to the first layer of the neural network. According to the previous first example, an audio sample value is applied to each neuron of the input layer.

[0149] During an S23 step, the different layers of the neural network are implemented in a conventional manner (in particular through a sequence of mathematical operations at the level of the different layers, etc.). Such implementation corresponds to the general knowledge of a person skilled in the art, and will not be detailed here.

[0150] During an S24 step, the artificial neural network outputs a prediction score for each output neuron, corresponding to an emotion label LEM. The number of output nodes can be arbitrary, and in all cases less than the number of training segments.

[0151] During an S25 step, the "softmax" (SM) function is used to generate a set of probabilities which are applied as input to the CE module.

[0152] During step S26 the CE module imposes the modification of at least one weight of the neural network, according for example to a backpropagation method of the gradient, so that the maximum probability corresponds to the expected value, i.e. the value of the LEMi index of the audio segment used for training.

[0153] This cycle can be repeated with other training audio segments, or the same ones, until the neural network training is complete, for example, when the classification accuracy (associating an audio segment with its LEMi index) no longer increases. This test can be performed during an optional S27 step.

[0154] The artificial neural network (ANN) trained following the implementation of the method is typically stored in the memory of a device such as the transmitting device itself, but can alternatively be stored externally, for example on a network gateway, a server, or a fixed or mobile terminal such as a personal computer, tablet, television, or smartphone.

[0155] Figure 8 represents, in schematic form, a database of emotions according to two embodiments of the invention.

[0156] The database BD / BD' is preferably populated by the encoder and used by the decoder. According to another embodiment, the two databases may be different.

[0157] The schematized database comprises emotions (EMi) accessible by an index, or key, or label, denoted LEMi, obtained at the encoder level via the output of the EMOT module, and at the decoder level via reading the received textual description. A label corresponds to a piece of speech context data. Access to the database can be made, for example, via an HTTP request. According to another example, the database of Data corresponds to storage on a hard drive. In another example, it is stored using a numerical index, which can take any known form (address, encrypted or unencrypted key, subscript, etc.). Generally speaking, the index is the element that allows access to the emotion, and if necessary, to a set of parameters associated with the emotion. Two implementations are shown in [Fig. 8].

[0158] According to a first embodiment, a first database, BD_1, is intended to be used for prosodic emotional speech synthesis. As described in support of [Fig. 5], step S14, a number of prosodic parameters representative of an emotion are stored. They are schematically represented in the figure by the columns labeled: - FREQ, which can represent, for example, a deviation from the average frequency, or pitch, of the speaker on the segment, - TEMPO, which can represent a variation in the speaking speed over the segment, - LOUDNESS which corresponds to an attenuation of the speaker's voice on the segment.

[0159] Thus, according to the example, the emotion "joy" is characterized by a 50% increase in frequency, 30% in tempo and a zero variation in attenuation.

[0160] Naturally, many other parameters are conceivable and easily accessible to a person skilled in the art to characterize an emotion.

[0161] According to another embodiment, an emotion synthesis based on a neural network is used, as described in the aforementioned paper by Tachibana et al. The neural network is selected from a plurality of networks, each trained on a corpus of different emotions (column "RNi" of Table 2). For example, a neural network has been trained with the emotion label "joy" and corresponds to a specific neural network pointed to by the LEM1 index of the emotion "joy". A reference to the neural network RNi is stored in the database BD_2.

[0162] According to another embodiment, a single neural network trained for all types of emotion is used, in which case the base is no longer needed.

[0163] At the receiver (DR), the emotion device receives as input a synthesized vocal segment and an emotion label (1), or an emotion (“joy”). It searches in the database for the parameters (FREQ, TEMPO, etc.) or the neural network (RNi) associated with the emotion and generates as output a vocal segment corrected by the use of these parameters.

[0164] At the issuer (DE), the database can be populated using any technique accessible to a person skilled in the art: neural network, statistical analysis of a corpus of test sequences, manual entry of features from information obtained from the literature, etc.

[0165] Naturally, this database can be external to the devices, for example it can be located in the network (cloud) or on an external hard drive, it can be distributed or not, etc. as long as the encoder-transmitter (DE) can access it for writing, and the decoder-receiver (DR) for reading.

[0166] It goes without saying that the embodiment described above has been given purely as an example and is in no way limiting, and that many modifications can easily be made by a person skilled in the art without going out of the scope of the invention.

[0167] In particular, in the case of a two-way conversation, each of the DE and DR terminals may include a text data generation module and an audio signal generation module from text data.< / segment> < / audio>

Claims

Claims

1. Method for generating textual data representative of an audio signal comprising at least one voice segment (SVC), the method being characterized in that it comprises, on a terminal (DE), the following steps: - identifying in the audio signal (SI) said at least one voice segment containing at least one piece of voice information; - extracting (S2) from said at least one voice segment at least one piece of sound context data, representative of the sound content of the voice segment; - extracting (S4) from the voice segment at least one piece of voice context data (LEMi), representative of an emotion associated with the voice information; preparing (S6, S7) a textual description of said voice segment, comprising at least said voice information and said voice context data and said sound context data.

2. Method for generating textual data according to claim 1, in which the audio signal further comprises at least one non-vocal segment (SNVC) and the method further comprises: - identifying (SI) said at least one non-vocal segment comprising at least one piece of sound context data, representative of the non-vocal content; - preparing (S2, S7) a textual description of said non-vocal segment, comprising at least said piece of sound context data:

3. A method of generating textual data according to one of the preceding claims, wherein the textual description is a data set in XML or JSON format.

4. A method of generating text data according to claim 1, wherein the voice context data is generated by a classification module.

5. A method for generating textual data according to one of the preceding claims, wherein the textual description is transmitted (S8) over a network.

6. A method for generating text data according to the preceding claim, wherein the transmission (S8) is carried out in the form of an SMS message.

7. Method for generating textual data according to claim 1, further comprising a training phase (SO) comprising the steps of: - receiving training audio segments (DEi) comprising at least one voice information; - receiving labeling data comprising at least one voice context data (LEMi) representative of an emotion associated with the voice information of the segment; - adapting a voice context detection system on the basis of the training audio segments and the labeling data received.

8. Device for generating textual data representative of an audio signal comprising at least one vocal segment, the device comprising a memory (MEM) and a processor (PROC) configured to: - identify in the audio signal said at least one vocal segment containing at least one piece of vocal information; - extract from said at least one vocal segment at least one piece of sound context data, representative of the sound content of the vocal segment; - extract from the vocal segment at least one piece of vocal context data, representative of an emotion associated with said vocal information; - prepare a textual description of said vocal segment, comprising at least said vocal information and said vocal context data and said sound context data.

9. A computer program capable of being implemented on a device as defined in claim 8, the program comprising code instructions which, when the program is executed by a processor, performs the steps of the method defined according to one of claims 1 to 7.

Images