METHOD FOR RECEIVING AND PROCESSING AUDIO DATA AND TRIGGERING ASSOCIATED SOUND EFFECTS BASED ON PROSODY AND / OR MOVEMENT

Abstract

METHOD FOR RECEIVING AND PROCESSING AUDIO DATA AND TRIGGERING ASSOCIATED SOUND EFFECTS BASED ON PROSODY AND / OR MOVEMENT The invention relates to a method for receiving and processing audio data comprising speech corresponding to the real-time reading of a source text for triggering sound effects synchronized with said reading of the text, characterized in that it comprises a step (110) of determining a speaker position index (12) in the source text, by detecting a correspondence between the received audio data and the source text, a step (112) of receiving at least one data point representing a prosody data point (10), and / or of receiving at least one movement data point, a step (140) of determining the sound effect to be triggered based on the position index (12), and based on the prosody data point and / or the movement data point,a step (150) for triggering the specified sound effect. Figure for the abbreviation: figure 1,

Description

Title of the invention: METHOD FOR RECEIVING, PROCESSING, AND TRIGGERING AUDIO DATA SOUND EFFECTS ASSOCIATED WITH PROSODY AND / OR MOVEMENT Technical field of the invention

[0001] The invention relates to a method for receiving and processing audio data, and a system for receiving and processing audio data. In particular, the invention relates to a method for triggering sound effects associated with a source text during the reading of the source text by a human speaker, taking into account the prosody of the reading and / or movement during the reading. Technological background

[0002] The invention falls within the field of reading and proposes to accompany the reading of a book by triggering sound effects related to the source text being read.

[0003] Various interactive reading techniques have already been proposed to enable the triggering of sound effects during reading.

[0004] Some techniques, for example, offer an estimation of the playback speed to allow the triggering and pre-loading of sound effects at the most opportune moment. These techniques are simple, but the risk of triggering a sound effect at the wrong time is high.

[0005] Other techniques are based on recognizing the reading of predetermined keywords in the text to trigger the associated sound effect. However, these word detection techniques are more prone to errors in recognition and reading tracking.

[0006] Furthermore, current techniques generally offer only one sound effect for each passage of the text and do not allow for variation in the reading experience with each new reading aloud of the source text. They also do not allow for adapting the sound effect to the speaker's emotions and reading style. Nor do they allow for triggering sound effects when the speaker moves the book to interact with it.

[0007] In particular, the sound effects used are based on a principle of "linear music" which uses sounds played in a loop and a transition from one loop to another by crossfading at the end of predetermined sequences.

[0008] The inventors thus sought to provide a method overcoming these drawbacks, by enabling the integration of a process similar to the principle of adaptive music when reading a text, in particular a book. Objectives of the invention

[0009] The invention aims to provide a method, a computer program product and a system for receiving and processing audio data including speech corresponding to the real-time reading of a source text by a speaker, for triggering sound effects synchronized with said reading of the text.

[0010] The invention aims in particular to provide, in at least one embodiment, a method, a computer program product and a system for receiving and processing audio data allowing the consideration of interactions and variations of emotions during the reading of the text, for a better renewal of the reading experience, in particular to adapt the sound effect to the emotion that the speaker wants to convey.

[0011] The invention also aims to provide, in at least one embodiment, a method, a computer program product and an audio data reception and processing system enabling better tracking of source text reading to ensure appropriate triggering of sound effects at the opportune time.

[0012] The invention also aims to provide, in at least one embodiment, a method, a computer program product and a system for receiving and processing audio data that can be embedded in a small portable device and can operate without an Internet connection. Description of the invention

[0013] To this end, the invention relates to a method for receiving and processing, in a receiving and processing system, audio data comprising speech corresponding to the real-time reading of a source text by at least one speaker, for triggering sound effects synchronized with said reading of the text,

[0014] characterized in that it comprises: - a step of determining a speaker's position index in the source text, by detecting a match between the received audio data and the source text recorded in a storage module of the reception and processing system, - a stage of receiving at least one piece of data representative of a prosodic value of the audio data, called prosodic data, and / or of receiving at least one piece of data representative of the presence or absence of movement from a motion recognition device during the reading of the text and representative of characteristics of a present movement, called motion data, - a step to determine the sound effect to be triggered based on the position index, and based on the prosody data and / or the movement data, - a triggering step for the determined sound effect.

[0015] A reception and processing method according to the invention thus allows for the consideration of prosody and / or movement data to determine whether a sound effect should be triggered when the position index corresponds to a triggering position for the sound effect. Taking this additional data into account allows for a renewed and more immersive and personalized reading experience, as a sound effect can be triggered, selected, or modified or not, taking into account variations in reading. The objective is to adapt the sound effect to the emotion that the speaker of the source text wishes to convey.

[0016] The triggering of sound effects is thus adaptive and can evolve in real time at each position index according to movements and / or prosody, which are respectively representative of actions and emotions. The operation is therefore similar to the principle of adaptive music, used particularly in video games where the music changes dynamically according to the player's activity. The sound effects are divided into fragments and several layers that can be selected and possibly combined according to movements and prosody.

[0017] In particular, taking into account the prosody of reading allows for the recognition of the speaker's emotions, for example, adapting the sound effects triggered to the reading volume, speed, tone, etc. A combination of these speech characteristics constituting the audio data constitutes prosody. More generally, prosody is defined as the set of non-verbal characteristics of spoken speech, in particular variations in rate, pitch (tone and intonation), and duration (accentuation and rhythm), which can be representative of an emotion conveyed in the words when reading the text. The same text can convey different emotions depending on the prosody of the speech during its reading by a speaker, without considering the meaning of the words and phrases spoken.Prosody can also characterize a sociolinguistic accent such as a regional accent and can also add context and meaning to a reading.

[0018] Prosody is detected and recognized through the use of at least one pre-configured dedicated algorithm, for example, implementing at least one learning model pre-trained with annotated prosody data, or for example, using predetermined prosody thresholds adjusted using speaker-specific audio data. The algorithm(s) take into account the combination of prosodic features of the audio data to determine a prosody of the audio data, which can be associated with an emotion conveyed when reading the text.

[0019] The algorithm is preferably fast to execute and autonomous, in particular requiring few computing resources and not requiring a connection to the Internet or any other external database.

[0020] Motion detection allows physical interactions on the motion recognition device during text reading to be associated with the reading process, and enables the linking of reading and movement to the triggering of sound effects. The motion data can be data indicating the absence of movement. The motion data can also include information representing the speed and / or acceleration of the movement.

[0021] The movement is detected and recognized through the use of at least one pre-configured dedicated algorithm, for example implementing at least one pre-trained learning model with annotated movement data, or for example using pre-determined movement models, adjusted using speaker-specific movement data and / or relative to the initial position of the motion recognition device. The algorithm(s) take into account the combination of movement characteristics detected by the motion recognition device to determine the recognized movement.

[0022] Advantageously and according to the invention, the source text is recorded in the processing device in the form of a list of phonemes, and the processing device is configured to detect in the audio data the presence of phonemes corresponding to the source text.

[0023] According to this aspect of the invention, decomposing the text into phonemes improves the accuracy of the position index as well as the calculation speed, and thus improves the triggering conditions. Phoneme detection provides better results than word detection and is more compatible with choppy reading, rewinding, pauses in reading, etc.

[0024] Furthermore, phoneme detection requires fewer resources than word detection and can therefore be implemented by a portable and / or embedded device, even without an internet connection. In particular, the number of different phonemes to be detected is much lower than the number of words that can be detected in word detection methods, and phoneme detection does not require deduction of the next word. The phoneme detection speed is thus on the order of 200 ms.

[0025] Advantageously and according to the invention, the speaker position index corresponds to a phoneme from the phoneme list of the source text, and the step of determining the speaker position index in the source text comprises: - a sub-step of receiving the current position of the index in the list of phonemes of the source text, - a sub-step comparing at least one phoneme detected in the audio data with at least one expected phoneme from the following phonemes in the source text phoneme list, - if no phoneme detected in the audio data corresponds with the following phonemes, a sub-step of receiving a plurality of phonemes detected in the audio data and a sub-step of searching in at least a part of the source text for a sequence of phonemes in the list of phonemes of the source text corresponding to said plurality of detected phonemes, - if the plurality of phonemes detected in the audio data does not correspond with any phoneme sequence in the source text's phoneme list, a sub-step will search the entire source text for a phoneme sequence in the source text's phoneme list that corresponds to said plurality of detected phonemes, - in the event of detection of a match of a phoneme or a plurality of detected phonemes with the list of phonemes of the source text, a sub-step of updating the index with said corresponding phoneme or the last phoneme of the corresponding sequence of phonemes.

[0026] According to this aspect of the invention, the phoneme correspondence tracking as described allows for an efficient determination of the position index for triggering sound effects.

[0027] Advantageously, if no phoneme detected in the audio data corresponds to the following phonemes, the step of determining the speaker's position index in the source text also includes a step of advancing the index to the next phoneme. Thanks to this step, the process can continue to advance through the list of phonemes to avoid losing the correspondence between the audio data and the list of phonemes.

[0028] Advantageously, and according to the invention, the comparison between the audio data and the phoneme list includes determining an index that represents the similarity between the phoneme detected in the audio data and the phoneme in the phoneme list. This index, which represents similarity, also represents the distance between these phonemes, depending on its value. For example, a value of zero indicates a perfect match, a value close to zero indicates similarity, and a higher positive value indicates a distance between the phonemes. For example, in French, phonemes beginning with the letters "p" and "b" are considered similar and therefore have a small distance, just like the letters "t" and "d". If the index value is below a predetermined threshold, a phoneme match can be considered, and the process can proceed. The index. In variants of the invention where a confidence index as described below is implemented, this confidence index can be lowered if the phoneme correspondence is not perfect (non-zero similarity index but below the predetermined threshold). Thus, advantageously and according to the invention, the substep of comparing at least one phoneme detected in the audio data with at least one expected phoneme from among the following phonemes in the source text phoneme list comprises determining a representative index of the similarity between the phoneme detected in the audio data and the phoneme in the phoneme list, and if the representative similarity index is below a predetermined threshold, a substep of updating the index with said detected phoneme and a substep of decreasing the confidence index.

[0029] Advantageously and according to the invention, the sound effect triggering step includes a sub-step of checking the value of a confidence index and in that the sound effect is triggered only if the confidence index is greater than a predetermined threshold.

[0030] According to this aspect of the invention, the confidence index thus makes it possible to avoid triggering an audible effect in the event of a lack of confidence in the position tracking. In particular, it makes it possible to avoid triggering an audible effect at an inopportune moment due to an incorrect determination of the position index.

[0031] Each sound effect can be linked to its own predetermined threshold.

[0032] Advantageously and according to the invention, the method includes a step of increasing the value of the confidence index in the event of detection of a correspondence of a phoneme or a plurality of detected phonemes with the list of phonemes of the source text and includes a step of decreasing the value of the confidence index if no phoneme detected in the audio data corresponds with the following phonemes.

[0033] According to this aspect of the invention, the increase and decrease of the confidence index is linked to the detection of phoneme correspondence, in particular the confidence index thus makes it possible to avoid the triggering of a sound effect if the detection of phoneme correspondence is not carried out with sufficient confidence.

[0034] Advantageously and according to the invention, all the sound effects associated with the source text are distributed across a plurality of time sequences associated with portions of the source text, each time sequence being associated with a sound effect or a group of sound effects comprising several sound effects, and when the position index corresponds to a time sequence associated with a group of sound effects, the step of determining the sound effect to be triggered comprises: a retrieval substep, based on the position index, of the sound effects group associated with the time sequence in which it is located

[0035]

[0036]

[0037]

[0038]

[0039]

[0040] the position index, - a sub-step of determining, based on prosody data and / or movement data, the sound effect to be triggered from among the sound effects in the sound effects group. According to this aspect of the invention, each temporal sequence is associated with one or more sound effects, and the triggered sound effects are adapted to specific prosody and / or movement data. The objective is to offer, for at least some temporal sequences, variant sound effects to be triggered for the same position index. Advantageously and according to the invention, the sound effects that can be triggered in each sound effect group can also depend on parameters predefined in advance by the speaker, or can vary according to prosody data and the confidence index in the associated variants. For example, the speaker can select a "day" or "night" mode, which prevents certain sound effects from being triggered within the sound effect group. Taking the confidence level into account also helps limit the risk of accidentally triggering a sound effect at the wrong time, for example, by delaying its triggering and / or preventing a specific sound effect from playing. The speaker can also enable or disable motion or prosody detection for reading. Advantageously and according to the invention, the prosody data comprises data or a combination of data from different types of data in the following list: - data on the intensity of speech in audio data, - data on the frequency and / or fundamental frequency of the speech in the audio data, - data on the intensity and / or frequency and / or fundamental frequency of consonants pronounced in the audio data, - data on the intensity and / or frequency and / or fundamental frequency of vowels pronounced in the audio data, - data on the length of vowels and / or spoken words in the audio data, - data on the length of pronounced consonants in the audio data, - data on the speech rate of audio data. According to this aspect of the invention, prosody depends on several factors that allow the speaker's emotion to be symbolized during the reading of the source text aloud. The prosody data can be based on an absolute value of each item in the list or on a temporal variation of its values, for example a gradual increase or decrease in sound intensity.

[0041] Advantageously and according to the invention, the motion data includes data relating to a displacement and / or a rotation in space and / or the speed of movement of the motion recognition device, and / or data relating to the correspondence of a movement or a combination of movements detected by the motion recognition device with a predetermined movement from a list of predetermined movements recorded in a storage module of the processing system.

[0042] According to this aspect of the invention, different types of movement can be taken into account for selecting the sound effect to be triggered. In particular, the method may include detecting a predetermined movement, for example, if the user makes a circle with the motion recognition device.

[0043] Motion in space is known to consist of one or a combination of the following motions: - vertical translation, - lateral translation, - longitudinal translation, - pitching, - shoelace, - rolling - jerking movement, - brief impacts.

[0044] The speed and / or acceleration of the motion can also be integrated into the motion data.

[0045] Advantageously and according to the invention, the sound effect triggering step is performed when the position index is at a position equal to or after a predetermined trigger index, or within a predetermined trigger window, and in that the sound effect can be emitted immediately as soon as the position index reaches or exceeds the trigger index or reaches the trigger window, or emitted on a delayed basis after the end of a sound effect in progress.

[0046] According to this aspect of the invention, the triggering of the sound effect can be immediate or delayed; for example, the specified sound effect can be triggered at the end of a time sequence. The timing of the triggering depends in particular on the position or trigger window in which the position index is located.

[0047] The invention also relates to a computer program product for receiving and processing audio data corresponding to the real-time reading of a source text by at least one speaker, for triggering synchronized sound effects with said reading of the text, said computer program product comprising program code instructions for the execution, when said computer program product is executed on a computer device, of the steps of the process according to the invention.

[0048] The computer program product is advantageously stored in the processing device, in particular in a portable computing device, preferably a smartphone, a digital tablet or a smartwatch, for example in the form of an application.

[0049] The invention also relates to a system for receiving and processing audio data corresponding to the real-time reading of a source text by at least one speaker, for triggering sound effects synchronized with said reading of the text, characterized in that it comprises: - a module for storing the source text, - an audio data reception module, - an audio data processing module, configured to determine a speaker position index in the source text, by detecting a match between the received audio data and the source text recorded in the storage module, - a motion recognition device configured to provide at least one data point representing the presence or absence of movement during text reading and representative of characteristics of any movement present, referred to as motion data, and / or an audio data prosody variation analysis module configured to provide at least one data point representing a prosody variation in the audio data - a module for determining the sound effect to be triggered based on the position index, and based on prosody data and / or movement data, - a sound emission device configured to emit the specified sound effect.

[0050] Advantageously, the reception and processing system according to the invention is configured to implement the reception and processing method according to the invention.

[0051] Advantageously and according to the invention, the receiving and processing method according to the invention is configured to be implemented by a receiving and processing system according to the invention.

[0052] Advantageously and according to the invention, the system comprises a portable computing device, preferably a smartphone, a digital tablet or a smartwatch, including the source text storage module, the audio data reception module, the audio data processing module, the motion recognition device and / or the prosody variation analysis module, the sound effect determination module and the sound emission device.

[0053] According to this aspect of the invention, a major part or all of the system components can be integrated into a portable computing device such as a smartphone, so as to combine the functionalities into a single system. The portable computing device can also display the source text or, preferably, be attached to a physical medium containing the source text, in particular where the source text is printed, such as a book.

[0054] The computer device is configured to implement the reception and processing method according to the invention in particular by means of a combination of one or more computer components such as a processor (CPU for Central Processing Unit), and / or a graphics processor (GPU for Graphics Processing Unit), and / or a digital signal processor (DSP for Digital Signal Processor), and / or one or more memories, and / or an analog / digital converter, a microphone, an accelerometer, a gyroscope, etc.

[0055] Advantageously and according to the invention, the system comprises a motion recognition device and a motion recognition device attachment, configured to allow the motion recognition device to be attached to a support on which the source text is printed, so that the motion recognition device is mechanically attached to said support when reading the support.

[0056] According to this aspect of the invention, the motion recognition device can be directly associated with the physical medium on which the source text is printed, such that a movement of the physical medium causes a movement of the motion recognition device. Thus, a movement of the physical medium is detected and taken into account in the motion data.

[0057] When the motion recognition device is integrated into a portable computing device, the attachment device is configured to attach the portable computing device to the physical support.

[0058] Advantageously and according to the invention, the attachment device is formed of one element or a combination of elements from the following list: - elastic elements connected to the support on which the source text is printed, configured to hold the motion recognition device in position, - a pocket arranged in the cover of said support, configured to ac- pick up the motion recognition device, - a compartment arranged in the cover of said support, configured to house the motion recognition device, - a permanent magnet arranged on said support and configured for magnetization with a magnetic element arranged on the motion recognition device, or a permanent magnet arranged on the motion recognition device and configured for magnetization with a magnetic element arranged on said support, - a pouch clipped onto said support, configured to accommodate the motion recognition device.

[0059] According to this aspect of the invention, these different attachment variants allow compatibility with different types of motion recognition devices, in particular when the latter is integrated into a portable computer device.

[0060] The invention also relates to a receiving and processing method, a computer program product, and a receiving and processing system characterized in combination by all or part of the features mentioned above or below. List of figures

[0061] Other objects, features and advantages of the invention will become apparent from the following description, given by way of non-limiting example only, and which refers to the accompanying figures in which:

[0062] [Fig. 1] is a schematic view of a method for receiving and processing audio data according to an embodiment of the invention,

[0063] [Fig.2] is a schematic view of a source text and a list of phonemes as recorded in a storage module of a reception and processing system according to an embodiment of the invention,

[0064] [Fig.3] is a graph schematically illustrating a variation in prosody of audio data processed by a reception and processing method according to an embodiment of the invention,

[0065] [Fig.4] is a graphic schematically illustrating three audio tracks comprising sound effects that can be triggered during the execution of a reception and processing method according to an embodiment of the invention,

[0066] [Fig.5] is a schematic view of an audio data reception and processing system according to an embodiment of the invention and according to a front and back view of a book forming the support for the source text.

[0067] Detailed description of an embodiment of the invention

[0068] In the figures, the scales and proportions are not strictly respected for the purposes of illustration and clarity.

[0069] Furthermore, identical, similar or analogous elements are designated by the same references in all figures.

[0070] Figure 1 schematically represents a method 100 for receiving and processing audio data according to an embodiment of the invention. The method enables the reception and processing of audio data comprising speech corresponding to the real-time reading of a source text by a speaker, for the triggering of sound effects synchronized with said reading of the text. The method is implemented here in an audio data reception and processing system, an embodiment of which is described below with reference to Figure 5.

[0071] The method includes a 110th step of determining a speaker position index in the source text, by detecting a match between the received audio data and the source text recorded in a storage module of the reception and processing system.

[0072] The source text is stored in the processing device as a list of phonemes, and the processing device is configured to detect in the audio data the presence of phonemes corresponding to the source text. The phoneme list is created beforehand, automatically, manually, or semi-automatically (for example, with manual correction of automatic preprocessing). The creation of the phoneme list is generally performed outside the system implementing the method according to the invention.

[0073] Figure 2 schematically represents an example 200 of a sentence in French from the source text as recorded in the storage module “En un éclair, le chat grimpa dans l'arbre.”, its transposition into phonemes “ em Afl é k le a le if æt gr im pa d enzhr b re * and the index assigned to each phoneme associated with this example sentence, the index being in this example between 1 and 19. Other index numbering may be used, for example the first index may correspond to an index zero, etc.

[0074] The speaker position index corresponds to the index of the corresponding phoneme in the phoneme list of the source text, and step 110 of determining the speaker position index in the source text comprises: - a sub-step 112 of receiving the current position of the index in the list of phonemes of the source text. Initially, the position index corresponds to the index of the first phoneme, for example index 1 in the example sentence of [Fig.2], - a sub-step 114 comparing at least one detected phoneme in the audio data with at least one expected phoneme from the following phonemes in the source text phoneme list. The following phonemes in the example sentence phoneme list are the phonemes « », “é”, “k” and “le”. - if no phoneme detected in the audio data corresponds to the following phonemes, a substep 116 of receiving a plurality of phonemes detected in the audio data and a substep 118 of searching in at least part of the source text for a sequence of phonemes in the list of phonemes in the source text corresponding to said plurality of detected phonemes. For example, in the example sentence in Figure 2, the phonemes "le f æt gr im pa" are detected in the audio data and the position index is then placed at the end of this sequence of phonemes in the source text, that is to say at index 12 which corresponds to the last phoneme of the sequence. - If the plurality of phonemes detected in the audio data does not correspond to any phoneme sequence in the source text's phoneme list, a substep 120 searches the entire source text for a phoneme sequence in the source text's phoneme list that corresponds to said plurality of detected phonemes. This substep allows for the search for a phoneme sequence throughout the entire source text, for example, sentences following the example sentence or sentences preceding the example sentence, which are not visible in the figures. - If a match is detected between one or more phonemes in the source text's phoneme list, a substep 122 updates the index with the corresponding phoneme or the last phoneme in the corresponding phoneme sequence. This final step of the cycle updates the index based on the match established in the preceding substeps. The updated index is retrieved during a subsequent execution of substep 112, which receives the current index position.

[0075] The method may also include other undescribed substeps to ensure the matching of phonemes detected in the audio text and phonemes as recorded by the source text.

[0076] The method then includes a step 130 of receiving at least one data representative of a prosody value of the audio data, called prosody data 10, and / or of receiving at least one data representative of the presence or absence of a movement of a motion recognition device during the reading of the text and representative of characteristics of a present movement, called motion data 12.

[0077] Prosody is representative of the emotion conveyed to the speaker of the source text during its reading. Taking movement into account allows for a interactivity while reading the source text.

[0078] Prosody and movement are taken into account independently or in combination in the rest of the process. This choice can be made by a user of the process, for example through an interface allowing them to select whether prosody and / or movement are taken into account in future readings.

[0079] The prosody data comprises data or a combination of data from different data types in the following list: - data on the intensity of speech in audio data, - data on the frequency and / or fundamental frequency of the speech in the audio data, - data on the intensity and / or frequency and / or fundamental frequency of consonants pronounced in the audio data, - data on the intensity and / or frequency and / or fundamental frequency of vowels pronounced in the audio data, - data on the length of vowels and / or spoken words in the audio data, - data on the length of pronounced consonants in the audio data, - data on the speech rate of audio data.

[0080] A combination of these data makes it possible to better characterize the emotion and adapt the triggering of sound effects accordingly.

[0081] Figure 3 is an illustrative graph representing an example of prosody as a function of time, as detected from audio data. For the sake of simplicity and purely illustrative purposes, the prosody in the graph only takes into account the intensity and fundamental frequency of the audio data. The value assigned to the prosody data evolves here in two dimensions, but in practice the value of the prosody data is characterized by the variation of several data points, for example, combined in a vector or a matrix.

[0082] During a first time interval, called the calibration interval 310, the intensity and fundamental frequency are measured to obtain a value that will be considered the average for the speaker of the audio data. The calibration interval 310 can be defined by a trigger, for example from a first predefined position index, and end from a second predefined position index.

[0083] The average prosody value determined during the calibration interval 310 is assigned to a zone called the average zone 312. If the prosody value varies over time, this value may fall into other zones: a high zone 314 defined by values ​​above a high threshold 315 and a low zone 316 defined by values ​​below a low threshold 317. The variation of the The value, for example, represents a percentage variation in the fundamental frequency and a percentage variation in intensity; these two values ​​can be weighted differently to give the total value. More zones can be defined depending on the data considered in the prosody, for example, four zones, five zones, or more. The zones thus described are primarily illustrative, and interpretations of prosody values ​​can be based on other classification methods, such as decision trees, etc.

[0084] Each zone then corresponds to an audio track containing the sound effects to be triggered. When a sound effect needs to be triggered, for example at times 318a, 318b, 318c, 318d represented by circles, the sound effect of the audio track associated with the zone containing the prosody value can be triggered. Here, times 318a, 318c, and 318d are linked to the high zone, and time 318b is linked to the low zone.

[0085] The triggering of the sound effect can of course be subject to other conditions, in particular to the movement input.

[0086] Motion data includes data relating to a displacement and / or rotation in space and / or the speed of movement of the motion recognition device, and / or data relating to the correspondence of a movement or combination of movements detected by the motion recognition device with a predetermined movement from a list of predetermined movements stored in a storage module of the processing system. The absence of movement is also motion data that can be interpreted by the method.

[0087] Further details on the motion data are described below with reference to [Fig.5] illustrating an audio data reception and processing system.

[0088] Returning to [Fig. 1], the method then includes a step 140 of determining the sound effect to be triggered based on the position index, and based on the prosody data and / or the movement data. The triggered sound effect is thus adapted according to the reading context provided by the prosody and / or the movements.

[0089] In particular, in one embodiment of the invention, all the sound effects associated with the source text are distributed across a plurality of time sequences associated with portions of the source text, each time sequence being associated with a sound effect or a group of sound effects comprising several sound effects, and in that when the position index corresponds to a time sequence associated with a group of sound effects, step 140 of determining the sound effect to be triggered comprises: - a retrieval substep 142, based on the position index, of the group of sound effects associated with the time sequence in which the position index is located, - a sub-step 144 of determining, based on the prosody data and / or the movement data, the sound effect to be triggered from among the sound effects in the sound effects group.

[0090] Figure 4 schematically represents a time-domain graph comprising three audio tracks: a mid-range track, a high-range track, and a low-range track, corresponding respectively to the sound effects to be triggered according to the mid, high, and low ranges described above. As described above, these tracks are primarily for illustrative purposes, and in practice, sound effect triggers may not be associated with specific tracks.

[0091] The audio tracks are read from left to right as the position index and playback progress. The sound effects of the audio tracks are grouped into sets of one to three sound effects associated with temporal sequences 421, 422, 423, 424, 425, 426, 427, 428, 429.

[0092] The first temporal sequence 421 is associated with a group that includes, for example, a sound effect 421a which is applied regardless of the area reached by the prosody value. The second temporal sequence 422 is associated with a group that also includes a sound effect 422a, in particular because these first two temporal sequences correspond to the calibration interval as described above.

[0093] The following two sequences 423 and 424 are associated with groups each comprising two sound effects, 423a and 423b on the one hand, and 424a and 424b on the other. Thus, if the prosody value is in the high zone, sound effects 423b and 424b will be played, while if the prosody value is in the medium or low zone, sound effects 423a and 424a will be played.

[0094] Finally, the following sequences 425 to 429 are associated with groups comprising a sound effect for each zone, referenced respectively as 425a, 425b and 425c, 426a, 426b and 426c, 427a, 427b and 427c, 428a, 428b and 428c, 429a, 429b and 429c. Thus, a sound effect is available for the high, middle and low zones reached by the prosody value.

[0095] The sound effect triggering step 150 is executed when the position index is at a position equal to or after a predetermined trigger index, or within a predetermined trigger window, and the sound effect can be emitted immediately as soon as the position index reaches or exceeds the trigger index or reaches the trigger window, or emitted with a delay after the end of a sound effect currently being emitted. If the position index reaches, for example, marker 430, and the prosody value has changed since the beginning of the temporal sequence 424, the effect The sound played can be modified immediately. Alternatively, if the position index reaches, for example, marker 431, and the prosody value has changed since the beginning of the temporal sequence 426, the sound effect played can only be modified at the beginning of the temporal sequence 427.

[0096] According to an embodiment not shown, these audio tracks correspond to a background sound, and additional sound effects can be added during playback based on prosody data and / or motion data and / or the position index. Furthermore, several sound effects can be combined, depending on the prosody and / or motion. For example, during the reading of a text, a first sound effect comprising the sound of a single musical instrument can be played, and if the sound intensity detected by determining the prosody increases, one or more other sound effects adding musical instruments can be combined with the first sound effect.

[0097] In addition, the number of sound tracks and sound zones may be greater than represented.

[0098] The method finally includes a step 150 of triggering the determined sound effect. The sound effect is emitted in particular by a sound emission device of the receiving and processing system. This sound emission device is, for example, a loudspeaker or headphones.

[0099] The emission of the sound effect thus increases immersion when reading the source text.

[0100] The sound effect triggering step 150 includes a substep 152 of checking the value of a confidence index and the sound effect is triggered only if the confidence index is greater than a predetermined threshold.

[0101] The confidence index is managed by a confidence index management subprocess 160, which includes a step 162 for increasing the confidence index value when a phoneme or a plurality of detected phonemes matches the phoneme list in the source text, and a step 164 for decreasing the confidence index value if no detected phoneme in the audio data matches subsequent phonemes, or if the detected phoneme is close enough to the expected phoneme to advance the position index but does not have an exact match. The confidence index thus prevents the sound effect from being triggered in case of excessive uncertainty regarding the accuracy of phoneme tracking. The confidence index can also be affected by other parameters.

[0102] In addition, a confidence index can also be applied independently to the prosody data.

[0103] Figure 5 schematically represents a 500 reception and processing system audio data according to an embodiment of the invention. The system allows the reception and processing of audio data corresponding to the real-time reading of a source text by at least one speaker, for the triggering of sound effects synchronized with said reading of the text, by implementing the steps of the reception and processing method described above.

[0104] The receiving system 500 herein comprises a portable computing device consisting of a 510 smart phone (more commonly known as a smartphone) including a source text storage module, an audio data reception module, an audio data processing module, a motion recognition device, a prosody variation analysis module, a sound effect determination module, and a sound emission device, adapted to implement the associated steps of the audio data reception and processing process. The motion recognition device is, for example, an accelerometer, a gyroscope, a magnetometer, or a combination thereof. The various modules are managed, for example, by the processor and the storage memory(ies) of the 510 smart phone.

[0105] The receiving system 500 also includes a motion recognition device attachment, more particularly in this embodiment an attachment device for the smart phone 510 comprising this motion recognition device, configured to allow the motion recognition device to be attached to a medium on which the source text is printed, such that the motion recognition device is mechanically fixed to said medium when the medium is read. The medium on which the source text is printed is here a book, of which [Fig. 5] shows a view 590a of the inside of the book and a view 590b of the outside of the book.

[0106] In particular, the attachment device here comprises an elastic band 512 passing through the back cover of the book via notches 514 in the attachment device. The smartphone 510 can, for example, be arranged on the last endpaper 592a of the book (also called the inside front cover, inside back cover, or inside back cover). Alternatively, but not shown, the smartphone 510 can be arranged with the same attachment device on the back cover 592b (also called the inside back cover).

[0107] Other types of fastening devices may be implemented: - elastic elements connected to the support on which the source text is printed, configured to hold the motion recognition device in position, - a pocket arranged in the cover of said support, configured to accommodate the motion recognition device, - a compartment arranged in the cover of said support, configured to house the motion recognition device, - a permanent magnet arranged on said support and configured for magnetization with a magnetic element arranged on the motion recognition device, or a permanent magnet arranged on the motion recognition device and configured for magnetization with a magnetic element arranged on said support, - a pouch clipped onto said support, configured to house the motion recognition device - etc.

[0108] The link between the motion recognition device and the support makes it possible to detect movement of the support, here the book, during reading and to trigger an associated sound effect. A movement could, for example, consist of: - a rotation of the book like a steering wheel, and trigger associated sound effects if the reading position index is associated with such a trigger, - tapping the book to mimic knocking on a door, or playing a percussion instrument, - shake the book to mimic the shaking of maracas, the movement of a fan, - lift the book, - Left / right pitching to mimic the rocking of an animal, a baby, or the movement of a rainstick, - Forward / backward roll to speed up / slow down the music, - etc.

[0109] The movement may also have to be executed a minimum number of times or with a minimum angle of rotation to validate the triggering of the sound effect.

[0110] The invention is not limited to the embodiments described. In particular: - the sound effects can be predetermined or generated in real time, for example via generative artificial intelligence, - the motion recognition device may not be attached to the source text medium if it is moving at the same time as the source text (if it is integrated into a smartwatch worn by the speaker, for example), - Detection of certain words in addition to phonemes can be implemented for the detection of special cases, in particular for the detection of custom words replacing or supplementing a portion of the source text, multiple speakers can be detected during reading of the source text and their respective prosody taken into account.

Claims

Claims

1. Method for receiving and processing, in a reception and processing system, audio data comprising words corresponding to the real-time reading of a source text by at least one speaker, for triggering sound effects synchronized with said reading of the text, characterized in that it comprises: - a step (110) of determining an index (12) of the speaker's position in the source text, by detecting a correspondence between the received audio data and the source text recorded in a storage module of the reception and processing system, - a step (112) of receiving at least one piece of data representative of a prosody value of the audio data, called prosody data (10),and / or receiving at least one piece of data (12) representative of the presence or absence of a movement of a movement recognition device during the reading of the text and representative of characteristics of a present movement, called movement data, - a step (140) of determining the sound effect to be triggered as a function of the position index (12), and as a function of the prosody data and / or the movement data, - a step (150) of triggering the determined sound effect.,

2. Receiving and processing method according to claim 1, characterized in that the source text is recorded in the processing device in the form of a list of phonemes, and the processing device is configured to detect in the audio data the presence of phonemes corresponding to the source text.

3. Reception and processing method according to claim 2, characterized in that the speaker position index (12) corresponds to a phoneme in the phoneme list of the source text, and in that the step (110) of determining the speaker position index in the source text comprises: - a sub-step (112) of receiving the current position of the index (12) in the list of phonemes of the source text, - a sub-step (114) of comparing at least one phoneme detected in the audio data with at least one phoneme expected from among the following phonemes in the list of phonemes of the source text, - if no phoneme detected in the audio data corresponds with the following phonemes, a sub-step (116) of receiving a plurality of phonemes detected in the audio data and a sub-step (118) of searching in at least part of the source text for a sequence of phonemes in the list of phonemes of the source text corresponding to said plurality of detected phonemes, - if the plurality of phonemes detected in the audio data do not correspond with any sequence of phonemes in the list of phonemes of the source text, a sub-step (120) of searching in the entire source text for a sequence of phonemes in the list of phonemes of the source text corresponding to said plurality of detected phonemes,- in the event of detection of a correspondence of a phoneme or of a plurality of phonemes detected with the list of phonemes of the source text, a sub-step (122) of updating the index with said corresponding phoneme or the last phoneme of the corresponding sequence of phonemes.,

4. Reception and processing method according to one of claims 1 to 3, characterized in that the step of triggering the sound effect comprises a sub-step (152) of verifying the value of a confidence index and in that the sound effect is triggered only if the confidence index is greater than a predetermined threshold.

5. Reception and processing method according to a combination of claims 3 and 4, characterized in that the sub-step (114) of comparing at least one phoneme detected in the audio data with at least one phoneme expected among the following phonemes in the list of phonemes of the source text comprises a determination of an index representative of the resemblance between the phoneme detected in the audio data and the phoneme in the list of phonemes, and in that if the index representative of the resemblance is less than a predetermined threshold, a sub-step (122) of updating the index with said detected phoneme and a sub-step of decreasing the confidence index.

6. Reception and processing method according to a combination of claims 2 and 4, characterized in that it comprises a step (162) of increasing the value of the confidence index in the event of detection of a correspondence of a phoneme or of a plurality of phonemes detected with the list of phonemes of the source text and in that it comprises a step (164) of decreasing the value of the confidence index if no phoneme detected in the audio data corresponds with the following phonemes.

7. Reception and processing method according to one of claims 1 to 6, characterized in that all of the sound effects associated with the source text are distributed in a plurality of time sequences associated with portions of the source texts, each time sequence being associated with a sound effect or a group of sound effects comprising several sound effects, and in that when the position index corresponds to a time sequence associated with a group of sound effects, the step of determining the sound effect to be triggered comprises: - a sub-step (142) of retrieving, as a function of the position index, the group of sound effects associated with the time sequence in which the position index is located, - a sub-step (144) of determining, as a function of the prosody data and / or the movement data, the sound effect to be triggered from among the sound effects of the group of sound effects.

8. Receiving and processing method according to one of claims 1 to 7, characterized in that the prosody data comprises data or a combination of data from different types of data in the following list: - data on the intensity of the words in the audio data, - data on the frequency and / or fundamental frequency of the words in the audio data, - data on the intensity and / or frequency and / or fundamental frequency of the consonants spoken in the audio data, - data on the intensity and / or frequency and / or fundamental frequency of vowels spoken in the audio data, - data on the length of vowels and / or words spoken in the audio data, - data on the length of consonants spoken in the audio data, - data on the speech rate of the audio data.

9. Reception and processing method according to one of claims 1 to 8, characterized in that the movement data comprises data relating to a movement and / or a rotation in space and / or to the speed of movement of the movement recognition device, and / or data relating to the correspondence of a movement or a combination of movements detected by the movement recognition device with a predetermined movement from a list of predetermined movements recorded in a storage module of the processing system.

10. A receiving and processing method according to one of claims 1 to 9, characterized in that the step (150) of triggering a sound effect is executed when the position index (12) is at a position equal to or after a predetermined trigger index, or in a predetermined trigger window, and in that the sound effect can be immediately emitted as soon as the position index reaches or exceeds the trigger index or reaches the trigger window, or emitted in a delayed manner after the end of a sound effect currently being emitted.

11. Computer program product for receiving and processing audio data corresponding to the real-time reading of a source text by at least one speaker, for triggering sound effects synchronized with said reading of the text, said computer program product comprising program code instructions for executing, when said computer program product is executed on a computing device, the steps of the method according to one of claims 1 to 10.

12. System for receiving and processing audio data corresponding to the real-time reading of a source text by at least one speaker, for triggering sound effects synchronized with said reading. of the text, characterized in that it comprises: - a module for storing the source text, - a module for receiving audio data, - an audio data processing module, configured for determining a position index of the speaker in the source text, by detecting a correspondence between the received audio data and the source text recorded in the storage module, - a motion recognition device configured to provide at least one piece of data representative of the presence or absence of a movement of a motion recognition device during the reading of the text and representative of characteristics of a present movement, called motion data,and / or a module for analyzing the variation in the prosody of the audio data configured to provide at least one data item representative of a variation in the prosody of the audio data - a module for determining the sound effect to be triggered as a function of the position index, and as a function of the prosody data and / or the movement data, - a sound emission device configured for the emission of the determined sound effect.,

13. System according to claim 12, characterized in that it comprises a portable computing device (510), preferably a smartphone, a digital tablet or a smart watch, comprising the source text storage module, the audio data reception module, the audio data processing module, the motion recognition device and / or the prosody variation analysis module, the sound effect determination module and the sound emission device.

14. System according to one of claims 12 or 13, characterized in that it comprises a motion recognition device and a device (512) for attaching the motion recognition device, configured to allow the motion recognition device to be attached to a support on which the source text is printed, so that the motion recognition device is mechanically secured to said support when reading the support.

15. System according to claim 14, characterized in that the attachment device is formed from an element or a combination of elements from the elements of the following list: - elastic elements (512) connected to the support on which the source text is printed, configured to hold the motion recognition device in position, - a pocket arranged in the cover of said support, configured to accommodate the motion recognition device, - a compartment arranged in the cover of said support, configured to accommodate the motion recognition device, - a permanent magnet arranged on said support and configured for magnetization with a magnetic element arranged on the motion recognition device, or a permanent magnet arranged on the motion recognition device and configured for magnetization with a magnetic element arranged on said support, - a pocket clipped onto said support, configured to accommodate the motion recognition device.

Images