Detection of a state of an environment based on audio frames captured in the environment

Abstract

A computer-implemented method, said method comprising: receiving a plurality of audio frames from one or more microphone located in an environment; using one or more sound detection supervised machine learning engine to detect one or more sound classes from said plurality of audio frames; using one or more state detection supervised machine learning engine to detect one or more state classes based at least on the output of said one or more sound detection supervised machine learning engine, each of said state classes being representative of a state of said environment and at least one of said state classes being representative of an alert state.

Description

[0001] Description

[0002] Title: Detection of a state of an environment based on audio frames captured in the environment

[0003] Technical Field

[0004] [1] This disclosure pertains to the field of audio processing. More specifically, the disclosure pertains to the field of analyzing audio frames captured in an environment to determine a state of the environment.

[0005] Background Art

[0006] [2] Remote monitoring of an environment relates to remotely verifying what is happening in a given environment. For example, remote monitoring solutions are used in places where fragile people are found, to automatically raise alarms when fragile people are in danger. Remote monitoring typically consists in using sensors to perform measurements in the environment and analyzing the measurements to detect the state of the environment, thereby detecting if there is need for intervention in the environment. The remote monitoring thus provides, through the automatic analysis of sensor measurements, an automatic monitoring of a large number of different environments by a limited number of human operators. A human intervention in a given environment can be performed only if the remote monitoring indicated that such an intervention may be useful.

[0007] [3] One typical example of environment that need to be remotely monitored is the rooms of retirement homes. Indeed, aged people may be subject to various hazards such as falls for example. Some aged people may also perform dangerous actions such as for example leaving their rooms unaccompanied. It is usually not possible in retirement home to provide a human supervision of each room. Therefore, remote monitoring system allows a central supervision of a large number of rooms by a limited number of operators, or, alternatively, improves the level of quality of care or assistance services provided by operators. As early as an alert is raised in one room, a human intervention can be provided to the room to verify if there indeed is hazard and provide assistance if needed to the occupant of the room.

[0008] [4] Other examples of remote monitoring of an environment are for example the remote monitoring of hospital rooms, medical homes or homes for disabled people. In this case also, a human supervision of each room is in practice not possible. Thus, remote supervision of the room allows detecting for example if a patient is in danger in one of the rooms, so as to trigger a human intervention. Additionally, it can preserve patients' sleep quality by, for example, informing nurses that it is not the right time to enter the room if it is not necessary.

[0009] [5] Another typical example of remote monitoring of an environment is the real-time assessment of its cleanliness. Whetherthe environment is residential, public, or industrial, maintaining cleanliness is a significant concern for health, safety, public perception, reputation, and productivity. Additionally, cleanliness is often tied to regulatory compliance, as many environments are subject to strict cleanliness regulations. So, investing in cleanliness is essential for creating a safe, efficient, and reputable environment. [6] Evaluating whether a cleaning activity has been properly executed, rather than directly assessing the cleanliness of a space, provides a more comprehensive understanding of how well the cleaning tasks were performed. Focusing on the process ensures that cleaning standards are consistently met. Additionally, it generates valuable data on performance metrics, such as time taken, areas covered, and adherence to protocols. This data can be used to optimize cleaning schedules and improve overall efficiency, which is not possible with a static assessment of cleanliness. Early alerts during the cleaning process may be triggered, allowing for human intervention to verify protocol adherence or check if any areas have been overlooked. Continuous monitoring of cleaning activities enables trend analysis, helping to identify patterns and areas that may require more attention or different cleaning strategies

[0010] [7] Another typical application of remote acoustic monitoring lies in the real-time supervision of individuals with mental health vulnerabilities in specialized care facilities. These environments, such as psychiatric hospitals or psychological support centers, host residents who may experience emotional instability, cognitive confusion, or behavioral disorders. Ensuring their safety, as well as the safety of staff and co-residents, is a priority both ethically and medically.

[0011] [8] Many other examples of environments may be remotely monitored. For example, a human may be monitored remotely by wearing sensor, a portion of a street may be remotely monitored to detect violent events, etc.

[0012] [9] Remote monitoring may be performed by different means. For example, it is most of the time performed by placing cameras in the environment, and automatically analyzing the images captured by the cameras.

[0013]

[0010] However, remote monitoring based on cameras has important drawbacks. The most important drawback is that the cameras only capture images in a limited field of view. It is extremely difficult to obtain a complete representation of an environment, for example a room, based only on cameras. Even when putting a large number of cameras in the room (which would render the remote monitoring system more expensive and resource consuming), unmonitored areas may still be present. . Details may also not be captured effectively. For instance, in cleanliness monitoring, certain spots such as corners or areas behind furniture can be difficult to observe remotely with vision-based systems. In mental health care settings, visually monitoring patients through cameras can be counterproductive, as the feeling of being constantly watched may worsen the psychological condition of vulnerable individuals. In contrast, sound-based monitoring is generally perceived as less intrusive than visual surveillance. Because it does not expose the person’s image or physical gestures, acoustic monitoring can provide valuable insights into behavior while preserving a greater sense of privacy and dignity — especially important in mental health care environments.

[0014]

[0011] Other drawbacks of remote monitoring based on cameras include the resource consumption. Indeed, video analysis is known to be very computationally expensive. Furthermore, when the captured videos are analyzed in a central server, the bandwidth requirements for transmitting the videos are important, especially in case of high-resolution videos. The resource issues are even more important when a plurality of cameras is located in the remotely controlled environments.

[0012] Using microphones instead of cameras alleviates these drawbacks. Indeed, a microphone placed in a room is able to capture all the sounds that are produced in the room. Thus, all a room, or a large portion of an outdoor environment, can be monitored using a single microphone. Furthermore, audio data is usually much smaller than video data, thereby significantly limiting the computing and bandwidth resources needed to process and analyze the microphone data compared to video.

[0015]

[0013] However, the audio data are much more difficult to automatically analyze than video data. Indeed, automatically detecting states or events solely based on audio data is extremely difficult. If machine learning engines are used to perform automatic analysis, the amount of training data necessary to train a machine learning engine to detect states of environments based on audio data is extremely high, so that such systems cannot in practice be deployed in prior art solutions.

[0016]

[0014] There is therefore a need of a method and system to remotely monitor an environment based on microphone data, which requires a limited amount of training data to be trained.

[0017] Summary

[0018]

[0015] This disclosure improves the situation.

[0019]

[0016] It is proposed a computer-implemented method, said method comprising: receiving a plurality of audio frames from one or more microphone located in an environment ; using one or more sound detection supervised machine learning engine to detect one or more sound classes from said plurality of audio frames ; using one or more state detection supervised machine learning engine to detect one or more state classes based at least from the output of said one or more sound detection supervised machine learning engine, each of said state classes being representative of a state of said environment and at least one of said state classes being representative of an alert state.

[0020]

[0017] By “an environment”, we designate a fixed or mobile location that is affected by human activity. For example, the environment may be an environment of a person such as for example a room where the person lives, a hospital bedroom where the person sleeps or even the person itself. The environment may also be a place that is affected by human activity without being tied to one specific person. For example, such an environment may be a classroom, a part of a street, a child playground, etc. Non limitative examples of environments comprise: a room : o a bedroom of a retirement home ; o a bedroom of a hospital ; o a bedroom in a house ; o a classroom ; o a factory floor; o a production area; o a manufacturing facility; o etc. a portion of an exterior place: o a playground ; o a part of a street; o etc. a vehicle, or portion of a vehicle: o the interior of a car ; o a wagon of a train ; o a cockpit of a plane ; o etc. a person or an animal: o an old person wearing a microphone to detect hazards ; o a sick person ; o a child ; o an animal ; o etc. a machine: o an automaton; o a cleaning appliance; o a robot; o a drone; o etc.. etc.

[0021]

[0018] By “state of an environment”, we designate an indication of what happens in the environment. For example, a state of an environment may represent : an action that is performed by one or more persons in an environment, such as “running”, “laughing”, “crying”, etc. an indication of the state of the elements of the environments, for example “TV playing”, “music playing”, “squeaky bed”, an indication of the some specific activities performed in the environments, for example "vacuum cleaner running", "mopping in progress", "floor being swept", "windows being ed", "dusting surfaces", "trash being emptied", "air freshener sprayed", "cleaning cart in the hallway"; etc. ;

[0022] An indication of some specific behaviors in mental health environments, for example “crying”, “screaming”, “talking to oneself’, “insulting others”, “sheets being torn”, “objects being thrown”, “pills being taken”, or “threatening language”, can provide early insights into emotional or behavioral crises. an indication of an event that just occurred in the environment, such as for example “person falling”, “person leaving the room”, etc. an indication of the presence or absence of person or animals ; when the environment is a person or an animal, a state of the person or the animal, such as for example “has fallen”, “long cry”, “repeated screaming”, “distress”, etc. etc.

[0023]

[0019] By “sound class” we designate a class that describes an individual sound that is recorded in an audio frame. A set of sound classes can be referred to as “a first set of classes”.

[0024]

[0020] By “state class” we designate a class that describes a state of the environment. A set of state classes can be referred to as “a second set of classes”.

[0025]

[0021] By “ machine learning engine”, we designate a model, or function, that can be trained on a training dataset, in order to generalize its learning on unseen data. The training phase of the machine learning engine allow determining parameters of the model or function that best fit the training set, while the inference phase makes use of the trained engine to perform predictions on unseen data.

[0026]

[0022] By “supervised machine learning engine”, we designate a type of machine learning engine where a specific type of output is predicted from an input type and wherein, and which is trained based on inputs associated to desired outputs called labels. Supervised machine learning engines encompass for examples machine learning engines such as artificial neural networks, decision trees, support vector machines or random forests. Supervised machine learning engines may typically be classifiers that aim at classifying input data in certain classes. In such case, the labels that are provided for the training can be the classes identified by a labeler in the training samples.

[0027]

[0023] By “sound detection supervised machine learning engine”, we designate a supervised machine learning engines that aims at detect sounds classes from audio frames.

[0028]

[0024] By “state detection supervised machine learning engine”, we designate a supervised machine learning engines that aims at detect states of environments.

[0029]

[0025] The sound detection supervised machine learning engine, and state detection supervised machine learning engine may detect the sound, respectively state classes in different ways. For example, each of the state and sound machine learning engines may perform one of the following tasks: a classification task: in such case, the machine learning engine can be qualified as “a classifier”, and is configured to detect the presence or absence of each class. In such case, a machine learning engine may for example output a vector indicating, for each class, the presence or absence of the class, and / or a confidence index of the presence or absence of the class , a regression task: in such case, the machine learning engine can be configured to predict, for each class, a ratio of positive labels associated to the class (e.g a ratio of predictions of the presence of the class among a plurality of predictions).

[0030]

[0026] By “ class representative of an alert state”, we designate a class that indicates an alert. For example, such class may be a class indicating that a persona has fallen, repeated screaming, patient distress, spilled liquid, etc.

[0031]

[0027] The state detection supervised machine learning engine takes as input a limited amount of input data to perform the state detection. In particular, the input data of the state detection supervised machine learning engine comprise sound classes (for example in the form of a binary prediction of presence / absence of each class, a confidence index of the presence / absence of each class, or a ratio of prediction of presence / absence of a class among a plurality of predictions), while the input data of the sound detection supervised machine learning engine comprises audio frames.

[0032]

[0028] Thus, the sound detection supervised machine learning engine is able to detect the sounds, and the state detection supervised machine learning engine can be a lightweight machine learning engine, that is able to perform an accurate detection of the states of the environment while a direct detection of the state of the environment from audio frames would be much harder to achieve.

[0033]

[0029] The method thus allows to use supervised machine learning engines for an accurate detection of the states of an environment using a limited amount of training data and labels.

[0034]

[0030] In another aspect, it is proposed a computer-implemented method comprising: receiving a plurality of audio frames from one or more microphone located in one or more environment ; for each audio frame of said plurality of received audio frames: causing one or more loudspeaker of one or more Man Machine Interface to play said audio frame ; obtaining one or more sound label belonging to a set of sound classes ; receiving, through said one or more Man Machine Interface, one or more state label belonging to a set of state classes, each of said state classes being representative of a state of said environment and at least one of said state classes being representative of an alert state ; enriching one or more sound detection training dataset with said audio frame associated to said one or more sound label ; using said one or more sound detection training dataset to train one or more sound detection supervised machine learning engine to detect one or more sounds classes from audio frames ; for each audio frame of said plurality of audio frames: enriching one or more state detection training dataset with a state detection training sample associating said one or more state label to one or more training sound classes ; using said state detection training dataset to train one or more state detection supervised machine learning engines to detect one or more state classes based at least from the output of said one or more sound detection supervised machine learning engine.

[0035]

[0031] The state detection supervised machine learning engine takes as input a limited amount of input data to perform the state detection. In particular, the input data of the state detection supervised machine learning engine comprise sound classes (for example in the form of a binary prediction of presence / absence of each class, a confidence index of the presence / absence of each class, or a ratio of prediction of presence / absence of a class among a plurality of predictions), while the input data of the sound detection supervised machine learning engine comprises audio frames.

[0036]

[0032] Thus, the sound detection supervised machine learning engine is able to detect the sounds, and the state detection supervised machine learning engine can be a lightweight machine learning engine, that is able to perform an accurate detection of the states of the environment while a direct detection of the state of the environment from audio frames would be much harder to achieve.

[0037]

[0033] Another advantage of the method is that a training can be performed for a specific environment and / or previously unseen sound or state classes with a limited amount of training data. Thus, the labelling burden for adapting the training to a new environment or new sound or state classes is limited. Furthermore, the detection solution can be quickly deployed to new environments while taking into account the specificities of the new environments (e.g background noise, specific sound or state classes to take into account, etc.).

[0038]

[0034] In another aspect, it is proposed a computer software comprising instructions to implement at least a part of a method as defined here when the software is executed by a processor.

[0039]

[0035] In another aspect, it is proposed a computer-readable non-transient recording medium on which a software is registered to implement the method as defined here when the software is executed by a processor.

[0040]

[0036] In another aspect, it is proposed one or more first computing device configured to: receive a plurality of audio frames from one or more microphone located in an environment ;use one or more sound detection supervised machine learning engine to detect one or more sound classes from said plurality of audio frames ; use one or more state detection supervised machine learning engine to detect one or more state classes based at least from the output of said one or more sound detection supervised machine learning engine, each of said state classes being representative of a state of said environment and at least one of said state classes being representative of an alert state.

[0041]

[0037] In another aspect, it is proposed a computing system comprising: one or more microphone; the one or more first computing device as defined here.

[0042]

[0038] In another aspect, it is proposed a computer system comprising: one or more Man Machine Interface comprising one or more loudspeaker ; one or more processing unit configured to: receive a plurality of audio frames from one or more microphone located in an environment ; for each audio frame of said plurality of audio frames: cause one or more loudspeaker of one or more Man Machine Interface to play said audio frame ; receive, through said one or more Man Machine Interface, one or more sound label belonging a set of sound classes ; receive, through said one or more Man Machine Interface, one or more state label belonging a set of state classes, each of said state classes being representative of a state of said environment at least one of said state classes being representative of an alert state ; enrich one or more sound detection training dataset with said audio frame associated to said one or more sound label ; use said one or more sound detection training dataset to train one or more sound detection supervised machine learning engine to detect one or more sounds classes from audio frames ; for each audio frame of said plurality of audio frames: enrich one or more state detection training dataset with a state detection training sample associating said one or more state label to one or more training sound classes ; use said state detection training dataset to train one or more state detection supervised machine learning engine to detect one or more state classes based at least from the output of said one or more sound detection supervised machine learning engine.

[0043]

[0039] The following features, can be optionally implemented, separately or in combination one with the others:

[0044]

[0040] In an embodiment, the method further comprises, when said one or more state detection supervised machine learning engine detects a class representative of an alert state, sending said alert to one or more user device configured to render said alert to a user.

[0045]

[0041] Thus, when an alert is detected in the environment, it is sent to a user device for processing the alert, and helping if needed the person.

[0046]

[0042] In an embodiment, said one or more training sound classes are said one or more sound label

[0047]

[0043] Said one or more training sound classes are said one or more sound labels. Thus, in the training phase the inputs of the state detection supervised machine learning engines are the same as the outputs of the sound detection supervised machine learning engines.

[0048]

[0044] Thus, the state detection training dataset can be trained based on the input (sound classes) and output (state classes) provided by the user. Thus, the training of the state detection supervised machine learning engine is based on labels that are as accurate as possible.

[0049]

[0045] In an embodiment, using said one or more sound detection training dataset to train said one or more sound detection supervised machine learning engine to detect one or more sounds classes from audio frames is performed prior to enriching said one or more state detection training dataset ; for each audio frame of said plurality of audio frames: the method comprises using said one or more trained sound detection supervised machine learning engine to detect one or more sound classes from said audio frame ; said one or more training sound classes are the one or more sound classes detected from said audio frame by the one or more sound detection supervised machine learning.

[0050]

[0046] In inference stage, the input of the state detection supervised machine learning engine comprises the output of the detection of the sound detection supervised machine learning engine.

[0047] In substance, the above features mean that when new audio frames are received and labeled, the sound detection supervised machine learning engine is first trained, then the state detection supervised machine learning engine is trained based upon the predictions of the sound detection supervised machine learning engine, and the state labels provided by the user. Thus, the training of the state detection supervised machine learning engine is performed based on input data that is as similar as possible to the input data that is received in inference phase. Thus, the general prediction of the states is more accurate.

[0051]

[0048] In an embodiment, audio frames are captured by said one or more microphone at successive time frames ; said one or more state detection supervised machine learning engine comprises a state detection supervised machine learning engine that takes into account detections of sound classes from the one or more sound detection supervised machine learning engine from a plurality of successive audio frames.

[0052]

[0049] By “takes into account detections of sound classes from the one or more sound detection supervised machine learning engine from a plurality of successive audio frames”, it is understood that the prediction of a state detection supervised machine learning engine for a given audio frame at a time frame depends not only on the predictions of sound classes from the one or more sound detection supervised machine learning engine for the given audio frame, but also from the predictions of sound classes of for the neighbors audio frames, for example the previous audio frames.

[0053]

[0050] Taking into account detections of sound classes from the one or more sound detection supervised machine learning engine from a plurality of successive audio frames may be performed in different ways, comprising for example: providing as input to the state detection supervised machine learning engine detections of sound classes from the one or more sound detection supervised machine learning engine from a plurality of successive audio frames. For example, when a state prediction is to be performed at a time step t, the state detection supervised machine learning engine may take as input the prediction of sound classes at time t, but also at times t-1 , t-2, etc. ; using a state detection supervised machine learning engine that maintains an internal (or hidden) state based upon the previous inputs and / or predictions. For example, when a state prediction is to be performed at a time step t, the state detection supervised machine learning engine may take as input only the prediction of sound classes at time t, but also takes into account an internal state that depends upon the predictions of sound classes that have been previously provided as input at times t-1 , t-2, etc. Thus the prediction of the state at time t takes a plurality of successive audio frames et times t, t-1 , t-2, etc..

[0054]

[0051] Taking into account detections of sound classes from the one or more sound detection supervised machine learning engine from a plurality of successive audio frames allows the state detection supervised machine learning engine to detect states of the environment according to a temporal succession of detected sounds.

[0052] Training a detection of states on a plurality of successive detection of sound classes over a plurality of successive audio frames allows detecting the state of the environment according to the audio captured in a time frame that corresponds to the sum of the durations of the plurality of audio frames.

[0055]

[0053] Thus, the detection of the state of the environment can be performed based upon the audio captured in a much longer duration than a single audio frame.

[0056]

[0054] Meanwhile, training the state detection supervised machine learning engine on detections of sound classes allows performing a much lighter training and requires a significantly lower amount of labels than training a state detection directly on audio frames in long time windows.

[0057]

[0055] Thus, an efficient detection of the state of the environment, and an efficient detection of alerts can be achieved, using a limited amount of training data.

[0058]

[0056] In an embodiment, said state detection supervised machine learning engine that takes into account detections of sound classes from the one or more sound detection supervised machine learning engine from the plurality of successive audio frames is a recurrent neural network.

[0059]

[0057] A recurrent neural network maintains a hidden state of the environment and updates this state when new observations are input to the recurrent neural network.

[0060]

[0058] Thus, at each time step, the recurrent neural network takes as input only the predictions of sound classes at the current time step, but takes into account the previous sound classes through its internal state. The interactions between the inputs, the predictions and the internal state of the recurrent neural network are optimized during the training phase, so that the recurrent neural network to perform accurate state predictions at each time step based on current but also past sound classes predictions.

[0061]

[0059] Thus, a recurrent neural network provides an efficient solution to detect the state of the environment based upon a temporal succession of audio frames.

[0062]

[0060] In an embodiment, an audio frame at a time frame is sent by said one or more microphone only upon a condition relative to a sound level from said audio frame; said state detection supervised machine learning engine further takes as input indications of absence of reception of audio frames

[0063]

[0061] Avoiding sending audio frames when no noticeable sound level has been recorded significantly reduces the amount of data sent by the microphones.

[0064]

[0062] Meanwhile, taking into account silent frames may be important for an accurate state detection.

[0065]

[0063] Therefore, the use by the state detection supervised machine learning engine of a further input indicating the absence of reception of audio frames allows providing an accurate state detection that takes into account the silences, while drastically limiting the amount of data sent by the microphones.

[0064] In an embodiment, said one or more state detection supervised machine learning engine comprises a state detection supervised machine learning engine that takes as input: the one or more sound classes detected by said one or more sound detection supervised machine learning engine from an audio frame ; one or more indicator of detection of words belonging to a predefined set of words by a speech recognition engine from said audio frame.

[0066]

[0065] Taking as input, for a same audio frame, detections of sound classes from the one or more sound detection supervised machine learning and indicators of detection of certain words in the audio frames allows the state detection supervised machine learning engine to correlate the detected sounds and words to provide an efficient state detection.

[0067]

[0066] In an embodiment, the execution of: the speech recognition engine; and the state detection supervised machine learning engine that further takes as input the one or more sound classes detected by said one or more sound detection supervised machine learning engine from an audio frame and words detected by the speech recognition engine from said audio frame ; are conditional upon a detection of said at least one of said state classes representative of an alert state by a gradient boosting machine learning engine that takes as input the one or more sound classes detected by said one or more sound detection supervised machine learning engine.

[0068]

[0067] Using a gradient boosting machine learning engine that takes as input the one or more sound classes detected by said one or more sound detection supervised machine learning engine allows a more focused detection of certain alert classes. The gradient boosting machine learning engine can be trained using the same labels as the state detection supervised machine learning engine.

[0069]

[0068] By “gradient boosting machine learning engine”, we designate a machine learning based on gradient boosting.

[0070]

[0069] Thus, the speech recognition engine, and state detection supervised machine learning engine are only executed when the gradient boosting machine learning engine detects a suspicion of presence of an alert state.

[0071]

[0070] The gradient boosting machine learning is significantly less complex than the speech recognition engine. For the vast majority of audio frames, the gradient boosting machine learning will be executed, but not the speech recognition engine. Thus, a significant reduction of the complexity of detection and power consumption can be achieved while preserving a good accuracy in the detection of alerts.

[0072]

[0071] In an embodiment, said speech recognition engine is not executed upon the detection of a sound class representative of a conversation between a plurality of persons.

[0073]

[0072] This promotes the privacy of the conversations, as the speech uttered is not detected in the audio frame when the audio frame relates to a conversation between a plurality of persons.

[0074]

[0073] In an embodiment, at least one among the one or more sound detection supervised machine learning engine and the one or more state detection supervised machine learning engine is configured to perform a regression task to predict, for each sound class, respectively state class, a ratio of received labels that indicate the presence or the absence of the sound class, respectively state class.

[0075]

[0074] When a plurality of labels are obtained for a same audio frame (for example if a plurality of labels and / or pseudo-labels are received respectively from a plurality of labelers and / or pre-trained machine engines for a sample), it is common that some labels differ (e.g some labelers / pre-trained model indicate that a given sound or state class is present from a given audio frame, while other indicate that the sound or state class is absent). It is thus possible to determine, for each class, a ratio of positive labels and negative labels (for example, 80% of labels indicate that the sound class “squeak” is present in a given audio frame).

[0076]

[0075] A ratio of received labels that indicates the presence or the absence of a (sound or state) class thus refers to a quantitative measure that represents the proportion of labels indicating the presence of a specific class relative to the total number of labels received for that class. This ratio is typically expressed as a fraction or percentage and is used to assess the likelihood that one or more labelers or pre-trained models indicate the class as present in the audio frame.

[0077]

[0076] For example, if an audio frame is labeled by multiple labelers, and 8 out of 10 labelers indicate the presence of the sound class "dog bark," the ratio of received labels indicating the presence of the "dog bark" sound class would be 0.8 (or 80%). Conversely, the ratio indicating the absence of the "dog bark" sound class would be 0.2 (or 20%).

[0078]

[0077] Predicting a ratio of received labels that indicate the presence or the absence of the sound class rather than a classification (presence / absence of a class, or index of confidence) allows providing more accurate results, because it allows preserving the complete information received in the creation of the training dataset. This is even more the case for the predictions of sound classes, because the ratios obtained as output of the sound detection engine can be directly provided as input to the state detection engine.

[0079]

[0078] For example, if 4 out of 5 labelers indicated that an audio frame contains the sound class “squeak”, training the sound prediction engine to predict that 80% of labels indicated the presence of the class retains more information that simply predicting that the class was present. The accuracy of the predictions is therefore increased with the same received labels.

[0080] Brief Description of Drawings

[0081]

[0079] Other features, details and advantages will be shown in the following detailed description and on the figures, on which:

[0082] Fig. 1

[0083]

[0080] [Fig. 1] is an example of a computing system to remotely monitor one or more environments according to an embodiment.

[0084] Fig. 2

[0085]

[0081] [Fig. 2] is an example of a computing system to train one or more state detection supervised machine learning engine. Fig. 3

[0086]

[0082] [Fig. 3] is an example of a method to detect states of an environment according to an embodiment.

[0087] Fig. 4

[0088]

[0083] [Fig. 4] is a computer-implemented training method according to an embodiment.

[0089] Fig. 5

[0090]

[0084] [Fig. 5] is a second example of a computer-implemented training method according to an embodiment.

[0091] Fig. 6

[0092]

[0085] [Fig. 6] is an architecture of machine learning engines to detect states of an environment in an embodiment.

[0093] Fig. 7

[0094]

[0086] [Fig. 7] is an example of a graphical interface to allow a labeler to associate sound labels to a sound frame according to an embodiment.

[0095] Fig. 8

[0096]

[0087] [Fig. 8] is an example of a graphical interface to allow a labeler to associate state labels to state frames according to an embodiment.

[0097] Fig. 9

[0098]

[0088] [Fig. 9] is an example of sound classes according to an embodiment.

[0099] Figs. 10A, 10B, 10C, 10D, 10E, 10F

[0100]

[0089] [Fig. 10A], [Fig. 10B], [Fig. 10C], [Fig. 10D], [Fig. 10E] and [Fig. 10F] are another example of sound classes according to an embodiment.

[0101] Figs. 11A, 11B

[0102]

[0090] [Fig. 11 A] and [Fig.11 B] are an example of state classes according to an embodiment.

[0103] Figs. 12A, 12B, 12C

[0104]

[0091] [Fig. 12A], [Fig.12B] and [Fig. 12C] are an example of a structured set of sound classes in the context of psychiatric or psychological care monitoring according to an embodiment.

[0105] Fig. 13

[0106]

[0092] [Fig. 13] is an example of a structured set of state classes that represents clinical alerts or behavioral states relevant to mental health monitoring

[0107] Description of Embodiments

[0108]

[0093] It is now referred to figure 1 .

[0094] Figure 1 is an example of a computing system Sys1 to remotely monitor one or more environments Env1 .1 , Env1 .2 according to an embodiment.

[0109]

[0095] In the example of figure 1 , the system Sys1 is a remote monitoring system of one or more retirement homes. In the example of figure 1 , the monitored environments are thus two bedrooms of a retirement home, noted respectively Env1.1 and Env1.2. Thus, the system Sys1 aims at automatically detecting, from the sound captured by the microphones, if something unusual happened in the bedrooms of the residents of the retirement homes.

[0110]

[0096] To this effect, one or more microphones are placed in each of the environments. In the example of figure 1 , one microphone Mid .1 is placed in the room Env1.1 , and one microphone Mid .2 is placed in the room Env1 .2.

[0111]

[0097] The microphones are configured to capture sounds in their respective environments and send the audio frames comprising the captured sound to a first computing device Dev1.1. In the example of figure 1 , the first microphone Mid .1 sends a first audio frame Snd1.1 , and the second microphone Mid .2 sends a second audio frame Snd1.2. the communications are performed between a communication endpoint and ComE1 .1 .1 of the first microphone Mid .1 , a communication endpoint ComE1 .1 .2 of the second microphone Mid .2, and a communication endpoint ComE1 .2 of one or more first computing device Devi .1 . The communication may be performed using any suitable communication means including wired or wireless telecommunication network. For example, Ethernet networks, 4G / 5G networks, optical fiber networks, etc. may be used.

[0112]

[0098] It is worth noting that the term “microphone” may refer to a device that comprises a microphone and a communication endpoint. For example, the term “microphone” may refer to a housing that comprises a microphone, a communication endpoint, and one or more processing units to process the signals from the microphone to create the audio frames to send via the communication endpoint.

[0113]

[0099] The one or more microphones may be associated to unique identifiers (which may be for example a unique identification number of the microphone, an indication of the location of the microphone, etc.), so that the environment to which each microphone belongs to can be traced during all the process of analyzing audio frames, then raising alerts, so that an alert can be associated with the right environment.

[0114]

[0100] The audio frames may be sent in different audio formats. For example, raw formats of compressed formats may be used. Different sound depths (8 bits, 16bits, 24bits, 32 bits for instance), or sound formats (mono, stereo for instance) may be used. Each microphone may integrate, or somehow associate, its unique identifier to the audio frames, so that the detection can be associated to a particular microphone and environment.

[0115]

[0101] Each frame may be associated with a timestamp, which may for example a frame identifier or a time of recording.

[0102] In a number of embodiments, the microphones may send the audio frames only upon a condition relative to a sound level in said audio frame. For example, a microphone may send an audio frame: only if the average sound intensity in the audio frame is above a predefined threshold ; or only if a maximum sound intensity in the audio frame is above a predefined threshold ; if a temporally close peak of sound intensity is identified near the audio frame. For example, the audio frames may be sent up to 3 seconds before and 7seconds after the sound intensity is equal to or higher than a predefined threshold, for example expressed in dB ; the sound intensity may be considered for a single microphone, or a plurality of microphones. In the latter case, the sound intensity may correspond to different combinations of the sound signals outputted by the microphones (e.g average of the sound signal or energy of the sound signals of a plurality of microphones, difference between the sound signals of two microphones or the energy of the sound signals of two microphones, etc.) etc.

[0116]

[0103] Stated otherwise, the microphones may send only the frames for which a sufficiently high sound level is detected, because the audio frames with only very low sound will not be considered as carrying information. Thus, the amount of data sent by the microphones is reduced and the system consumes a lower amount of bandwidth, without reducing the accuracy of the detection.

[0117]

[0104] In addition, in order to ensure that a minimum number of audio frames are transmitted, an audio frame can be transmitted at regular intervals even in absence of noticeable sounds. For example, an audio frame can be transmitted every 10 minutes even in absence of noticeable sound. In an embodiment, a longer frame (for example an audio frame of 2 minutes instead of 10 seconds) can also be sent from time to time, for example every 7 hours.

[0118]

[0105] When an audio frame is not transmitted by a microphone to the device Devi .1 , the absence of transmission may be signaled in different ways. In an embodiment, no data is transmitted. The detection of the absence of reception of audio frame at a given timestamp then indicates that the frame has not been transmitted. Alternatively, an explicit signaling indicating the absence of transmission may be sent.

[0119]

[0106] The one or more first computing device Dev.1 .1 comprises one or more memories Mem1 .1 , and one or more processing units Prod .1 .

[0120]

[0107] “Processing unit” means an electronic component capable of performing electronic or computer calculations for a function. A processing unit can designate any type of processor or electronic component capable of performing digital calculations. For example, a processing unit can be an integrated circuit, an ASIC (from the English acronym "Application-Specific Integrated Circuit", literally in French "integrated circuit specific to an application", a microcontroller or microcontroller unit (MCU), a microprocessor, a Digital Signal Processor (DSP), a processor, a Graphical Processing Unit (GPU). A processing unit according to the invention is not limited to a particular type of calculation architecture. For example, a processor can implement a Harvard or Von Neumann type architecture.

[0121]

[0108] “Memory” means a digital electronic device used to store data. Different types of memories can be used in the invention, such as read only memory, random access memory, volatile memory or flash memory. A device according to the invention can be equipped with one or more non-volatile memories which can be of different types such as mass memories, flash memory, read only memories or SSD memories. A device according to the invention may also comprise one or more random access memories such as RAM, DRAM, SRAM, DPRREAM, VRAM, eDRAM or 1T-SRAM.

[0122]

[0109] The one or more processing units Proc1.1 are configured to detect states of the environments based upon the received audio frames. To this effect, the one or more processing unit Proc1.1 is configured to analyze the audio frames, and deduce the states of the environments. For example, the first processing unit may detect the states of the rooms of the retirement home, to determine for example if the resident of the room is fallen, left the room unaccompanied, if the TV is playing, etc. When the audio frames are associated to a unique identifier of a microphone, the state can be associated to the corresponding environment.

[0123]

[0110] To this effect, the states of the environments can be classified into one or more classes, and some of the classes may be representative of an alert state (for example a fall or a unaccompanied leaving of the room for an old person). When a class representative of an alert state is detected, an alert can be raised to the personal of the retirement home.

[0124]

[0111] In the example of figure 1 , in order to raise an alert: the alert is sent by a communication endpoint ComE1 .3 of the one or more computing device to a communication endpoint of one or more second computing device ; the one or more second computing device determines a user device UDev1 .3 to which the alert is to be transmitted, and transmit, from a communication endpoint ComE1 .5 of the one or more second computing device, the alert to a communication endpoint ComE1.6 of the user device UDev1 .3 where the alert is to be dispatched.

[0125]

[0112] To this effect, the one or more second computing device is equipped with one or more processing unit Prod .2 and one or more memory Mend .2. The one or more processing unit Prod .2 is configured to determine to which user device the alert is to be dispatch depending upon the environment that is concerned by the alert and / or the alert type. For example, the one or more processing unit Prod .2 pay determine, based on the alert type and an identifier of the microphone from which the audio frames that caused the alert were sent.

[0126]

[0113] For example, the user devices UDev1 .3 may be tablets, smartphones or other user devices of the personal of the retirement homes where the rooms Env1.1 and Env1.2 are found. Thus, the one or more computing device may determine to which retirement home, and optionally to which personal, the alert is to be sent. For example, certain rooms may be associated to a specific personal that is in charge of the room. The personal to which the alert is sent may also depend upon the type of alert. Some alerts may be for example medical emergency, that are sent directly to a medic, some other may be sent to caregivers, etc.

[0127]

[0114] Once an alert is received by a user device UDev1.3, the alert may be rendered to the user of the user device. For example, a visual indication may be displayed of the screen of the user device, indicating the type of alert, as well as an indication of the environment, such as for example the room number, or the name of the resident of the room. Other alert means may be rendered to the user, such as for example beeps, sound alerts, etc.

[0128]

[0115] Thus, the system Sys1 allows detecting different states and events of the rooms, and dispatch alerts as early as possible so that the personal can perform interventions as early as possible. As the alerts are based upon the sound data that is captured by the microphones, everything that happens in the room can be monitored. The system Sys1 therefore provides an effective means of remotely monitoring a large number of rooms of retirement homes and perform interventions as early as possible when aged people require assistance.

[0129]

[0116] In some embodiments, some classes that are not representative of alert classes can also be transmitted. Thus, the personal of the retirement homes can benefit from a remote monitoring that is not limited to alerts. For example, they can know which residents are singing, watching TV, etc.

[0130]

[0117] The example of figure 1 is provided by means of non limitative examples only of a computing system of the disclosure. The disclosure is however not limited to this system. For example: the number of rooms that are monitored may be different from 2. For example, a single room or more than two rooms may be monitored. The monitored room may belong to one or a plurality of different retirement homes ; the monitored environments may be different from a retirement homes. For example: o fixed environments such as hospital rooms, classrooms, children playground, portions of streets may be monitored ; o mobile environments such as a vehicle, a portion of a vehicle of a person can be monitored. In such cases, position information can be send, for example by the microphones, in addition to the audio frames or in audio frames so that, when an alert is raised, an intervention can be performed at the right location (for example the current location of the vehicle or the person). a monitored environment may be an environment of a person, for example a bedroom of that person in a retirement home or a hospital, or an environment that is not specifically tied to a person, such as for example a classroom or a portion of street ; more than one microphone can be placed in a given environment ; the computing system may be a computing system that is internal to a single structure (for example, a computing system of a single retirement home), or a computer system that encompasses a plurality of structures (for example, a plurality of different retirement homes). In the former case, the computing devices may be located in the single structure, and local communication networks such as Wi-Fi or Ethernet may be used. In the latter case, the computing devices may be external to the structures. The computing devices may be for example servers of a remote monitoring service provider, and global communication networks such as optical fiber or 4G / 5G networks may be used for the communication between the microphones and the computing devices ; even if a single computing device Dev1.1 is represented is figure 1 , a plurality of first computing devices may be used, for example a server farm ; even if a single computing device Dev1.2 is represented is figure 1 , a plurality of second computing devices may be used, for example a server farm ; even if a single user device UDev1.3 is represented is figure 1 , a plurality of user devices may be used ; even if a single processing unit Proc1.1 is represented is figure 1 , a plurality of processing units may be used by the one or more first computing device Devi .1 , for example a plurality of cores of a multi-core processor, a plurality of processors, a plurality of processors spread across a plurality of computing device (in case distributed computing is used), etc. even if a single processing unit Prod .2 is represented is figure 1 , a plurality of processing units may be used by the one or more second computing device Devi .2, for example a plurality of cores of a multi-core processor, a plurality of processors, a plurality of processors spread across a plurality of computing device (in case distributed computing is used), etc. the alerts that are raised are dependent upon the environment that is monitored. The dispatch of the alert through the one or more second computing devices and user devices is purely illustrative, and any suitable means of dispatching alerts can be used according to various embodiments of the invention ; the dispatch of alerts through the one or more second computing device Dev1.2, and the user device UDev3.1 is provided by means of purely illustrative example only. For example: o the one or more first computing device Dev1.1 can directly transmit alerts to user devices ; o the alerts can be transmitted to central service rather than user devices ; o the monitoring of states is not necessarily done to raise alert, but may for example be done for obtaining statistics relative to the actions performed in the environment o etc.

[0131]

[0118] The system Sys1 is thus provided by means of illustrative example only of a computing system of the disclosure. It is apparent that the objective of the invention is to detect the states of the environments, and that many different uses of the state classes can be performed.

[0119] It is in particular worth noting that the invention is not restricted to detection of states relative to an environment of a person. For, the detected state classes may be classes for a cleaning monitoring application.

[0132]

[0120] More generally, the invention is not restricted to a particular purpose such as the monitoring of an environment of a person, or of a cleaning application, but may be more generally to any case where audio recordings captured in an environment can be analyzed to determine the state of an environment and raise an alert.

[0133]

[0121] It is now referred to figure 2.

[0134]

[0122] Figure 2 is an example of a computing system to train one or more state detection supervised machine learning engine.

[0135]

[0123] The computing system Sys2 may be for example a personal computer, a tablet, a server or more generally any kind of computing system that has access to one or more Man Machine Interface. The computing system Sys2 thus comprises one or more Man Machine Interface MMI2, and one or more processing unit Proc2.

[0136]

[0124] The Man Machine Interface may be at least partially located in the same computing device than the one or more processing unit Proc2 (which is for example the case of an internal loudspeaker or an internal microphone of a tablet).

[0137]

[0125] The one or more Man Machine Interface MMI2 may be also be located outside the computing device where the one or more processing unit are located. This may be for example the case of a Man Machine Interface in a wired or wireless local connection with the computing device, such as for example a mouse, keyboard, or a microphone or loudspeakers connected through a wire or a Bluetooth connection to the computing device. This may also be the case for example if the interface with the user is performed in a local device, and the training in a distant computing device in connection with the local device.

[0138]

[0126] The one or more Man Machine Interface MMI2 comprises one or more Loudspeakers Ldspk2. The one or more Loudspeakers Ldspk2 allow rendering audio frames to a user of the system sys2. The one or more Man Machine Interface MMI2 may further comprises other interfaces such as one or more screen Scr2 and / or one or more mouse Ms2 and or one or more keyboard Keyb2. Other types of Man Machine Interfaces, not represented in the figure 2, may also be considered. For example, a microphone may be used. It is apparent that the system Sys2 is not restricted to specific types of Man Machine Interfaces, provided that the user is able to listen to audio frames, and associate labels to the audio frames.

[0139]

[0127] In general, the user can associate each audio frame to : one or more sound labels belonging a set of sound classes. The sound labels represent the sounds that can be heard in the audio frames. For examples, the sound classes may define different types of sounds such as noises, breathing, clapping, flush, dog, bird, cat, phone ring, sing laugh, etc. ; one or more state labels belonging to a set of state classes. The state labels represent the states of the environment where the audio frame are recorded. The state labels may for example represent an event, an action or a feeling of a resident that can be inferred from the sounds. For example, in a context of a room in a retirement home, state classes may for example be “call for help”, “repeated screams”, “happy”, “stress”, “left the room alone”, “speaking to telephone”, “speaks alone”, etc.

[0140]

[0128] The provision by the user of sound and state labels in response to rendering audio frames on the loudspeakers allow enriching, on one or more memory Mem2 of the system Sys2: one or more sound detection training dataset SndDS2.1 , SndDS2.2 where the audio frames are associated to sound labels ; one or more state detection training dataset StDS2.1 , StDS2.2 where the audio frames are associated to state labels.

[0141]

[0129] The one or more sound detection training dataset SndDS2.1 , SndDS2.2 are used to train one or more sound detection supervised machine learning engine (also called more shortly “sound detection engines”) SndMLEng2.1 , SndMLEng2.2 to detect one or more sounds classes in audio frames.

[0142]

[0130] The one or more state detection training dataset StDS2.1 , StDS2.2 are used to train one or more state detection supervised machine learning engine (also called more shortly “state detection engines”) StMLEng2.1 , StMLEng2.2 to detect one or more state classes.

[0143]

[0131] The sound detection engines and state detection engines may for example be classifiers.

[0144]

[0132] The sound detection engines SndMLEng2.1 and SndMLEng2.2 may thus be classifiers that are configured to output respectively a vector having one element for each class of a set of sound classes, wherein each element provides an indication of the detection of a class. The indication of the detection of a class may be for example a Boolean value indicating the detection or lack of detection of a class, a confidence value of the presence of a class, etc. Thus, no sound class, or more than one sound classes may be detected.

[0145]

[0133] Conversely, the state detection engines StMLEng2.1 , StMLEng2.2 may be classifiers that are configured to output respectively a vector having one element for each class of a set of state classes, wherein each element provides an indication of the detection of a class. The indication of the detection of a class may be for example a Boolean value indicating the detection or lack of detection of a class, a confidence value of the presence of a class, etc. Thus, no sound class, or more than one state classes may be detected.

[0146]

[0134] Alternatively, the sound detection engines and state detection engines may perform a regression task to predict a ratio of labels indicating the presence or absence of each class. Such a label may for example be expressed as a percentage, or a natural number in an interval [0; 1]. In such case, the output of the sound detection engines would be a vector having one element for each class of a set of sound classes, wherein each element provides a ratio of labels indicating the presence or absence of the detection of the class, while the output of the state detection engines would be a vector having one element for each class of a set of state classes, wherein each element provides a ratio of labels indicating the presence or absence of the detection of the class

[0147]

[0135] Of course, the sound detection engines and state detection engines may belong to different types of engines. For example, the sound detection engines may perform a regression task, while the state detection engines may be classifiers that perform a classification task.

[0148]

[0136] The operations performed by the computing system Sys2 in order to train the machine learning engines are described in more details with reference to figures 4 and 5.

[0149]

[0137] It is now referred to figure 3.

[0150]

[0138] Figure 3 is an example of a method P3 to detect states of an environment according to an embodiment.

[0151]

[0139] The method P3 may for example be implemented by a computing device such as the device Devi .1 represented in figure 1 .

[0152]

[0140] The method P3 comprises a first step S3.1 of receiving a plurality of audio frames from one or more microphone located in an environment.

[0153]

[0141] In the example of figure 1 , the step S3.1 may for example correspond to receiving the audio frames Snd1 .1 , Snd1 .2 from the one or more microphone Mid .1 ; Mid .2 located in the environments Env1.1 , Env1.2.

[0154]

[0142] The method P3 further comprises a second step S3.2 of using one or more sound detection supervised machine learning engine to detect one or more sound classes from said plurality of audio frames. The one or more sound detection machine learning engine may for example be the machine learning engines SndMLEng2.1 and SndMLEng2.2 represented in figure 2.

[0155]

[0143] As already explained, a sound detection machine learning engine is a supervised machine learning engine that takes as input an audio frames and delivers as output a prediction of sound classes that are present from the audio frame.

[0156]

[0144] The prediction of sound classes may for example take the form of a vector that comprises for each sound class an indication of presence or absence of the sound class (e.g Boolean value, confidence of presence, etc.), if the sound detection machine learning engine is a classifier.

[0157]

[0145] The prediction of sound classes may also in another example take the form of a vector that comprises for each sound class a ratio that indicates what would be the ratio of labels representative of presence or absence of the sound class among a plurality of labels received for the audio frame.

[0158]

[0146] The method P3 further comprises a third step S3.3 of using one or more state detection supervised machine learning engine to detect one or more state classes based at least on the output of said one or more sound detection supervised machine learning engine, each of said state classes being representative of a state of said environment and at least one of said state classes being representative of an alert state. The one or more state detection machine learning engine may for example be the machine learning engines StMLEng2.1 and StMLEng2.2 represented in figure 2.

[0159]

[0147] Thus, a state detection machine learning engine is a supervised machine learning engine that takes as input the prediction of sound classes delivered by the one or more sound detection supervised machine learning engine and predicts a state of the environment based at least from the prediction of sound classes. It is worth noting that depending upon the embodiments of the invention, the state detection machine learning engines may take as input predictions of sound classes corresponding to a single audio frame, or predictions of sound classes corresponding to a temporal succession of a plurality of successive audio frames.

[0160]

[0148] According to various embodiments of the invention, the detection machine learning engine may for example thus take as input: a vector that comprises for each sound class an indication of presence or absence of the sound class ; or a vector that comprises for each sound class a ratio that indicates what would be the ratio of labels representative of presence or absence of the sound class among a plurality of labels received for the audio frame.

[0161]

[0149] As explained above, the state detection machine learning engine thus takes as input a limited number of data corresponding to the predictions of the one or more sound detection machine learning engine. The state detection machine learning engine can thus take as input an amount of data that is much more limited that the content of the audio frames. The state detection machine learning engine can thus be a lightweight machine learning engine that can provide accurate detection while being trained with a limited training set.

[0162]

[0150] The output of the state detection machine learning engine may also belong to different types.

[0163]

[0151] The prediction of state classes may for example take the form of a vector that comprises for each state class an indication of presence or absence of the sound class (e.g Boolean value, confidence of presence, etc.), if the state detection machine learning engine is a classifier.

[0164]

[0152] The prediction of state classes may also in another example take the form of a vector that comprises for each state class a ratio that indicates what would be the ratio of labels representative of presence or absence of the state class among a plurality of labels received for the audio frame.

[0165]

[0153] The presence of a given class may be ultimately assessed in different ways depending upon the outputs of the state detection engine. For example:

[0166] If the state detection engine returns a vector providing for each sound class a binary value of absence or presence of the class, the binary value can be directly read to determine if the class is present or absent ;

[0167] If the state detection engine returns a vector providing for each sound class an index of confidence of the presence of the class, the index of confidence can be compared to a threshold to determine if the class is present or absent (e.g the class can be considered as present if an index of confidence of presence is higher than a predefined threshold, and absent otherwise) ;

[0168] If the state detection engine returns a vector providing for each sound class a ratio that indicates what would be the ratio of labels representative of presence or absence of the state class among a plurality of labels received for the audio frame, the ratio can be compared to a threshold to determine if the class is present or absent. For example, if a ratio of 50% is applied, a class can be considered as present if the state detection engine predicts that more that 50% of the labelers would label the class as present.

[0169]

[0154] The method P3 thus allows to detect, based on the audio frames received from one or more microphone in an environment, classes representative of the state of the environment. As one or more class is representative of an alert state, the method P3 allows raising an alerts based upon measurements that come only from a microphone.

[0170]

[0155] Thus, when said one or more state detection supervised machine learning engine detect a class representative of an alert state, the method P3 can raise the corresponding alert.

[0171]

[0156] An alert can raise in many different ways depending upon the environment and the alert type.

[0172]

[0157] For example:

[0173] If the environment is the room of a resident of a retirement home, alerts may for example indicate that the resident is in danger or require assistance. Such alert may for example indicate that the resident has fallen, has left the room alone, is repeatedly screaming, etc. In such case, the alert can be sent to the personal of the retirement home. For example, in the context of figure 1 , the alert may be sent to the one or more user device UDev1.3 that is configured to render the alert to a user (which may be the personal of the retirement home). The rendering of the alert may for example comprise displaying the alert on a screen of the one or more user device UDev1 .3, playing an alert sound on the loudspeakers, etc. The alert may be accompanied by indications relative to the room or resident (for example a room number, resident name, etc. that is concerned by the alert ; if the environment is a hospital room, alerts may for example indicate that the patient is in danger or require assistance. Such alerts may for example comprise classes representative of repeated screaming, severe coughs, falls, etc. Such alerts can be sent to the personal of the hospital in order to provide immediately assistance to the patient ; if the environment is a portion of a street (e.g a microphone is placed in a specific location in a street), alerts may for example indicate that there is a fire, an aggression, a car accident, etc. Depending upon the alert type, it can be dispatched to the relevant service in order to perform a quick intervention. For example, a fire alert can be sent to the firemen, an aggression alert to the police, a car accident alert to the ambulance, etc. if the environment is a person (e.g a person is wearing a microphone), alerts may for example indicate that the person is attacked, has fallen, is facing a health crisis. Here again, the alert can be dispatch to the relevant service (e.g police, ambulance, etc.) depending upon the alert type.

[0174]

[0158] In cases where the environment is fixed (room, playground, portion of street, etc.), it is sufficient to know the environment that is concerned by the alert and the alert type to perform the right intervention. In the case of mobile environments (a person, vehicle, portion of a vehicle, etc.), it is also necessary to determine the current position of the environment, that may for example be sent by the microphone, for example integrated in the audio frame, or through another channel.

[0175]

[0159] It is now referred to figure 4.

[0176]

[0160] Figure 4 represents a computer-implemented training method P4 according to an embodiment.

[0177]

[0161] The method P4 aims at training the sound detection supervised machine learning engines (such as SndMLEng2.1 or SndMLEng2.2), and the state detection supervised machine learning engines (such as StMLEng2.1 or StMLEng2.2). The method P4 can for example be implemented by the computing system Sys2.

[0178]

[0162] It is first worth noting that the trained sound detection machine learning engine, and state detection machine learning engine may be trained for one or more environment. For example: a sound detection machine learning engine and a state detection machine learning engine may be specifically trained for a given environment (e.g a specific room, car, person, portion of street, etc.) ; a sound detection machine learning engine and a state detection machine learning engine may be specifically trained for a given group of environments. Such groups of environments may either relate to environments that belong to a same structure (for example all the rooms of a given retirement home, all the rooms of a given hospital etc) ; a sound detection machine learning engine and a state detection machine learning engine may be specifically trained for a given type of environments. A type of environments may refer to environments that share some degrees of similarity. For example, rooms of retirement homes in general, rooms of hospital in general, rooms that are near a street, etc.

[0179]

[0163] The computer-implemented method P4 comprises a first step S4.1 of receiving a plurality of audio frames (for example Snd1 .1 and / or Snd1 .2) from one or more microphone (for example Mid .1 and / or Mid .2) located in one or more environment (for example Env1 .1 and / or Env1 .2).

[0180]

[0164] The microphones from which the audio frames are received may depend upon the environments for which the training is performed. For example: if the training is performed for a single environment (for example Env1 .1), only audio frames from the microphones in that environment may be received (for example Mid .1) ; if the training is performed for a group of environments, only audio frames from the microphones in that group of environment may be received ; if the training is performed for a type of environments, only audio frames from microphones in environments belonging to this type may be received.

[0181]

[0165] Then, the steps S4.2 to S4.5 are executed for each of audio frame of the plurality of received audio frames.

[0182]

[0166] The step S4.2 consists in causing one or more loudspeaker (such as the loudspeakers Ldspk2 for example) of one or more Man Machine Interface (such as the Man Machine Interface MMI2 for example) to play the audio frame.

[0183]

[0167] The step S4.3 consists in obtaining one or more sound label belonging to a set of sound classes.

[0184]

[0168] For example, the one or more sound label may be received through the one or more Man Machine Interface. In this case, the one or more sound label can be provided by one or more labeler after having listened to the audio frame. For example, the one or more sound label may be received may be received through an interface such as the interface shown in figure 7.

[0185]

[0169] Alternatively, the sound labels may be provided at step S4.3 by a pre-trained model. For example, such a model may be a specialized model trained to detect a specific sound class, a a general model that has been previously trained. For example, the sound labels may be provided by a very large model to train a lightweight model. Thus, accurate labels may be provided forthe training, but the model that runs in inference phase has a low resource consumption.

[0186]

[0170] The step S4.4 consists in receiving, through said one or more Man Machine Interface, one or more state label belonging to a set of state classes.

[0187]

[0171] In substance, the steps S4.2 to S4.4 thus allow one or more user of system such as the system Sys2 for example to label the training examples, by listening the audio frames and associating the audio frames to sound and state classes. For the sake of simplicity, the figure 4 will be discussed with reference to “a user” that performs the labelling. However, the method is also applicable to a plurality of user that perform the labelling, for example a plurality of users that label different audio frames. The user of the labeling system may be called “labeler” in the subsequent parts of the description.

[0188]

[0172] Of course, the sound and even more state classes may be dependent on the environment. For example, the state classes may be different if the environment is a room of a retirement home or a child playground.

[0189]

[0173] The labelling at steps S4.3 and S4.4 may be performed in different ways. For example, the user may click of buttons representative of each class that he / she has detected, type the noun of the classes, etc. The user may label the sound, then the state classes, or may label the sound and state classes simultaneously, by providing at the same time for a given audio frame both sound and state labels. The figures 7 and 8 provide examples of interfaces for labelling the audio frames in an embodiment of the invention. As explained above, at step S4.3 the labels may also be obtained by applying a pre-trained model on the audio frames, rather than through manual labeling;

[0174] It is also worth noting that, if the sound labels are provided for each audio frame independently frame the other ones, the state labels may be provided on a continuous basis, where the labeler indicates when a given state is detected upon continuously playing the successive audio frame as explained for example with reference to figure 8.

[0190]

[0175] The step S4.5 consist in enriching one or more sound detection training dataset (for example the training datasets SndDS2.1 or SndDS2.2) with the audio frame associated to said one or more sound label.

[0191]

[0176] The enriching consists in adding a training example comprising the audio frame and the associated one or more sound label. Thus the one or more sound detection training dataset can be enriched with new training examples thanks to the labels provided by the labeler or pretrained model. The enriching can be performed in different ways.

[0192]

[0177] In a number of embodiments, a single label is received for each audio frame and sound class. Stated otherwise, the audio frame is labeled by a single labeler or pseudo-labeled by a single pre-trained model. In such case, the training sample will simply associate the audio frame to the received label (indication or presence or absence) of each class.

[0193]

[0178] In other embodiments, a plurality of labels or pseudo-labels are received, from different labelers and / or pre-trained models, for each audio frame and sound class. In such case, the enriching comprises adding the label received, and, when all the expected labels are received, combining them into a single label.

[0194]

[0179] For example, a plurality of labelers, pre-trained models or a combination thereof may label or pseudo-label each audio frame. In such case, a training sample may be provided in different ways. For example:

[0195] If the sound detection engine is a classifier, the plurality of labels may be combined into a single label of presence of absence of the classes. For example, a ratio of labels of presence can be calculated and compared to a threshold. For example, if a threshold of 50% is used, and 5 labelers label a frame, a sound class can be considered as present if at least 3 labelers among the 5 (60% > 50%) indicate that the sound class is present ;

[0196] If the sound detection engine performs a regression task, a ratio is calculated to indicate the ratio or percentage of labelers that considered the class as present or absent. For example, if 3 out of 5 labelers indicated that a given sound class is present, a ratio of 0.6 or 60% can be associated to the class.

[0197]

[0180] After step S4.5, in an embodiment a step S4.6 consists in verifying if a subsequent received audio frame to label exists. If yes, the steps S4.2 to S4.5 are repeated for the subsequent audio frame to label. Otherwise, if all the received audio frames have already been processed, the sound detection training dataset has been completely enriched, and the step S4.7 is executed.

[0198]

[0181] The step S4.7 consists in using the one or more sound detection training dataset to train one or more sound detection supervised machine learning engine (for example SndMLEng2.1 or SndMLEng2.2) to detect one or more sounds classes from audio frames. Thus, the one or more sound detection supervised machine learning engine can be trained to infer, from the audio frames, the sound labels as defined in the training samples.

[0199]

[0182] In particular, in cases where a plurality of sound detection training dataset are enriched, each sound detection training dataset can be used to train a different sound detection supervised machine learning engine. For example, the sound training dataset SndDS2.1 can be used to train the sound detection supervised machine learning engine SndMLEng2.1 , and the sound training dataset SndDS2.2 can be used to train the sound detection supervised machine learning engine SndMLEng2.2.

[0200]

[0183] The method P4 further comprises, for each audio frame of said plurality of audio frames, a step S4.8 of enriching one or more state detection training dataset (such as for example StDS1 .2 or StDS2.2) with a state detection training sample associating said one or more state label to one or more training sound classes.

[0201]

[0184] Thus, the state detection training samples associate one or more training sound classes to one or more state labels, in order to allow training a state detection supervised machine learning to detect the states of the environment based solely on the previously detected sound classes.

[0202]

[0185] As discussed above, in a state detection training sample the one or more training sound classes may be provided in the same format as the output of the sound detection engines. Thus, the one or more training sound classes may correspond to indications of absence / presence of the classes, ratios of predictions of absence / presence of the classes, etc.

[0203]

[0186] In a number of embodiments, a single label is received for each audio frame and state class. Stated otherwise, the audio frame can be labeled by a single labeler or pseudo-labeled by a single pre-trained model. In such case, the training sample will simply associate the one or more training sound classes to the received label (indication or presence or absence) of each class. A labeler may provide state labels based on listening on the audio frame and / or other sources such as video, alerts that are raised in equipments, etc. Alternatively or in addition, certain state labels may be provided automatically according to the detection of certain events, such as for example a machine failure detected by a failure message sent by the machine, or the reception of a message by a technician.

[0204]

[0187] In other embodiments, a plurality of state labels are received, from different labelers and / or pre-trained models, for each audio frame and state class. In such case, the enriching comprises adding the label received, and, when all the expected labels are received, combining them into a single label.

[0205]

[0188] For example, a plurality of labelers, pre-trained models or a combination thereof may label or pseudo-label each audio frame. In such case, a training sample may be provided in different ways. For example:

[0206] If the state detection engine is a classifier, the plurality of state labels may be combined into a single label of presence of absence of the classes. For example, a ratio of labels of presence can be calculated and compared to a threshold. For example, if a threshold of 50% is used, and 5 labelers label a frame, a state class can be considered as present if at least 3 labelers among the 5 (60% > 50%) indicate that the sound class is present ;

[0207] If the state detection engine performs a regression task, a ratio is calculated to indicate the ratio or percentage of labelers that considered the class as present or absent. For example, if 3 out of 5 labelers indicated that a given sound class is present, a ratio of 0.6 or 60% can be associated to the class.

[0208]

[0189] In a number of embodiments, data augmentation techniques may be applied to the sound and / or state training samples in orderto increase the amount of training samples based upon a same amount of labels provided by the labelers. For example, additional training samples may be formed from an initial training sample by accelerating / decelerating audio signals, modifying the pitch of the audio signal, mixup the audio signal, performing a circular shift, etc. Thus, additional audio frames can be created, that can be associated with the same sound and state labels than the initial training sample in order to create additional sound and / or state training samples without requiring any additional labelling burden.

[0209]

[0190] State labels may also be, at least in part, obtained from other sources than a labeler. For example, in the labels representative from some or all the alert classes represented in figures 11 A and 11 B, that relate to cleanliness alerts, may be obtained from external sources. For example, a state label “equipment malfunction” may be assigned when an operator of the equipment, or an equipment itself, raises an alert. Certain states may also be detected automatically, such as for example the state “team message” that may be detected automatically when a message is emitted / received.

[0210]

[0191] State labels in mental health environments may also, at least in part, originate from sources other than manual annotation. For instance, labels corresponding to clinical alerts — such as “agitation episode”, “verbal aggression”, or “risk of self-harm” — may be derived from external sources like caregiver reports, digital nursing logs, or alerts triggered by safety monitoring systems. Additionally, some states can be inferred automatically, such as a “night-time wandering” label derived from a sequence of detected steps and door interactions during restricted hours. This hybrid labelling approach strengthens the reliability of the behavioral monitoring framework.

[0211]

[0192] After step S4.8, in an embodiment a step S4.9 consists in verifying if a subsequent received audio frame to label exists. If yes, the step S4.8 is repeated for the subsequent audio frame to label. Otherwise, if all the received audio frames have already been processed, the sound detection training dataset has been completely enriched, and the step S4.7 is executed.

[0212]

[0193] The training sound classes of the state detection training samples may be obtained in different ways.

[0213]

[0194] In an embodiment, the one or more training sound classes are the one or more sound label provided by the user. Stated otherwise, for each received audio frame: the user or a pre-trained model provides one or more sound labels at step S4.3 ; the user provides one or more state labels at step S4.4 ; a sound detection training sample is formed at step S4.5, that associated the received audio frame to the one or more sound labels provided by the user; a state detection training sample is formed at step S4.8, that associates the one or more sound labels provided by the user to the one or more state labels provided by the user.

[0214]

[0195] Thus, all the labels in the training datasets are provided by the user, so that the labels are as accurate as possible.

[0215]

[0196] In such case, the order of the steps may in an embodiment be slightly different from the order presented in figure 4. For example, all the steps S4.2, S4.3, S4.4, S4.5 and S4.8 may be performed in the same loop, before the step S4.7 of training the sound detection machine learning engine is performed.

[0216]

[0197] In another example presented in figure 5, the sound labeling at step S4.8 comes from a prediction performed by the sound detection machine learning engine.

[0217]

[0198] When the enrichment of the one or more state detection training dataset is completed, the method P4 comprises a step S4.9 of using said state detection training dataset to train one or more state detection supervised machine learning engines to detect one or more state classes based at least from the output of said one or more sound detection supervised machine learning engine.

[0218]

[0199] The step S4.9 consists in using the one or more state detection training dataset to train one or more state detection supervised machine learning engine (for example StMLEng2.1 or StMLEng2.2) to detect one or more state classes from one or more training sound classes. Thus, the one or more state detection supervised machine learning engine can be trained to infer, from an input comprising sound classes, the state labels that have been provided by the user.

[0219]

[0200] In particular, in cases where a plurality of state detection training dataset are enriched, each state detection training dataset can be used to train a different sound detection supervised machine learning engine. For example, the state training dataset StDS2.1 can be used to train the state detection supervised machine learning engine StMLEng2.1 , and the state training dataset StDS2.2 can be used to train the state detection supervised machine learning engine StMLEng2.2.

[0220]

[0201] According to various embodiments of the invention, each of the sound detection and state detection training datasets can be either completely created during by the method P4, an existing training dataset can be enriched, or a new training dataset can be derived from an existing one. The considerations below are valid for both the sound detection training datasets and the state detection training datasets.

[0221]

[0202] In some embodiments, a training dataset is formed only from the training samples created with the audio frames received at step S4.1 . Thus, a complete creation of a training dataset is performed.

[0222]

[0203] In other examples, a pre-existing training dataset is enriched with new training sample. For example: a generic dataset can be enriched with training samples from a specific environment to create a training dataset adapted to this specific environment. For example, a generic dataset of sound and states from rooms of retirement homes can be enriched with training samples from a given room of a given retirement home, or all the rooms of a retirement homes, to create a dataset specifically adapted to a given room or retirement home ; a pre-exiting training dataset associated to a given environment, group of environments or type of environments is enriched with new training samples from that environment, group of environments or type of environments. Thus, the training dataset can be expanded and / or take into account new sound or state classes.

[0223]

[0204] It is worth noting that the sizes of the initial training datasets may be different between the one or more sound detection training dataset and the one or more state detection training dataset. For example, the one or more sound detection training dataset may be a large and / or generic dataset, comprising a large number of training samples from a large number of environments, while the one or more state detection training dataset may be created during the method, or be smaller and / or more specific datasets that relate to a specific group or type of environment. For example, a sound training dataset may be a general training dataset, while a state training dataset may be a dataset specific to a single environment, a group of environments (for example the rooms of a given a retirement home), or a type of environments (for example the rooms of retirement homes). This is because sound detection is a general purpose that is applies to a large number of environments, while the state detection is specific to an environment, and the state classes are very different between the different environments.

[0224]

[0205] The training steps at steps S4.7 and S4.9 can be performed using any known method for training a supervised machine learning engine. For example, if a machine learning engines relies on an artificial neural network, the training may be performed using gradient descent methods. For example, one or more of the following below may be performed for the trainings: a plurality of iterations of optimizing parameters of the of the state and / or sound detection supervised machine learning engines using for example a gradient descent (for example a stochastic gradient descent such as described by Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980.) in order to optimize a loss function such as for example a Binary Cross Entropy “BCE” or Kullback Leibler “KL” divergence loss function. The loss function may for example be applied to a binary classification ; a plurality of optimization of hyperparameters of the state and / or sound detection supervised machine learning engines, for example a Bayesian optimization using Parzen Tree Estimators ; etc.

[0225]

[0206] As already explained, training a state detection engine based on sound classes rather than audio frames themselves allow obtaining an accurate state classification using a limited training set. Thus, the method P4 allows training an accurate state detection with a limited labelling burden. Furthermore, the training of a new or updated state detection supervised machine learning engine (for example to take into account additional state classes, or to create a state classifier for a given environment or group of environments) can be performed quickly, because a limited number of training samples is required. It is therefore possible to obtain rapidly a state detection supervised machine learning engines that is optimized for a specific environment.

[0226]

[0207] The method P4 is provided by means of non-limitative example only of a method according to an embodiment of the invention. In a number of embodiments of the invention. A training method according to the invention may slightly differ from the method that is represented in figure 4.

[0227]

[0208] For example, the steps may be performed in a different order that the order presented in figure 4. For example:

[0228] The order of the steps S4.3 and S4.4 may be inversed, or they may be performed simultaneously ;

[0229] The sound labeling and state labeling (i.e steps S4.3 and s4.4) may be performed in two different loops. For example: o a first loop may be performed by playing each audio frame independently of the others, and receiving sound labels each audio frame independently as shown in figure 7 ; o a second loop may be performed by playing a temporal succession of audio frames from a same microphone, and continuously receiving the state labels, for example by a labeler indicating in real time which are the active states, as shown for example in figure 8 ;

[0230] The step S4.8 may be included in the same loop than steps S4.3 to S4.5 ;

[0231] The trainings at step S4.7 and S4.10 may be performed simultaneously ;

[0232] - Etc.

[0233]

[0209] It is now referred to figure 5.

[0234]

[0210] Figure 5 is a second example of a computer-implemented training method P5 according to an embodiment according to an embodiment.

[0235]

[0211] The method P5 comprises all the steps of the method P4. In the method P5, the training sound classes that are provided as input in the state training samples are not the sound labels provided by the user, but predictions of sound classes performed by the training one or more sound detection supervised machine learning engine.

[0236]

[0212] To this effect, the step S4.7 of using said one or more sound detection training dataset to train said one or more sound detection supervised machine learning engine to detect one or more sounds classes from audio frames is performed prior to the step S4.8 enriching said one or more state detection training dataset.

[0213] Furthermore, once the step S4.7 is performed, for each received audio frame: the method comprises a step S5.1 of using said one or more trained sound detection supervised machine learning engine to detect one or more sound classes from said audio frame ; for the purpose of the step S4.8 of enriching the state detection training dataset with a state detection training sample, the one or more training sound classes of the state detection training sample are the one or more sound classes detected from said audio frame by the one or more sound detection supervised machine learning engine.

[0237]

[0214] Stated otherwise: one or more sound detection supervised machine learning engine is first trained using the one or more training dataset that is enriched with the labels provided by the user for each audio frame; then the trained one or more sound detection supervised machine learning engine is used to perform predictions of sound classes for each audio frame. It is worth noting that the predictions performed by the sound detection supervised machine learning engine may be slightly different from the sound labels provided by the labeler, or plurality of users and / or pre-trained models; the state detection training samples are formed by associating, for a given audio frame, the one or more state label provided by the user to the predictions of sound classes provided by the one or more sound detection supervised machine learning engine for the given audio frame.

[0238]

[0215] The accuracy of detection of states may be thus increased. Indeed, as shown in figure 3, in inference phase, the state detection supervised machine learning engine takes as input the predictions of sound classes outputted by the trained sound detection supervised machine learning engine. Thus, when these predictions rather than sound labels provided by the user are used for forming the state detection training samples during the training phase as exemplified in the method P5; the state detection supervised machine learning engine is trained on state training samples as similar to the input data in inference phase. The accuracy of the state detection is thus increased.

[0239]

[0216] It is now referred to figure 6.

[0240]

[0217] Figure 6 represents an architecture of machine learning engines to detect states of an environment in an embodiment.

[0241]

[0218] As already explained with reference to the previous figures, the invention relies mainly on two types of machine learning engines: sound detection supervised machine learning engines, and state detection supervised machine learning engines that take as input the sound classes outputted by the sound detection supervised machine learning engines.

[0242]

[0219] According to various embodiments of the invention, the detection of the state of the environments may be performed using: a single sound detection supervised machine learning engine, or a plurality of different sound detection supervised machine learning engines (in case of a plurality of different sound detection supervised machine learning engines, the different sound detection supervised machine learning engines may be different) ; a single state detection supervised machine learning engine, or a plurality of different state detection supervised machine learning engines, (in case of a plurality of different state detection supervised machine learning engines, the different state detection supervised machine learning engines may be different).

[0243]

[0220] The figure 6 represents an example using two different sound detection machine learning engines SndMLEng2.1 and SndMLEng2.2, and two different state detection supervised machine learning engines StMLEng2.1 and StMLEng2.2. The arrows in figure 6 represent the relationship between the outputs of different engines and the inputs of the subsequent ones.

[0244]

[0221] The figure 6 is however provided by means of example only, and many different combinations (single sound detection machine learning engine / single state detection machine learning engine; plurality of sound detection machine learning engines / single state detection machine learning engine; single sound detection machine learning engine / plurality of state detection machine learning engines; plurality of sound detection machine learning engines / plurality of state detection machine learning engines; a plurality may be more than two machine learning engines, etc.).

[0245]

[0222] It is worth noting that a state detection supervised machine learning engine can take as input the sound classes detected by a plurality of different sound detection supervised machine learning engines, provided that the data received as input during the inference phase is consistent with the data received as input during the training phase. For example, in figure 6, the state detection engines StMLENg2.1 and StMLENg2.2 both receive as input the sound classes detected by the sound detection engines SndMLEng2.1 and SndMLEng2.2. Thus, if the machine learning engines are trained by a method such as the method P5: the sound detection engines SndMLEng2.1 and SndMLEng2.2 can be trained at step S4.7 ; the trained sound detection engines SndMLEng2.1 and SndMLEng2.2 can be used to perform sound classes predictions at step S5.1 ; at step S4.8, the state training samples can be formed using as input the predictions of both SndMLEng2.1 and SndMLEng2.2 ; at step S4.10 the training can thus be performed based on the training dataset thus formed.

[0246]

[0223] The state detection engines StMLENg2.1 and StMLENg2.2 are thus trained to perform detection based on inputs from two (or more different) sound detection engines SndMLEng2.1 and SndMLEng2.2 instead of one. This may allow benefiting from sound detections that have different properties.

[0224] One example of different properties is the handling of an extended set of sound classes from an existing training dataset. In such an example, prior to the training phase, an initial sound training dataset (that may for convenience be called SndDS2.0) is based on an initial set of sound classes. During the training phase, an extended set of sound classes, that comprises the initial set of sound classes and additional classes is used. Stated otherwise, the user will during the training phase generate additional sound training samples by providing for each audio frames sound labels in the extended set of sound classes. Each sound label may thus belong either to one of the classes of the initial set, or one of the additional classes. Thus two different training sets can be formed: a first sound training set SndDS2.1 to train the sound detection engine SndMLEng2.1 , that comprises the initial training set SndDS2.0, as well as the additional sound training samples. The first sound training set SndDS2.1 will thus provide an accurate prediction for the initial set of classes, because it benefits from a large number of previous training samples, but may provide a less specific prediction of the additional sound classes, because the number of training samples where the additional sound classes have been taken into account is low compared to the size of the training set ; a second sound training set SndDS2.2 to train the sound detection engine SndMLEng2.2, that comprises only the additional sound training samples. The first sound training set SndDS2.1 will thus provide less accurate prediction for the initial set of classes, because the size of the training set will be much lower than SndDS2.1 , but will provide a much better prediction for the additional sound classes, because they have be taken into account for all the training samples of SndMLEng2.2.

[0247]

[0225] In this example the first sound detection engine SndMLEng2.1 and the second sound detection engine SndMLEng2.2 will thus be complementary, the former being more accurate for the initial classes and the latter being more accurate for the additional classes.

[0248]

[0226] During the training phase, the state detection engines will also automatically learn to take the best benefits from the outputs from the first sound detection engine SndMLEng2.1 and the second sound detection engine SndMLEng2.2. It is for example in certain cases possible that certain state classes will be better predicted by the predictions from SndMLEng2.1 and other classes by the predictions from SndMLEng2.2.

[0249]

[0227] This example demonstrates one advantage for the state detection engines for taking as input the predictions of a plurality of different sound prediction engines. Of course, it is provided by means of example only, and other cases may be considered, such as for example the use of a single sound detection engine, or the use of a plurality of different sound detection machine engines that may have other distinct properties.

[0250]

[0228] The sound detection engine SndMLEng2.1 and the sound detection engine SndMLEng2.2 may be of different types. For example they may be Convolutional Neural Network (CNNs) such as for example one of the versions of the “MobilNet” Convolutional Neural Network described by: Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., ... & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861. ;

[0251] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L. C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4510-4520).. ;

[0252] Howard, A., Sandler, M., Chu, G., Chen, L. C., Chen, B., Tan, M., ... & Adam, H. (2019). Searching for mobilenetv3. In Proceedings of the IEEE / CVF international conference on computer vision (pp. 1314-1324);

[0253]

[0229] According to various embodiments of the invention, an audio frame may be input to the sound detection engine in different ways: the waveform (e.g succession of values of audio samples) can be directly fed to the sound detection engine, for example if the sound detection engine is based on a CNN in 1 D (e.g a CNN that takes as input data in 1 dimension such as a 1 -column or 1-row vector, that may be a waveform for example). This solution preserves the whole content of the audio frame and may provide precise results, especially when a large training dataset has been built ; a spectrogram of the audio frame may be provided as input to the sound detection engine. For example, a 2D-image of the spectrogram of the frame may be constructed, and provided as input to the sound detection engine, for example if the sound detection engine is 2D CNN (e.g a CNN that takes as input data in 2 dimensions such as an image for example). This solution may provide satisfactory results, even when only a small training dataset can be used.

[0254]

[0230] More generally, a sound detection engine may take as input the audio frame itself, or an input derived from the audio frame, such as for example a frequency transform of the audio frame, or a spectrogram of the audio frame. The type of the input may also depend upon the inputs that are accepted by the sound detection engine, especially in terms of number of dimensions.

[0255]

[0231] In the example of the figure 6, two branches of detection are considered:

[0256] A first branch comprises the state detection engine StMLEng2.1 that takes as input the sound predictions of SndMLEng2.1 and SndMLEng2.2 over a plurality of successive time frames ,in order to perform a state detection over a time window that is longer than a single audio frame ;

[0257] In a second branch, the sound detection is enriched with speech to text transcription of the audio frames. To this effect, the sound predictions of SndMLEng2.1 and SndMLEng2.2 are successively provided to the processing engines (or modules) GrBstMLEng6, SpRecEng6 and StMLEng2.2 that will be described hereinafter.

[0258]

[0232] Thus, the use of the two branches allows benefiting from both two predictions that benefit respectively from a long-term perspective, and an analysis of the speech uttered. Of course, these two branches are provided by means of non limitative examples of different prediction branches, and many other different prediction branches may be used according to various embodiments.

[0259]

[0233] In the first branch, successive audio frames are received from the microphones, and processed by the sound prediction engines SndMLEng2.1 and SndMLEng2.2, then the state prediction engine StMLEng2.1 takes into account detections of sound classes from SndMLEng2.1 and SndMLEng2.2 from a plurality of successive audio frames.

[0260]

[0234] This allows inferring the states of the environment according to a plurality of successive sounds in a time longer than the duration of audio frames. For example, if the audio frame has a fixed duration of 10 seconds, using as input the predictions of sound classes for a plurality of successive audio frames may allow predicting the states of an environment in a time window of few minutes or more. As already discussed, training dataset required for an accurate prediction is much lower than if one tried to directly predict the states from audio frames of few minutes.

[0261]

[0235] Taking into account the prediction of sounds for a plurality of audio frames also allows detecting state classes that exhibit a much higher level of information than a single sound. For example, a series of sound detections where a plurality of screams is detected can allow detecting a state “repeated screaming” that triggers an alert, while a single scream may not be considered as an alert

[0262]

[0236] In another example, the detection of a sound “door slam” followed by no sound in the context of a retirement home may cause the detection of a state indicating that the resident has left his room alone, which can be a dangerous situation raising an alert, while the detection of a sound “door slam” followed by detection of speeches may not trigger an alert, because it means that the resident is still in his room.

[0263]

[0237] The state prediction engine StMLEng2.1 may belong to different types of machine learning engines and takes into account a plurality of audio frames in different ways.

[0264]

[0238] In a number of embodiments, the sound classes predictions for a plurality of audio frames are provided as input to the state prediction engine StMLEng2.1 , so that the prediction can be performed while taking into account a plurality of successive audio frames. For example, in order to predict a state at time step t, the sound predictions at time step t but also t-1 , t-2, etc. may be fed as input to StMLEng2.1 . In such examples, StMLEng2.1 may belong to different types of classifiers such as for example artificial neural networks.

[0265]

[0239] Another option is to consider for StMLEng2.1 a state prediction engine that maintains an internal (or hidden) state. Thus, for the prediction of a state at time step t, only the sound predictions at time step t needs to be fed as input to StMLEng2.1 , but the state prediction will also take into account the hidden state that depends upon the sound predictions at the previous time steps. For example, StMLEng2.1 may be a recurrent neural network, such as for example a Long Short Term Memory (LSTM) neural network.

[0266]

[0240] A recurrent neural network is an efficient solution for detecting the states over successive temporal samples, because it maintains a hidden state of the environment, that is updated with new audio frames. Such a hidden state of the environment allows disambiguating the sound predictions from the new audio frames. For example, if the hidden state indicates that TV is playing, the detection of repeated screams may not cause the detection of a “repeated scream” class, because the screams may be caused by the TV playing. Such association may have been learned during the training of StMLEng2.1 based upon the state labels provided by the labeler. The hidden state may have different sizes. For example, in a LSTM the hidden state may be a vector having a length comprised between 20 and 50.

[0267]

[0241] As explained with reference to figure 1 , in some embodiments an audio frame at a time frame is sent by said one or more microphone only upon a condition relative to a sound level from said audio frame (an audio frame being for example sent only when the sound has a sufficiently high amplitude).

[0268]

[0242] In such cases, the state detection supervised machine learning engine StMLEng2.1 further takes as input indications of absence of reception of audio frames. Stated otherwise, at each time step, the state detection supervised machine learning engine StMLEng2.1 takes as input either the sound classes predictions of the sound prediction machine learning engines, or an indication that no audio frame has been sent.

[0269]

[0243] Thus, the bandwidth can be saved by limiting the amount of audio frames to send. Meanwhile, the state detection machine learning engines can accurately take into account the existence of silences between the audio frames that have been sent in a time window.

[0270]

[0244] Of course, the indication that no audio frame has been sent can be used as input of the state detection supervised machine learning engine StMLEng2.1 during the inference phase, but also during the training phase, so that the supervised machine learning engine StMLEng2.1 can be trained during the training phase to associate sequences of audio frames and silences to certain states.

[0271]

[0245] As already explained, the sound detection engines SndMLEng2.1 and SndMLEng2.2 may provide different outputs, for example: a vector where each element is representative of the detection, or lack of detection, of a given sound class (each element being for example a Boolean value, an index of prediction confidence, etc.); a vector where each element is representative a ratio of labels indicating the presence of given sound class among the labels or pseudo-labels from a plurality of labelers and / or pretrained models.

[0272]

[0246] The vectors of sound classes may then be fed as input to the state supervised machine learning engine StMLEng2.1. Of course, the invention is not restricted to a specific form of output from the sound learning engines, and other outputs than vectors may be considered. When the output formats of the sound detection engines are different from the input format of the state detection engine StMLEng2.1 , they may be transtyped to match the input format of StMLEng2.1 . For example, if a sound machine learning engine outputs a vector of ratios of labels indicating the presence of sound classes, and a state learning engine expects as input a vector of binary values of presence or absence of classes, a threshold can be applied to the ratios to obtain a binary value (e.g if more than a predefined ratio of labelers would predict the presence of a given sound classes, the given sound class is predicted as present).

[0273]

[0247] As already explained above, The state detection engine StMLEng2.1 may take into account the temporal succession of sound classes detection in different ways. For example series of vectors corresponding to different timestamps may be provided as input to the state detection engine StMLEng2.1. When the state detection engine StMLEng2.1 is a recurrent neural network, it is sufficient to provide the sound classes detection corresponding to a single audio frame as input to StMLEng2.1 , because the succession of the different audio frames is taken into account by the hidden state of the recurrent neural network.

[0274]

[0248] Referring now to the second branch represented in figure 6, the second branch is formed of a gradient boosting machine learning engine GrBstMLEng6 ; a speech recognition engine SpRecEng6 ; the state detection engine StMLEng2.2.

[0275]

[0249] In the second branch, the audio frames are processed independently. Thus, each received audio frame causes the execution of the second branch.

[0276]

[0250] The sound classes detected by the sound detection engines SndMLEng2.1 and SndMLEng2.2 are first provided for each audio frame to the gradient boosting machine learning engine GrBstMLEng6 that is trained to detect if the sound classes are representative of an alert.

[0277]

[0251] The gradient boosting machine learning engine GrBstMLEng6 may typically have been trained by an additional training of the method P4 and P5 to detect if the sound classes detected by the sound detection engines SndMLEng2.1 and SndMLEng2.2 are likely associated with an alert state provided by the user.

[0278]

[0252] The gradient boosting machine learning engine GrBstMLEng6 provide a focused detection of the alert classes. For example, the gradient boosting machine learning engine GrBstMLEng6 may be trained using a training dataset where each training samples associate, for a single audio frame, sound classes (either sound labels obtained at step S4.3, or predictions from the trained sound detection engines) to state labels received at step S4.4. In order to better focus on the alert classes, only certain state labels, for example the state labels corresponding to alert classes, could be taken into account.

[0279]

[0253] Thus, during the inference phase, if the gradient boosting machine learning engine GrBstMLEng6 does not detect a possible alert state, the execution of the second branch is stopped, because the detection of an alert is unlikely. Thus, a significant reduction of the complexity of detection and power consumption can be achieved while preserving a good accuracy in the detection of alerts.

[0254] Conversely, if the gradient boosting machine learning engine GrBstMLEng6 detects a possible alert state ,the speech recognition engine SpRecEng6 and state detection engine StMLEng2.2 are executed.

[0280]

[0255] The use of the gradient boosting machine learning engine GrBstMLEng6 is used only in certain embodiments of the invention. In other embodiments not represented in figure 6, the speech recognition engine SpRecEng6 and state detection engine StMLEng2.2 are executed for each received audio frame. Other means of detecting the likelihood of the detection of an alert class may also be considered.

[0281]

[0256] The speech recognition engine SpRecEng6 is configured to detect words uttered in an audio frame. Any suitable Automatic Speech Recognition “ASR” technique can be used by the speech recognition engine SpRecEng6. Some keywords such as for example “help” can be extracted from the speech recognition. Then, one or more indicator of detection of words belonging to a predefined set of words can be built. Typically, the predefined set of words may comprise words that are known to be correlated with an alert. For example, the predefined set of words may comprise words such as “help”, “ouch”, etc. Some words that are not by themselves indicative of an alert, but have a sonority close to an alert words and may be detected instead can also be present in the predefined set of words. For example, the word “kelp” may be present, because the speech recognition engine SpRecEng6 may sometimes interpret the word “help” as “kelp”. The set of words whose detection is to be assessed can thus be predefined by one or more user to comprise words that are known to provide an indication of alert.

[0282]

[0257] The one or more indicator of detection of words may typically be Boolean values indicating if a given words has been detected or not. Alternatively, the one or more indicator of detection of words may be an alert score can be defined according to the detected word. For example, the word “help” may be associated to a high alert score, while the words “kelp” may be associated to a lower alert score, because there exist an uncertainty whether the word “help” or “kelp” was actually uttered.

[0283]

[0258] Then, the state detection supervised machine learning StMLEng2.2 takes as input for each frame the sound classes outputted by SndMLEng2.1 and SndMLEng2.2 in combination to the indicators of detection of words belonging to the predefined set of words. For example, the state detection supervised machine learning StMLEng2.2 may take as input a vector wherein: certain elements represent the detection of a sound class, for example through a binary value, an index of confidence of the detection, or a ratio ; certain elements represent the detection of a given word by the speech recognition engine SpRecEng6, for example though a binary value for each word to detect, or an alert score.

[0284]

[0259] The state detection supervised machine learning StMLEng2.2 can thus be trained to predict the presence of state classes, in particular state classes representative of an alert, through a correlation between the indicators of presence of words and the detected sound class. These combined inputs provide a valuable input for detecting the state classes, in particular alert classes.

[0260] By means of example, the keyword “help” may in general cause an alert, except when certain sound classes such as for example “TV playing” are detected. Indeed, when TV is playing, the word “help” will likely come from the TV. Stated otherwise, the state detection supervised machine learning StMLEng2.2 can be trained during the training phase to discover how detected sounds and words can be correlated to perform an efficient state / alert detection.

[0285]

[0261] To this effect, the state detection supervised machine learning StMLEng2.2 may belong to different types. It may for example be a random forest or a gradient boosting engine. Indeed, these types of machine learning engines are well suited to take as input heterogeneous data.

[0286]

[0262] Of course, during a training phase to train SndMLEng2.2 such as for example the method P4 and P5, the speech recognition engine SpRecEng6 may run on the training audio frames, so that the state detection supervised machine learning StMLEng2.2 can be trained based on the sound detection and speech recognition, i.e the same kind of inputs that it receives during the inference phase.

[0287]

[0263] According to various embodiments of the invention, one or more of the engines of the second branch may be executed under different conditions. Fer example:

[0288] The second branch formed of the gradient boosting machine learning engine GrBstMLEng6, speech recognition engine SpRecEng6 and state detection engine StMLEng2.2 may be executed systematically, as early as new sound classes have been detected by the sound detection engines SndMLEng2.1 and / or SndMLEng2.2 ;

[0289] The gradient boosting machine learning engine GrBstMLEng6, speech recognition engine SpRecEng6 and / or state detection engine StMLEng2.2 may on the contrary, in certain embodiments, be executed only when certain conditions are fulfilled, for example under the detection, or conversely absence of detection, of certain classes.

[0290]

[0264] For example, certain of the sound classes output by the sound detection engines SndMLEng2.1 and SndMLEng2.2 may be representative of a conversation between a plurality of persons. This is for example the case of the classes “ Resident / AS conversation” , “Storage AS / AS”, “ Resident / resident conversation” and “Telephone conversation” represented in figure 9. In such case, the classes and audio frames may be sent to the speech recognition engine SpRecEng6 only if a class that is not representative of a conversation between a plurality of person is detected. Taking again the example of figure 9, the sound classes and associated audio frames may be sent to the speech recognition engine SpRecEng6 (and more generally the second branch) for all classes except the classes “Resident / AS conversation” , “Storage AS / AS”, “Resident / resident conversation” and “Telephone conversation” that indicate that a conversation between a plurality of person occur.

[0291]

[0265] In more details, this distinction can be implemented as follows:

[0292] In an embodiment: o if a sound class detected by the sound detection engines SndMLEng2.1 and / or SndMLEng2.2 is not representative of a conversation between a plurality of persons, the second branch formed by the boosting machine learning engine GrBstMLEng6, speech recognition engine SpRecEng6 and state detection engine StMLEng2.2 is executed based on the detected sound class and associated audio frame ; o conversely, if the sound class detected by the sound detection engines SndMLEng2.1 and / or SndMLEng2.2 is representative of a conversation between a plurality of persons, the second branch is not executed ;

[0293] In another embodiment, the boosting machine learning engine GrBstMLEng6 is executed and : o if a class detected by the boosting machine learning engine GrBstMLEng6 is not representative of a conversation between a plurality of persons, the rest of the second branch formed by the speech recognition engine SpRecEng6 and state detection engine StMLEng2.2 is executed based upon the detected sound class and associated audio frame o conversely, if the class detected by the boosting machine learning engine GrBstMLEng6 is representative of a conversation between a plurality of persons, the rest of second branch is not executed

[0294]

[0266] Thus, in both cases the speech recognition engine SpRecEng6 is not executed if a sound class (detected either by the sound detection engine SndMLEng2.1 , sound detection engine SndMLEng2.2 or boosting machine learning engine GrBstMLEng6) is representative of a conversation between a plurality of persons.

[0295]

[0267] This therefore maintains the privacy of conversations, because the speech associated to the conversations is not detected. Thus, speech detection is only performed for detecting the speech uttered by a resident alone to detect calls to help, without exposing speeches uttered within private conversation. In certain embodiment the speech recognition engine SpRecEng6 is implemented by a third party service provider. In such case, the non-execution of the speech recognition engine SpRecEng6 for classes representative of conversation further allows not to send audio frames that contain conversation to a third party device, which further promotes the privacy of conversations.

[0296]

[0268] The figure 6 demonstrates an example where the states are detected from sound using two different branches, one that considers a temporal succession of sound detections, and one that takes into account the words that are uttered in the audio frame. These examples are however provided by means of non limitative example only of the detection of states based on the detected sounds. In other embodiments of the invention, only one of the two branches, or yet other ways of detecting states from the detected sounds, may be implemented.

[0297]

[0269] It is now referred to figure 7.

[0298]

[0270] Figure 7 is an example of a graphical interface Grlnt7 to allow a labeler to associate sound labels to a sound frame according to an embodiment.

[0271] The graphical interface Grlnt7 is an example of graphical interface that could be used be the system Sys2 to allow a labeler to enrich a sound detection training dataset with sound detection training samples associating sound labels to audio frame.

[0299]

[0272] To this effect, the labeler can use the button PI7 to play an audio frame, Snd7 in the example of figure 7. Then, the labeler can click to buttons SndLbls representative of sound classes. In the example of figure 7, the labeler can add labels of classes such has “schock”, “alarm”, “squeak”, etc. Some sound classes may comprise sub-classes. For example, the class “Vocalization” has in this example 9 sub-classes. When the labeler clicks on the button “Vocalization”, a window SndLblVoc appears, which comprises the sub-classes. The labeler can then provide labels of more precise vocalizations, such as for example “sing”, “laugh”, “shout”, etc. When the labeler has finished to provide the labels, he can click on the button Val7 to validate. The labeler can also click on the button Conf7 to indicate that he is confident with the labeling, so that more weight can be given to this training sample.

[0300]

[0273] When the labeler validates the labeling, a sound training sample can be formed comprising:

[0301] The audio frame Snd7 ;

[0302] The labels provided by the user.

[0303]

[0274] The labels may be typically provided in the expected output format of the sound detection supervised machine learning engine to train. For example, if the sound detection supervised machine learning engine to train outputs a vector indicating the classes that are detected from an audio frame, such a vector can be formed with the labels provided by the user. For example, if the vector comprises Boolean values, a Boolean values indicating that a sound class is present is added for all the labels that have been entered by the labeler, and a Boolean values indicating that a sound class is absent otherwise. If the vector comprises confidence level of the presence of a sound class, a high confidence value can be added for the classes that have been labeled by the labeler (and even a higher or maximal confidence value if the labeler has indicated through the button Conf7 that he is confident in the labels), and a low or minimal value otherwise. If no sound has been identified be the labeler, each element of the vector will comprise a label representative of the absence of corresponding class. The sound labels for an audio frame could be provided by a single labeler or be a combination of labels provided by a plurality of labelers for a given audio frame.

[0304]

[0275] The figure 7 is provided by means of example only of a graphical interface to provide sound labels according to an embodiment. Other interfaces could be foreseen, provided that they allow a labeler to listen to an audio frame and provide the labels corresponding to the sound classes that have been heard.

[0305]

[0276] The figure 7 allows labelling training samples into some of the sound classes represented in figure 9. The sound classes represented in figure 9 are focused on the activity of residents of retirement homes. This interface is however provided by means of example only, and similar interfaces may allow labelling training samples into other sets of sound classes, possibly for other purposes. For example, a similar interface may allow labelling training samples into the set of classes represented in figures 10A to 10F, which is focused on cleaning sounds, or the set of classes represented into figures 12A, 12B and 12C, that relate to classifications into classes representative of mental healthcare.

[0306]

[0277] It is now referred to figure 8.

[0307]

[0278] Figure 8 is an example of a graphical interface Grlnt8 to allow a labeler to associate state labels to sound frames according to an embodiment.

[0308]

[0279] The graphical interface Grlnt8 is an example of graphical interface that could be used be the system Sys2 to allow a labeler to enrich a state detection training dataset with state detection training samples associating state labels to audio frames.

[0309]

[0280] To this effect, the labeler can use the button PI8 to play a plurality of successive audio frames, Snd8 in the example of figure 8. The plurality of audio frames may also include silences if some audio frames have not been transmitted. The plurality of audio frames thus forms a continuous recording of the sound in an environment.

[0310]

[0281] Then, the labeler can click to buttons StLbls representative of state classes. In the example of figure 8, the environment is a room of a retirement home. The states thus represent the state of the room, for example the actions that are performed or the activity of the resident of the home. The labeler can for example add labels of state classes such has “left accompanied”, “has just returned accompanied”, “moves / stand”, “happy”, “stress”, “call for help”, “is on ground”, “has fallen”, “respiratory distress”, etc.

[0311]

[0282] The labeler, while listening to the audio frames, activates a state by clicking on the button corresponding to the state. Once he considers that the state has ended, he can click again on the button to deactivate the state. For example, during the play of the audio frames, the labeler can indicate, by clicking a first then a second time on the button “call for help”, a time window at which the resident of the room was calling for help.

[0312]

[0283] A continuous set of states St8 can thus be defined for the audio frames, each state being active during a time window indicated by the labeler. It is possible that at certain times two states are active, or conversely no state is active. When the labeler has finished to provide the labels, he can click on the button Val8 to validate.

[0313]

[0284] When the labeler validates the labeling, a plurality of state training sample can be formed comprising, for each audio frame of Snd8 (or indication that no audio frame was received / sent):

[0314] The audio frame (or indication that no audio frame was received / sent);

[0315] The states provided by the user during the duration of playing of the audio frame. For example, the audio frame during the time window Win8 can be associated to the states “speech” and “snoring”.

[0316]

[0285] The state labels may be typically provided in the expected output format of the state detection supervised machine learning engine to train. For example, if the state detection supervised machine learning engine to train outputs a vector indicating the state classes that are detected from an audio frame, such a vector can be formed with the labels provided by the user. For example, if the vector comprises Boolean values, a Boolean values indicating that a state class is present is added for all the labels that have been entered by the labeler, and a Boolean values indicating that a state class is absent otherwise. If the vector comprises confidence level of the presence of a state class, a high confidence value can be added for the classes that have been determined by the labeler, and a low or minimal value otherwise. If no active state has been identified be the labeler, each element of the vector will comprise a label representative of the absence of corresponding class. As the sound labels, the state labels for an audio frame could be provided by a single labeler or be a combination of labels provided by a plurality of labelers for a given audio frame.

[0317]

[0286] In embodiments, not shown in figure 8, the graphical interface may further show the sound classes that have been detected for each audio frame, in order for the labeler to have an additional information allowing him to provide state labels. Thus, an overview comprising the sound labels for the plurality of audio frames in the represented time frames can be provided to the labeler.

[0318]

[0287] The figure 8 is provided by means of example only of a graphical interface to provide state labels according to an embodiment. Other interfaces could be foreseen, provided that they allow a labeler to listen to an audio frame and provide the labels corresponding to the sound classes that have been heard.

[0319]

[0288] The figure 8 allows a labelling training samples into state classes that are focused on the state and alerts relative to the activity of residents of retirement homes. The interface of figure 8 is however provided by means of example only, and similar interfaces may allow labelling training samples into other sets of state classes, possibly for other purposes. For example, a similar interface may allow labelling training samples into the set of state classes represented in figures 11A and 11 B, or figures 12A, 12B and 12C, which are respectively focused on state classes representative of alerts in a context of cleaning and in a context of mental health care settings.

[0320]

[0289] It is also worth noting that, in the example of figure 8, the user is able to determine state labels only by listening to the training sample. In other examples, the user may use, in addition to the sound, further inputs to perform the labelling into state classes. For example, in the example of figures 11 A and 11 B, the user may use, in addition to the audio frame, further inputs such as camera pictures to verify the cleaning quality, or alerts that have been raised by the machines or cleaners. By means of example, the state class “equipment malfunction” may be labelled according to the play of the audio frame, a camera showing the equipment, an alert raised by the equipment itself, an alert raised by a user of the equipment, etc. or a combination thereof.

[0321]

[0290] Although the figure 8 show an interface to allow a user to provide state labels, this example is not limitative. In some cases, certain state labels may be provided automatically, for example upon the detection of an event, as will be explained with reference to figures 11 A and 11 B.

[0322]

[0291] Annotation may also come from data entered by users of the place that is being cleaned. For example, some places like toilets in train stations or airport are equipped with devices that allow users to enter their satisfaction relative to the cleanness of the place. These devices may for example take the form of a device where the user enter a grade, or a satisfaction level after using toilets. For example, a plurality of buttons may indicate the relative satisfaction of the user relative to the cleanness of the place. For example a green button with happy face may represent satisfaction, a yellow button with neutral face mixed feelings, and a red button with angry face strong dissatisfaction. Such kinds of inputs may be used to label automatically events relative to cleanness. The labels may represent the instantaneous state of cleanness, as well as a dynamic state. For example a dynamic state may be represented by a sudden or gradual shift of satisfaction.

[0323]

[0292] Another example of annotation is an annotation provided by a user using a user device such as the user device UDev1 .3 upon the reception of an alert. For example, in aa use case of a retirement home, when an alert is raised on a user device UDev1 .3, the healthcare professional can move to the room which the alert relates to, then confirm the state representative of the alert, or invalidate the state representative of the alert. The healthcare professional can also enter other labels corresponding to other states. For example, if an alert relative to the class “falling person” (indicating that a person is falling in a room), the healthcare professional goes to room the tied to the alert, and can confirm or invalidate the alert. For example, if the shock heard was actually a shock in a film played on TV, the healthcare professional can invalidate the detection of the state “falling person”, and indicate instead that a state of the room should be “playing TV”. These labels can be associated to the corresponding audio frame to enrich the training databases for subsequent training phases. More generally, in an embodiment, the healthcare professionals can enter state labels using the user devices UDev1.3 when they consider that a specific state occur. This improves the training of the state detection supervised machine learning engines, because the labels submitted by the healthcare professionals in real situations are highly reliable.

[0324]

[0293] The labeling by user of the user devices where alerts are raised are also possible in other embodiments than the monitoring of retirement homes. This principle of letting the users modify the detected states, or enter states of environments when they consider appropriate can more generally be applied to all the embodiments using the applicable states.

[0325]

[0294] When an audio frame is played both in an interface such as the interface of figure 7, and an interface such as the interface of figure 8, the labeler has been able to provide both sound labels and state labels for the audio frame.

[0326]

[0295] Is now referred to figure 9.

[0327]

[0296] Figure 9 is an example of sound classes in an embodiment.

[0328]

[0297] The figure 9 represents an example of a set of sound possible classes. The set of sound classes thus correspond to the sound classes that can be detected, and the labels that can be entered in the training phase.

[0329]

[0298] Some sound classes comprise sub-classes, that are represented under the main class in figure 9. For example, the class “squeak” has 3 sub-classes “bed squeak”, “door squeak”, and “shutter squeak”. The set of classes represented in figure 9 is adapted to an environment that is a room of a retirement homes and represent various sounds that can be heard in that context. Of course, different sets of classes can be defined according to the environment that is analyzed, and the sets of sound classes would be different in other environments, for example for a playground. It is of course possible to define the most relevant set of classes for a given environment.

[0330]

[0299] It is now described some examples of the sound classes represented in figure 9, and the relevance to detect such types of sounds. The description below is provided by means of non- limitative example only of possible purposes of detecting certain of the sound classes represented in figure 9. The purposes listed below are non-limitative. As it is also apparent, at least from the examples provided in figure 9, many other sound classes could be considered within the present disclosure.

[0331]

[0300] Sighs: Sighs are involuntary respiratory events that can provide valuable insights into an individual's physical and emotional state. In vulnerable populations, such as the elderly or those with chronic conditions, sighs may indicate underlying health issues or emotional distress or discomfort.

[0332] Frequent sighing may be associated with respiratory problems or other medical conditions. Sighs can be a form of non-verbal communication for those who struggle to express themselves verbally.

[0333] For vulnerable adults who may have difficulty articulating their needs or feelings, sighs can provide valuable insights into their emotional state. By recognizing and responding to these non-verbal cues, caregivers and professionals can better support vulnerable individuals in making informed choices and expressing their needs. Also, on a pure medical point of view, it has been found that sighing patterns differ between Parkinson Disease and multiple system atrophy (MSA), a related neurodegenerative disorder. Excessive sighing during sleep, particularly in slow-wave sleep, is more characteristic of MSA than PD (see Vargas Gonzalez, E., Yang, Z., Dodet, P. etal. Increased sighing during sleep as a marker of multiple system atrophy, npj Parkinsons Dis. 10, 176 (2024). Current monitoring systems often overlook these subtle cues, potentially missing early warning signs of deteriorating health or emotional well-being.

[0334] Detecting sighs thanks to acoustic monitoring can improve the quality of life for vulnerable people.

[0335]

[0301] Aspiration events: Detecting aspiration events in vulnerable individuals, particularly the elderly, holds significant medical importance. Aspiration, often resulting from the accumulation of saliva or other substances (solid or liquid, foods or drugs, pills) in the mouth that are inhaled into the airways, can lead to violent coughing fits. While these episodes are generally benign and resolved quickly, they can become more problematic with age as the individual's ability to clear the airways diminishes. This can result in the accumulation of fluid or other substances in the lungs, increasing the risk of pulmonary infections. Additionally, the aging process affects swallowing quality, further elevating the risk of aspiration events. Repeated aspiration episodes significantly increase the likelihood of pulmonary infections, which can be life-threatening. Monitoring the frequency and severity of aspiration events in vulnerable individuals can provide valuable insights for healthcare providers. This information can aid in anticipating and mitigating the risk of pulmonary infections, as well as enhancing patient care, such as providing nursing assistance during meals to reduce the occurrence of such incidents. By detecting and addressing aspiration events early, healthcare providers can improve the overall well-being and quality of life for vulnerable patients.

[0302] Scratching (skin) : In a monitored environment, frequent skin scratching can be easily detected through acoustic monitoring, providing significant benefits for vulnerable individuals. Frequent scratching can indicate underlying dermatological conditions like eczema or psoriasis, the presence of parasites or infections, or even mental health issues such as stress, anxiety, or obsessive-compulsive disorder (OCD). Early detection allows for prompt diagnosis and treatment, preventing skin lesions, secondary infections, and scarring, which can be particularly problematic for elderly or medically fragile individuals. Caregivers receiving alerts thanks to such a system can take immediate actions, such as assessing the individual, applying topical creams, ensuring a clean environment, and providing psychosocial support to address underlying stress or anxiety. Documenting the frequency and severity of scratching episodes aids in long-term monitoring and follow-up, ensuring that persistent issues are addressed effectively. Overall, this approach enhances patient care, prevents complications, and improves the overall well-being of vulnerable individuals.

[0336]

[0303] Urination on the floor : This type of sound is specific and can be easily detected with acoustic monitoring. In specialized facilities, some individuals may experience neurological and cognitive disorders that lead them to urinate voluntarily or involuntarily on the floor. Detecting such events and alerting caregivers can be highly beneficial. Prompt notification allows for immediate cleanup, reducing the risk of falls and injuries from slippery surfaces. Additionally, quick intervention can prevent the spread of bacteria and odors, maintaining a hygienic environment. For individuals with cognitive impairments, frequent urination incidents may indicate underlying medical issues that require attention. Alerting caregivers enables timely assessment and appropriate care, ensuring the safety and well-being of the residents.

[0337]

[0304] Tracheostomy tube obstructions: these events can occur due to various factors, including the accumulation of mucus plugs, blood clots, or thick secretions within the tube. Other causes include the formation of granulomas, improper tube positioning, or pressure from the airway walls. These obstructions can rapidly lead to life-threatening respiratory compromise, making early detection crucial. In a healthcare setting, particularly where resources may be limited, detecting such obstructions is vital for patient safety. Early detection allows for prompt intervention by caregivers, potentially preventing serious complications such as respiratory distress, atelectasis, or lung abscesses. By implementing our alert system based on acoustic monitoring for healthcare providers, obstructions can be addressed quickly through appropriate measures like suctioning, tube repositioning, or replacement, thereby reducing the risk of emergencies and improving overall patient care.

[0338]

[0305] The method and system described here are not only able to accurately detect the sound classes described above, but also to provide an accurate detection of alert states that are linked to those classes, the training phase allowing to determine the most relevant correlations between the detected sounds and states.

[0339]

[0306] It is now referred to figures 10A to 10F.

[0340]

[0307] Figures 10A to 10F (10A, 10B, 10C, 10D, 10E and 10F) are an example of sound classes in an embodiment for cleaning monitoring application.

[0308] The figures 10A to 10F represent an example of a set of sound possible classes in cleaning monitoring. The set of sound classes thus correspond to the sound classes that can be detected, and the labels that can be entered in the training phase.

[0341]

[0309] The figures 10A to 10F represent a structured graph of various sounds explained as follows.

[0342]

[0310] - General Cleaning Noises

[0343] - Floor Cleaning

[0344] - Vacuum Cleaner Running: A continuous, loud humming or whirring sound.

[0345] - Mopping, drying: A repetitive, wet, slapping sound as the mop hits the floor or is being dried.

[0346] - Sweeping: A rhythmic, scratching or brushing sound.

[0347] - Floor Buffing: A loud, mechanical humming or buzzing sound.

[0348]

[0311] - Surface Cleaning

[0349] - Dusting: A soft, brushing or swishing sound.

[0350] - Wiping: A gentle, rubbing or squeaking sound.

[0351] - Scrubbing: A vigorous, scratching or rubbing sound.

[0352] - Polishing: A smooth, buffing or rubbing sound, sometimes accompanied by a low hum if a machine is used.

[0353]

[0312] - Chemical Application

[0354] - Spraying: A hissing or misting sound as liquid is sprayed from a bottle.

[0355] - Chemical Mixing: A pouring or stirring sound, sometimes with a slight fizzing if chemicals react.

[0356]

[0313] - Waste Management

[0357] - Trash Being Emptied: A rustling or crinkling sound of plastic bags, followed by a thud as the bag hits the bottom of the trash can.

[0358]

[0314] - Movement and Communication

[0359] - Moving Furniture: A scraping, dragging, or thumping sound as furniture is moved.

[0360] - Dishwashing noise: A combination of water running, dishes clinking, and scrubbing sounds.

[0361] - Eating meals : Sounds of chewing, utensils clinking against plates, and occasional conversation.

[0362] - Coughing : A sudden, sharp, and often repetitive sound.

[0363] - Other Presence Noises: Various sounds that indicate the presence of people, such as footsteps, rustling of clothes, and background conversations or other sound activities previously described

[0364] - Footsteps: A rhythmic, walking sound that varies based on the type of flooring.

[0365] - Conversations: Human voices talking, sometimes with laughter or other vocalizations, speech with specific meanings related to cleaning:

[0366] - “very dirty here I...”

[0367] - “Dusty”

[0368] - “Clean” - etc..

[0369]

[0315] - Specific to Residential or Public Spaces

[0370] - Kitchen and Bathroom Cleaning

[0371] - Running Water: A continuous, flowing or splashing sound.

[0372] - Dishwashing: A combination of water running, dishes clinking, and scrubbing sounds.

[0373] - Laundry

[0374] - Laundry Machines: A mechanical humming or spinning sound, sometimes with a clicking or beeping at the end of the cycle.

[0375] - Floor Care

[0376] - Carpet Cleaning: A loud, mechanical humming or suction sound, sometimes with a wet, sloshing sound if water is involved.

[0377]

[0316] - Specific to Industrial Settings

[0378] - Heavy-Duty Cleaning

[0379] - Power Washing: A loud, pressurized spraying sound, often with a high- pitched whine.

[0380] - Industrial Vacuums: A very loud, mechanical humming or whirring sound.

[0381] - Steam Cleaning: A hissing or steaming sound, sometimes with a mechanical hum.

[0382] - Machinery Maintenance

[0383] - Machinery Maintenance: A variety of mechanical sounds, including clanking, banging, and the hum of power tools.

[0384] - Safety and Alerts

[0385] - Safety Alarms: A loud, repetitive beeping or buzzing sound designed to attract attention.

[0386]

[0317] - Incidental Noises

[0387] - Accidental Events

[0388] - Dropping Items: A sudden, sharp clattering or thudding sound.

[0389] - Knocking Over Objects: A crashing or clattering sound as objects fall.

[0390] - Spilled Liquid: A splashing or dripping sound, sometimes with a subsequent wiping or mopping sound.

[0391] - Door and Window Operations

[0392] - Door Opening / Closing: A creaking, clicking, or slamming sound.

[0393]

[0318] - Environmental Noises

[0394] - HVAC Systems

[0395] - Air Conditioning: A continuous, low humming or whirring sound.

[0396] - Ventilation Systems: A steady, blowing or whooshing sound.

[0397] - Background Audio

[0398] - Background Music: Various musical sounds, often at a low volume.

[0399]

[0319] - Equipment Malfunctions - Cleaning Equipment Issues

[0400] - Equipment Malfunctions: Unusual mechanical noises, such as grinding, screeching, or irregular humming, indicating a problem with the equipment.

[0401]

[0320] The sound classes represented in figures 10A to 10F are provided by means of example only of a set of sound classes in the context of cleanliness. Of course, other sound classes may be considered in the context of a cleaning monitoring application. A set of sound classes may also comprise both sound classes relative to the context of cleaning monitoring application such as those represented in figure 9 for example, and other sound classes such as for example some or all the sound classes represented in figure 8.

[0402]

[0321] Is now referred to figures 11 A and 11 B.

[0403]

[0322] Figures 1 1 A and 11 B are an example of state classes in an embodiment.

[0404]

[0323] Figures 11A and 1 1 B are an example of state classes in an embodiment for cleaning monitoring application. The state classes represented in figures 11A and 11 B are representative of alerts in cleaning monitoring.

[0405]

[0324] The figures 11A and 11 B are represented on 2 figures only for the purpose of intelligibility of the figures. The figures 11 A and 11 B shall thus be read as one example of a set of possible state classes that comprises all the state classes represented in figures 11 A and 11 B.

[0406]

[0325] Figures 11A and 11 B represent an example of a set of possible classes of alerts in cleaning monitoring. The set of alerts thus corresponds to the alert classes that can be pushed to actors of the cleaning process, and the labels that can be entered in the training phase. As explained above, the labels relative to the classes represented in the figures 11 A and 11 B can be for example provided: by a labeler, based on one or more of an audio frame captured during cleaning, a video of the cleaning area, alerts raised by equipments, etc. ; and / or automatically, through detection of events during cleaning, for example an alert raised by a machine.

[0407]

[0326] Of course, the sate labels may be obtained based on a combination of the above, and / or other means, provided that the state labeling provides a time accurate labeling of alerts that were caused during the cleaning process.

[0408]

[0327] The figures 11A and 11 B represent a structured graph of various alerts for professional cleaning process or industrial environment, explained as follows (non exhaustive examples):

[0409]

[0328] - Safety Alerts

[0410] - Falling Object Detection: "Alert: An object has been detected as falling in Zone X. Check for any damages or potential risks."

[0411] - Spilled Liquid: "Alert: Liquid has been spilled in Zone Y. Immediate intervention required to clean up and prevent slips."

[0412] - Safety Alarm: "Emergency: A safety alarm has been triggered in Zone Z. Check the situation immediately."

[0329] - Maintenance Alerts

[0413] - Equipment Malfunction: "Alert: A malfunction has been detected in the cleaning equipment in Zone A. Check and repair if necessary."

[0414] - Preventive Maintenance: "Reminder: It's time to check and clean the filters of the industrial vacuum cleaner."

[0415]

[0330] - Cleaning Quality Alerts

[0416] - Uncleaned Area: "Alert: Zone B has not been cleaned according to the schedule. Intervention required to complete the cleaning."

[0417] - Cleaning necessary: “Alert: Zone B has experienced some human activities - intervention is required”

[0418] Unnecessary cleaning: "Alert: Zone B has not been used for some time, scheduled cleaning is not necessary after all”

[0419] - Incomplete Cleaning: "Alert: The cleaning in Zone C appears incomplete. Check and finish the cleaning if necessary."

[0420]

[0331] - Productivity Alerts

[0421] - Task in Progress: "Info: Floor cleaning is in progress in Zone D. Avoid this area for now."

[0422] - Task Completed: "Info: Surface cleaning in Zone E is completed. You can move on to the next task."

[0423]

[0332] - Communication Alerts

[0424] - Team Message: "Message: The cleaning team needs your help in Zone F. Respond as soon as possible."

[0425] - Meeting Reminder: "Reminder: Team meeting at 2:00 PM in the conference room."

[0426]

[0333] - Compliance Alerts

[0427] - Upcoming Inspection: "Reminder: A compliance inspection is scheduled for tomorrow. Ensure all areas are ready."

[0428] - Non-Compliance Detected: "Alert: Non-compliance detected in Zone I. Correct immediately."

[0429]

[0334] - Employee Safety Alerts

[0430] - Fall Detected: "Emergency: A fall has been detected in Zone J. Check the employee's condition immediately."

[0431] - Lack of Movement: "Alert: No movement detected in Zone K for the past 10 minutes. Check the situation."

[0432]

[0335] - A combination of sound classes such as those represented in figures 10A to 10F, and state classes such as those represented in figures 11A and 11 B thus allow raising alerts relative to a cleaning process based only upon the sound received from one or more microphones. Even if some or all of the state labels have not been entered solely by listening to an audio frame, the training phase allows the state detection machine learning engine to determine alerts relative to the cleaning process based upon the detected sound classes.

[0336] Therefore, a training using sets of sound classes and state classes relative to a cleaning monitoring application allow obtaining models that are able to raised alerts relative to a cleaning application using only sounds captured by microphones.

[0433]

[0337] Figures 12A, 12B to 12 C provide an example of a structured set of sound classes in the context of psychiatric or psychological care monitoring. These types of sound classes or similar classes could be applied, to some extent, for correctional facilities or detention centers.

[0434]

[0338] These sound classes correspond to types of sounds that may be detected in such environments and used as labels during the training phase of a machine learning model.

[0435]

[0339] . The structured set of sound classes takes the form of a tree graph where three super classes are broken down into nested sub-classes. By means of non-limitative example, sound classes relative to psychiatric or psychological care monitoring, generally denoted by the super class “General psychiatric center sound” under the numeral reference A in the 3 figures 12A, 12B and 12C, are organized as follows (non-exhaustive examples):

[0436]

[0340] A first category could be sounds caused by events of imminent danger, represented by the super-class “Revealing imminent danger” comprising the following sub-classes :

[0437] Emotional Vocalizations o Screaming: this class corresponds to Loud, high-pitched, piercing, and urgent vocalization ; o Prolonged howling ; o crying: this class corresponds to Repetitive sobbing or sharp emotional sounds; o Moaning or lamenting: this class corresponds to Long, low-frequency vocalizations expressing discomfort or despair;

[0438] Aggressive Speech o Violent Verbal insults: this class corresponds to Detectable offensive or aggressive language, for example statements like “I’m going to beat him”, “you’ll see I”, “I will kill him” o Explicit aggressive intent: this class corresponds to Statements like “I’ll kill him” or “I’ll hurt myself’. “I want to die now” o Shouting matches: this class corresponds to Two or more people yelling at each other.

[0439] Self-directed or Risk Behaviors non-speech sounds o Head banging: this class corresponds to Rhythmic impact sounds against wall or floor. o Tearing fabric: this class corresponds to Sheets or clothing being ripped apart. o Swallowing pills: this class corresponds to Series of gulping and blister-pack manipulation. o Impact sounds: this class corresponds to Sudden crashes or loud body movement sounds.

[0341] A second category could be sounds caused by events of announcing potential dangerous behavior, represented by the super-class “Announcing potential dangerous behavior” comprising the following sub-classes :

[0440] Addiction-related Sounds o Lighter flicking / match striking. o Rolling a cigarette or joint: this class corresponds to Paper, friction, etc. o Injection preparation: this class corresponds to Sounds of ampoules, syringes. o Alcohol Drinking: this class corresponds to Gulping from a bottle, cork popping, can opening.

[0441] Suicidal ideation o Vague suicidal intent verbalization: this class corresponds to the uttering by the residents of a speech relative to suicide, such as for example “I want to end” o Vague escape intent verbalization: this class corresponds to the uttering by the residents of a speech relative to escapeSudden vercal delirium, such as for example “I want it to leave”

[0442] Disorganized speech o Sudden verbal delirium o Paranoid monologue

[0443] Environmental Interaction Sounds o Furniture dragging / movement. o Door slamming. o Footsteps at abnormal hours. o Unusual silence in occupied spaces.

[0444]

[0342] A third category could be signs of mental degradation or apathy, comprising the following sub-classes :

[0445] Disorganized speech

[0446] Frequent sighting

[0447] Recurrent whispering

[0448] Word repetition

[0449] Inappropriate laughter

[0450] Talking to absent voices

[0451] Repetitive words or sounds

[0452] Repetitive vocal tics

[0453] Monotone voice

[0454] Flat or faint voice

[0455] Slow speech with long pauses

[0456] Flat voice

[0457] Slow speech rate

[0343] Figure 13 represents an example of a structure set of state classes that represents clinical alerts or behavioral states relevant to mental health monitoring.

[0458]

[0344] The structured set of state classes takes the form of a tree graph where three super classes are broken down into nested sub-classes

[0459]

[0345] These state classes can be: manually entered by caregivers or observers, for example through a user interface similar to the user interface shown in figure 8, automatically derived from structured events (e.g., footsteps at night, repeated threats), or provided by external systems (e.g., digital nursing logs, access control systems, etc.).

[0460]

[0346] The state classes are grouped as follows (non-exhaustive examples):

[0461] Immediate Risk Alerts o Active self-harm in progress o Verbal aggression toward others o Physical aggression or object destruction o Suicidal gesture or verbalization

[0462] Vigilance Alerts (Early Warning Signs) o Verbalized despair or suicidal ideation o Agitation episode o Preparatory behaviors (fleeing, substance use) o Hallucinatory behavior or disorganized talk

[0463] Behavioral Monitoring / Long-Term Update o Signs of apathy o Nighttime activity or disturbed sleep. o Repetitive, compulsive behaviors o Onset of confusion or cognitive disorganization

[0464]

[0347] - A combination of sound classes such as those represented in figures 12A to 12C, and state classes such as those represented in figure 13 thus allow raising alerts relative to psychiatric or psychological events based only upon the sound received from one or more microphones. Even if some or all of the state labels have not been entered solely by listening to an audio frame, the training phase allows the state detection machine learning engine to determine alerts relative to the cleaning process based upon the detected sound classes.

[0465]

[0348] Therefore, a training using sets of sound classes and state classes relative to a psychiatric or psychological monitoring application allows obtaining models that are able to raise alerts relative to psychiatric or psychological events using only sounds captured by microphones

[0466]

[0349] This disclosure is not limited to the method, system devices and computer programs described here, which are only examples. The invention encompasses every alternative that a person skilled in the art would envisage when reading this text.

Claims

Claims

1. A computer-implemented method (P3), said method comprising: receiving (S3.1) a plurality of audio frames (Snd1 .1 , Snd1 .2) from one or more microphone (Mid .1 ; Mid .2) located in an environment (Env1 .1 , Env1 .2); using (S3.2)one or more sound detection supervised machine learning engine (SndMLEng2.1 , SndMLEng2.2) to detect one or more sound classes from said plurality of audio frames ; using (S3.3) one or more state detection supervised machine learning engine (StMLEng2.1 , StMLEng2.2) to detect one or more state classes based at least from the output of said one or more sound detection supervised machine learning engine, each of said state classes being representative of a state of said environment and at least one of said state classes being representative of an alert state.

2. The computer-implemented method of claim 1 , further comprising, when said one or more state detection supervised machine learning engine detects a class representative of an alert state, sending said alert to one or more user device (UDev1.3) configured to render said alert to a user.

3. A computer-implemented method (P4; P5) comprising: receiving (S4.1) a plurality of audio frames (Snd1 .1 , Snd1 .2) from one or more microphone (Mid .1 ; Mid .2) located in one or more environment (Env1 .1 , Env1 .2); for each audio frame of said plurality of received audio frames: causing (S4.2) one or more loudspeaker (Ldspk2) of one or more Man Machine Interface (MMI2) to play said audio frame ; obtaining (S4.3) one or more sound label (SndLbl) belonging to a set of sound classes ; receiving (S4.4), through said one or more Man Machine Interface, one or more state label (StLbl) belonging to a set of state classes, each of said state classes being representative of a state of said environment and at least one of said state classes being representative of an alert state ; enriching (S4.5) one or more sound detection training dataset (SndDS2.1 ; SndDS2.2) with said audio frame associated to said one or more sound label ; using said one or more sound detection training dataset to train (S4.7) one or more sound detection supervised machine learning engine (SndMLEng2.1 , SndMLEng2.2) to detect one or more sounds classes from audio frames ; for each audio frame of said plurality of audio frames:enriching (S4.8) one or more state detection training dataset (StDS1 .2; StDS2.2) with a state detection training sample associating said one or more state label to one or more training sound classes ; using said state detection training dataset to train (S4.9) one or more state detection supervised machine learning engines (StMLEng2.1 , StMLEng2.2) to detect one or more state classes based at least from the output of said one or more sound detection supervised machine learning engine.

4. The computer-implemented method (P5) of claim 3, wherein said one or more training sound classes are said one or more sound label.

5. The computer-implemented method (P5) of claim 3, wherein : using said one or more sound detection training dataset to train said one or more sound detection supervised machine learning engine to detect one or more sounds classes from audio frames is performed prior to enriching said one or more state detection training dataset ; for each audio frame of said plurality of audio frames: the method comprises using said one or more trained sound detection supervised machine learning engine to detect one or more sound classes (S5.1) from said audio frame ; said one or more training sound classes are the one or more sound classes detected from said audio frame by the one or more sound detection supervised machine learning.

6. The computer-implemented method of any of the preceding claims, wherein audio frames are captured by said one or more microphone at successive time frames ; said one or more state detection supervised machine learning engine comprises a state detection supervised machine learning engine (StMLEng2.1) that takes into account detections of sound classes from the one or more sound detection supervised machine learning engine from a plurality of successive audio frames.

7. The computer-implemented method of claim 6, wherein said state detection supervised machine learning engine that takes into account detections of sound classes from the one or more sound detection supervised machine learning engine from the plurality of successive audio frames is a recurrent neural network.

8. The computer-implemented method of any one of claims 6 and 7, wherein: an audio frame at a time frame is sent by said one or more microphone only upon a condition relative to a sound level from said audio frame ; said state detection supervised machine learning engine further takes as input indications of absence of reception of audio frames.

9. The computer-implemented method of any of the preceding claims, wherein said one or more state detection supervised machine learning engine comprises a state detection supervised machine learning (StMLEng2.2) engine that takes as input: the one or more sound classes detected by said one or more sound detection supervised machine learning engine from an audio frame ; one or more indicator of detection of words belonging to a predefined set of words by a speech recognition engine (SpRecEng6) from said audio frame.

10. The computer-implemented method of claim 9, wherein the execution of: the speech recognition engine; and the state detection supervised machine learning engine that further takes as input the one or more sound classes detected by said one or more sound detection supervised machine learning engine from an audio frame and words detected by the speech recognition engine from said audio frame ; are conditional upon a detection of said at least one of said state classes representative of an alert state by a gradient boosting machine learning engine (GrBstMLEng6) that takes as input the one or more sound classes detected by said one or more sound detection supervised machine learning engine.

11. The computer-implemented method of one of claims 9 and 10, wherein said speech recognition engine (SpRecEng6) is not executed upon the detection of a sound class representative of a conversation between a plurality of persons.

12. The computer-implemented method of any one of the preceding claims, wherein at least one among the one or more sound detection supervised machine learning engine and the one or more state detection supervised machine learning engine is configured to perform a regression task to predict, for each sound class, respectively state class, a ratio of received labels that indicate the presence or the absence of the sound class, respectively state class.

13. Computer software comprising instructions to implement at least a part of a method according to any one of the preceding claims when the software is executed by a processor.

14. Computer-readable non-transient recording medium on which a software is registered to implement a method according to one of claims 1 to 12 when the software is executed by a processor.

15. One or more first computing device (Devi .1) configured to: receive (S3.1) a plurality of audio frames (Snd1.1 , Snd1.2) from one or more microphone (Mic1 a.1 ; Mid a.2) located in an environment ; use one or more sound detection supervised machine learning engine (SndMLEng2.1 , SndMLEng2.2) to detect one or more sound classes (S3.2) from said plurality of audio frames ; use one or more state detection supervised machine learning engine (StMLEng2.1 , StMLEng2.2) to detect one or more state classes (S3.3) based at least from the output of said one or more sound detection supervised machine learning engine, each of said state classes being representative of a state of said environment and at least one of said state classes being representative of an alert state.

16. A computing system (Sys1) comprising: one or more microphone (Mid .1 ; Mid .2 ) ; the one or more first computing device (Dev1.1) of claim 15.

17. A computer system (Sys2) comprising: one or more Man Machine Interface (MMI2) comprising one or more loudspeaker (Ldspk2) ; one or more processing unit (Proc2) configured to: receive (S4.1) a plurality of audio frames (Snd1 .1 , Snd1 .2) from one or more microphone (Mic1 a.1 ; Mid a.2) located in an environment ; for each audio frame of said plurality of audio frames: cause (S4.2) one or more loudspeaker (Ldspk2) of one or more Man Machine Interface (MMI2) to play said audio frame ; receive (S4.3), through said one or more Man Machine Interface, one or more sound label (SndLbl) belonging a set of sound classes ;receive (S4.4), through said one or more Man Machine Interface, one or more state label (StLbl) belonging a set of state classes, each of said state classes being representative of a state of said environment at least one of said state classes being representative of an alert state ; enrich (S4.5) one or more sound detection training dataset (SndDS1.2;SndDS2.2) with said audio frame associated to said one or more sound label use said one or more sound detection training dataset to train (S4.7) one or more sound detection supervised machine learning engine (SndMLEng2.1 , SndMLEng2.2)to detect one or more sounds classes from audio frames ; for each audio frame of said plurality of audio frames: enrich (S4.8) one or more state detection training dataset (StDS1.2; StDS2.2) with a state detection training sample associating said one or more state label to one or more training sound classes ; use said state detection training dataset to train (S4.9) one or more state detection supervised machine learning engine (StMLEng2.1 , StMLEng2.2) to detect one or more state classes based at least from the output of said one or more sound detection supervised machine learning engine.

Images