Voice virtual assistant for home automation and personal security with integrated emotional and decision-making recognition and method of operation

Abstract

The voice virtual assistant for home automation and personal security with integrated emotional recognition and decision-making is an advanced voice virtual assistant for home automation and personal security. The system comprises a client device with a microprocessor and various sensors, a cloud server with artificial intelligence models, and several API gateways for the connection with external services and home automation devices. One of a kind, the assistant is capable of an emotional recognition of sounds coming from the environment and is able to autonomously decide to raise the alarm in the event of detected dangers. It can provide personalized voice responses based on the analysis of the user's emotions and environmental situation, execute specific home automation commands and activate safety or emergency protocols. The system is designed to be highly inter-operable and can integrate with existing home automation platforms.

Description

[0001] Voice virtual assistant for home automation and personal security with integrated emotional and decision-making recognition and method of operation.

[0002] Description

[0003] Field of the invention

[0004] The present invention relates to the field of home automation, artificial intelligence and speech recognition, with particular attention to applications which offer decisional support and improved personal security. More specifically, the patent application describes a voice virtual assistant equipped with advanced speaker recognition, language interpretation and emotional analysis functions for controlling home environments and responding to situations of danger or difficulty for the user.

[0005] Prior art

[0006] In the context of increasing digitalization and global connectivity, speech recognition systems have gained considerable importance and have developed in various application areas. These range from the simple automation and control of electronic devices, to complex biometric authentication systems which can identify a person through unique voice characteristics. Improvements in artificial intelligence techniques, such as deep learning and convolutional neural networks, have enabled an increasingly accurate and versatile analysis of spoken language. At the same time, the evolution of network infrastructures, including fiber optics and 5G cellular networks, has facilitated the transmission and processing of data at increasingly lower latencies, making it possible to use powerful Al algorithms through cloud servers. Despite significant progress, the current systems still have several limitations which hinder their use for more advanced and sensitive applications. For example, while most systems can interpret verbal commands and synthesize responses, the ability to understand emotional context or act autonomously is still largely unexplored or unreliable. Moreover, the integration with other technologies, such as home automation or security systems, is often fragmented and requires a high degree of customization, which may be beyond the reach of the average user.

[0007] Recognizing sentiment and emotion in language represents a further level of complexity and opportunity. Artificial intelligence is now able to analyze not only the semantic content of language, but also tone, timbre, and other subtleties that can indicate the emotional or mental state of the individual. This capability has enormous potential for applications ranging from mental health to home security monitoring. However, even in this emerging field, there is a gap in integrating such features into a coherent and user-friendly system.

[0008] In terms of industrial patents, US patent US 11323281B2 claims a method for remotely managing the delivery of voice-controlled services to a user's device; the gateway devices using the service management system implicate the creation of dual communications, receiving endpoint device data and performing wireless services.

[0009] US20150162006A1 instead relates to a home automation management system which uses voice commands. The system not only recognizes the voice command to control a specific device in the home automation network, but also identifies who gave the command through voice analysis. After determining the speaker's identity and the device to be controlled, the system verifies whether the user has the necessary permissions to control that device. If the user has the necessary permissions, the system proceeds with the control of the device as requested. This invention lacks the emotional identification and decision -making processes integrated into the device, which is the subject of this application.

[0010] Patent JP4546555B2 also describes a speech recognition (VR) system which uses a combination of speaker-independent (SI) and speaker-dependent (SD) acoustic models. The idea is to use at least a SI acoustic model in combination with at least a SD acoustic model to achieve a speech recognition performance which is at least as good as that of a purely SI model. The proposed system is able to continuously update the SD acoustic models through unsupervised training, thus improving the speech recognition performance over time. In practice, the system tries to combine the advantages of the SI and SD models to achieve more accurate and adaptive speech recognition. Yet even this patent, despite being defined as “adaptive”, ignores the emotional characteristics of the speaker.

[0011] The object of the present industrial patent application is therefore to introduce a new and alternative solution, proposing a voice virtual assistant for home automation and personal security with integrated emotional recognition and decision-making, which has been carefully designed to fill these gaps. Unlike traditional speech recognition and virtual assistant systems, our device offers a full suite of integrated features. This includes not only basic speech recognition and human interaction, but also advanced emotion detection and decision-making capabilities. In fact, in situations recognized as emergency or dangerous, the device is programmed to autonomously assess the severity of the situation and take appropriate measures, such as activating alarms or reporting to emergency services.

[0012] The solution is able to flexibly integrate with existing home automation platforms, including the well-known open-source system Home Assistant, thus allowing an efficient and centralized management of various home functions. Furthermore, it has been specially optimized to provide assistance to specific groups of users, such as the elderly or people with motor disabilities, or in specific cases of danger, such as the entry of thieves or the occurrence of domestic violence, thus offering a significant improvement in quality of life and public safety.

[0013] Description of the invention

[0014] With the present patent application for an industrial invention, it is intended to describe and claim a device provided with at least a new and alternative solution to the solutions known so far and / or to satisfy one or more needs perceived in the art and in particular deduced from what has been reported above. To achieve this object, the inventors have developed an integrated system, designed to acquire, process and analyze audio signals, in particular the human voice, so as to perform biometric recognition of the user's language and emotions to assist them in both everyday life and in dangerous situations. The device also integrates a decision-making system and the ability to interface with home automation systems, which is able to intervene in case of emergencies or other situations which require rapid and authenticated intervention. Specifically, the voice virtual assistant for home automation and personal security with integrated emotional recognition and decision-making comprises:

[0015] - at least a Client device, which includes a microprocessor, a network card, a GSM / GPRS card, a microphone array and a speaker. This device is responsible for collecting realtime data from the surrounding environment, including voice commands and ambient sounds;

[0016] - a plurality of API Gateways, each with a specific function:

[0017] ■ first API Gateway, called A, for the connection with the home automation control panel;

[0018] ■ second API Gateway, called B, for the connection with the instant messaging services available on the network;

[0019] ■ third API Gateway, called C, for the connection with Cloud Artificial Intelligence services;

[0020] - Cloud servers where the artificial intelligence models which manage the various functions of the assistant are hosted;

[0021] - control unit, or home automation panel, capable of controlling connected devices such as, for example, lighting, alarm, air conditioning, heating, door and window automation, telephone network and more.

[0022] The Client device is configured to recognize pre-registered users by means of a speech recognition model (“speaker recognition”). Once authenticated, users can execute voice commands to control home automation functions (e.g., kitchen lights, outdoor siren, etc.). This information is transmitted to the home automation control panel through the API Gateway A. The system not only executes commands upon user request, but also has the ability to act as a supervisor. Using emotional recognition, it can identify potentially dangerous or emergency situations and take autonomous measures, which can include generating alarms or notifications. These communications can be made through a call or SMS using a GSM / GPRS card or through instant messaging services connected through the API Gateway B.

[0023] The assistant is particularly well suited to contexts in which there are individuals who require special assistance, such as the elderly or people with reduced mobility. It is able to detect various dangerous conditions, authenticate users and their emotional state, and activate appropriate alarm or signaling devices.

[0024] The Client device, using its microphone system, detects the presence of vocal sounds and sends the data to the Cloud Server through the API Gateway C. The server, functioning as the “brain” of the system, processes this data using various artificial intelligence algorithms. Once the data has been processed, the server sends the relevant information to the Client device's microprocessor, which can in turn communicate with the various API Gateways (A, B or C) or activate its own peripherals to execute specific actions such as sending messages or activating control / alarm systems.

[0025] The connectors for the API Gateways can be developed in various programming languages, including JAVA. The connection can be made using standard APIs or through the development of dedicated software code.

[0026] The voice virtual assistant for home automation and personal security with integrated emotional recognition and decision-making, described in the present industrial patent application, operates following a specific sequence of operations which comprises audio signal acquisition, audio signal processing, extraction of specific characteristics and, lastly, biometric recognition of language and emotions. In the audio signal acquisition step, the signal is sampled through an integrated acquisition system at a predetermined frequency, preferably 44.1 kHz. This allows the high-precision capture of the wide range of human vocal frequencies and other related sounds which may be important for biometric recognition.

[0027] Several operations are included in the audio signal processing step. Initially, the signals are subjected to band-pass filtering to eliminate all the unnecessary or uninteresting frequencies. Subsequently, the signals can be truncated to extract a reduced period of time, so as to speed up the data analysis and processing. This truncation step uses a division of the signal into time frames of approximately 20-30 ms with an overlapping of approximately 10 ms. After truncation, the signal is smoothed to eliminate any anomalous peaks, using a filter with a moving average and a smoothing coefficient specific to the application. Finally, the signals are normalized with respect to their maximum and minimum peak values, to ensure that they are comparable to each other.

[0028] In the step of extracting the indicators, or features, specific statistical indicators are selected from the audio signal in the time, frequency and time-frequency domains. Prosodic indicators, which best represent human speech, are extracted from the frequency and time-frequency domains.

[0029] Automatic recognition is implemented through various machine learning methods, including supervised and unsupervised learning. In particular, the supervised learning processes aim to predict the class to which a set of attributes belongs through the use of classification algorithms such as Support Vector Machine (SVM), DecisionTree, Artificial Neural Network (ANN), among others. Similarly, the unsupervised learning processes, which use clustering algorithms such as K-Means, Hierarchical Clustering, Principal Component Analysis (PCA), aim to determine homogeneous groupings of records.

[0030] The user-side device, known as the Client, detects the presence of a human voice and sends an audio stream to the remote computing server until a no-voice condition is detected. This voice audio stream is then processed to extract the indicators which feed the speaker, speech and emotion recognition models present on the server.

[0031] The speaker recognition model is customized to the users' voice and requires a guided training step. If the user is recognized as an “impostor,” or “unauthorized,” they are disabled from executing the functions which will then be interpreted by the speech recognition model. The speech recognition model is responsible for recognizing the user's speech and interpreting the home automation requests. The emotion recognition model is responsible for classifying the user's emotional state.

[0032] The Artificial Intelligence gateway manages the interaction between the Client device and the cloud server through appropriate APIs. The audio signal is sent to the server, where it is processed to query the forecasting models. A separate database is provided for each Client, which contains the names of the users and their audio track acquired during the training step. Every time this database is updated with a new audio track, the recognition algorithm is reformed, releasing a new version of the recognition model.

[0033] In addition to the recognition and interpretation functions, the device is equipped with an advanced decision-making module which, based on the information acquired and processed, is able to implement specific actions autonomously. This decision-making module is programmed to assess the severity of the situation, such as a potential emergency or danger, and act accordingly. The actions can range from simple push notifications to more drastic interventions, such as activating alarms, calling emergency services, or manipulating home automation devices such as lights, locks, and security systems.

[0034] The system is also designed to flexibly integrate with existing home automation platforms, such as Home Assistant, thus allowing an efficient and centralized management of home functions and security measures. This integration is made possible through the use of standardized communication protocols and open APIs, making the device compatible with multiple home automation systems and platforms.

[0035] The proposed system offers an unprecedented level of interaction and assistance, thanks to the combination of biometric recognition, emotional evaluation, autonomous decision-making and integration with home automation systems. Its modular architecture and flexibility make it an ideal solution for a wide range of applications, from assistance to the elderly and people with disabilities, to the prevention and management of emergencies.

[0036] The advantages offered by the present invention are evident in the light of the description presented thus far and will be even clearer thanks to the attached figures and the related detailed description.

[0037] Description of the figures

[0038] The previous advantages, as well as other advantages and characteristics of the present invention, will be illustrated by referring to the attached figures, which are to be considered purely illustrative and not limiting or binding for the purposes of the present patent application, in which:

[0039] - FIGURE 1 shows the possible operation of a Client 10, comprising a microprocessor 1, a microphone 2, a speaker 3, a network card 4 and a GSM card 5, and of the three main API Gateways: a Gateway A to a home automation panel 100, a Gateway B to a messaging system 200 and a Gateway C to and from a Cloud Server 300;

[0040] - FIGURE 2 illustrates a possible operating diagram between a Client 10 and a Cloud Server 300;

[0041] - FIGURE 3 shows the possible operating diagram of said Cloud Server 300 in its three functions of Speaker Recognition 310, Speech Recognition 320 and Emotion Recognition 330.

[0042] Detailed description of the invention

[0043] It will be immediately obvious that countless variations and modifications can be made to what has been described (for example relating to shape, dimensions, arrangements and parts with equivalent functionality) without departing from the scope of protection of the invention as appears in the attached claims.

[0044] With reference to Fig. 1, the invention in question is illustrated as a Client 10, comprising a microprocessor 1, a microphone 2, a speaker 3, a network card 4 and a GSM card 5, which detects the presence of voices through the microphone 2 and sends the recording to the Cloud Server 300 through said API Gateway C; the signal processing algorithms and the automatic recognition models reside on said Cloud Server 300. Through the Gateway C, the Cloud Server 300 provides a response to the microprocessor 1 which in turn can communicate with the Gateway A, towards the home automation panel 100, or B, towards a messaging service 200. With reference to Fig. 2, the interaction between said Client 10, said Cloud Server 300, said home automation panel 100 and said messaging system 200 is shown. The user-side device (i.e., said Client 10), through a Voice Activity Detection system 11 connected to said microphone 2, detects the presence of a human voice and sends an audio stream to said Cloud Server 300 until a condition of no voice is detected. The voice audio stream is processed to extract the indicators which feed the speaker 310, speech 320, and emotion 330 recognition models present on the server 300. The speaker recognition model 310 requires being customized to the users' voice, for which a guided training step is provided upon the first startup of the system. The speaker recognition model 310 is also responsible for verifying the user's authorization: if the user is recognized as "unauthorized", then they are disabled from performing the functions; otherwise, the requests will be interpreted by the speech recognition model 320 and then sent to a control system 12 which sends them to the home automation panel 100 for the activation of the peripherals. Finally, the emotion recognition model 330 recognizes and classifies the emotional state of the user. If the user's emotional state is deemed negative (difficulty, fear, etc.), the Client 10 will activate the signaling and alarm peripherals connected to said messaging system 200.

[0045] With reference to Fig. 3, lastly, a diagram of the operation of said Cloud Server 300 is illustrated in its three functions of Speaker Recognition 310, Speech Recognition 320 and Emotion Recognition 330. The API Gateway C is responsible for managing the interaction between the Client device 10 and the Cloud Server 300, which contains the recognition models, through appropriate APIs. The audio signal is sent, by means of API, to the server 300 where it is processed in order to query the forecasting models 310, 320 and 330. In particular, the speaker recognition model 310 must be customized for each user, for which a database A is provided which contains the names of the users and their audio track acquired during the guided training at the first start-up of the device. Every time the database A is updated with a new audio track, the recognition algorithm B is retrained, releasing a new version C of the recognition model. The speech 320 and emotion recognition 330 models are instead independent of the specific user and are selected from a list of models based on the nationality or even the geographical area of belonging, defined during the start-up step of the device. This is because the single speech model 320 is trained on a specific nationality language (Italian, English, . . .) and the emotion model 330 could, in general, be trained on the geographical area for abetter accuracy of the model itself (in fact, a Sicilian user could express emotions differently from a Venetian user). These models can also undergo updates, but these are driven by the server manager and independently of the client.

[0046] The server returns in output, by means of API, the data needed to identify the speaker, execute control actions and interpret the emotional state for autonomous supervision actions.

[0047] It is clear that modifications, additions or variations that are obvious to a person skilled in the art can be made to the invention described so far, without thereby departing from the scope of protection provided by the attached claims.

Claims

Claims1. Voice virtual assistant for home automation and personal security with integrated emotional recognition and decision-making, characterized in that it comprises at least:- a Client device (10) equipped with a microprocessor (1), a network card (4), a GSM / GPRS card (5), a microphone array (2) and a speaker (3), positioned in an environment to control and adapted to collect a plurality of data in real time from the surrounding environment and communicate with the peripherals described below;- a Cloud server (300) hosting artificial intelligence models for managing functions;- a first specific API Gateway for the connection between said Client (10) and a home automation panel (100), called API Gateway A;- a second API Gateway specific for the connection between said Client (10) and the instant messaging services (200) available on the network, called API Gateway B;- a third specific API Gateway for the connection between said Client (10) and the Artificial Intelligence services hosted on said Cloud server (300), called API Gateway C; said third API Gateway C being suitable for automatic recognition, implemented through supervised and unsupervised learning processes; the supervised learning processes having the aim of predicting the class to which a set of attributes belongs through the use of classification algorithms such as, by way of non-binding example, Support Vector Machine, DecisionTree, Artificial Neural Network; and unsupervised learning processes using clustering algorithms such as, for example, K-Means, Hierarchical Clustering, Principal Component Analysis (PCA), with the aim of determining groupings of homogeneous records;- a control unit, or home automation panel (100), adapted to control connected devices such as, for example, lighting, alarm, air conditioning, heating, door and window automation, telephone network and more;said voice virtual assistant being adapted to provide a personalized vocal response based on the analysis of the user's emotions and environmental situation, to execute specific home automation control commands based on said emotional and environmental recognition, and to connect securely with messaging and / or emergency services through the GSM / GPRS card (5) in the event of identified risk or danger situations.

2. Voice virtual assistant for home automation and personal security with integrated emotional recognition and decision-making, according to the previous claim, characterized in that said Cloud Server (300) comprises at least one Speaker Recognition model (310), suitable at least for recognition of the voice and the attribution of voice permissions associated with the user for accessing and modifying the system.

3. Voice virtual assistant for home automation and personal security with integrated emotional recognition and decision-making, according to claim 1 or 2, characterized in that said Cloud Server (300) comprises at least one Speech Recognition model (320), adapted to interpret the user’s requests when they are deemed authorized and send them to a home automation panel (100) to activate the peripherals.

4. Voice virtual assistant for home automation and personal security with integrated emotional recognition and decision-making, according to any of the previous claims, characterized in that said Cloud Server (300) comprises at least one emotion recognition model (330) suitable for the emotional interpretation of the user’s requests with the aim of autonomously activating a messaging service (200) when a dangerous situation is detected.

5. Voice virtual assistant for home automation and personal security with integrated emotional recognition and decision-making according to the previous claims, characterized in that it can be integrated with existing home automation platforms, such as, by way of a non-binding example, “Home Assistant”, through the use of standardized communication protocols and open APIs.

6. Method of operation of a voice virtual assistant for home automation and personalsecurity with integrated emotional recognition and decision-making according to any of the previous claims, characterized in that it comprises the steps of: (i) acquisition of the audio signal; (ii) audio signal processing; (iii) extraction of predetermined features and, finally, (iv) biometric recognition of language and emotions.

7. Method according to the previous claim 6, characterized in that during the acquisition of the audio signals (i) the signal is sampled through an integrated acquisition system at a predetermined frequency, preferably 44.1 kHz; this allowing to capture with high precision the wide range of human vocal frequencies and other related sounds that may be of importance for biometric recognition.

8. Method according to the previous claim 6 or 7, characterized in that during the processing of the audio signals (ii) the signals are initially subjected to band-pass filtering to eliminate all unnecessary frequencies; subsequently, they are truncated using a division into time frames preferably of 20-30 ms with a preferable overlapping of approximately 10 ms; next the method proceeds with the smoothing of the signal to eliminate any anomalous peaks, using a filter with a moving average and a smoothing coefficient specific to the application; and finally, normalizing the signals with respect to their maximum and minimum peak values, to ensure that they are comparable to each other.

9. Method according to any of the previous claims 6-8, characterized in that it comprises a step of sending the signal to said Cloud server (300) by said API Gateway C, said Cloud server (300) querying said forecast models (310, 320 and 330); a separate database (A) being provided for each user, which contains the names and audio fracks acquired during training; said database (A) being continuously updated; an algorithm (B) being consequently reformed at each update; and a new version of the recognition model (C) being released every cycle.

10. Method, according to any of the previous claims 6-9, characterized in that it comprises a step of implementing predetermined actions by an advanced decisionmaking module which, on the basis of the results of said emotion recognition model (330), is adapted to assess the severity of the situation, such as a potentialemergency or danger, and act accordingly; such actions can vary from sending notifications to the messaging service (200), up to interventions such as the activation of alarms and / or the manipulation of home automation devices such as lights, locks and security systems, by means of access to said home automation panel (100).

Images