System and method for generating and providing real-time textual interpretations of surrounding sounds

Abstract

A system and method for generating and providing in real-time ongoing textual interpretations or textual transcription or description of surrounding sounds. The system and the method includes the means and steps of utilizing an existing smartphone and smartwatch, activating a microphone of the smartphone to continuously capture surrounding sounds, transmitting the sounds to an Al processor on a remote server or feeding an existing processor on the smartphone, analyzing the captured sounds using an artificial intelligence (Al) or a computer program to identify and interpret, describe, or transcribe the sounds in real time, generating textual interpretations or transcriptions or description of the identified sounds, transmitting or feeding the textual results to the smartphone, and displaying it on the smartwatch.

Description

[0001] System and Method for Generating and Providing Real-Time Textual Interpretations of Surrounding Sounds

[0002] Field of the Invention

[0003] The present invention relates to systems and methods for assisting individuals with hearing impairments by providing real-time textual interpretations of surrounding sounds. More specifically, it leverages the existing smartphone and smartwatch of the user to capture, analyze, and display textual interpretations of sounds, allowing users to better understand their environment, especially in emergency situations.

[0004] Background of the Invention

[0005] Current systems designed to assist deaf and hard of hearing individuals often rely on dedicated hardware components, such as specialized wristbands or microphones, to capture and interpret surrounding sounds. These systems are typically expensive, cumbersome, and require users to carry additional devices, which may not be seamlessly integrated into their daily lives.

[0006] There is a need for a more accessible and user-friendly solution that can perform similar functions without the requirement for additional dedicated hardware. The present invention addresses this need by utilizing the user's existing smartphone and smartwatch, which are already widely adopted and integrated into everyday life, to provide real-time textual interpretations of surrounding sounds.

[0007] Summary of the Invention

[0008] The present invention provides a system and method that leverages the existing smartphone and smartwatch of the user to capture, analyze, and display textual interpretations of surrounding sounds in real time. The invention does not require any dedicated external hardware, making it a more accessible and cost-effective solution for individuals with hearing impairments.

[0009] The system comprises a smartphone configured to activate its microphone (16) at regular intervals to capture surrounding sounds. The captured sounds are then transmitted to an Al processor, which may reside either on the smartphone or on a remote server. The Al processor analyzes the captured sounds using an artificial intelligence model to identify and interpret them, generating corresponding textual interpretations. These textual interpretations are transmitted back to the smartphone, which then displays them on a connected smartwatch. The smartwatch also may activate a vibration alert to notify the user when a new textual interpretation is available.

[0010] DESCRIPTION OF THE DRAWINGS

[0011] The intention of the drawings attached to the application is not to limit the scope of the invention and its application. The drawings are intended only to illustrate the invention and they constitute only one of its many possible implementations.

[0012] FIG. 1 is a schematic depiction of the system (10).

[0013] Detailed Description of the Invention

[0014] System Overview

[0015] The system (10) is composed of three primary components: a smartphone (11) with an existing processor (12), an Al processor (13), and a smartwatch (14).

[0016] Smartphone:

[0017] The smartphone is an existing device owned by the user and serves as the primary interface for capturing sounds. It activates its built-in microphone to capture ambient sounds continuously. These captured sounds are then transmitted to the Al processor for analysis.

[0018] Al Processor:

[0019] The Al processor, which may be integrated within the smartphone (means: using the existing processors inside to smartphone to be used as the Al processor) or hosted on a remote server (15), analyzes the captured sounds using a trained artificial intelligence model. This model is capable of identifying a wide range of sounds, such as voices, alarms, or environmental noises. The term sounds in this disclosure and the claims means anything a person can hear, so this term means any kind of sound and including voices and noises.

[0020] The processor generates textual interpretations of these sounds in real time. The term "interpretation" in this context indicates that the method or system does not simply transcribe spoken words or provide direct descriptions of sounds. Instead, it generates a deeper understanding of the situation, and not just displaying "knocking on the door" when knocks are heard or "police car siren" when a siren is detected. The generated interpretation is derived through artificial intelligence, allowing the system to convey more complex scenarios. For example, the system might display "Missile attack. Missile hit nearby" based on the combination of sirens, noises, and voices. The system offers an interpretation akin to how a human brain would process and understand the same sounds in the surrounding environment.

[0021] Smartwatch: The smartwatch, also an existing device hold by the user, receives the textual interpretations from the smartphone. It displays these interpretations on its screen and activates a vibration alert to notify the user when new information is available. It is possible that the textual interpretations will be displayed on the smartphone, on the smartwatch, or on both, depends on the setting done by the user. The communication protocol between the smartwatch and the smartphone is known and it is in many cases Bluetooth connection.

[0022] Smart Device: The functions described under the foregoing paragraph "Smartwatch" may be applied instead or in addition to a smartwatch to any other device which is capable to the deployment of any part of such functions, including but not limited to: car screen, smart ring or any other smart device. The term “smartwatch” in this disclosure and in the claims refers to traditional smartwatch and also to other kind of devices that provide relevant functions such as displaying information on a display, and receiving information wirelessly.

[0023] Direct Deployment: It is possible that instead of using the smartphone as the receiver and transmitter of the information, the implementation of the present invention may be directly to the smart device (such as smartwatch), instead or in addition to the smartphone, such that an existing smart device is independent to demonstrate all or part of the functions detailed under the Smartphone description above.

[0024] Innovative Aspects:

[0025] One of the key innovative aspects of the present invention is its ability to perform complex sound analysis and interpretation using only the existing smartphone and smartwatch of the user. Unlike known systems that require dedicated hardware such as specialized wristbands or external microphones, the present invention eliminates the need for any additional devices, making it more accessible and easier to use.

[0026] Operational Flow:

[0027] Sound Capture: The smartphone's microphone is activated to continuously capture surrounding sounds.

[0028] Sound Transmission: The captured sounds are transmitted to the Al processor for real-time analysis.

[0029] Sound Analysis and Interpretation: The Al processor analyzes the sounds using an Al model, generating textual interpretations.

[0030] Textual Transmission: The textual interpretations are transmitted back to the smartphone.

[0031] Display and Notification: The smartphone displays the interpretations on a connected smartwatch, which also activates a vibration alert to notify the user.

[0032] Advantages:

[0033] Cost-Effective: By using the user’s existing smartphone and smartwatch, the invention avoids the need for additional, expensive hardware.

[0034] User-Friendly: The system integrates seamlessly with devices that users are already familiar with, making it easy to adopt.

[0035] Real-Time Processing: The Al processor performs real-time analysis and interpretation, ensuring that users receive timely information about their environment. Accessibility: The elimination of dedicated hardware makes the system more accessible to a wider range of users, particularly those who may not be able to afford specialized devices.

[0036] Textual interpretations: known systems and method provides transcription of the sound into text, while the present invention provides deep interpretations.

[0037] The present invention discloses a method for generating and providing real-time textual interpretations of surrounding sounds that includes the following steps: Utilizing the existing smartphone and smartwatch; Activating the microphone of the smartphone to continuously capture surrounding sounds; Transmitting, by the smartphone, the captured sounds to an Al processor located on the smartphone or a remote server; Analyzing, by the Al processor, the captured sounds using an artificial intelligence (Al) model to identify and interpret the sounds in real time; Generating, by the Al processor, textual interpretations of the identified sounds; Transmitting, by the Al processor, the textual interpretations to the smartphone; Displaying, by the smartphone, the textual interpretations on a connected smartwatch; and Activating, by the smartwatch, a vibration alert to notify the user upon receipt of the textual interpretations.

[0038] The present invention also discloses the system for generating and providing the realtime textual interpretations of the surrounding sounds. The system includes:

[0039] The smartphone that is configured to activate the microphone to continuously capture surrounding sounds, to transmit the captured sounds to the Al processor located on the smartphone or the remote server, to receive the textual interpretations generated by the Al processor, and to display the textual interpretations on the connected smartwatch.

[0040] The Al processor that is configured to analyze the captured sounds using the artificial intelligence (Al) model to identify and interpret the sounds in real time, and to generate the textual interpretations of the identified sounds.

[0041] The smartwatch that is connected to the smartphone and configured to receive the textual interpretations from the smartphone, to display the textual interpretations on its display, and to activate a vibration alert to notify the user upon receipt of the textual interpretations.

[0042] The system and method are designed not only for individuals who are deaf or hard of hearing but also for those in noisy environments. A primary objective of this invention is to provide the user with textual interpretations of various sounds or events occurring in their surroundings. These interpretations may also include drawings, illustrations, and other visual symbols, as the term "textual” interpretations in this disclosure and in the claims encompass all such visual representations. The system ensures that individuals with hearing difficulties can understand what is happening around them in real time and in a sequential manner. Connection to an External Network:

[0043] The system operates in two modes: a limited mode, which functions without Internet and / or Bluetooth with restricted capabilities, and a full mode, which operates with Internet and Bluetooth, offering full functionality. The underlying assumption is that an Al processor on a remote network can perform more efficiently than one on the smartphone.

[0044] Sensory Indications:

[0045] The system is designed to provide distinct sensory indications for different types of sounds using various means, such as varying vibration patterns or visual cues. For example, different colors might be used for different alerts: red for very dangerous situations, orange for imminent danger, and yellow for general attention. These examples are illustrative and not exhaustive, meant to clarify the concept.

[0046] Sound and Event Detection:

[0047] The system primarily receives information through the smartphone's microphone but can also integrate data from other sources, such as online reports about nearby events. The indications can relate to a wide range of situations, not necessarily limited to emergencies. In emergencies, the system can provide textual interpretations for situations like war alarms, instructions to seek shelter from missile attacks, alerts about hazardous materials, tsunami, earthquake, or hurricane warnings, fire alarms, and more.

[0048] For everyday scenarios, the system might include alerts such as:

[0049] A person is stepping towards you.

[0050] Between 3 to 4 People are talking about yesterday's basketball final-four, would you like to follow their conversation?

[0051] Car horns from behind.

[0052] Police, ambulance, or fire truck sirens.

[0053] Doorbell or intercom chimes at home.

[0054] The ringing of a microwave oven, cooking timer, or stove alarm.

[0055] A baby's cry detector.

[0056] An alert for a dripping faucet when a deaf person forgets to turn it off. Vibrations indicating a phone call. The sound of a dog barking.

[0057] This system is designed to ensure that users, whether they have hearing impairments or are in a noisy environment, can receive real-time, contextual notifications of their surroundings, enhancing their awareness and safety.

[0058] The system and the method may work optimally when the smartphone has an active internet connection with the remote server to enable the analysis of the captured sounds. However, the system may work also without an internet connection by using the existing processor on the smartphone itself that may include a computer program for transcribing the sounds into textual transcription or description or poor Al abilities that may be installed on those kinds of existing processors on standard smartphones, to serve war- situation, or serving an emergency situation during which an internet connection is not available.

[0059] Using the smartwatch to provide the alerts and the textual interpretations, descriptions or transcriptions is important for deaf and people with hearing difficulties, due to the fact that the smartwatch is always on them and vibrates on their wrist, while a smartphone is not always on them (so they will not feel anything), and even when it is on them it is most likely in their pocket where vibrating may be damped and therefore be missed.

[0060] The system may use several kinds of inputs: Sounds captured by the smartphone's microphone, information collected from the Internet (such as connectivity to governmental alert internet notification or other services / information available on the internet), Radio waves, satellite communication, Cellular network (such as alerts by SMS or otherwise), and as also Cell Broadcast or Wireless Emergency Alerts (WEA). The last option, WEA, allows notifications and information to be sent to mobile devices without requiring a traditional cellular network connection for each individual phone. Instead, it uses a one-to-many broadcast model. This system can push alerts and messages to smartphones within a specific geographic area, even when cellular networks are congested or unavailable, as long as the device is within reach of a broadcasting tower. It may be a regular software processing of sounds, and may be not an Al processing.

[0061] In summary, the present invention discloses a system and a method for generating and providing in real-time ongoing textual interpretations or textual transcription and description of surrounding sounds. The system and the method include one or more of the following means and steps: Utilizing an existing smartphone (11) and smartwatch (14), activating the microphone of the smartphone to continuously capture surrounding sounds, transmitting, by the smartphone, the captured sounds to the Al processor located on the remote server or feeding the existing processor (12) on the smartphone, analyzing, by the Al processor or by the existing processor, the captured sounds using an artificial intelligence (Al) model or a computer program to identify and interpret , describe, or transcribe the sounds in real time; generating, by the Al processor or by the existing processor, textual interpretations or textual transcriptions or description of the identified sounds, transmitting or feeding, by the Al processor or by the existing processor, the textual interpretations or description to the smartphone, displaying, by the smartphone, the textual interpretations or the textual transcription or description on the smartwatch connected to the smartphone, and activating, by the smartwatch, a vibration alert to notify the user upon receipt of textual interpretations or textual transcription or description.

[0062] Figure 1 depicts the system (10) that includes the smartphone (11) with an existing processor (12) and microphone (16), a remote server (15) with an Al processor (13), and a smartwatch (14).

Claims

ClaimsWhat is claimed is:

1. A method for generating and providing in real-time ongoing textual interpretations or textual transcription or description of surrounding sounds, comprising: utilizing an existing smartphone and smartwatch, activating a microphone of the smartphone to continuously capture surrounding sounds; transmitting, by the smartphone, the captured sounds to an Al processor located on a remote server or feeding an existing processor on the smartphone; analyzing, by the Al processor or by the existing processor, the captured sounds using an artificial intelligence (Al) model or a computer program to identify and interpret , describe, or transcribe the sounds in real time; generating, by the Al processor or by the existing processor, textual interpretations or textual transcriptions or description of the identified sounds; transmitting or feeding, by the Al processor or by the existing processor, the textual interpretations or description to the smartphone; displaying, by the smartphone, the textual interpretations or the textual transcription or description on the smartwatch connected to the smartphone ; and activating, by the smartwatch, a vibration alert to notify the user upon receipt of textual interpretations or textual transcription or description.

2. A system for generating and providing real-time textual interpretations or textual transcription or description of surrounding sounds, comprising:(a) a smartphone configured to: activate a microphone to continuously capture surrounding sounds; transmit the captured sounds to an Al processor located on a remote server or to an existing processor on the smartphone;receive textual interpretations or textual transcription or description generated by the Al processor or the existing processor; and display the textual interpretations or the textual transcription or description on a connected smartwatch.(b) an Al processor or an existing processor configured to: analyze the captured sounds using an artificial intelligence (Al) model to identify and interpret the sounds in real-time or using a computer program for transcribing the captured sounds; and generate textual interpretations or textual transcription or description of the identified sounds.(c) a smartwatch connected to the smartphone and configured to: receive the textual interpretations or the textual transcription or description from the smartphone; display the textual interpretations or the textual transcription or description on its display; and activate a vibration alert to notify the user upon receipt of the textual interpretations or the textual transcription or description.

Images