System

Abstract

To eliminate the need of a person with hearing disabilities to rely on sign language or writing for smooth communication.SOLUTION: A system that assists communication of a person with hearing disabilities, includes: voice recognition means which acquires voice information which is voice of a speaker through voice recognition, and converts the voice information into text data; learning means which integrates vibration data of a bone-conduction microphone with video data from a camera embedded in a smartphone to learn correlation between the voice and movement of the mouth of the speaker; and a loudspeaker that cooperates with the smartphone to emit synthetic voice corresponding to the text data recognized by the voice recognition means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to eliminate the need for hearing-impaired people to rely on sign language or writing to communicate smoothly. [Means for solving the problem]

[0005] A system for assisting communication for the hearing impaired, comprising: a voice recognition means for acquiring audio information, which is a speaker's voice, through voice recognition and converting the audio information into text data; a learning means for integrating vibration data from a bone conduction microphone and video data from a smartphone's built-in camera to learn the correlation between the speaker's mouth movements and voice; and a speaker that works in conjunction with the smartphone to play a synthesized voice corresponding to the text data recognized by the voice recognition means. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0015] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0016] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0017] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0018] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicative of the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicative of the user input.

[0019] The output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0020] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0021] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0022] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0023] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0024] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0025] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a ChatGPT server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, or a home appliance) owned by a user. Next, the identification process by the identification processing unit 290 of the data processing device 12 will be described.

[0026] (Example 1) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by the vibration of a bone conduction microphone or the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the other party's voice is acquired using the microphone of the smartphone and converted into text using voice recognition technology. In addition, the movement of the mouth is detected by the vibration of the bone conduction microphone or the built-in camera of the smartphone, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the content of the other party's speech is provided as a voice to the hearing-impaired person from the speaker for playing the synthetic voice in cooperation with the smartphone. The acquisition means is realized by acquiring the voice of the other party using the microphone 38B of the smart device 14, and transmitting the voice data to the data processing device 12 by the control unit 46A. The artificial intelligence is a learning means realized by the specific processing unit 290 of the data processing device 12, and analyzes data from the camera 42 and the bone conduction microphone of the smart device 14. The speaker is realized as the output device 40B of the smart device 14. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means can use an advanced voice recognition algorithm to process voice data acquired from a microphone built into the smart device in real time and convert it into text data. The voice recognition means also has a keyword spotting function that analyzes voice data and identifies specific keywords and phrases, and can also instantly extract important information. The voice recognition means also has a text conversion function that converts the voice acquired from the microphone of the smart device into a digital signal and converts the digital signal into text data, and this text conversion can eliminate ambient noise by a voice separation function and extract only clear voice signals. Furthermore, it includes a semantic analysis function for identifying words and phrases with important meanings from the voice data, which makes it possible to understand the context of the dialogue and instantly capture necessary information. The learning means integrates vibration data from the bone conduction microphone and video data from the smart device's built-in camera to learn the speaker's mouth movements and pronunciation characteristics. The learning means can also fuse the vibration patterns from the bone conduction microphone with the visual information from the smart device's camera. This fusion process learns the correlation between the speaker's facial movements and voice, and more accurately interprets the content of the voice by using lip reading as an aid even when the voice signal is unclear. This process includes a face tracking function that tracks facial movements and a vibration analysis function that analyzes vibration patterns, which deepen learning based on data collected from multiple speakers. The speaker conveys the contents of the other person's speech to the hearing-impaired person by generating and outputting synthetic speech. The speaker also has an output function that generates synthetic speech through a speech synthesis process and conveys the speech to the user. The speech synthesis process includes a waveform generation function that synthesizes speech waveforms based on text data, and an articulation function that adjusts the articulation of the synthesized speech. This converts the text data into natural speech that resembles a human voice and provides it in a form that can be easily understood by the hearing-impaired person. A manual data collection method includes a function in which a user inputs text information using a touch screen of a smart device, and a speech synthesis means generates speech based on the text. An example of data collection that does not include a sensor is when a user directly inputs text via a touch screen of a smart device, and the text data is output as speech by a speech synthesis means. This method is suitable for users who use sign language as an alternative to speech recognition, and helps them communicate. This method is also effective in situations where speech data is not available or when you do not want to rely on speech recognition. The text information entered by the user is converted into a clear and easy-to-understand speech message by a speech synthesis process. It is possible to expand the functions of the speech recognition means and the learning means by artificial intelligence in the present invention. For example, the speech recognition means can be added with a speaker identification function that identifies the characteristics of an individual's voice and determines who is speaking. This makes it possible to accurately grasp the content of each speaker's speech even in a situation where multiple people are talking, and to provide clearer communication to hearing-impaired people. Furthermore, by incorporating an emotion recognition algorithm into the speech recognition means, it is possible to capture the emotional nuances of the speech and reflect the emotional expression in the synthetic speech. The learning means can be combined with face recognition technology to learn real-time changes in mouth movements and subtle changes in facial expressions to achieve more accurate lip reading. This makes it easier to accurately interpret the speaker's intention even when the audio signal is unclear. The learning means also has a multilingual learning function to accommodate different languages ​​and dialects, making it effective in international communication situations. The speaker has an automatic volume adjustment function that adjusts the volume according to the user's hearing level, and provides optimal audio output according to the usage situation of the hearing impaired.The speaker also has an environmental adaptation function that automatically adjusts the clarity of the sound according to the surrounding noise level, and always provides the best hearing experience while responding to changes in the external environment. Manual input data collection methods may include predictive text entry to allow users to easily edit selected text and custom dictionary features for technical terms and abbreviations that are difficult to recognize through speech, allowing users to enter information more quickly and accurately, resulting in more accurate speech. In addition, the system can be used not only by the hearing impaired, but also as a multilingual interpretation device, using voice recognition to translate foreign language conversations in real time, thereby reducing communication barriers between people who speak different languages ​​and promoting international exchange. It can also be incorporated with the ability to provide real-time subtitles during video calls and online meetings with remote locations, helping the hearing impaired to participate more actively in social activities. The voice recognition means of this embodiment further includes an environmental sound recognition function, which detects important sounds other than conversation, such as emergency vehicle sirens and fire alarms, and provides them to the hearing impaired as visual alerts or vibration signals. This improves safety and enables rapid response to dangerous situations. In addition, the voice recognition means incorporates a function that automatically measures the surrounding noise level and optimizes the voice recognition accuracy based on that information, thereby achieving highly reliable recognition even in noisy environments. The AI ​​learning method can learn from a user's usage and speech habits to provide more personalized lip reading assistance. For example, it can recognize patterns of words and phrases commonly used by a particular user and improve the accuracy of lip reading based on that. It can also detect a user's emotional state and stress level and provide feedback to improve the quality of communication. The speaker can add richer speech expressions to the synthesized speech to make it easier for the hearing impaired to understand as a more natural speech. For example, it can adjust the tone and intonation of the speech to reflect the speaker's emotion or emphasis. It can also add a personalization feature that allows users to record their own voice and incorporate that voice quality into the synthesized speech. For manual data collection methods, incorporating gesture control and eye-tracking technology will make text entry easier for users with disabilities or in situations where touchscreen operation is difficult, and providing audio feedback during text entry will make it easier to correct input errors and further improve communication efficiency. These functions are designed to respond to various scenarios that users face, making it easier for the hearing impaired to become more independent and participate in society. They also function as support tools to reduce communication barriers in the daily lives of the hearing impaired and facilitate smooth communication with people with normal hearing. This will enable the hearing impaired to live a richer social life and contribute to realizing an open communication environment for all. (Example 2) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by vibration of a bone conduction microphone or the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the movement of the other party's mouth is detected in real time using the built-in camera of the smartphone, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the voice of the other party is converted into text using voice recognition technology and displayed on the smartphone screen. The acquisition means is realized by detecting the movement of the other person's mouth using the camera 42 of the smart device 14, and transmitting image data to the data processing device 12 by the control unit 46A. The artificial intelligence is a learning means realized by the specific processing unit 290 of the data processing device 12, and analyzes the voice data from the microphone 38B of the smart device 14. The speaker is realized as the output device 40B of the smart device 14. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means includes a function for capturing and analyzing voice signals using advanced voice processing algorithms. This means combines a noise canceling function for reducing environmental noise with a speaker identification function for identifying the speaker's voice to accurately capture the other party's voice. The voice recognition means also includes a highly sensitive microphone for capturing the speaker's voice and a voice analysis processor for analyzing the captured voice signal. This means can utilize echo cancellation and beamforming techniques to achieve high recognition accuracy even in complex acoustic environments. The voice recognition means can also analyze the captured signal and use it as data for converting the spoken content into text. The acquisition method is to detect mouth movements using the smartphone's built-in camera, and also to acquire vibrations using a bone conduction microphone. The built-in camera detects mouth movements using high-speed, highly accurate image processing technology and analyzes the captured video data. This analysis combines face recognition technology and lip reading technology to predict the content of the other person's speech from their mouth movements. The bone conduction microphone uses a vibration sensor to detect the speaker's jaw movements and vocal cord vibrations, and can acquire these signals as voice data. The speaker as an output means outputs synthetic speech generated through a speech synthesis engine. The speech synthesis engine includes a prosody generation function that adjusts the pitch and speed of the speech in order to convert text information into natural speech. The pronunciation model can also generate pronunciation patterns for various words and phrases so that the synthetic speech sounds closer to the voice of an actual speaker. The AI ​​uses deep learning networks and statistical learning models as a learning vehicle to extract speech features from large amounts of audio and visual data. This learning is used to identify the acoustic characteristics of a particular language or dialect, aiming to accurately recognize and understand the speech. The learning process uses approaches such as supervised learning and reinforcement learning, and new data is incorporated to continuously improve the model's performance. The AI ​​learning vehicle is also trained on samples taken from an extensive audio database to extract features from the audio data and gain the ability to identify a particular language or dialect, as well as understand the characteristics of an individual user's speaking style and accent. The learning vehicle can update the model using feedback data for continuous performance improvement. The manual input data collection method provides an interface for users to directly input text information. Users input information using the touch screen keyboard or voice input function of their smartphone, which is then received by the system. This input data is processed like any other speech recognition result and can be used for text display or voice output. To expand the functionality of the acquisition means, an acceleration sensor that analyzes vibrations can be incorporated in addition to the bone conduction microphone. This allows the speaker's subtle head and body movements to be detected, and information related to speech to be acquired from a wider range of angles. Furthermore, by using a wearable device to capture hand movements and changes in facial expression, additional communication information can be collected from gestures and facial expressions, helping to understand the content of speech. In addition, by using the built-in camera of a smartphone, it is possible to track the speaker's eye movements and blinks to infer attention and intent, enhancing contextual awareness using visual information and enabling more natural and intuitive interactions. To improve the speaker's functionality, a function can be added that automatically adjusts the frequency and volume of the sound to suit the user's hearing characteristics, providing optimal audibility according to environmental noise and the user's hearing characteristics. The AI ​​learning method will deeply analyze the correlation between voice data and visual data to more accurately understand non-verbal communication elements, and multi-modal learning will combine both voice and visual information for complex learning to improve the accuracy and naturalness of the dialogue system. For manual input, we will utilize the smartphone's motion sensors and gesture recognition to provide input methods other than keyboard input. For example, we will build an input system using air gestures and eye tracking to allow users to provide data to the system more intuitively. By combining these features, users can communicate not only what they say, but also non-verbal information such as gestures and facial expressions to the system, and the system can synthesize this information to achieve more accurate communication. These additional features can also be customized according to specific situations and user needs, providing the flexibility to accommodate a variety of usage scenarios. In terms of acquisition methods, more advanced voice separation technology will be introduced to selectively separate the speaker's voice from surrounding noise and other voices to acquire clearer voice data. In addition, it will have the ability to analyze environmental sounds in real time and understand the atmosphere and situation of the place. This will improve the accuracy of voice recognition and enrich the information obtained from voice data. The speaker, which serves as the output means, has the ability to learn the user's hearing characteristics and automatically adjust the sound quality accordingly. This allows for audio that is easy for each user to hear, making communication more comfortable. In addition, by incorporating spatial audio technology, audio can be heard from the direction of the speaker, providing a more natural conversation experience. The AI ​​is trained with an extensive voice database to handle different languages ​​and accents, making it suitable for global communication. It also learns the speaking style of each individual user to enable personalized dialogue. In addition, emotion recognition technology is introduced to identify the speaker's emotions and respond accordingly, making dialogue more human-like. Manual input data collection adds the ability to recognize a user's handwriting or sketches and convert them into text and graphical information, allowing users to provide information to the system through natural writing. These extensions enable the system to grasp not only the user's speech but also their emotions and intentions more accurately, and to gain a deeper understanding of the user's experience. These features also increase the value of the system as an accessible communication tool for users with special needs, such as those with physical or speech disabilities. This allows a wider range of users to enjoy communication with society. (Example 3) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the vibration of a bone conduction microphone and the movement of the mouth by the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the bone conduction microphone of the smartphone is used to detect the vibration of the other party's voice, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the content of the other party's speech is provided as a voice to the hearing impaired person from the speaker for playing the synthetic voice in cooperation with the smartphone. The acquisition means is realized by detecting vibrations of the other person's voice using a bone conduction microphone of the smart device 14, and transmitting the vibration data to the data processing device 12 by the control unit 46A. The artificial intelligence is a learning means realized by the specific processing unit 290 of the data processing device 12, and analyzes image data from the built-in camera 42 of the smart device 14. The speaker is realized as the output device 40B of the smart device 14. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means has a function of removing environmental sounds and surrounding noises using an advanced voice processing algorithm and extracting only the other party's voice with high accuracy. The voice recognition means also includes an automatic voice recognition system that converts a voice signal into text, and this system is trained using a deep learning model and has the flexibility to accommodate various accents and speaking styles. The voice recognition means also removes background noise through an environmental noise reduction function and highlights the other party's voice using a target voice extraction function. The extracted voice signal is analyzed for frequency characteristics and amplitude fluctuations through a waveform analysis function, and is converted into features reflecting the speaker's voice quality and speech characteristics through a voice feature extraction function. This information is transmitted to a data processing device via a data transfer function. The automatic voice recognition system also includes a data-driven learning process that utilizes a learning model based on a deep neural network to adapt to various speech patterns and accents. The vibration detection means using a bone conduction microphone detects the vibration of the skull that occurs when speaking and converts it into an electrical signal. The vibration detection means using a bone conduction microphone also analyzes the characteristics of the vibration transmitted through the skull and converts it into a digital signal using an analog-to-digital conversion function. The vibration is analyzed to extract features related to the content of the speech through the vibration pattern analysis means. Meanwhile, the mouth movement detection means using a built-in camera analyzes the mouth movements captured by the camera using image processing technology and tracks the shape and movement of the mouth during speaking. The mouth movement detection means also precisely analyzes the mouth movements captured by the video image processing function using a mouth shape analysis function and a movement tracking function. These data are fused by the integrated analysis function to perform a comprehensive speech analysis that combines voice recognition and visual information. The AI ​​extracts features related to the speech content based on the vibration data and lip movement data, and compares them with a learning database using a matching means. The matching means utilizes a database containing a large amount of speech data and determines the speech content using a machine learning algorithm. The AI ​​also compares the features of the vibration data and lip movement data with an existing database through a database matching function, and identifies the speech content using a pattern matching function. In this process, a data fusion function with voice recognition data is used to improve the accuracy of speech recognition. The matching results are interpreted by the speech understanding function and used to provide appropriate responses and information. The voice output means by the speaker provides the speech content by synthetic voice to the hearing impaired. The voice output means by the speaker generates a natural voice that reflects the speaker's intention through a synthetic voice generation function, and converts this voice signal into a reproducible analog signal through a digital-to-analog conversion function. The voice synthesis means generates a voice having natural pronunciation and intonation according to the speech content, and outputs this from the speaker. The voice synthesis means uses text-to-speech (TTS) technology that converts text to speech, and has a built-in multilingual model that supports various languages ​​and dialects. The voice synthesis means generates a voice having pronunciation and intonation that supports different languages ​​and dialects through a multilingual text-to-speech function. An example of data collection that does not include sensors would be a user manually inputting speech via a keyboard or touch screen. This manual input method is particularly useful in noisy environments or when the speaker does not have access to a microphone. The manual user input method may also include a function where the user directly provides text information through an input interface, and the manually input information is subjected to analysis by the artificial intelligence via a text processing function. The manually input text data is similarly processed by the artificial intelligence and output as synthetic speech. The manually input text is also processed through a text analysis function and fed into a speech synthesis process for output as synthetic speech. For example, various functions can be added to each of the means of this embodiment. For example, the acquisition means can be added with a superdirectional microphone for instantaneous voice detection and analysis, thereby selectively acquiring only a specific person's voice and separating it from surrounding noise. The acquisition means can also incorporate a function for monitoring text-based communication from social media and messaging apps in real time and reproducing it as voice utterances. This allows the hearing impaired to directly receive online conversations as voice. The speech recognition means can utilize a multilingual model to handle multiple languages ​​and dialects, and can understand the speech of users with different cultural backgrounds. Furthermore, the speech recognition means can be equipped with an emotion analysis function to capture subtle changes in the speaker's voice in order to more accurately grasp the speaker's emotions and intentions. This makes it possible to reflect the speaker's emotions in the voice, rather than simply reading out the text, resulting in more human-like communication. The artificial intelligence means has the capability of continuous learning and self-improvement, and analyzes the speaker's speech characteristics and lip movement patterns more accurately over time. The artificial intelligence means can also be customized to the user's usage and preferences to provide the most natural and understandable speech analysis for each individual user. Such customizable AI can respond to the individual needs of each user, improving usability. Emotion recognition means can provide richer emotional information by adding a function to analyze not only the tone of a speaker's voice, but also subtle changes in facial expressions and body language. For example, it is possible to estimate the speaker's intention and the intensity of his or her emotions from the speaker's eye movements and hand gestures, achieving more accurate emotion recognition. Output means can be more diverse, using not only speakers, but also vibration and light signals to provide information to the hearing impaired. For example, smart devices can warn with vibration patterns or color changes depending on the speaker's emotional state and the urgency of what is being said. Information transmission systems can also be developed that use tactile information rather than synthetic voice to accommodate the visually impaired. These additional features can be implemented through hardware and software updates to existing smart devices, significantly improving the user experience. They can also be linked to cloud-based services to facilitate remote updates and customization, providing users with the latest technology at all times. Furthermore, these features can be widely applied as communication support tools in education, work, and daily life, and are expected to benefit a diverse range of users, including not only the hearing impaired, but also those with speech disabilities and non-native speakers. To further improve the acquisition method, an external noise canceling function can be added to collect clearer voice data. In addition, a real-time language conversion function can be introduced to facilitate communication with people who speak different languages. This will further strengthen the AI's ability to understand multiple languages ​​and expand its use in international communication. The AI ​​means will be further developed to have the ability to memorize what the user has said in the past and learn the individual's unique speaking style, which will enable the AI ​​to reproduce the user's speech more naturally and improve the quality of communication. It will also have an emotion-adaptive synthetic voice generation function that can read the user's emotions and respond with an appropriate tone. The speaker-based audio output means can be combined with a bone conduction headset or a vibrating wearable device so that the hearing impaired can experience sound more naturally. This allows the hearing impaired to intuitively grasp the rhythm and strength of the speaker's voice through the vibrations, enabling them to enjoy a richer communication experience. In addition, by adding customization features that allow users to adjust the volume and speed of audio output depending on the situation, it can be used to meet individual needs. It is also possible to incorporate a safety measure function that automatically increases the volume in an emergency. By integrating these functions, devices can be provided that are easier to use and more effective for the hearing impaired, as well as the elderly and other users who need to communicate in a variety of situations. Furthermore, these technologies are expected to be applied as educational support tools, remote communication, and multilingual dialogue systems, helping to promote social inclusion.

[0027] The process flow of each embodiment will be described below.

[0028] (Example 1) Step 1: Use your smartphone's microphone to pick up the other person's voice. Step 2: Use voice recognition technology to convert the other person's voice into text. Step 3: Mouth movements are detected using vibrations from the bone conduction microphone and the smartphone's built-in camera, and the content of what the other person is saying is understood by comparing this with data that the artificial intelligence has learned. Step 4: The speaker, which works in conjunction with a smartphone to play synthesized speech, provides the hearing-impaired person with audio of what the other person is saying. (Example 2) Step 1: Use the smartphone's built-in camera to detect the other person's mouth movements in real time. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. (Example 3) Step 1: The smartphone's bone conduction microphone is used to detect the vibrations of the other person's voice. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: The speaker, which works in conjunction with a smartphone to play synthetic speech, provides the hearing-impaired person with audio of what the other person is saying.

[0029] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0030] (Example 1) The embodiment of the present invention includes an acquisition means for acquiring the voice of the other party by a voice recognition means, an artificial intelligence for learning the movement of the mouth by the vibration of a bone conduction microphone and the built-in camera of a smartphone, a speaker for playing synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotions. Specifically, the other party's voice is acquired using the microphone of the smartphone and converted into text using voice recognition technology. In addition, the vibration of the bone conduction microphone and the built-in camera of the smartphone are detected, and the movement of the mouth is compared with the data learned by the artificial intelligence to understand the content of the other party's speech. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the speaker for playing synthetic voice in cooperation with the smartphone provides the other party's speech content and emotional state as voice to the hearing impaired person. The acquisition means is realized by acquiring the other person's voice using the microphone 38B of the smart device 14 and transmitting the voice data to the data processing device 12 by the control unit 46A. The artificial intelligence is realized by the specific processing unit 290 of the data processing device 12. The speaker is realized by the output device 40B of the smart device 14. The emotion engine is realized by the specific processing unit 290 of the data processing device 12. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The acquisition means has a function of accurately acquiring the voice of the other party as digital data by utilizing voice recognition technology. For example, the acquisition means acquires clear voice data while eliminating surrounding noise by using a high-sensitivity microphone equipped with a noise cancellation function. The acquisition means can also acquire the speech of the other party as digital voice data by using the voice recognition means via a microphone on the smart device. This voice data can clearly capture the human voice by the high-sensitivity microphone while reducing environmental noise by a noise canceling function. The voice recognition means performs processing to convert the spoken content into text data based on the acquired voice data, thereby achieving highly accurate text conversion. This process includes feature extraction from the voice signal, segmentation of utterance units, and recognition processing based on a voice model. The voice recognition means can also identify the speaker's voiceprint and speech pattern using a characteristic analysis function of the voice signal, and perform preprocessing to improve the accuracy of voice recognition. The acquired voice data can be subjected to voice-to-text conversion processing by extracting features of the voice data, performing segmentation, and converting the voice into text. The artificial intelligence means includes analyzing vibration patterns of a bone conduction microphone and capturing and analyzing mouth movements using a built-in camera. The means performs processing to infer the speaker's speech content from the acquired vibration data and video data. The artificial intelligence means can also capture the speaker's mouth movements and head vibrations using the bone conduction microphone and the built-in camera. The artificial intelligence means can analyze these data and understand the speaker's speech content. The bone conduction data analysis means analyzes the head vibrations captured by the bone conduction microphone and associates them with the voice to understand the speech content. The mouth movement analysis means can use video data captured by the smartphone camera to track the speaker's mouth movements and associate them with the speech content. These means are trained using machine learning techniques to learn the speaker's individual speech characteristics and achieve high-precision speech understanding in real time. The emotion recognition means has a function of estimating the emotional state of a speaker by using both voice data and video data. This means includes a voice emotion analysis means based on the analysis of the tone and intonation of the voice, and a facial expression analysis means that captures changes in facial expressions. The voice emotion analysis means analyzes the pitch, intensity, acoustic characteristics, etc. of the voice data to estimate the emotional state of the speaker. The emotion engine also has both a voice emotion analysis function and a facial expression analysis function, and can estimate the emotional state of the speaker. The facial expression analysis means detects the facial expressions of the speaker from the video data captured by the built-in camera, and estimates the emotional state by capturing subtle changes in facial expressions. These means are trained using an emotion recognition model, and by identifying emotional patterns, grasp the speaker's emotions in real time. The output means includes a speaker that cooperates with the smartphone to generate synthetic voice and provide it to the user. The means includes a text-to-speech (TTS) function that converts text to voice, and the tone and intonation of the synthesized voice are adjusted to reflect the speaker's spoken content and emotions. Furthermore, the output voice has clear pronunciation and an appropriate speed so that the hearing impaired can easily understand it. The output means through the speaker can also generate synthetic voice and provide the hearing impaired with the other party's spoken content and emotional state by voice. The voice flowing from the speaker is designed so that the user can easily grasp the other party's spoken content and emotions. This allows the user to grasp the other party's words not only by hearing but also by combining them with visual information. As an extension of the functions related to the acquisition means, it is possible to selectively pick up the voice of a specific speaker by using a directional microphone that is resistant to external noise. In addition, by utilizing a combination of a smart device and a wearable device, the user's body movements and gestures can be acquired to assist in understanding non-verbal communication. Furthermore, it is conceivable to utilize beamforming technology through the acquisition means to identify individual speakers even in situations where multiple people are talking and accurately grasp the content of each speaker's speech. In order to improve the accuracy of lip reading, the artificial intelligence means uses a high-resolution camera to capture more detailed mouth movements. In addition, by developing an algorithm that analyzes the intention of communication not only from the speaker's mouth movements but also from facial expressions and body language, a richer understanding of communication is realized. The artificial intelligence means can build a lip reading model that supports multiple languages ​​and can also handle international communication. Regarding emotion recognition, more accurate emotion recognition can be achieved by incorporating a biofeedback sensor that obtains physiological indicators such as heart rate and skin electrical conductivity and uses them to estimate the emotional state. In addition, a customized emotion recognition model can be built by learning an individual's emotional patterns, providing emotion recognition adapted to each individual user. The emotion recognition means can also capture subtle differences in emotions expressed by users in different contexts and social situations. As for the output means, a personalized audio system that adjusts the frequency band according to the user's hearing characteristics can be introduced, making it possible to provide audio tailored to each individual hearing-impaired person. In addition, by incorporating a function into a smartphone application that uses visual information to display the other person's speech in text in real time, it becomes possible to obtain information from both audio and vision. Furthermore, by combining synthetic voice with an expressive animated avatar, visual communication can be enhanced and more intuitive understanding can be supported. The acquisition method can be supplemented with an active noise canceling function that analyzes the sound environment around the user in real time and performs optimal noise reduction. In addition, by utilizing bone conduction earphone technology that allows users to hear sound without blocking their ears, it is possible to provide users with clear sound while still being aware of the surrounding environmental sounds. Furthermore, by linking with devices other than smartphones, such as smart watches and smart glasses, voice data can be acquired using multiple devices, creating a more comprehensive communication environment. Regarding artificial intelligence means, multitask learning will be introduced to simultaneously analyze not only lip reading but also non-verbal information such as the speaker's posture and hand movements to gain a deeper understanding of the context of communication. In addition, by using edge computing technology to perform part of the data processing on the smart device side, it is possible to reduce communication delays and improve real-time performance. Furthermore, a function will be developed that uses artificial intelligence to automatically identify and translate the language used by the speaker, and to support communication between different languages. The emotion recognition method combines machine vision technology to capture subtle changes in facial expressions and skin color, improving the accuracy of emotion analysis. In addition, bio-signals obtained from wearable devices are used for emotion recognition to more accurately evaluate the user's psychological state, such as stress level and fatigue. In addition, situation recognition technology is introduced to adjust the emotion recognition algorithm according to the user's environment and situation, allowing for more appropriate emotion estimation. As for output means, a visual assist function will be added that adjusts the size and contrast of text display to suit the user's visual acuity characteristics. In addition, an avatar-based sign language translation system will be incorporated that recognizes and generates sign language and gestures necessary for the hearing impaired to communicate with others. An emotional expression assist function will also be developed that allows users to convey their emotional state to others, realizing rich communication that includes emotions. Through these additional functions, the invention can provide a more comfortable and effective means of communication not only for the hearing impaired but also for users with various communication disabilities. In addition, by combining these functions, it is possible to reduce misunderstandings that occur during communication and realize smoother dialogue. (Example 2) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by vibration of a bone conduction microphone and the built-in camera of a smartphone, a speaker for playing a synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotion. Specifically, the smartphone's built-in camera is used to detect the other party's mouth movement in real time, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the other party's voice is converted into text using voice recognition technology and displayed on the smartphone screen. The acquisition means is realized by detecting the movement of the other person's mouth using the camera 42 of the smart device 14 and transmitting image data to the data processing device 12 by the control unit 46A. The artificial intelligence is realized by the specific processing unit 290 of the data processing device 12. The speaker is realized by the output device 40B of the smart device 14. The emotion engine is realized by the specific processing unit 290 of the data processing device 12. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means includes a voice data acquisition function using advanced voice signal processing technology. This function uses a noise suppression algorithm to remove ambient noise and clearly capture the other party's voice. The noise suppression algorithm distinguishes between human voices and background noise through frequency analysis and subtracts noise components in real time. The voice recognition means also uses a machine learning voice recognition model to identify the speaker and the contents of the speech from the voice. This model has the ability to learn large amounts of voice data and identify the voice characteristics and speech patterns of a specific speaker. The voice data acquisition function, which uses voice recognition technology to acquire and analyze voice data obtained from the other party's voice, also utilizes an advanced signal processing algorithm to remove ambient noise and clearly identify the speaker's voice. This function improves the quality of communication by identifying and removing noise components in the voice data. In addition, as a speaker identification function, a model based on machine learning is used to learn the voice characteristics of each speaker and identify who the speaker is from the voice data. The acquisition means includes a mouth movement detection function using the camera of the smart device. This function combines image recognition technology and machine learning algorithms to infer the content of the speech from the shape and movement of the other person's mouth. The detected mouth movement data is sent to a data processing device in real time and analyzed by artificial intelligence. In this analysis process, the data is compared with a trained data set and the speech content is understood through lip reading. The artificial intelligence uses face recognition technology to identify a specific individual, learns the characteristics of the person's mouth movement, and builds a correspondence with the speech content. In addition, the mouth movement detection function uses the camera of the smart device to acquire an image of the other person's face, and uses lip reading technology to infer the speech content from the movement. This function detects the shape and movement of the mouth from the face image, and uses a learned lip reading model to analyze the words spoken by the speaker. In this process, the face recognition function is also utilized to recognize the mouth movement patterns of a specific individual and assist in understanding the speech content. The speaker means converts the text data into a voice having natural pronunciation and intonation using synthetic voice generation technology. This technology can reflect not only the accuracy of pronunciation but also the emotion and intention of the speaker in the text-to-voice conversion. The speaker may be built into the smart device, or may be an external wireless speaker that transmits an audio signal. In either case, an acoustic processing technology is employed to ensure a sound quality and volume that is easy for the user to hear. The speaker means for playing the synthetic voice includes a text-to-voice conversion function, which converts the text data into a voice having natural pronunciation and intonation using synthetic voice generation technology. This technology can reflect the emotion and intention of the speaker in the voice, and depending on the speaker used, a sound quality adjustment function and a volume adjustment function are also used in combination. The emotion recognition means has a function of estimating emotions from audio signals and video data. This function uses a voice emotion analysis algorithm that analyzes changes in voice tone, speech rate, and intonation, and a facial expression recognition algorithm that captures changes in facial expressions and eye movements. The emotion recognition means integrates these data to infer the emotional state of the other person. The emotion estimation result becomes important information in the context of communication and is provided as feedback to help the user understand the emotions of the other person. The emotion recognition means also estimates the emotional state of the speaker by analyzing the audio signals and video data. This means includes a voice emotion analysis algorithm that monitors changes in tone, speech rate, and intonation. It also includes a facial expression recognition algorithm that detects changes in facial expressions and eye movements from the video data. By integrating these pieces of information, the other person's emotional state is estimated in real time and the result is provided to the user. An example of data collection that does not involve sensors is when a user inputs a text message via a keyboard or voice input function of a smart device and transmits the message through a communication means. This method allows the user to communicate with the other party without relying on voice or video. Another example of data collection that does not involve sensors is text-based communication that is manually input by a user. The user inputs his / her own words as text using a keyboard or voice input function of a smart device and transmits the text to the other party. This method allows direct text communication without requiring a voice recognition means or a lip reading means. The voice recognition means, acquisition means, speaker means, and emotion engine in the embodiments of the present invention can further add other functions. For example, the voice recognition means can employ binaural recording technology and reproduce a more natural acoustic space using two microphones positioned at the position of human ears. This allows the user to easily sense the direction and distance of the other person's voice, allowing the user to enjoy a more realistic communication experience. Furthermore, the voice recognition means can be used to realize the operation of smart devices and the control of home automation systems by voice commands. The acquisition means may incorporate a feedback function to assist the user in speaking. The feedback function may provide guidance on mouth movements and pronunciation when the user speaks. Furthermore, the acquisition means may include a noise canceling function to effectively eliminate external noise and improve the accuracy of detecting mouth movements. The speaker means can have a function to automatically adjust the sound quality according to the user's hearing condition and hearing preferences. Advanced digital signal processing technology is utilized to provide the listener with an optimal hearing experience. The speaker means can also be added with a function to link with external smart home devices and in-car systems to expand the reach of the voice wherever the user is. The emotion engine can incorporate sensors to detect the user's physiological reactions and add a function to more accurately estimate the user's emotional state from changes in heart rate and electrical resistance of the skin. Such biofeedback functions can monitor the user's stress level and excitement in real time, deepening the understanding of emotions in communication. With the addition of these functions, the embodiment of the present invention becomes a more advanced communication support tool, enabling users to have richer and more emotionally rich conversations with others. These functions also provide effective support for users with disabilities or when communication is difficult in a particular environment. Furthermore, as smart devices evolve, these functions are expected to be expanded and optimized in the future to accommodate the needs of a variety of users. The voice recognition means of this embodiment can be equipped with a sensitivity enhancement function that accurately captures the contents of speech even when the user whispers or speaks at a very low volume. This function combines a special microphone array with an algorithm that analyzes ambient noise to make the speaker's voice stand out. In addition, a directional pickup function can be provided to selectively capture the voice of a specific speaker in a situation where multiple speakers are speaking at the same time. Acquisition methods can also be enhanced with real-time translation capabilities to instantly translate utterances spoken in different languages ​​into the user's native language. This functionality works with multilingual databases to reduce global communication barriers. Acquisition methods can also employ region-specific models to identify speaker accents and dialects for more accurate transcription and translation. The speaker means may be added with a personalized equalization function to analyze the user's hearing adaptation and provide an optimal hearing experience. This function realizes audio output that is tailored to the individual ear characteristics and hearing preferences of the user. The speaker means may also provide a virtual reality experience that enhances spatial awareness by adding 3D audio effects to the voice guidance. The emotion engine can be equipped with an emotion response function that tracks the user's emotional changes during the conversation and provides supportive and encouraging messages at the right time. The emotion engine can also estimate the user's psychological stress level from their behavior and speech patterns and suggest advice and relaxation techniques to reduce stress. These additional functions will provide users with a more detailed and personalized communication experience. Furthermore, these technologies are expected to be applied in fields such as education, business, and healthcare, contributing to improving the quality of communication and promoting accessibility. In addition, by providing interactive feedback according to the user's behavior and situation, more natural human-machine dialogue will be realized. The evolution of these technologies will promote a deeper level of user engagement in the future, laying the foundation for providing more valuable services to users. (Example 3) The embodiment of the present invention includes an acquisition means for acquiring the voice of the other party by a voice recognition means, an artificial intelligence for learning the vibration of a bone conduction microphone and the movement of the mouth by the built-in camera of a smartphone, a speaker for playing synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotions. Specifically, the bone conduction microphone of the smartphone is used to detect the vibration of the other party's voice, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the speaker for playing synthetic voice in cooperation with the smartphone provides the other party's speech content and emotional state as voice to the hearing impaired person. The acquisition means is realized by acquiring the other person's voice using the microphone 38B of the smart device 14 and transmitting the voice data to the data processing device 12 by the control unit 46A. The artificial intelligence is realized by the specific processing unit 290 of the data processing device 12. The speaker is realized by the output device 40B of the smart device 14. The emotion engine is realized by the specific processing unit 290 of the data processing device 12. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means has a function of acquiring surrounding voice data through a microphone built into the smart device, and the acquired voice data is pre-processed by digital signal processing, such as noise removal and amplification, and converted into clear voice information. The voice recognition means also has a voice acquisition function for capturing voice from the surroundings, and applies a voice pre-processing function to the acquired voice data. This pre-processing includes the removal of environmental noise by a noise reduction function and the optimization of the volume by a gain adjustment function. This makes the voice signal clearer and converts it into a state suitable for subsequent processing. Next, based on this pre-processed voice data, a voice analysis algorithm extracts specific voice features and improves the accuracy of voice recognition. The extracted voice features include the frequency, amplitude, pitch, duration, etc. of the voice, which are important indicators for understanding the characteristics of the speaker's voice and the content of the words. The voice analysis algorithm also extracts features such as the frequency characteristics and temporal patterns of the voice through a feature extraction function, and grasps the meaning of the voice data using a speaker identification function and a speech content analysis function. The AI ​​learns the vibration patterns of speech detected by the bone conduction microphone as a data set and identifies the content of speech. This learning process uses multiple machine learning algorithms including deep learning technology. In particular, convolutional neural networks are suitable for extracting features from image data and are used to analyze mouth movements captured by the built-in camera of a smartphone. The AI ​​integrates both audio data and visual data and performs multimodal information processing, enabling accurate identification and understanding of the content of speech. The AI ​​also combines the analysis function of vibration data from the bone conduction microphone with the visual data analysis function to accurately identify the content of speech. The vibration data analysis function captures the physical characteristics of the voice obtained by bone conduction and compares it with the audio data. The visual data analysis function identifies the patterns of mouth movements obtained by the smartphone camera and integrates and processes this information to enable more advanced recognition from multimodal information. The speaker is built into the smart device and generates synthetic speech to provide audio information to the user in an easy-to-listen format. The synthetic speech is generated using text-to-speech technology and is designed to reproduce not only the content of the speech, but also changes in tone and intonation that express emotions. This enables the hearing impaired to understand not only the content of the speech of the other person, but also its emotional nuances. The speaker also converts text information into speech using a synthetic speech generation function, providing easy-to-listen audio information to the user. The synthetic speech adjustment function mimics the emotional elements of speech, including tone and intonation, to reproduce the richness of emotional expression during communication. The emotion recognition means performs a composite emotion analysis by integrating acoustic features such as tone, pitch, and speed of the voice with visual information such as changes in facial expressions captured by the smartphone camera. In this process, a machine learning model specialized in emotion recognition is used, and the user's emotions can be classified into multiple categories such as positive, negative, surprise, and sadness. As a result, the emotion recognition means estimates the user's emotional state and provides an emotional context in the communication. In addition, the emotion recognition means extracts emotion information from the voice data and visual data using an emotion analysis algorithm. This analysis includes a voice emotion analysis function and a facial expression analysis function, which are integrated to perform a multidimensional analysis of the user's emotional state, and the emotion estimation function understands the user's emotions. This makes it easier for the user to grasp the other person's emotions, improving the quality of communication. In implementing this embodiment, it is useful to consider further functional expansion. For example, it is possible to introduce beamforming technology into the acquisition means, which selectively acquires a specific voice from among surrounding sounds. This makes it possible to clearly capture the speaker's voice even in a noisy environment and improve the accuracy of voice recognition. In addition, a multilingual processing function for handling different languages ​​and dialects can be added to the artificial intelligence to accommodate a wide range of users. It is also possible to reduce communication barriers with foreign language speakers by incorporating a real-time translation function using artificial intelligence. As for the speakers, a function to adjust the frequency characteristics of the sound to suit the user's hearing characteristics will be added, providing even more personalized listening ease. The emotion engine will be able to convey emotional fluctuations to the user in line with the flow of the conversation by capturing emotional changes in real time. In addition, a context-aware function will be added to the emotion engine, which will estimate emotions based on the context of the conversation, achieving more accurate emotion recognition. Furthermore, it is conceivable to incorporate a biosensor function that acquires the user's biometric information and uses that information to more precisely estimate the other person's emotions and state. This will further improve the quality of communication. Also, adding an assistive communication function that learns the user's communication patterns and prompts them to speak at the appropriate time will enable smoother conversations. In addition, collaboration with wearable devices and smart glasses will be strengthened to expand communication support using visual information. By providing visual feedback such as text, emojis, and videos through smart glasses, the hearing impaired will be able to better understand the emotions and intentions of others. The smart glasses will also be equipped with an audio visualization function that visualizes the surrounding environmental sounds, allowing users to visually understand the direction and strength of sounds. This will make it easier for users to understand the surrounding sound environment from visual information. With these functional extensions, this embodiment is expected to greatly improve the communication experience of users, and contribute to the realization of a more inclusive society by providing a customizable communication means that meets the individual needs of each user. In addition to the acquisition means of this embodiment, it is useful to consider a function for capturing bone conduction vibrations more accurately using a wearable device worn near the user's ear. This wearable device is designed to fit the ear and is structured to be less susceptible to external noise. Furthermore, by developing technology to improve the sensitivity of the bone conduction microphone itself, it is possible to detect more subtle vibrations of speech and increase the accuracy of understanding the content of speech. The AI ​​training dataset will include a wide variety of languages ​​and dialects, as well as the voice patterns of speakers with specific disabilities, to accommodate a wider range of users. In addition, when performing real-time speech recognition, it will incorporate context-based predictive modeling to more accurately grasp the intent of the spoken content. This will enable the system to accurately understand the meaning of words in the context of the dialogue. The synthesized voice from the speaker will be equipped with a function that allows the pitch and speed of the voice to be adjusted according to the user's level of hearing impairment and preferences. This will allow the user to hear voices that are easy to understand. In addition, the emotional expressions of the synthesized voice will be able to reflect emotional characteristics extracted from recordings of actual human voices, making it possible to convey emotions more naturally and realistically. The emotion engine will be able to perform more comprehensive emotion recognition by incorporating sensor technology that analyzes not only language but also non-verbal elements such as gestures and body postures. It will also be able to realize more personalized emotion recognition by taking into account the relationship between the user and the other party and past conversation history. To improve the user's communication experience, a smartphone app will be built with an interface that conveys information visually. This interface will include a function that displays voice information as text and a function that expresses emotions with colors and patterns. This will allow the hearing impaired to communicate through visual information as well. This interface can also be customized according to the user's usage situation, making it more intuitive and easy to use. These functional enhancements will broaden the scope of communication for the hearing impaired and promote their fuller participation in society. Technological developments that reduce the communication barriers faced by the hearing impaired and enable smooth dialogue in a variety of situations will contribute to the advancement of inclusion in society.

[0031] The process flow of each embodiment will be described below.

[0032] (Example 1) Step 1: Use your smartphone's microphone to pick up the other person's voice. Step 2: Use voice recognition technology to convert the other person's voice into text. Step 3: Mouth movements are detected using vibrations from the bone conduction microphone and the smartphone's built-in camera, and the content of what the other person is saying is understood by comparing this with data that the artificial intelligence has learned. Step 4: The emotion engine analyzes the other person's tone of voice and changes in facial expressions to estimate their emotional state. Step 5: A speaker that works in conjunction with a smartphone to play synthetic speech provides the hearing-impaired person with audio of the other person's speech and emotional state. (Example 2) Step 1: Use the smartphone's built-in camera to detect the other person's mouth movements in real time. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. Step 4: The emotion engine analyzes the other person's tone of voice and changes in facial expressions to estimate their emotional state. Step 5: A speaker that works in conjunction with a smartphone to play synthetic speech provides the hearing-impaired person with audio of the other person's speech and emotional state. (Example 3) Step 1: The smartphone's bone conduction microphone is used to detect the vibrations of the other person's voice. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: The emotion engine analyzes the other person's tone of voice and changes in facial expressions to estimate their emotional state. Step 4: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. Step 5: A speaker that works in conjunction with a smartphone to play synthetic speech provides the hearing-impaired person with audio of the other person's speech and emotional state.

[0033] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0034] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing while using the data generation model 58.

[0035] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0036] [Second embodiment]

[0037] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0038] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0039] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0040] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0041] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0042] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0043] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0044] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0045] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0046] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0047] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0048] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a ChatGPT server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, or a home appliance) owned by a user. Next, the identification process by the identification processing unit 290 of the data processing device 12 will be described.

[0049] (Example 1) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by the vibration of a bone conduction microphone or the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the other party's voice is acquired using the microphone of the smartphone and converted into text using voice recognition technology. In addition, the movement of the mouth is detected by the vibration of the bone conduction microphone or the built-in camera of the smartphone, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the content of the other party's speech is provided as a voice to the hearing-impaired person from the speaker for playing the synthetic voice in cooperation with the smartphone. The acquisition means is realized by the specific processing unit 290 of the data processing device 12, and for example, acquires the other party's voice using a microphone of a smartphone and converts it into text using voice recognition technology. The artificial intelligence has a learning function realized by the specific processing unit 290 of the data processing device 12, detects vibrations of the bone conduction microphone and mouth movements using the built-in camera of the smartphone, and compares them with learning data. The speaker is realized by the speaker 240 of the smart glasses 214 as a means for playing synthetic voice in cooperation with the smartphone. The correspondence between each means and the device or control unit is not limited to the above example, and various changes are possible. The voice recognition means can use an advanced voice recognition algorithm to process voice data acquired from a microphone built into the smart device in real time and convert it into text data. The voice recognition means also has a keyword spotting function that analyzes voice data and identifies specific keywords and phrases, and can also instantly extract important information. The voice recognition means also has a text conversion function that converts the voice acquired from the microphone of the smart device into a digital signal and converts the digital signal into text data, and this text conversion can eliminate ambient noise by a voice separation function and extract only clear voice signals. Furthermore, it includes a semantic analysis function for identifying words and phrases with important meanings from the voice data, which makes it possible to understand the context of the dialogue and instantly capture necessary information. The learning means integrates vibration data from the bone conduction microphone and video data from the smart device's built-in camera to learn the speaker's mouth movements and pronunciation characteristics. The learning means can also fuse the vibration patterns from the bone conduction microphone with the visual information from the smart device's camera. This fusion process learns the correlation between the speaker's facial movements and voice, and more accurately interprets the content of the voice by using lip reading as an aid even when the voice signal is unclear. This process includes a face tracking function that tracks facial movements and a vibration analysis function that analyzes vibration patterns, which deepen learning based on data collected from multiple speakers. The speaker conveys the contents of the other person's speech to the hearing-impaired person by generating and outputting synthetic speech. The speaker also has an output function that generates synthetic speech through a speech synthesis process and conveys the speech to the user. The speech synthesis process includes a waveform generation function that synthesizes speech waveforms based on text data, and an articulation function that adjusts the articulation of the synthesized speech. This converts the text data into natural speech that resembles a human voice and provides it in a form that can be easily understood by the hearing-impaired person. A manual data collection method includes a function in which a user inputs text information using a touch screen of a smart device, and a speech synthesis means generates speech based on the text. An example of data collection that does not include a sensor is when a user directly inputs text via a touch screen of a smart device, and the text data is output as speech by a speech synthesis means. This method is suitable for users who use sign language as an alternative to speech recognition, and helps them communicate. This method is also effective in situations where speech data is not available or when you do not want to rely on speech recognition. The text information entered by the user is converted into a clear and easy-to-understand speech message by a speech synthesis process. It is possible to expand the functions of the speech recognition means and the learning means by artificial intelligence in the present invention. For example, the speech recognition means can be added with a speaker identification function that identifies the characteristics of an individual's voice and determines who is speaking. This makes it possible to accurately grasp the content of each speaker's speech even in a situation where multiple people are talking, and to provide clearer communication to hearing-impaired people. Furthermore, by incorporating an emotion recognition algorithm into the speech recognition means, it is possible to capture the emotional nuances of the speech and reflect the emotional expression in the synthetic speech. The learning means can be combined with face recognition technology to learn real-time changes in mouth movements and subtle changes in facial expressions to achieve more accurate lip reading. This makes it easier to accurately interpret the speaker's intention even when the audio signal is unclear. The learning means also has a multilingual learning function to accommodate different languages ​​and dialects, making it effective in international communication situations. The speaker has an automatic volume adjustment function that adjusts the volume according to the user's hearing level, and provides optimal audio output according to the usage situation of the hearing impaired.The speaker also has an environmental adaptation function that automatically adjusts the clarity of the sound according to the surrounding noise level, and always provides the best hearing experience while responding to changes in the external environment. Manual input data collection methods may include predictive text entry to allow users to easily edit selected text and custom dictionary features for technical terms and abbreviations that are difficult to recognize through speech, allowing users to enter information more quickly and accurately, resulting in more accurate speech. In addition, the system can be used not only by the hearing impaired, but also as a multilingual interpretation device, using voice recognition to translate foreign language conversations in real time, thereby reducing communication barriers between people who speak different languages ​​and promoting international exchange. It can also be incorporated with the ability to provide real-time subtitles during video calls and online meetings with remote locations, helping the hearing impaired to participate more actively in social activities. The voice recognition means of this embodiment further includes an environmental sound recognition function, which detects important sounds other than conversation, such as emergency vehicle sirens and fire alarms, and provides them to the hearing impaired as visual alerts or vibration signals. This improves safety and enables rapid response to dangerous situations. In addition, the voice recognition means incorporates a function that automatically measures the surrounding noise level and optimizes the voice recognition accuracy based on that information, thereby achieving highly reliable recognition even in noisy environments. The AI ​​learning method can learn from a user's usage and speech habits to provide more personalized lip reading assistance. For example, it can recognize patterns of words and phrases commonly used by a particular user and improve the accuracy of lip reading based on that. It can also detect a user's emotional state and stress level and provide feedback to improve the quality of communication. The speaker can add richer speech expressions to the synthesized speech to make it easier for the hearing impaired to understand as a more natural speech. For example, it can adjust the tone and intonation of the speech to reflect the speaker's emotion or emphasis. It can also add a personalization feature that allows users to record their own voice and incorporate that voice quality into the synthesized speech. For manual data collection methods, incorporating gesture control and eye-tracking technology will make text entry easier for users with disabilities or in situations where touchscreen operation is difficult, and providing audio feedback during text entry will make it easier to correct input errors and further improve communication efficiency. These functions are designed to respond to various scenarios that users face, making it easier for the hearing impaired to become more independent and participate in society. They also function as support tools to reduce communication barriers in the daily lives of the hearing impaired and facilitate smooth communication with people with normal hearing. This will enable the hearing impaired to live a richer social life and contribute to realizing an open communication environment for all. (Example 2) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by vibration of a bone conduction microphone or the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the movement of the other party's mouth is detected in real time using the built-in camera of the smartphone, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the voice of the other party is converted into text using voice recognition technology and displayed on the smartphone screen. The acquisition means is realized by the camera 42 of the smart glasses 214 as a means for detecting the other party's mouth movements in real time using the built-in camera of the smartphone. The artificial intelligence has a learning function realized by the specific processing unit 290 of the data processing device 12, and compares the detected mouth movements with learning data. The display means is realized by the display 343 of the smart glasses 214 as a means for converting the other party's voice into text using voice recognition technology and displaying it on the smartphone screen. The correspondence between each means and the device or control unit is not limited to the above example, and various changes are possible. The voice recognition means includes a function for capturing and analyzing voice signals using advanced voice processing algorithms. This means combines a noise canceling function for reducing environmental noise with a speaker identification function for identifying the speaker's voice to accurately capture the other party's voice. The voice recognition means also includes a highly sensitive microphone for capturing the speaker's voice and a voice analysis processor for analyzing the captured voice signal. This means can utilize echo cancellation and beamforming techniques to achieve high recognition accuracy even in complex acoustic environments. The voice recognition means can also analyze the captured signal and use it as data for converting the spoken content into text. The acquisition method is to detect mouth movements using the smartphone's built-in camera, and also to acquire vibrations using a bone conduction microphone. The built-in camera detects mouth movements using high-speed, highly accurate image processing technology and analyzes the captured video data. This analysis combines face recognition technology and lip reading technology to predict the content of the other person's speech from their mouth movements. The bone conduction microphone uses a vibration sensor to detect the speaker's jaw movements and vocal cord vibrations, and can acquire these signals as voice data. The speaker as an output means outputs synthetic speech generated through a speech synthesis engine. The speech synthesis engine includes a prosody generation function that adjusts the pitch and speed of the speech in order to convert text information into natural speech. The pronunciation model can also generate pronunciation patterns for various words and phrases so that the synthetic speech sounds closer to the voice of an actual speaker. The AI ​​uses deep learning networks and statistical learning models as a learning vehicle to extract speech features from large amounts of audio and visual data. This learning is used to identify the acoustic characteristics of a particular language or dialect, aiming to accurately recognize and understand the speech. The learning process uses approaches such as supervised learning and reinforcement learning, and new data is incorporated to continuously improve the model's performance. The AI ​​learning vehicle is also trained on samples taken from an extensive audio database to extract features from the audio data and gain the ability to identify a particular language or dialect, as well as understand the characteristics of an individual user's speaking style and accent. The learning vehicle can update the model using feedback data for continuous performance improvement. The manual input data collection method provides an interface for users to directly input text information. Users input information using the touch screen keyboard or voice input function of their smartphone, which is then received by the system. This input data is processed like any other speech recognition result and can be used for text display or voice output. To expand the functionality of the acquisition means, an acceleration sensor that analyzes vibrations can be incorporated in addition to the bone conduction microphone. This allows the speaker's subtle head and body movements to be detected, and information related to speech to be acquired from a wider range of angles. Furthermore, by using a wearable device to capture hand movements and changes in facial expression, additional communication information can be collected from gestures and facial expressions, helping to understand the content of speech. In addition, by using the built-in camera of a smartphone, it is possible to track the speaker's eye movements and blinks to infer attention and intent, enhancing contextual awareness using visual information and enabling more natural and intuitive interactions. To improve the speaker's functionality, a function can be added that automatically adjusts the frequency and volume of the sound to suit the user's hearing characteristics, providing optimal audibility according to environmental noise and the user's hearing characteristics. The AI ​​learning method will deeply analyze the correlation between voice data and visual data to more accurately understand non-verbal communication elements, and multi-modal learning will combine both voice and visual information for complex learning to improve the accuracy and naturalness of the dialogue system. For manual input, we will utilize the smartphone's motion sensors and gesture recognition to provide input methods other than keyboard input. For example, we will build an input system using air gestures and eye tracking to allow users to provide data to the system more intuitively. By combining these features, users can communicate not only what they say, but also non-verbal information such as gestures and facial expressions to the system, and the system can synthesize this information to achieve more accurate communication. These additional features can also be customized according to specific situations and user needs, providing the flexibility to accommodate a variety of usage scenarios. In terms of acquisition methods, more advanced voice separation technology will be introduced to selectively separate the speaker's voice from surrounding noise and other voices to acquire clearer voice data. In addition, it will have the ability to analyze environmental sounds in real time and understand the atmosphere and situation of the place. This will improve the accuracy of voice recognition and enrich the information obtained from voice data. The speaker, which serves as the output means, has the ability to learn the user's hearing characteristics and automatically adjust the sound quality accordingly. This allows for audio that is easy for each user to hear, making communication more comfortable. In addition, by incorporating spatial audio technology, audio can be heard from the direction of the speaker, providing a more natural conversation experience. The AI ​​is trained with an extensive voice database to handle different languages ​​and accents, making it suitable for global communication. It also learns the speaking style of each individual user to enable personalized dialogue. In addition, emotion recognition technology is introduced to identify the speaker's emotions and respond accordingly, making dialogue more human-like. Manual input data collection adds the ability to recognize a user's handwriting or sketches and convert them into text and graphical information, allowing users to provide information to the system through natural writing. These extensions enable the system to grasp not only the user's speech but also their emotions and intentions more accurately, and to gain a deeper understanding of the user's experience. These features also increase the value of the system as an accessible communication tool for users with special needs, such as those with physical or speech disabilities. This allows a wider range of users to enjoy communication with society. (Example 3) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the vibration of a bone conduction microphone and the movement of the mouth by the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the bone conduction microphone of the smartphone is used to detect the vibration of the other party's voice, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the content of the other party's speech is provided as a voice to the hearing impaired person from the speaker for playing the synthetic voice in cooperation with the smartphone. The acquisition means is realized by the microphone 238 of the smart glasses 214 as a means for detecting the vibration of the other person's voice using the bone conduction microphone of the smartphone. The artificial intelligence has a learning function realized by the specific processing unit 290 of the data processing device 12, and compares the detected vibration with the learning data. The speaker is realized by the speaker 240 of the smart glasses 214 as a means for playing synthetic voice in cooperation with the smartphone. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means has a function of removing environmental sounds and surrounding noises using an advanced voice processing algorithm and extracting only the other party's voice with high accuracy. The voice recognition means also includes an automatic voice recognition system that converts a voice signal into text, and this system is trained using a deep learning model and has the flexibility to accommodate various accents and speaking styles. The voice recognition means also removes background noise through an environmental noise reduction function and highlights the other party's voice using a target voice extraction function. The extracted voice signal is analyzed for frequency characteristics and amplitude fluctuations through a waveform analysis function, and is converted into features reflecting the speaker's voice quality and speech characteristics through a voice feature extraction function. This information is transmitted to a data processing device via a data transfer function. The automatic voice recognition system also includes a data-driven learning process that utilizes a learning model based on a deep neural network to adapt to various speech patterns and accents. The vibration detection means using a bone conduction microphone detects the vibration of the skull that occurs when speaking and converts it into an electrical signal. The vibration detection means using a bone conduction microphone also analyzes the characteristics of the vibration transmitted through the skull and converts it into a digital signal using an analog-to-digital conversion function. The vibration is analyzed to extract features related to the content of the speech through the vibration pattern analysis means. Meanwhile, the mouth movement detection means using a built-in camera analyzes the mouth movements captured by the camera using image processing technology and tracks the shape and movement of the mouth during speaking. The mouth movement detection means also precisely analyzes the mouth movements captured by the video image processing function using a mouth shape analysis function and a movement tracking function. These data are fused by the integrated analysis function to perform a comprehensive speech analysis that combines voice recognition and visual information. The AI ​​extracts features related to the speech content based on the vibration data and lip movement data, and compares them with a learning database using a matching means. The matching means utilizes a database containing a large amount of speech data and determines the speech content using a machine learning algorithm. The AI ​​also compares the features of the vibration data and lip movement data with an existing database through a database matching function, and identifies the speech content using a pattern matching function. In this process, a data fusion function with voice recognition data is used to improve the accuracy of speech recognition. The matching results are interpreted by the speech understanding function and used to provide appropriate responses and information. The voice output means by the speaker provides the speech content by synthetic voice to the hearing impaired. The voice output means by the speaker generates a natural voice that reflects the speaker's intention through a synthetic voice generation function, and converts this voice signal into a reproducible analog signal through a digital-to-analog conversion function. The voice synthesis means generates a voice having natural pronunciation and intonation according to the speech content, and outputs this from the speaker. The voice synthesis means uses text-to-speech (TTS) technology that converts text to speech, and has a built-in multilingual model that supports various languages ​​and dialects. The voice synthesis means generates a voice having pronunciation and intonation that supports different languages ​​and dialects through a multilingual text-to-speech function. An example of data collection that does not include sensors would be a user manually inputting speech via a keyboard or touch screen. This manual input method is particularly useful in noisy environments or when the speaker does not have access to a microphone. The manual user input method may also include a function where the user directly provides text information through an input interface, and the manually input information is subjected to analysis by the artificial intelligence via a text processing function. The manually input text data is similarly processed by the artificial intelligence and output as synthetic speech. The manually input text is also processed through a text analysis function and fed into a speech synthesis process for output as synthetic speech. For example, various functions can be added to each of the means of this embodiment. For example, the acquisition means can be added with a superdirectional microphone for instantaneous voice detection and analysis, thereby selectively acquiring only a specific person's voice and separating it from surrounding noise. The acquisition means can also incorporate a function for monitoring text-based communication from social media and messaging apps in real time and reproducing it as voice utterances. This allows the hearing impaired to directly receive online conversations as voice. The speech recognition means can utilize a multilingual model to handle multiple languages ​​and dialects, and can understand the speech of users with different cultural backgrounds. Furthermore, the speech recognition means can be equipped with an emotion analysis function to capture subtle changes in the speaker's voice in order to more accurately grasp the speaker's emotions and intentions. This makes it possible to reflect the speaker's emotions in the voice, rather than simply reading out the text, resulting in more human-like communication. The artificial intelligence means has the capability of continuous learning and self-improvement, and analyzes the speaker's speech characteristics and lip movement patterns more accurately over time. The artificial intelligence means can also be customized to the user's usage and preferences to provide the most natural and understandable speech analysis for each individual user. Such customizable AI can respond to the individual needs of each user, improving usability. Emotion recognition means can provide richer emotional information by adding a function to analyze not only the tone of a speaker's voice, but also subtle changes in facial expressions and body language. For example, it is possible to estimate the speaker's intention and the intensity of his or her emotions from the speaker's eye movements and hand gestures, achieving more accurate emotion recognition. Output means can be more diverse, using not only speakers, but also vibration and light signals to provide information to the hearing impaired. For example, smart devices can warn with vibration patterns or color changes depending on the speaker's emotional state and the urgency of what is being said. Information transmission systems can also be developed that use tactile information rather than synthetic voice to accommodate the visually impaired. These additional features can be implemented through hardware and software updates to existing smart devices, significantly improving the user experience. They can also be linked to cloud-based services to facilitate remote updates and customization, providing users with the latest technology at all times. Furthermore, these features can be widely applied as communication support tools in education, work, and daily life, and are expected to benefit a diverse range of users, including not only the hearing impaired, but also those with speech disabilities and non-native speakers. To further improve the acquisition method, an external noise canceling function can be added to collect clearer voice data. In addition, a real-time language conversion function can be introduced to facilitate communication with people who speak different languages. This will further strengthen the AI's ability to understand multiple languages ​​and expand its use in international communication. The AI ​​means will be further developed to have the ability to memorize what the user has said in the past and learn the individual's unique speaking style, which will enable the AI ​​to reproduce the user's speech more naturally and improve the quality of communication. It will also have an emotion-adaptive synthetic voice generation function that can read the user's emotions and respond with an appropriate tone. The speaker-based audio output means can be combined with a bone conduction headset or a vibrating wearable device so that the hearing impaired can experience sound more naturally. This allows the hearing impaired to intuitively grasp the rhythm and strength of the speaker's voice through the vibrations, enabling them to enjoy a richer communication experience. In addition, by adding customization features that allow users to adjust the volume and speed of audio output depending on the situation, it can be used to meet individual needs. It is also possible to incorporate a safety measure function that automatically increases the volume in an emergency. By integrating these functions, devices can be provided that are easier to use and more effective for the hearing impaired, as well as the elderly and other users who need to communicate in a variety of situations. Furthermore, these technologies are expected to be applied as educational support tools, remote communication, and multilingual dialogue systems, helping to promote social inclusion.

[0050] The process flow of each embodiment will be described below.

[0051] (Example 1) Step 1: Use your smartphone's microphone to pick up the other person's voice. Step 2: Use voice recognition technology to convert the other person's voice into text. Step 3: Mouth movements are detected using vibrations from the bone conduction microphone and the smartphone's built-in camera, and the content of what the other person is saying is understood by comparing this with data that the artificial intelligence has learned. Step 4: The speaker, which works in conjunction with a smartphone to play synthesized speech, provides the hearing-impaired person with audio of what the other person is saying. (Example 2) Step 1: Use the smartphone's built-in camera to detect the other person's mouth movements in real time. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. (Example 3) Step 1: The smartphone's bone conduction microphone is used to detect the vibrations of the other person's voice. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: The speaker, which works in conjunction with a smartphone to play synthetic speech, provides the hearing-impaired person with audio of what the other person is saying.

[0052] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0053] (Example 1) The embodiment of the present invention includes an acquisition means for acquiring the voice of the other party by a voice recognition means, an artificial intelligence for learning the movement of the mouth by the vibration of a bone conduction microphone and the built-in camera of a smartphone, a speaker for playing synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotions. Specifically, the other party's voice is acquired using the microphone of the smartphone and converted into text using voice recognition technology. In addition, the vibration of the bone conduction microphone and the built-in camera of the smartphone are detected, and the movement of the mouth is compared with the data learned by the artificial intelligence to understand the content of the other party's speech. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the speaker for playing synthetic voice in cooperation with the smartphone provides the other party's speech content and emotional state as voice to the hearing impaired person. The acquisition means is realized by the microphone 238 of the smart glasses 214, and acquires the user's voice. The artificial intelligence is realized by the specific processing unit 290 of the data processing device 12, and learns the vibration of the bone conduction microphone and the mouth movement by the built-in camera of the smartphone. The speaker is realized by the speaker 240 of the smart glasses 214, and plays synthetic voice. The emotion engine is realized by the specific processing unit 290 of the data processing device 12, and recognizes the user's emotion. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The acquisition means has a function of accurately acquiring the voice of the other party as digital data by utilizing voice recognition technology. For example, the acquisition means acquires clear voice data while eliminating surrounding noise by using a high-sensitivity microphone equipped with a noise cancellation function. The acquisition means can also acquire the speech of the other party as digital voice data by using the voice recognition means via a microphone on the smart device. This voice data can clearly capture the human voice by the high-sensitivity microphone while reducing environmental noise by a noise canceling function. The voice recognition means performs processing to convert the spoken content into text data based on the acquired voice data, thereby achieving highly accurate text conversion. This process includes feature extraction from the voice signal, segmentation of utterance units, and recognition processing based on a voice model. The voice recognition means can also identify the speaker's voiceprint and speech pattern using a characteristic analysis function of the voice signal, and perform preprocessing to improve the accuracy of voice recognition. The acquired voice data can be subjected to voice-to-text conversion processing by extracting features of the voice data, performing segmentation, and converting the voice into text. The artificial intelligence means includes analyzing vibration patterns of a bone conduction microphone and capturing and analyzing mouth movements using a built-in camera. The means performs processing to infer the speaker's speech content from the acquired vibration data and video data. The artificial intelligence means can also capture the speaker's mouth movements and head vibrations using the bone conduction microphone and the built-in camera. The artificial intelligence means can analyze these data and understand the speaker's speech content. The bone conduction data analysis means analyzes the head vibrations captured by the bone conduction microphone and associates them with the voice to understand the speech content. The mouth movement analysis means can use video data captured by the smartphone camera to track the speaker's mouth movements and associate them with the speech content. These means are trained using machine learning techniques to learn the speaker's individual speech characteristics and achieve high-precision speech understanding in real time. The emotion recognition means has a function of estimating the emotional state of a speaker by using both voice data and video data. This means includes a voice emotion analysis means based on the analysis of the tone and intonation of the voice, and a facial expression analysis means that captures changes in facial expressions. The voice emotion analysis means analyzes the pitch, intensity, acoustic characteristics, etc. of the voice data to estimate the emotional state of the speaker. The emotion engine also has both a voice emotion analysis function and a facial expression analysis function, and can estimate the emotional state of the speaker. The facial expression analysis means detects the facial expressions of the speaker from the video data captured by the built-in camera, and estimates the emotional state by capturing subtle changes in facial expressions. These means are trained using an emotion recognition model, and by identifying emotional patterns, grasp the speaker's emotions in real time. The output means includes a speaker that cooperates with the smartphone to generate synthetic voice and provide it to the user. The means includes a text-to-speech (TTS) function that converts text to voice, and the tone and intonation of the synthesized voice are adjusted to reflect the speaker's spoken content and emotions. Furthermore, the output voice has clear pronunciation and an appropriate speed so that the hearing impaired can easily understand it. The output means through the speaker can also generate synthetic voice and provide the hearing impaired with the other party's spoken content and emotional state by voice. The voice flowing from the speaker is designed so that the user can easily grasp the other party's spoken content and emotions. This allows the user to grasp the other party's words not only by hearing but also by combining them with visual information. As an extension of the functions related to the acquisition means, it is possible to selectively pick up the voice of a specific speaker by using a directional microphone that is resistant to external noise. In addition, by utilizing a combination of a smart device and a wearable device, the user's body movements and gestures can be acquired to assist in understanding non-verbal communication. Furthermore, it is conceivable to utilize beamforming technology through the acquisition means to identify individual speakers even in situations where multiple people are talking and accurately grasp the content of each speaker's speech. In order to improve the accuracy of lip reading, the artificial intelligence means uses a high-resolution camera to capture more detailed mouth movements. In addition, by developing an algorithm that analyzes the intention of communication not only from the speaker's mouth movements but also from facial expressions and body language, a richer understanding of communication is realized. The artificial intelligence means can build a lip reading model that supports multiple languages ​​and can also handle international communication. Regarding emotion recognition, more accurate emotion recognition can be achieved by incorporating a biofeedback sensor that obtains physiological indicators such as heart rate and skin electrical conductivity and uses them to estimate the emotional state. In addition, a customized emotion recognition model can be built by learning an individual's emotional patterns, providing emotion recognition adapted to each individual user. The emotion recognition means can also capture subtle differences in emotions expressed by users in different contexts and social situations. As for the output means, a personalized audio system that adjusts the frequency band according to the user's hearing characteristics can be introduced, making it possible to provide audio tailored to each individual hearing-impaired person. In addition, by incorporating a function into a smartphone application that uses visual information to display the other person's speech in text in real time, it becomes possible to obtain information from both audio and vision. Furthermore, by combining synthetic voice with an expressive animated avatar, visual communication can be enhanced and more intuitive understanding can be supported. The acquisition method can be supplemented with an active noise canceling function that analyzes the sound environment around the user in real time and performs optimal noise reduction. In addition, by utilizing bone conduction earphone technology that allows users to hear sound without blocking their ears, it is possible to provide users with clear sound while still being aware of the surrounding environmental sounds. Furthermore, by linking with devices other than smartphones, such as smart watches and smart glasses, voice data can be acquired using multiple devices, creating a more comprehensive communication environment. Regarding artificial intelligence means, multitask learning will be introduced to simultaneously analyze not only lip reading but also non-verbal information such as the speaker's posture and hand movements to gain a deeper understanding of the context of communication. In addition, by using edge computing technology to perform part of the data processing on the smart device side, it is possible to reduce communication delays and improve real-time performance. Furthermore, a function will be developed that uses artificial intelligence to automatically identify and translate the language used by the speaker, and to support communication between different languages. The emotion recognition method combines machine vision technology to capture subtle changes in facial expressions and skin color, improving the accuracy of emotion analysis. In addition, bio-signals obtained from wearable devices are used for emotion recognition to more accurately evaluate the user's psychological state, such as stress level and fatigue. In addition, situation recognition technology is introduced to adjust the emotion recognition algorithm according to the user's environment and situation, allowing for more appropriate emotion estimation. As for output means, a visual assist function will be added that adjusts the size and contrast of text display to suit the user's visual acuity characteristics. In addition, an avatar-based sign language translation system will be incorporated that recognizes and generates sign language and gestures necessary for the hearing impaired to communicate with others. An emotional expression assist function will also be developed that allows users to convey their emotional state to others, realizing rich communication that includes emotions. Through these additional functions, the invention can provide a more comfortable and effective means of communication not only for the hearing impaired but also for users with various communication disabilities. In addition, by combining these functions, it is possible to reduce misunderstandings that occur during communication and realize smoother dialogue. (Example 2) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by vibration of a bone conduction microphone and the built-in camera of a smartphone, a speaker for playing a synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotion. Specifically, the smartphone's built-in camera is used to detect the other party's mouth movement in real time, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the other party's voice is converted into text using voice recognition technology and displayed on the smartphone screen. The acquisition means is realized by the camera 42 of the smart glasses 214, and detects the movement of the other person's mouth in real time. The artificial intelligence is realized by the specific processing unit 290 of the data processing device 12, and understands the content of the other person's speech by comparing it with the data being learned. The emotion engine is realized by the specific processing unit 290 of the data processing device 12, and estimates the other person's emotional state. The display means is realized by the control unit 46A of the smart glasses 214, and converts the other person's voice into text and displays it on the smartphone screen. The correspondence between each means and the device or control unit is not limited to the above example, and various changes are possible. The voice recognition means includes a voice data acquisition function using advanced voice signal processing technology. This function uses a noise suppression algorithm to remove ambient noise and clearly capture the other party's voice. The noise suppression algorithm distinguishes between human voices and background noise through frequency analysis and subtracts noise components in real time. The voice recognition means also uses a machine learning voice recognition model to identify the speaker and the contents of the speech from the voice. This model has the ability to learn large amounts of voice data and identify the voice characteristics and speech patterns of a specific speaker. The voice data acquisition function, which uses voice recognition technology to acquire and analyze voice data obtained from the other party's voice, also utilizes an advanced signal processing algorithm to remove ambient noise and clearly identify the speaker's voice. This function improves the quality of communication by identifying and removing noise components in the voice data. In addition, as a speaker identification function, a model based on machine learning is used to learn the voice characteristics of each speaker and identify who the speaker is from the voice data. The acquisition means includes a mouth movement detection function using the camera of the smart device. This function combines image recognition technology and machine learning algorithms to infer the content of the speech from the shape and movement of the other person's mouth. The detected mouth movement data is sent to a data processing device in real time and analyzed by artificial intelligence. In this analysis process, the data is compared with a trained data set and the speech content is understood through lip reading. The artificial intelligence uses face recognition technology to identify a specific individual, learns the characteristics of the person's mouth movement, and builds a correspondence with the speech content. In addition, the mouth movement detection function uses the camera of the smart device to acquire an image of the other person's face, and uses lip reading technology to infer the speech content from the movement. This function detects the shape and movement of the mouth from the face image, and uses a learned lip reading model to analyze the words spoken by the speaker. In this process, the face recognition function is also utilized to recognize the mouth movement patterns of a specific individual and assist in understanding the speech content. The speaker means converts the text data into a voice having natural pronunciation and intonation using synthetic voice generation technology. This technology can reflect not only the accuracy of pronunciation but also the emotion and intention of the speaker in the text-to-voice conversion. The speaker may be built into the smart device, or may be an external wireless speaker that transmits an audio signal. In either case, an acoustic processing technology is employed to ensure a sound quality and volume that is easy for the user to hear. The speaker means for playing the synthetic voice includes a text-to-voice conversion function, which converts the text data into a voice having natural pronunciation and intonation using synthetic voice generation technology. This technology can reflect the emotion and intention of the speaker in the voice, and depending on the speaker used, a sound quality adjustment function and a volume adjustment function are also used in combination. The emotion recognition means has a function of estimating emotions from audio signals and video data. This function uses a voice emotion analysis algorithm that analyzes changes in voice tone, speech rate, and intonation, and a facial expression recognition algorithm that captures changes in facial expressions and eye movements. The emotion recognition means integrates these data to infer the emotional state of the other person. The emotion estimation result becomes important information in the context of communication and is provided as feedback to help the user understand the emotions of the other person. The emotion recognition means also estimates the emotional state of the speaker by analyzing the audio signals and video data. This means includes a voice emotion analysis algorithm that monitors changes in tone, speech rate, and intonation. It also includes a facial expression recognition algorithm that detects changes in facial expressions and eye movements from the video data. By integrating these pieces of information, the other person's emotional state is estimated in real time and the result is provided to the user. An example of data collection that does not involve sensors is when a user inputs a text message via a keyboard or voice input function of a smart device and transmits the message through a communication means. This method allows the user to communicate with the other party without relying on voice or video. Another example of data collection that does not involve sensors is text-based communication that is manually input by a user. The user inputs his / her own words as text using a keyboard or voice input function of a smart device and transmits the text to the other party. This method allows direct text communication without requiring a voice recognition means or a lip reading means. The voice recognition means, acquisition means, speaker means, and emotion engine in the embodiments of the present invention can further add other functions. For example, the voice recognition means can employ binaural recording technology and reproduce a more natural acoustic space using two microphones positioned at the position of human ears. This allows the user to easily sense the direction and distance of the other person's voice, allowing the user to enjoy a more realistic communication experience. Furthermore, the voice recognition means can be used to realize the operation of smart devices and the control of home automation systems by voice commands. The acquisition means may incorporate a feedback function to assist the user in speaking. The feedback function may provide guidance on mouth movements and pronunciation when the user speaks. Furthermore, the acquisition means may include a noise canceling function to effectively eliminate external noise and improve the accuracy of detecting mouth movements. The speaker means can have a function to automatically adjust the sound quality according to the user's hearing condition and hearing preferences. Advanced digital signal processing technology is utilized to provide the listener with an optimal hearing experience. The speaker means can also be added with a function to link with external smart home devices and in-car systems to expand the reach of the voice wherever the user is. The emotion engine can incorporate sensors to detect the user's physiological reactions and add a function to more accurately estimate the user's emotional state from changes in heart rate and electrical resistance of the skin. Such biofeedback functions can monitor the user's stress level and excitement in real time, deepening the understanding of emotions in communication. With the addition of these functions, the embodiment of the present invention becomes a more advanced communication support tool, enabling users to have richer and more emotionally rich conversations with others. These functions also provide effective support for users with disabilities or when communication is difficult in a particular environment. Furthermore, as smart devices evolve, these functions are expected to be expanded and optimized in the future to accommodate the needs of a variety of users. The voice recognition means of this embodiment can be equipped with a sensitivity enhancement function that accurately captures the contents of speech even when the user whispers or speaks at a very low volume. This function combines a special microphone array with an algorithm that analyzes ambient noise to make the speaker's voice stand out. In addition, a directional pickup function can be provided to selectively capture the voice of a specific speaker in a situation where multiple speakers are speaking at the same time. Acquisition methods can also be enhanced with real-time translation capabilities to instantly translate utterances spoken in different languages ​​into the user's native language. This functionality works with multilingual databases to reduce global communication barriers. Acquisition methods can also employ region-specific models to identify speaker accents and dialects for more accurate transcription and translation. The speaker means may be added with a personalized equalization function to analyze the user's hearing adaptation and provide an optimal hearing experience. This function realizes audio output that is tailored to the individual ear characteristics and hearing preferences of the user. The speaker means may also provide a virtual reality experience that enhances spatial awareness by adding 3D audio effects to the voice guidance. The emotion engine can be equipped with an emotion response function that tracks the user's emotional changes during the conversation and provides supportive and encouraging messages at the right time. The emotion engine can also estimate the user's psychological stress level from their behavior and speech patterns and suggest advice and relaxation techniques to reduce stress. These additional functions will provide users with a more detailed and personalized communication experience. Furthermore, these technologies are expected to be applied in fields such as education, business, and healthcare, contributing to improving the quality of communication and promoting accessibility. In addition, by providing interactive feedback according to the user's behavior and situation, more natural human-machine dialogue will be realized. The evolution of these technologies will promote a deeper level of user engagement in the future, laying the foundation for providing more valuable services to users. (Example 3) The embodiment of the present invention includes an acquisition means for acquiring the voice of the other party by a voice recognition means, an artificial intelligence for learning the vibration of a bone conduction microphone and the movement of the mouth by the built-in camera of a smartphone, a speaker for playing synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotions. Specifically, the bone conduction microphone of the smartphone is used to detect the vibration of the other party's voice, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the speaker for playing synthetic voice in cooperation with the smartphone provides the other party's speech content and emotional state as voice to the hearing impaired person. The acquisition means is realized by the microphone 238 of the smart glasses 214, and detects the vibrations of the other person's voice. The artificial intelligence is realized by the specific processing unit 290 of the data processing device 12, and understands the content of the other person's voice by comparing it with the data being learned. The emotion engine is realized by the specific processing unit 290 of the data processing device 12, and estimates the other person's emotional state. The speaker is realized by the speaker 240 of the smart glasses 214, and plays synthetic voice. The correspondence between each means and the device or control unit is not limited to the above example, and various changes are possible. The voice recognition means has a function of acquiring surrounding voice data through a microphone built into the smart device, and the acquired voice data is pre-processed by digital signal processing, such as noise removal and amplification, and converted into clear voice information. The voice recognition means also has a voice acquisition function for capturing voice from the surroundings, and applies a voice pre-processing function to the acquired voice data. This pre-processing includes the removal of environmental noise by a noise reduction function and the optimization of the volume by a gain adjustment function. This makes the voice signal clearer and converts it into a state suitable for subsequent processing. Next, based on this pre-processed voice data, a voice analysis algorithm extracts specific voice features and improves the accuracy of voice recognition. The extracted voice features include the frequency, amplitude, pitch, duration, etc. of the voice, which are important indicators for understanding the characteristics of the speaker's voice and the content of the words. The voice analysis algorithm also extracts features such as the frequency characteristics and temporal patterns of the voice through a feature extraction function, and grasps the meaning of the voice data using a speaker identification function and a speech content analysis function. The AI ​​learns the vibration patterns of speech detected by the bone conduction microphone as a data set and identifies the content of speech. This learning process uses multiple machine learning algorithms including deep learning technology. In particular, convolutional neural networks are suitable for extracting features from image data and are used to analyze mouth movements captured by the built-in camera of a smartphone. The AI ​​integrates both audio data and visual data and performs multimodal information processing, enabling accurate identification and understanding of the content of speech. The AI ​​also combines the analysis function of vibration data from the bone conduction microphone with the visual data analysis function to accurately identify the content of speech. The vibration data analysis function captures the physical characteristics of the voice obtained by bone conduction and compares it with the audio data. The visual data analysis function identifies the patterns of mouth movements obtained by the smartphone camera and integrates and processes this information to enable more advanced recognition from multimodal information. The speaker is built into the smart device and generates synthetic speech to provide audio information to the user in an easy-to-listen format. The synthetic speech is generated using text-to-speech technology and is designed to reproduce not only the content of the speech, but also changes in tone and intonation that express emotions. This enables the hearing impaired to understand not only the content of the speech of the other person, but also its emotional nuances. The speaker also converts text information into speech using a synthetic speech generation function, providing easy-to-listen audio information to the user. The synthetic speech adjustment function mimics the emotional elements of speech, including tone and intonation, to reproduce the richness of emotional expression during communication. The emotion recognition means performs a composite emotion analysis by integrating acoustic features such as tone, pitch, and speed of the voice with visual information such as changes in facial expressions captured by the smartphone camera. In this process, a machine learning model specialized in emotion recognition is used, and the user's emotions can be classified into multiple categories such as positive, negative, surprise, and sadness. As a result, the emotion recognition means estimates the user's emotional state and provides an emotional context in the communication. In addition, the emotion recognition means extracts emotion information from the voice data and visual data using an emotion analysis algorithm. This analysis includes a voice emotion analysis function and a facial expression analysis function, which are integrated to perform a multidimensional analysis of the user's emotional state, and the emotion estimation function understands the user's emotions. This makes it easier for the user to grasp the other person's emotions, improving the quality of communication. In implementing this embodiment, it is useful to consider further functional expansion. For example, it is possible to introduce beamforming technology into the acquisition means, which selectively acquires a specific voice from among surrounding sounds. This makes it possible to clearly capture the speaker's voice even in a noisy environment and improve the accuracy of voice recognition. In addition, a multilingual processing function for handling different languages ​​and dialects can be added to the artificial intelligence to accommodate a wide range of users. It is also possible to reduce communication barriers with foreign language speakers by incorporating a real-time translation function using artificial intelligence. As for the speakers, a function to adjust the frequency characteristics of the sound to suit the user's hearing characteristics will be added, providing even more personalized listening ease. The emotion engine will be able to convey emotional fluctuations to the user in line with the flow of the conversation by capturing emotional changes in real time. In addition, a context-aware function will be added to the emotion engine, which will estimate emotions based on the context of the conversation, achieving more accurate emotion recognition. Furthermore, it is conceivable to incorporate a biosensor function that acquires the user's biometric information and uses that information to more precisely estimate the other person's emotions and state. This will further improve the quality of communication. Also, adding an assistive communication function that learns the user's communication patterns and prompts them to speak at the appropriate time will enable smoother conversations. In addition, collaboration with wearable devices and smart glasses will be strengthened to expand communication support using visual information. By providing visual feedback such as text, emojis, and videos through smart glasses, the hearing impaired will be able to better understand the emotions and intentions of others. The smart glasses will also be equipped with an audio visualization function that visualizes the surrounding environmental sounds, allowing users to visually understand the direction and strength of sounds. This will make it easier for users to understand the surrounding sound environment from visual information. With these functional extensions, this embodiment is expected to greatly improve the communication experience of users, and contribute to the realization of a more inclusive society by providing a customizable communication means that meets the individual needs of each user. In addition to the acquisition means of this embodiment, it is useful to consider a function for capturing bone conduction vibrations more accurately using a wearable device worn near the user's ear. This wearable device is designed to fit the ear and is structured to be less susceptible to external noise. Furthermore, by developing technology to improve the sensitivity of the bone conduction microphone itself, it is possible to detect more subtle vibrations of speech and increase the accuracy of understanding the content of speech. The AI ​​training dataset will include a wide variety of languages ​​and dialects, as well as the voice patterns of speakers with specific disabilities, to accommodate a wider range of users. In addition, when performing real-time speech recognition, it will incorporate context-based predictive modeling to more accurately grasp the intent of the spoken content. This will enable the system to accurately understand the meaning of words in the context of the dialogue. The synthesized voice from the speaker will be equipped with a function that allows the pitch and speed of the voice to be adjusted according to the user's level of hearing impairment and preferences. This will allow the user to hear voices that are easy to understand. In addition, the emotional expressions of the synthesized voice will be able to reflect emotional characteristics extracted from recordings of actual human voices, making it possible to convey emotions more naturally and realistically. The emotion engine will be able to perform more comprehensive emotion recognition by incorporating sensor technology that analyzes not only language but also non-verbal elements such as gestures and body postures. It will also be able to realize more personalized emotion recognition by taking into account the relationship between the user and the other party and past conversation history. To improve the user's communication experience, a smartphone app will be built with an interface that conveys information visually. This interface will include a function that displays voice information as text and a function that expresses emotions with colors and patterns. This will allow the hearing impaired to communicate through visual information as well. This interface can also be customized according to the user's usage situation, making it more intuitive and easy to use. These functional enhancements will broaden the scope of communication for the hearing impaired and promote their fuller participation in society. Technological developments that reduce the communication barriers faced by the hearing impaired and enable smooth dialogue in a variety of situations will contribute to the advancement of inclusion in society.

[0054] The process flow of each embodiment will be described below.

[0055] (Example 1) Step 1: Use your smartphone's microphone to pick up the other person's voice. Step 2: Use voice recognition technology to convert the other person's voice into text. Step 3: Mouth movements are detected using vibrations from the bone conduction microphone and the smartphone's built-in camera, and the content of what the other person is saying is understood by comparing this with data that the artificial intelligence has learned. Step 4: The emotion engine analyzes the other person's tone of voice and changes in facial expressions to estimate their emotional state. Step 5: A speaker that works in conjunction with a smartphone to play synthetic speech provides the hearing-impaired person with audio of the other person's speech and emotional state. (Example 2) Step 1: Use the smartphone's built-in camera to detect the other person's mouth movements in real time. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. Step 4: The emotion engine analyzes the other person's tone of voice and changes in facial expressions to estimate their emotional state. Step 5: A speaker that works in conjunction with a smartphone to play synthetic speech provides the hearing-impaired person with audio of the other person's speech and emotional state. (Example 3) Step 1: The smartphone's bone conduction microphone is used to detect the vibrations of the other person's voice. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: The emotion engine analyzes the other person's tone of voice and changes in facial expressions to estimate their emotional state. Step 4: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. Step 5: A speaker that works in conjunction with a smartphone to play synthetic speech provides the hearing-impaired person with audio of the other person's speech and emotional state.

[0056] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0057] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing while using the data generation model 58.

[0058] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0059] [Third embodiment]

[0060] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0061] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0062] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0063] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0064] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0065] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0066] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0067] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0068] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0069] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0070] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0071] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a ChatGPT server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, or a home appliance) owned by a user. Next, the identification process by the identification processing unit 290 of the data processing device 12 will be described.

[0072] (Example 1) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by the vibration of a bone conduction microphone or the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the other party's voice is acquired using the microphone of the smartphone and converted into text using voice recognition technology. In addition, the movement of the mouth is detected by the vibration of the bone conduction microphone or the built-in camera of the smartphone, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the content of the other party's speech is provided as a voice to the hearing-impaired person from the speaker for playing the synthetic voice in cooperation with the smartphone. The acquisition means is realized by the microphone 238 of the headset terminal 314, and acquires the voice of the other party. It may also be realized by the specific processing unit 290 of the data processing device 12. The artificial intelligence is realized by the specific processing unit 290 of the data processing device 12, and learns the vibration of the bone conduction microphone and the mouth movement by the built-in camera of the smartphone. The speaker is realized by the speaker 240 of the headset terminal 314, and plays a synthetic voice. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means can use an advanced voice recognition algorithm to process voice data acquired from a microphone built into the smart device in real time and convert it into text data. The voice recognition means also has a keyword spotting function that analyzes voice data and identifies specific keywords and phrases, and can also instantly extract important information. The voice recognition means also has a text conversion function that converts the voice acquired from the microphone of the smart device into a digital signal and converts the digital signal into text data, and this text conversion can eliminate ambient noise by a voice separation function and extract only clear voice signals. Furthermore, it includes a semantic analysis function for identifying words and phrases with important meanings from the voice data, which makes it possible to understand the context of the dialogue and instantly capture necessary information. The learning means integrates vibration data from the bone conduction microphone and video data from the smart device's built-in camera to learn the speaker's mouth movements and pronunciation characteristics. The learning means can also fuse the vibration patterns from the bone conduction microphone with the visual information from the smart device's camera. This fusion process learns the correlation between the speaker's facial movements and voice, and more accurately interprets the content of the voice by using lip reading as an aid even when the voice signal is unclear. This process includes a face tracking function that tracks facial movements and a vibration analysis function that analyzes vibration patterns, which deepen learning based on data collected from multiple speakers. The speaker conveys the contents of the other person's speech to the hearing-impaired person by generating and outputting synthetic speech. The speaker also has an output function that generates synthetic speech through a speech synthesis process and conveys the speech to the user. The speech synthesis process includes a waveform generation function that synthesizes speech waveforms based on text data, and an articulation function that adjusts the articulation of the synthesized speech. This converts the text data into natural speech that resembles a human voice and provides it in a form that can be easily understood by the hearing-impaired person. A manual data collection method includes a function in which a user inputs text information using a touch screen of a smart device, and a speech synthesis means generates speech based on the text. An example of data collection that does not include a sensor is when a user directly inputs text via a touch screen of a smart device, and the text data is output as speech by a speech synthesis means. This method is suitable for users who use sign language as an alternative to speech recognition, and helps them communicate. This method is also effective in situations where speech data is not available or when you do not want to rely on speech recognition. The text information entered by the user is converted into a clear and easy-to-understand speech message by a speech synthesis process. It is possible to expand the functions of the speech recognition means and the learning means by artificial intelligence in the present invention. For example, the speech recognition means can be added with a speaker identification function that identifies the characteristics of an individual's voice and determines who is speaking. This makes it possible to accurately grasp the content of each speaker's speech even in a situation where multiple people are talking, and to provide clearer communication to hearing-impaired people. Furthermore, by incorporating an emotion recognition algorithm into the speech recognition means, it is possible to capture the emotional nuances of the speech and reflect the emotional expression in the synthetic speech. The learning means can be combined with face recognition technology to learn real-time changes in mouth movements and subtle changes in facial expressions to achieve more accurate lip reading. This makes it easier to accurately interpret the speaker's intention even when the audio signal is unclear. The learning means also has a multilingual learning function to accommodate different languages ​​and dialects, making it effective in international communication situations. The speaker has an automatic volume adjustment function that adjusts the volume according to the user's hearing level, and provides optimal audio output according to the usage situation of the hearing impaired.The speaker also has an environmental adaptation function that automatically adjusts the clarity of the sound according to the surrounding noise level, and always provides the best hearing experience while responding to changes in the external environment. Manual input data collection methods may include predictive text entry to allow users to easily edit selected text and custom dictionary features for technical terms and abbreviations that are difficult to recognize through speech, allowing users to enter information more quickly and accurately, resulting in more accurate speech. In addition, the system can be used not only by the hearing impaired, but also as a multilingual interpretation device, using voice recognition to translate foreign language conversations in real time, thereby reducing communication barriers between people who speak different languages ​​and promoting international exchange. It can also be incorporated with the ability to provide real-time subtitles during video calls and online meetings with remote locations, helping the hearing impaired to participate more actively in social activities. The voice recognition means of this embodiment further includes an environmental sound recognition function, which detects important sounds other than conversation, such as emergency vehicle sirens and fire alarms, and provides them to the hearing impaired as visual alerts or vibration signals. This improves safety and enables rapid response to dangerous situations. In addition, the voice recognition means incorporates a function that automatically measures the surrounding noise level and optimizes the voice recognition accuracy based on that information, thereby achieving highly reliable recognition even in noisy environments. The AI ​​learning method can learn from a user's usage and speech habits to provide more personalized lip reading assistance. For example, it can recognize patterns of words and phrases commonly used by a particular user and improve the accuracy of lip reading based on that. It can also detect a user's emotional state and stress level and provide feedback to improve the quality of communication. The speaker can add richer speech expressions to the synthesized speech to make it easier for the hearing impaired to understand as a more natural speech. For example, it can adjust the tone and intonation of the speech to reflect the speaker's emotion or emphasis. It can also add a personalization feature that allows users to record their own voice and incorporate that voice quality into the synthesized speech. For manual data collection methods, incorporating gesture control and eye-tracking technology will make text entry easier for users with disabilities or in situations where touchscreen operation is difficult, and providing audio feedback during text entry will make it easier to correct input errors and further improve communication efficiency. These functions are designed to respond to various scenarios that users face, making it easier for the hearing impaired to become more independent and participate in society. They also function as support tools to reduce communication barriers in the daily lives of the hearing impaired and facilitate smooth communication with people with normal hearing. This will enable the hearing impaired to live a richer social life and contribute to realizing an open communication environment for all. (Example 2) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by vibration of a bone conduction microphone or the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the movement of the other party's mouth is detected in real time using the built-in camera of the smartphone, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the voice of the other party is converted into text using voice recognition technology and displayed on the smartphone screen. The acquisition means is realized by the microphone 238 of the headset terminal 314, and acquires the voice of the other party. It may also be realized by the specific processing unit 290 of the data processing device 12. The artificial intelligence is realized by the specific processing unit 290 of the data processing device 12, and learns the mouth movements using the built-in camera of the smartphone. The display means is realized by the display 343 of the headset terminal 314, and converts the voice of the other party into text and displays it. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means includes a function for capturing and analyzing voice signals using advanced voice processing algorithms. This means combines a noise canceling function for reducing environmental noise with a speaker identification function for identifying the speaker's voice to accurately capture the other party's voice. The voice recognition means also includes a highly sensitive microphone for capturing the speaker's voice and a voice analysis processor for analyzing the captured voice signal. This means can utilize echo cancellation and beamforming techniques to achieve high recognition accuracy even in complex acoustic environments. The voice recognition means can also analyze the captured signal and use it as data for converting the spoken content into text. The acquisition method is to detect mouth movements using the smartphone's built-in camera, and also to acquire vibrations using a bone conduction microphone. The built-in camera detects mouth movements using high-speed, highly accurate image processing technology and analyzes the captured video data. This analysis combines face recognition technology and lip reading technology to predict the content of the other person's speech from their mouth movements. The bone conduction microphone uses a vibration sensor to detect the speaker's jaw movements and vocal cord vibrations, and can acquire these signals as voice data. The speaker as an output means outputs synthetic speech generated through a speech synthesis engine. The speech synthesis engine includes a prosody generation function that adjusts the pitch and speed of the speech in order to convert text information into natural speech. The pronunciation model can also generate pronunciation patterns for various words and phrases so that the synthetic speech sounds closer to the voice of an actual speaker. The AI ​​uses deep learning networks and statistical learning models as a learning vehicle to extract speech features from large amounts of audio and visual data. This learning is used to identify the acoustic characteristics of a particular language or dialect, aiming to accurately recognize and understand the speech. The learning process uses approaches such as supervised learning and reinforcement learning, and new data is incorporated to continuously improve the model's performance. The AI ​​learning vehicle is also trained on samples taken from an extensive audio database to extract features from the audio data and gain the ability to identify a particular language or dialect, as well as understand the characteristics of an individual user's speaking style and accent. The learning vehicle can update the model using feedback data for continuous performance improvement. The manual input data collection method provides an interface for users to directly input text information. Users input information using the touch screen keyboard or voice input function of their smartphone, which is then received by the system. This input data is processed like any other speech recognition result and can be used for text display or voice output. To expand the functionality of the acquisition means, an acceleration sensor that analyzes vibrations can be incorporated in addition to the bone conduction microphone. This allows the speaker's subtle head and body movements to be detected, and information related to speech to be acquired from a wider range of angles. Furthermore, by using a wearable device to capture hand movements and changes in facial expression, additional communication information can be collected from gestures and facial expressions, helping to understand the content of speech. In addition, by using the built-in camera of a smartphone, it is possible to track the speaker's eye movements and blinks to infer attention and intent, enhancing contextual awareness using visual information and enabling more natural and intuitive interactions. To improve the speaker's functionality, a function can be added that automatically adjusts the frequency and volume of the sound to suit the user's hearing characteristics, providing optimal audibility according to environmental noise and the user's hearing characteristics. The AI ​​learning method will deeply analyze the correlation between voice data and visual data to more accurately understand non-verbal communication elements, and multi-modal learning will combine both voice and visual information for complex learning to improve the accuracy and naturalness of the dialogue system. For manual input, we will utilize the smartphone's motion sensors and gesture recognition to provide input methods other than keyboard input. For example, we will build an input system using air gestures and eye tracking to allow users to provide data to the system more intuitively. By combining these features, users can communicate not only what they say, but also non-verbal information such as gestures and facial expressions to the system, and the system can synthesize this information to achieve more accurate communication. These additional features can also be customized according to specific situations and user needs, providing the flexibility to accommodate a variety of usage scenarios. In terms of acquisition methods, more advanced voice separation technology will be introduced to selectively separate the speaker's voice from surrounding noise and other voices to acquire clearer voice data. In addition, it will have the ability to analyze environmental sounds in real time and understand the atmosphere and situation of the place. This will improve the accuracy of voice recognition and enrich the information obtained from voice data. The speaker, which serves as the output means, has the ability to learn the user's hearing characteristics and automatically adjust the sound quality accordingly. This allows for audio that is easy for each user to hear, making communication more comfortable. In addition, by incorporating spatial audio technology, audio can be heard from the direction of the speaker, providing a more natural conversation experience. The AI ​​is trained with an extensive voice database to handle different languages ​​and accents, making it suitable for global communication. It also learns the speaking style of each individual user to enable personalized dialogue. In addition, emotion recognition technology is introduced to identify the speaker's emotions and respond accordingly, making dialogue more human-like. Manual input data collection adds the ability to recognize a user's handwriting or sketches and convert them into text and graphical information, allowing users to provide information to the system through natural writing. These extensions enable the system to grasp not only the user's speech but also their emotions and intentions more accurately, and to gain a deeper understanding of the user's experience. These features also increase the value of the system as an accessible communication tool for users with special needs, such as those with physical or speech disabilities. This allows a wider range of users to enjoy communication with society. (Example 3) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the vibration of a bone conduction microphone and the movement of the mouth by the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the bone conduction microphone of the smartphone is used to detect the vibration of the other party's voice, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the content of the other party's speech is provided as a voice to the hearing impaired person from the speaker for playing the synthetic voice in cooperation with the smartphone. The acquisition means is realized by the microphone 238 of the headset terminal 314, and acquires the voice of the other party. It may also be realized by the specific processing unit 290 of the data processing device 12. The artificial intelligence is realized by the specific processing unit 290 of the data processing device 12, and learns the vibration of the bone conduction microphone. The speaker is realized by the speaker 240 of the headset terminal 314, and plays a synthetic voice. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various modifications are possible. The voice recognition means has a function of removing environmental sounds and surrounding noises using an advanced voice processing algorithm and extracting only the other party's voice with high accuracy. The voice recognition means also includes an automatic voice recognition system that converts a voice signal into text, and this system is trained using a deep learning model and has the flexibility to accommodate various accents and speaking styles. The voice recognition means also removes background noise through an environmental noise reduction function and highlights the other party's voice using a target voice extraction function. The extracted voice signal is analyzed for frequency characteristics and amplitude fluctuations through a waveform analysis function, and is converted into features reflecting the speaker's voice quality and speech characteristics through a voice feature extraction function. This information is transmitted to a data processing device via a data transfer function. The automatic voice recognition system also includes a data-driven learning process that utilizes a learning model based on a deep neural network to adapt to various speech patterns and accents. The vibration detection means using a bone conduction microphone detects the vibration of the skull that occurs when speaking and converts it into an electrical signal. The vibration detection means using a bone conduction microphone also analyzes the characteristics of the vibration transmitted through the skull and converts it into a digital signal using an analog-to-digital conversion function. The vibration is analyzed to extract features related to the content of the speech through the vibration pattern analysis means. Meanwhile, the mouth movement detection means using a built-in camera analyzes the mouth movements captured by the camera using image processing technology and tracks the shape and movement of the mouth during speaking. The mouth movement detection means also precisely analyzes the mouth movements captured by the video image processing function using a mouth shape analysis function and a movement tracking function. These data are fused by the integrated analysis function to perform a comprehensive speech analysis that combines voice recognition and visual information. The AI ​​extracts features related to the speech content based on the vibration data and lip movement data, and compares them with a learned database using a matching means. The matching means utilizes a database containing a large amount of speech data and determines the speech content using a machine learning algorithm. The AI ​​also compares the features of the vibration data and lip movement data with an existing database through a database matching function, and identifies the speech content using a pattern matching function. In this process, a data fusion function with voice recognition data is used to improve the accuracy of speech recognition. The matching results are interpreted by the speech understanding function and used to provide appropriate responses and information. The voice output means by the speaker provides the speech content by synthetic voice to the hearing impaired. The voice output means by the speaker generates a natural voice that reflects the speaker's intention through a synthetic voice generation function, and converts this voice signal into a reproducible analog signal through a digital-to-analog conversion function. The voice synthesis means generates a voice having natural pronunciation and intonation according to the speech content, and outputs this from the speaker. The voice synthesis means uses text-to-speech (TTS) technology that converts text to speech, and has a built-in multilingual model that supports various languages ​​and dialects. The voice synthesis means generates a voice having pronunciation and intonation that supports different languages ​​and dialects through a multilingual text-to-speech function. An example of data collection that does not include sensors would be a user manually inputting speech via a keyboard or touch screen. This manual input method is particularly useful in noisy environments or when the speaker does not have access to a microphone. The manual user input method may also include a function where the user directly provides text information through an input interface, and the manually input information is subjected to analysis by the artificial intelligence via a text processing function. The manually input text data is similarly processed by the artificial intelligence and output as synthetic speech. The manually input text is also processed through a text analysis function and fed into a speech synthesis process for output as synthetic speech. For example, various functions can be added to each of the means of this embodiment. For example, the acquisition means can be added with a superdirectional microphone for instantaneous voice detection and analysis, thereby selectively acquiring only a specific person's voice and separating it from surrounding noise. The acquisition means can also incorporate a function for monitoring text-based communication from social media and messaging apps in real time and reproducing it as voice utterances. This allows the hearing impaired to directly receive online conversations as voice. The speech recognition means can utilize a multilingual model to handle multiple languages ​​and dialects, and can understand the speech of users with different cultural backgrounds. Furthermore, the speech recognition means can be equipped with an emotion analysis function to capture subtle changes in the speaker's voice in order to more accurately grasp the speaker's emotions and intentions. This makes it possible to reflect the speaker's emotions in the voice, rather than simply reading out the text, resulting in more human-like communication. The artificial intelligence means has the capability of continuous learning and self-improvement, and analyzes the speaker's speech characteristics and lip movement patterns more accurately over time. The artificial intelligence means can also be customized to the user's usage and preferences to provide the most natural and understandable speech analysis for each individual user. Such customizable AI can respond to the individual needs of each user, improving usability. Emotion recognition means can provide richer emotional information by adding a function to analyze not only the tone of a speaker's voice, but also subtle changes in facial expressions and body language. For example, it is possible to estimate the speaker's intention and the intensity of his or her emotions from the speaker's eye movements and hand gestures, achieving more accurate emotion recognition. Output means can be more diverse, using not only speakers, but also vibration and light signals to provide information to the hearing impaired. For example, smart devices can warn with vibration patterns or color changes depending on the speaker's emotional state and the urgency of what is being said. Information transmission systems can also be developed that use tactile information rather than synthetic voice to accommodate the visually impaired. These additional features can be implemented through hardware and software updates to existing smart devices, significantly improving the user experience. They can also be linked to cloud-based services to facilitate remote updates and customization, providing users with the latest technology at all times. Furthermore, these features can be widely applied as communication support tools in education, work, and daily life, and are expected to benefit a diverse range of users, including not only the hearing impaired, but also those with speech disabilities and non-native speakers. To further improve the acquisition method, an external noise canceling function can be added to collect clearer voice data. In addition, a real-time language conversion function can be introduced to facilitate communication with people who speak different languages. This will further strengthen the AI's ability to understand multiple languages ​​and expand its use in international communication. The AI ​​means will be further developed to have the ability to memorize what the user has said in the past and learn the individual's unique speaking style, which will enable the AI ​​to reproduce the user's speech more naturally and improve the quality of communication. It will also have an emotion-adaptive synthetic voice generation function that can read the user's emotions and respond with an appropriate tone. The speaker-based audio output means can be combined with a bone conduction headset or a vibrating wearable device so that the hearing impaired can experience sound more naturally. This allows the hearing impaired to intuitively grasp the rhythm and strength of the speaker's voice through the vibrations, enabling them to enjoy a richer communication experience. In addition, by adding customization features that allow users to adjust the volume and speed of audio output depending on the situation, it can be used to meet individual needs. It is also possible to incorporate a safety measure function that automatically increases the volume in an emergency. By integrating these functions, devices can be provided that are easier to use and more effective for the hearing impaired, as well as the elderly and other users who need to communicate in a variety of situations. Furthermore, these technologies are expected to be applied as educational support tools, remote communication, and multilingual dialogue systems, helping to promote social inclusion.

[0073] The process flow of each embodiment will be described below.

[0074] (Example 1) Step 1: Use your smartphone's microphone to pick up the other person's voice. Step 2: Use voice recognition technology to convert the other person's voice into text. Step 3: Mouth movements are detected using vibrations from the bone conduction microphone and the smartphone's built-in camera, and the content of what the other person is saying is understood by comparing this with data that the artificial intelligence has learned. Step 4: The speaker, which works in conjunction with a smartphone to play synthesized speech, provides the hearing-impaired person with audio of what the other person is saying. (Example 2) Step 1: Use the smartphone's built-in camera to detect the other person's mouth movements in real time. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. (Example 3) Step 1: The smartphone's bone conduction microphone is used to detect the vibrations of the other person's voice. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: The speaker, which works in conjunction with a smartphone to play synthetic speech, provides the hearing-impaired person with audio of what the other person is saying.

[0075] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0076] (Example 1) The embodiment of the present invention includes an acquisition means for acquiring the voice of the other party by a voice recognition means, an artificial intelligence for learning the movement of the mouth by the vibration of a bone conduction microphone and the built-in camera of a smartphone, a speaker for playing synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotions. Specifically, the other party's voice is acquired using the microphone of the smartphone and converted into text using voice recognition technology. In addition, the vibration of the bone conduction microphone and the built-in camera of the smartphone are detected, and the movement of the mouth is compared with the data learned by the artificial intelligence to understand the content of the other party's speech. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the speaker for playing synthetic voice in cooperation with the smartphone provides the other party's speech content and emotional state as voice to the hearing impaired person. The acquisition means is realized by the microphone 238 of the headset type terminal 314, acquires the voice of the other party, and converts it into voice data by the processor 46. The artificial intelligence is realized by, for example, the specific processing unit 290 of the data processing device 12, learns the vibration of the bone conduction microphone and the movement of the mouth by the built-in camera of the smartphone, and grasps the contents of the speech. The speaker is realized by the speaker 240 of the headset type terminal 314, and plays a synthetic voice. The emotion engine is realized by, for example, the specific processing unit 290 of the data processing device 12, analyzes the tone of the other party's voice and changes in facial expression, and estimates the emotional state. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The acquisition means has a function of accurately acquiring the voice of the other party as digital data by utilizing voice recognition technology. For example, the acquisition means acquires clear voice data while eliminating surrounding noise by using a high-sensitivity microphone equipped with a noise cancellation function. The acquisition means can also acquire the speech of the other party as digital voice data by using the voice recognition means via a microphone on the smart device. This voice data can clearly capture the human voice by the high-sensitivity microphone while reducing environmental noise by a noise canceling function. The voice recognition means performs processing to convert the spoken content into text data based on the acquired voice data, thereby achieving highly accurate text conversion. This process includes feature extraction from the voice signal, segmentation of utterance units, and recognition processing based on a voice model. The voice recognition means can also identify the speaker's voiceprint and speech pattern using a characteristic analysis function of the voice signal, and perform preprocessing to improve the accuracy of voice recognition. The acquired voice data can be subjected to voice-to-text conversion processing by extracting features of the voice data, performing segmentation, and converting the voice into text. The artificial intelligence means includes analyzing vibration patterns of a bone conduction microphone and capturing and analyzing mouth movements using a built-in camera. The means performs processing to infer the speaker's speech content from the acquired vibration data and video data. The artificial intelligence means can also capture the speaker's mouth movements and head vibrations using the bone conduction microphone and the built-in camera. The artificial intelligence means can analyze these data and understand the speaker's speech content. The bone conduction data analysis means analyzes the head vibrations captured by the bone conduction microphone and associates them with the voice to understand the speech content. The mouth movement analysis means can use video data captured by the smartphone camera to track the speaker's mouth movements and associate them with the speech content. These means are trained using machine learning techniques to learn the speaker's individual speech characteristics and achieve high-precision speech understanding in real time. The emotion recognition means has a function of estimating the emotional state of a speaker by using both voice data and video data. This means includes a voice emotion analysis means based on the analysis of the tone and intonation of the voice, and a facial expression analysis means that captures changes in facial expressions. The voice emotion analysis means analyzes the pitch, intensity, acoustic characteristics, etc. of the voice data to estimate the emotional state of the speaker. The emotion engine also has both a voice emotion analysis function and a facial expression analysis function, and can estimate the emotional state of the speaker. The facial expression analysis means detects the facial expressions of the speaker from the video data captured by the built-in camera, and estimates the emotional state by capturing subtle changes in facial expressions. These means are trained using an emotion recognition model, and by identifying emotional patterns, grasp the speaker's emotions in real time. The output means includes a speaker that cooperates with the smartphone to generate synthetic voice and provide it to the user. The means includes a text-to-speech (TTS) function that converts text to voice, and the tone and intonation of the synthesized voice are adjusted to reflect the speaker's spoken content and emotions. Furthermore, the output voice has clear pronunciation and an appropriate speed so that the hearing impaired can easily understand it. The output means through the speaker can also generate synthetic voice and provide the hearing impaired with the other party's spoken content and emotional state by voice. The voice flowing from the speaker is designed so that the user can easily grasp the other party's spoken content and emotions. This allows the user to grasp the other party's words not only by hearing but also by combining them with visual information. As an extension of the functions related to the acquisition means, it is possible to selectively pick up the voice of a specific speaker by using a directional microphone that is resistant to external noise. In addition, by utilizing a combination of a smart device and a wearable device, the user's body movements and gestures can be acquired to assist in understanding non-verbal communication. Furthermore, it is conceivable to utilize beamforming technology through the acquisition means to identify individual speakers even in situations where multiple people are talking and accurately grasp the content of each speaker's speech. In order to improve the accuracy of lip reading, the artificial intelligence means uses a high-resolution camera to capture more detailed mouth movements. In addition, by developing an algorithm that analyzes the intention of communication not only from the speaker's mouth movements but also from facial expressions and body language, a richer understanding of communication is realized. The artificial intelligence means can build a lip reading model that supports multiple languages ​​and can also handle international communication. Regarding emotion recognition, more accurate emotion recognition can be achieved by incorporating a biofeedback sensor that obtains physiological indicators such as heart rate and skin electrical conductivity and uses them to estimate the emotional state. In addition, a customized emotion recognition model can be built by learning an individual's emotional patterns, providing emotion recognition adapted to each individual user. The emotion recognition means can also capture subtle differences in emotions expressed by users in different contexts and social situations. As for the output means, a personalized audio system that adjusts the frequency band according to the user's hearing characteristics can be introduced, making it possible to provide audio tailored to each individual hearing-impaired person. In addition, by incorporating a function into a smartphone application that uses visual information to display the other person's speech in text in real time, it becomes possible to obtain information from both audio and vision. Furthermore, by combining synthetic voice with an expressive animated avatar, visual communication can be enhanced and more intuitive understanding can be supported. The acquisition method can be supplemented with an active noise canceling function that analyzes the sound environment around the user in real time and performs optimal noise reduction. In addition, by utilizing bone conduction earphone technology that allows users to hear sound without blocking their ears, it is possible to provide users with clear sound while still being aware of the surrounding environmental sounds. Furthermore, by linking with devices other than smartphones, such as smart watches and smart glasses, voice data can be acquired using multiple devices, creating a more comprehensive communication environment. Regarding artificial intelligence means, multitask learning will be introduced to simultaneously analyze not only lip reading but also non-verbal information such as the speaker's posture and hand movements to gain a deeper understanding of the context of communication. In addition, by using edge computing technology to perform part of the data processing on the smart device side, it is possible to reduce communication delays and improve real-time performance. Furthermore, a function will be developed that uses artificial intelligence to automatically identify and translate the language used by the speaker, and to support communication between different languages. The emotion recognition method combines machine vision technology to capture subtle changes in facial expressions and skin color, improving the accuracy of emotion analysis. In addition, bio-signals obtained from wearable devices are used for emotion recognition to more accurately evaluate the user's psychological state, such as stress level and fatigue. In addition, situation recognition technology is introduced to adjust the emotion recognition algorithm according to the user's environment and situation, allowing for more appropriate emotion estimation. As for output means, a visual assist function will be added that adjusts the size and contrast of text display to suit the user's visual acuity characteristics. In addition, an avatar-based sign language translation system will be incorporated that recognizes and generates sign language and gestures necessary for the hearing impaired to communicate with others. An emotional expression assist function will also be developed that allows users to convey their emotional state to others, realizing rich communication that includes emotions. Through these additional functions, the invention can provide a more comfortable and effective means of communication not only for the hearing impaired but also for users with various communication disabilities. In addition, by combining these functions, it is possible to reduce misunderstandings that occur during communication and realize smoother dialogue. (Example 2) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by vibration of a bone conduction microphone and the built-in camera of a smartphone, a speaker for playing a synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotion. Specifically, the smartphone's built-in camera is used to detect the other party's mouth movement in real time, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the other party's voice is converted into text using voice recognition technology and displayed on the smartphone screen. The acquisition means is realized by the microphone 238 of the headset terminal 314, acquires the voice of the other party, and converts it into voice data by the processor 46. The artificial intelligence is realized, for example, by the specific processing unit 290 of the data processing device 12, detects the movement of the mouth by the built-in camera of the smartphone in real time, and grasps the contents of the speech. The emotion engine is realized, for example, by the specific processing unit 290 of the data processing device 12, analyzes the tone of the other party's voice and changes in facial expression, and estimates the emotional state. The display means is realized by the display 343 of the headset terminal 314, and converts the other party's voice into text and displays it on the smartphone screen. The correspondence between each means and the device or control unit is not limited to the above example, and various changes are possible. The voice recognition means includes a voice data acquisition function using advanced voice signal processing technology. This function uses a noise suppression algorithm to remove ambient noise and clearly capture the other party's voice. The noise suppression algorithm distinguishes between human voices and background noise through frequency analysis and subtracts noise components in real time. The voice recognition means also uses a machine learning voice recognition model to identify the speaker and the contents of the speech from the voice. This model has the ability to learn large amounts of voice data and identify the voice characteristics and speech patterns of a specific speaker. The voice data acquisition function, which uses voice recognition technology to acquire and analyze voice data obtained from the other party's voice, also utilizes an advanced signal processing algorithm to remove ambient noise and clearly identify the speaker's voice. This function improves the quality of communication by identifying and removing noise components in the voice data. In addition, as a speaker identification function, a model based on machine learning is used to learn the voice characteristics of each speaker and identify who the speaker is from the voice data. The acquisition means includes a mouth movement detection function using the camera of the smart device. This function combines image recognition technology and machine learning algorithms to infer the content of the speech from the shape and movement of the other person's mouth. The detected mouth movement data is sent to a data processing device in real time and analyzed by artificial intelligence. In this analysis process, the data is compared with a trained data set and the speech content is understood through lip reading. The artificial intelligence uses face recognition technology to identify a specific individual, learns the characteristics of the person's mouth movement, and builds a correspondence with the speech content. In addition, the mouth movement detection function uses the camera of the smart device to acquire an image of the other person's face, and uses lip reading technology to infer the speech content from the movement. This function detects the shape and movement of the mouth from the face image, and uses a learned lip reading model to analyze the words spoken by the speaker. In this process, the face recognition function is also utilized to recognize the mouth movement patterns of a specific individual and assist in understanding the speech content. The speaker means converts the text data into a voice having natural pronunciation and intonation using synthetic voice generation technology. This technology can reflect not only the accuracy of pronunciation but also the emotion and intention of the speaker in the text-to-voice conversion. The speaker may be built into the smart device, or may be an external wireless speaker that transmits an audio signal. In either case, an acoustic processing technology is employed to ensure a sound quality and volume that is easy for the user to hear. The speaker means for playing the synthetic voice includes a text-to-voice conversion function, which converts the text data into a voice having natural pronunciation and intonation using synthetic voice generation technology. This technology can reflect the emotion and intention of the speaker in the voice, and depending on the speaker used, a sound quality adjustment function and a volume adjustment function are also used in combination. The emotion recognition means has a function of estimating emotions from audio signals and video data. This function uses a voice emotion analysis algorithm that analyzes changes in voice tone, speech rate, and intonation, and a facial expression recognition algorithm that captures changes in facial expressions and eye movements. The emotion recognition means integrates these data to infer the emotional state of the other person. The emotion estimation result becomes important information in the context of communication and is provided as feedback to help the user understand the emotions of the other person. The emotion recognition means also estimates the emotional state of the speaker by analyzing the audio signals and video data. This means includes a voice emotion analysis algorithm that monitors changes in tone, speech rate, and intonation. It also includes a facial expression recognition algorithm that detects changes in facial expressions and eye movements from the video data. By integrating these pieces of information, the other person's emotional state is estimated in real time and the result is provided to the user. An example of data collection that does not involve sensors is when a user inputs a text message via a keyboard or voice input function of a smart device and transmits the message through a communication means. This method allows the user to communicate with the other party without relying on voice or video. Another example of data collection that does not involve sensors is text-based communication that is manually input by a user. The user inputs his / her own words as text using a keyboard or voice input function of a smart device and transmits the text to the other party. This method allows direct text communication without requiring a voice recognition means or a lip reading means. The voice recognition means, acquisition means, speaker means, and emotion engine in the embodiments of the present invention can further add other functions. For example, the voice recognition means can employ binaural recording technology and reproduce a more natural acoustic space using two microphones positioned at the position of human ears. This allows the user to easily sense the direction and distance of the other person's voice, allowing the user to enjoy a more realistic communication experience. Furthermore, the voice recognition means can be used to realize the operation of smart devices and the control of home automation systems by voice commands. The acquisition means may incorporate a feedback function to assist the user in speaking. The feedback function may provide guidance on mouth movements and pronunciation when the user speaks. Furthermore, the acquisition means may include a noise canceling function to effectively eliminate external noise and improve the accuracy of detecting mouth movements. The speaker means can have a function to automatically adjust the sound quality according to the user's hearing condition and hearing preferences. Advanced digital signal processing technology is utilized to provide the listener with an optimal hearing experience. The speaker means can also be added with a function to link with external smart home devices and in-car systems to expand the reach of the voice wherever the user is. The emotion engine can incorporate sensors to detect the user's physiological reactions and add a function to more accurately estimate the user's emotional state from changes in heart rate and electrical resistance of the skin. Such biofeedback functions can monitor the user's stress level and excitement in real time, deepening the understanding of emotions in communication. With the addition of these functions, the embodiment of the present invention becomes a more advanced communication support tool, enabling users to have richer and more emotionally rich conversations with others. These functions also provide effective support for users with disabilities or when communication is difficult in a particular environment. Furthermore, as smart devices evolve, these functions are expected to be expanded and optimized in the future to accommodate the needs of a variety of users. The voice recognition means of this embodiment can be equipped with a sensitivity enhancement function that accurately captures the contents of speech even when the user whispers or speaks at a very low volume. This function combines a special microphone array with an algorithm that analyzes ambient noise to make the speaker's voice stand out. In addition, a directional pickup function can be provided to selectively capture the voice of a specific speaker in a situation where multiple speakers are speaking at the same time. Acquisition methods can also be enhanced with real-time translation capabilities to instantly translate utterances spoken in different languages ​​into the user's native language. This functionality works with multilingual databases to reduce global communication barriers. Acquisition methods can also employ region-specific models to identify speaker accents and dialects for more accurate transcription and translation. The speaker means may be added with a personalized equalization function to analyze the user's hearing adaptation and provide an optimal hearing experience. This function realizes audio output that is tailored to the individual ear characteristics and hearing preferences of the user. The speaker means may also provide a virtual reality experience that enhances spatial awareness by adding 3D audio effects to the voice guidance. The emotion engine can be equipped with an emotion response function that tracks the user's emotional changes during the conversation and provides supportive and encouraging messages at the right time. The emotion engine can also estimate the user's psychological stress level from their behavior and speech patterns and suggest advice and relaxation techniques to reduce stress. These additional functions will provide users with a more detailed and personalized communication experience. Furthermore, these technologies are expected to be applied in fields such as education, business, and healthcare, contributing to improving the quality of communication and promoting accessibility. In addition, by providing interactive feedback according to the user's behavior and situation, more natural human-machine dialogue will be realized. The evolution of these technologies will promote a deeper level of user engagement in the future, laying the foundation for providing more valuable services to users. (Example 3) The embodiment of the present invention includes an acquisition means for acquiring the voice of the other party by a voice recognition means, an artificial intelligence for learning the vibration of a bone conduction microphone and the movement of the mouth by the built-in camera of a smartphone, a speaker for playing synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotions. Specifically, the bone conduction microphone of the smartphone is used to detect the vibration of the other party's voice, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the speaker for playing synthetic voice in cooperation with the smartphone provides the other party's speech content and emotional state as voice to the hearing impaired person. The acquisition means is realized by the microphone 238 of the headset type terminal 314, acquires the voice of the other party, and converts it into voice data by the processor 46. The artificial intelligence is realized by, for example, the specific processing unit 290 of the data processing device 12, detects vibrations of the bone conduction microphone, and grasps the contents of the speech. The speaker is realized by the speaker 240 of the headset type terminal 314, and plays synthetic voice. The emotion engine is realized by, for example, the specific processing unit 290 of the data processing device 12, and analyzes changes in the tone of the other party's voice and facial expression, and estimates the emotional state. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various modifications are possible. The voice recognition means has a function of acquiring surrounding voice data through a microphone built into the smart device, and the acquired voice data is pre-processed by digital signal processing, such as noise removal and amplification, and converted into clear voice information. The voice recognition means also has a voice acquisition function for capturing voice from the surroundings, and applies a voice pre-processing function to the acquired voice data. This pre-processing includes the removal of environmental noise by a noise reduction function and the optimization of the volume by a gain adjustment function. This makes the voice signal clearer and converts it into a state suitable for subsequent processing. Next, based on this pre-processed voice data, a voice analysis algorithm extracts specific voice features and improves the accuracy of voice recognition. The extracted voice features include the frequency, amplitude, pitch, duration, etc. of the voice, which are important indicators for understanding the characteristics of the speaker's voice and the content of the words. The voice analysis algorithm also extracts features such as the frequency characteristics and temporal patterns of the voice through a feature extraction function, and grasps the meaning of the voice data using a speaker identification function and a speech content analysis function. The AI ​​learns the vibration patterns of speech detected by the bone conduction microphone as a data set and identifies the content of speech. This learning process uses multiple machine learning algorithms including deep learning technology. In particular, convolutional neural networks are suitable for extracting features from image data and are used to analyze mouth movements captured by the built-in camera of a smartphone. The AI ​​integrates both audio data and visual data and performs multimodal information processing, enabling accurate identification and understanding of the content of speech. The AI ​​also combines the analysis function of vibration data from the bone conduction microphone with the visual data analysis function to accurately identify the content of speech. The vibration data analysis function captures the physical characteristics of the voice obtained by bone conduction and compares it with the audio data. The visual data analysis function identifies the patterns of mouth movements obtained by the smartphone camera and integrates and processes this information to enable more advanced recognition from multimodal information. The speaker is built into the smart device and generates synthetic speech to provide audio information to the user in an easy-to-listen format. The synthetic speech is generated using text-to-speech technology and is designed to reproduce not only the content of the speech, but also changes in tone and intonation that express emotions. This enables the hearing impaired to understand not only the content of the speech of the other person, but also its emotional nuances. The speaker also converts text information into speech using a synthetic speech generation function, providing easy-to-listen audio information to the user. The synthetic speech adjustment function mimics the emotional elements of speech, including tone and intonation, to reproduce the richness of emotional expression during communication. The emotion recognition means performs a composite emotion analysis by integrating acoustic features such as tone, pitch, and speed of the voice with visual information such as changes in facial expressions captured by the smartphone camera. In this process, a machine learning model specialized in emotion recognition is used, and the user's emotions can be classified into multiple categories such as positive, negative, surprise, and sadness. As a result, the emotion recognition means estimates the user's emotional state and provides an emotional context in the communication. In addition, the emotion recognition means extracts emotion information from the voice data and visual data using an emotion analysis algorithm. This analysis includes a voice emotion analysis function and a facial expression analysis function, which are integrated to perform a multidimensional analysis of the user's emotional state, and the emotion estimation function understands the user's emotions. This makes it easier for the user to grasp the other person's emotions, improving the quality of communication. In implementing this embodiment, it is useful to consider further functional expansion. For example, it is possible to introduce beamforming technology into the acquisition means, which selectively acquires a specific voice from among surrounding sounds. This makes it possible to clearly capture the speaker's voice even in a noisy environment and improve the accuracy of voice recognition. In addition, a multilingual processing function for handling different languages ​​and dialects can be added to the artificial intelligence to accommodate a wide range of users. It is also possible to reduce communication barriers with foreign language speakers by incorporating a real-time translation function using artificial intelligence. As for the speakers, a function to adjust the frequency characteristics of the sound to suit the user's hearing characteristics will be added, providing even more personalized listening ease. The emotion engine will be able to convey emotional fluctuations to the user in line with the flow of the conversation by capturing emotional changes in real time. In addition, a context-aware function will be added to the emotion engine, which will estimate emotions based on the context of the conversation, achieving more accurate emotion recognition. Furthermore, it is conceivable to incorporate a biosensor function that acquires the user's biometric information and uses that information to more precisely estimate the other person's emotions and state. This will further improve the quality of communication. Also, adding an assistive communication function that learns the user's communication patterns and prompts them to speak at the appropriate time will enable smoother conversations. In addition, collaboration with wearable devices and smart glasses will be strengthened to expand communication support using visual information. By providing visual feedback such as text, emojis, and videos through smart glasses, the hearing impaired will be able to better understand the emotions and intentions of others. The smart glasses will also be equipped with an audio visualization function that visualizes the surrounding environmental sounds, allowing users to visually understand the direction and strength of sounds. This will make it easier for users to understand the surrounding sound environment from visual information. With these functional extensions, this embodiment is expected to greatly improve the communication experience of users, and contribute to the realization of a more inclusive society by providing a customizable communication means that meets the individual needs of each user. In addition to the acquisition means of this embodiment, it is useful to consider a function for capturing bone conduction vibrations more accurately using a wearable device worn near the user's ear. This wearable device is designed to fit the ear and is structured to be less susceptible to external noise. Furthermore, by developing technology to improve the sensitivity of the bone conduction microphone itself, it is possible to detect more subtle vibrations of speech and increase the accuracy of understanding the content of speech. The AI ​​training dataset will include a wide variety of languages ​​and dialects, as well as the voice patterns of speakers with specific disabilities, to accommodate a wider range of users. In addition, when performing real-time speech recognition, it will incorporate context-based predictive modeling to more accurately grasp the intent of the spoken content. This will enable the system to accurately understand the meaning of words in the context of the dialogue. The synthesized voice from the speaker will be equipped with a function that allows the pitch and speed of the voice to be adjusted according to the user's level of hearing impairment and preferences. This will allow the user to hear voices that are easy to understand. In addition, the emotional expressions of the synthesized voice will be able to reflect emotional characteristics extracted from recordings of actual human voices, making it possible to convey emotions more naturally and realistically. The emotion engine will be able to perform more comprehensive emotion recognition by incorporating sensor technology that analyzes not only language but also non-verbal elements such as gestures and body postures. It will also be able to realize more personalized emotion recognition by taking into account the relationship between the user and the other party and past conversation history. To improve the user's communication experience, a smartphone app will be built with an interface that conveys information visually. This interface will include a function that displays voice information as text and a function that expresses emotions with colors and patterns. This will allow the hearing impaired to communicate through visual information as well. This interface can also be customized according to the user's usage situation, making it more intuitive and easy to use. These functional enhancements will broaden the scope of communication for the hearing impaired and promote their fuller participation in society. Technological developments that reduce the communication barriers faced by the hearing impaired and enable smooth dialogue in a variety of situations will contribute to the advancement of inclusion in society.

[0077] The process flow of each embodiment will be described below.

[0078] (Example 1) Step 1: Use your smartphone's microphone to pick up the other person's voice. Step 2: Use voice recognition technology to convert the other person's voice into text. Step 3: Mouth movements are detected using vibrations from the bone conduction microphone and the smartphone's built-in camera, and the content of what the other person is saying is understood by comparing this with data that the artificial intelligence has learned. Step 4: The emotion engine analyzes the other person's tone of voice and changes in facial expressions to estimate their emotional state. Step 5: A speaker that works in conjunction with a smartphone to play synthetic speech provides the hearing-impaired person with audio of the other person's speech and emotional state. (Example 2) Step 1: Use the smartphone's built-in camera to detect the other person's mouth movements in real time. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. Step 4: The emotion engine analyzes the other person's tone of voice and changes in facial expressions to estimate their emotional state. Step 5: A speaker that works in conjunction with a smartphone to play synthetic speech provides the hearing-impaired person with audio of the other person's speech and emotional state. (Example 3) Step 1: The smartphone's bone conduction microphone is used to detect the vibrations of the other person's voice. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: The emotion engine analyzes the other person's tone of voice and changes in facial expressions to estimate their emotional state. Step 4: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. Step 5: A speaker that works in conjunction with a smartphone to play synthetic speech provides the hearing-impaired person with audio of the other person's speech and emotional state.

[0079] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0080] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing while using the data generation model 58.

[0081] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314. [Fourth embodiment]

[0082] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0083] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0084] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0085] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0086] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0087] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0088] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0089] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0090] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0091] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0092] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0093] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0094] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a ChatGPT server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, or a home appliance) owned by a user. Next, the identification process by the identification processing unit 290 of the data processing device 12 will be described.

[0095] (Example 1) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by the vibration of a bone conduction microphone or the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the other party's voice is acquired using the microphone of the smartphone and converted into text using voice recognition technology. In addition, the movement of the mouth is detected by the vibration of the bone conduction microphone or the built-in camera of the smartphone, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the content of the other party's speech is provided as a voice to the hearing-impaired person from the speaker for playing the synthetic voice in cooperation with the smartphone. The acquiring means is realized, for example, by the specific processing unit 290 of the data processing device 12, and converts the other party's voice into text using voice recognition technology. Also, a means for detecting the vibration of a bone conduction microphone or the movement of the mouth using a built-in camera of a smartphone and collating it with data learned by the artificial intelligence to grasp the contents of the other party's speech may be realized, for example, by the control unit 46A of the robot 414. A means for playing a synthetic voice from a speaker is realized, for example, by the speaker 240 of the robot 414. The correspondence between each means and the device or control unit is not limited to the above example, and various changes are possible. The voice recognition means can use an advanced voice recognition algorithm to process voice data acquired from a microphone built into the smart device in real time and convert it into text data. The voice recognition means also has a keyword spotting function that analyzes voice data and identifies specific keywords and phrases, and can also instantly extract important information. The voice recognition means also has a text conversion function that converts the voice acquired from the microphone of the smart device into a digital signal and converts the digital signal into text data, and this text conversion can eliminate ambient noise by a voice separation function and extract only clear voice signals. Furthermore, it includes a semantic analysis function for identifying words and phrases with important meanings from the voice data, which makes it possible to understand the context of the dialogue and instantly capture necessary information. The learning means integrates vibration data from the bone conduction microphone and video data from the smart device's built-in camera to learn the speaker's mouth movements and pronunciation characteristics. The learning means can also fuse the vibration patterns from the bone conduction microphone with the visual information from the smart device's camera. This fusion process learns the correlation between the speaker's facial movements and voice, and more accurately interprets the content of the voice by using lip reading as an aid even when the voice signal is unclear. This process includes a face tracking function that tracks facial movements and a vibration analysis function that analyzes vibration patterns, which deepen learning based on data collected from multiple speakers. The speaker conveys the contents of the other person's speech to the hearing-impaired person by generating and outputting synthetic speech. The speaker also has an output function that generates synthetic speech through a speech synthesis process and conveys the speech to the user. The speech synthesis process includes a waveform generation function that synthesizes speech waveforms based on text data, and an articulation function that adjusts the articulation of the synthesized speech. This converts the text data into natural speech that resembles a human voice and provides it in a form that can be easily understood by the hearing-impaired person. A manual data collection method includes a function in which a user inputs text information using a touch screen of a smart device, and a speech synthesis means generates speech based on the text. An example of data collection that does not include a sensor is when a user directly inputs text via a touch screen of a smart device, and the text data is output as speech by a speech synthesis means. This method is suitable for users who use sign language as an alternative to speech recognition, and helps them communicate. This method is also effective in situations where speech data is not available or when you do not want to rely on speech recognition. The text information entered by the user is converted into a clear and easy-to-understand speech message by a speech synthesis process. It is possible to expand the functions of the speech recognition means and the learning means by artificial intelligence in the present invention. For example, the speech recognition means can be added with a speaker identification function that identifies the characteristics of an individual's voice and determines who is speaking. This makes it possible to accurately grasp the content of each speaker's speech even in a situation where multiple people are talking, and to provide clearer communication to hearing-impaired people. Furthermore, by incorporating an emotion recognition algorithm into the speech recognition means, it is possible to capture the emotional nuances of the speech and reflect the emotional expression in the synthetic speech. The learning means can be combined with face recognition technology to learn real-time changes in mouth movements and subtle changes in facial expressions to achieve more accurate lip reading. This makes it easier to accurately interpret the speaker's intention even when the audio signal is unclear. The learning means also has a multilingual learning function to accommodate different languages ​​and dialects, making it effective in international communication situations. The speaker has an automatic volume adjustment function that adjusts the volume according to the user's hearing level, and provides optimal audio output according to the usage situation of the hearing impaired.The speaker also has an environmental adaptation function that automatically adjusts the clarity of the sound according to the surrounding noise level, and always provides the best hearing experience while responding to changes in the external environment. Manual input data collection methods may include predictive text entry to allow users to easily edit selected text and custom dictionary features for technical terms and abbreviations that are difficult to recognize through speech, allowing users to enter information more quickly and accurately, resulting in more accurate speech. In addition, the system can be used not only by the hearing impaired, but also as a multilingual interpretation device, using voice recognition to translate foreign language conversations in real time, thereby reducing communication barriers between people who speak different languages ​​and promoting international exchange. It can also be incorporated with the ability to provide real-time subtitles during video calls and online meetings with remote locations, helping the hearing impaired to participate more actively in social activities. The voice recognition means of this embodiment further includes an environmental sound recognition function, which detects important sounds other than conversation, such as emergency vehicle sirens and fire alarms, and provides them to the hearing impaired as visual alerts or vibration signals. This improves safety and enables rapid response to dangerous situations. In addition, the voice recognition means incorporates a function that automatically measures the surrounding noise level and optimizes the voice recognition accuracy based on that information, thereby achieving highly reliable recognition even in noisy environments. The AI ​​learning method can learn from a user's usage and speech habits to provide more personalized lip reading assistance. For example, it can recognize patterns of words and phrases commonly used by a particular user and improve the accuracy of lip reading based on that. It can also detect a user's emotional state and stress level and provide feedback to improve the quality of communication. The speaker can add richer speech expressions to the synthesized speech to make it easier for the hearing impaired to understand as a more natural speech. For example, it can adjust the tone and intonation of the speech to reflect the speaker's emotion or emphasis. It can also add a personalization feature that allows users to record their own voice and incorporate that voice quality into the synthesized speech. For manual data collection methods, incorporating gesture control and eye-tracking technology will make text entry easier for users with disabilities or in situations where touchscreen operation is difficult, and providing audio feedback during text entry will make it easier to correct input errors and further improve communication efficiency. These functions are designed to respond to various scenarios that users face, making it easier for the hearing impaired to become more independent and participate in society. They also function as support tools to reduce communication barriers in the daily lives of the hearing impaired and facilitate smooth communication with people with normal hearing. This will enable the hearing impaired to live a richer social life and contribute to realizing an open communication environment for all. (Example 2) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by vibration of a bone conduction microphone or the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the movement of the other party's mouth is detected in real time using the built-in camera of the smartphone, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the voice of the other party is converted into text using voice recognition technology and displayed on the smartphone screen. The acquisition means is realized by the specific processing unit 290 of the data processing device 12 as a means for grasping the contents of the other party's speech by detecting the movement of the other party's mouth in real time using, for example, a built-in camera of the smartphone and comparing it with data learned by the artificial intelligence. A means for converting the other party's voice into text using a voice recognition technology and displaying it on the smartphone screen may be realized by, for example, the control unit 46A of the robot 414. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means includes a function for capturing and analyzing voice signals using advanced voice processing algorithms. This means combines a noise canceling function for reducing environmental noise with a speaker identification function for identifying the speaker's voice to accurately capture the other party's voice. The voice recognition means also includes a highly sensitive microphone for capturing the speaker's voice and a voice analysis processor for analyzing the captured voice signal. This means can utilize echo cancellation and beamforming techniques to achieve high recognition accuracy even in complex acoustic environments. The voice recognition means can also analyze the captured signal and use it as data for converting the spoken content into text. The acquisition method is to detect mouth movements using the smartphone's built-in camera, and also to acquire vibrations using a bone conduction microphone. The built-in camera detects mouth movements using high-speed, highly accurate image processing technology and analyzes the captured video data. This analysis combines face recognition technology and lip reading technology to predict the content of the other person's speech from their mouth movements. The bone conduction microphone uses a vibration sensor to detect the speaker's jaw movements and vocal cord vibrations, and can acquire these signals as voice data. The speaker as an output means outputs synthetic speech generated through a speech synthesis engine. The speech synthesis engine includes a prosody generation function that adjusts the pitch and speed of the speech in order to convert text information into natural speech. The pronunciation model can also generate pronunciation patterns for various words and phrases so that the synthetic speech sounds closer to the voice of an actual speaker. The AI ​​uses deep learning networks and statistical learning models as a learning vehicle to extract speech features from large amounts of audio and visual data. This learning is used to identify the acoustic characteristics of a particular language or dialect, aiming to accurately recognize and understand the speech. The learning process uses approaches such as supervised learning and reinforcement learning, and new data is incorporated to continuously improve the model's performance. The AI ​​learning vehicle is also trained on samples taken from an extensive audio database to extract features from the audio data and gain the ability to identify a particular language or dialect, as well as understand the characteristics of an individual user's speaking style and accent. The learning vehicle can update the model using feedback data for continuous performance improvement. The manual input data collection method provides an interface for users to directly input text information. Users input information using the touch screen keyboard or voice input function of their smartphone, which is then received by the system. This input data is processed like any other speech recognition result and can be used for text display or voice output. To expand the functionality of the acquisition means, an acceleration sensor that analyzes vibrations can be incorporated in addition to the bone conduction microphone. This allows the speaker's subtle head and body movements to be detected, and information related to speech to be acquired from a wider range of angles. Furthermore, by using a wearable device to capture hand movements and changes in facial expression, additional communication information can be collected from gestures and facial expressions, helping to understand the content of speech. In addition, by using the built-in camera of a smartphone, it is possible to track the speaker's eye movements and blinks to infer attention and intent, enhancing contextual awareness using visual information and enabling more natural and intuitive interactions. To improve the speaker's functionality, a function can be added that automatically adjusts the frequency and volume of the sound to suit the user's hearing characteristics, providing optimal audibility according to environmental noise and the user's hearing characteristics. The AI ​​learning method will deeply analyze the correlation between voice data and visual data to more accurately understand non-verbal communication elements, and multi-modal learning will combine both voice and visual information for complex learning to improve the accuracy and naturalness of the dialogue system. For manual input, we will utilize the smartphone's motion sensors and gesture recognition to provide input methods other than keyboard input. For example, we will build an input system using air gestures and eye tracking to allow users to provide data to the system more intuitively. By combining these features, users can communicate not only what they say, but also non-verbal information such as gestures and facial expressions to the system, and the system can synthesize this information to achieve more accurate communication. These additional features can also be customized according to specific situations and user needs, providing the flexibility to accommodate a variety of usage scenarios. In terms of acquisition methods, more advanced voice separation technology will be introduced to selectively separate the speaker's voice from surrounding noise and other voices to acquire clearer voice data. In addition, it will have the ability to analyze environmental sounds in real time and understand the atmosphere and situation of the place. This will improve the accuracy of voice recognition and enrich the information obtained from voice data. The speaker, which serves as the output means, has the ability to learn the user's hearing characteristics and automatically adjust the sound quality accordingly. This allows for audio that is easy for each user to hear, making communication more comfortable. In addition, by incorporating spatial audio technology, audio can be heard from the direction of the speaker, providing a more natural conversation experience. The AI ​​is trained with an extensive voice database to handle different languages ​​and accents, making it suitable for global communication. It also learns the speaking style of each individual user to enable personalized dialogue. In addition, emotion recognition technology is introduced to identify the speaker's emotions and respond accordingly, making dialogue more human-like. Manual input data collection adds the ability to recognize a user's handwriting or sketches and convert them into text and graphical information, allowing users to provide information to the system through natural writing. These extensions enable the system to grasp not only the user's speech but also their emotions and intentions more accurately, and to gain a deeper understanding of the user's experience. These features also increase the value of the system as an accessible communication tool for users with special needs, such as those with physical or speech disabilities. This allows a wider range of users to enjoy communication with society. (Example 3) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the vibration of a bone conduction microphone and the movement of the mouth by the built-in camera of a smartphone, and a speaker for playing a synthetic voice in cooperation with the smartphone. Specifically, the bone conduction microphone of the smartphone is used to detect the vibration of the other party's voice, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Then, the content of the other party's speech is provided as a voice to the hearing impaired person from the speaker for playing the synthetic voice in cooperation with the smartphone. The acquisition means is realized by the specific processing unit 290 of the data processing device 12 as a means for grasping the contents of the other party's speech by detecting vibrations of the other party's speech using, for example, a bone conduction microphone of a smartphone and comparing it with data learned by the artificial intelligence. The means for playing the synthetic voice from a speaker is realized by, for example, the speaker 240 of the robot 414. The correspondence between each means and the device or control unit is not limited to the above example, and various changes are possible. The voice recognition means has a function of removing environmental sounds and surrounding noises using an advanced voice processing algorithm and extracting only the other party's voice with high accuracy. The voice recognition means also includes an automatic voice recognition system that converts a voice signal into text, and this system is trained using a deep learning model and has the flexibility to accommodate various accents and speaking styles. The voice recognition means also removes background noise through an environmental noise reduction function and highlights the other party's voice using a target voice extraction function. The extracted voice signal is analyzed for frequency characteristics and amplitude fluctuations through a waveform analysis function, and is converted into features reflecting the speaker's voice quality and speech characteristics through a voice feature extraction function. This information is transmitted to a data processing device via a data transfer function. The automatic voice recognition system also includes a data-driven learning process that utilizes a learning model based on a deep neural network to adapt to various speech patterns and accents. The vibration detection means using a bone conduction microphone detects the vibration of the skull that occurs when speaking and converts it into an electrical signal. The vibration detection means using a bone conduction microphone also analyzes the characteristics of the vibration transmitted through the skull and converts it into a digital signal using an analog-to-digital conversion function. The vibration is analyzed to extract features related to the content of the speech through the vibration pattern analysis means. Meanwhile, the mouth movement detection means using a built-in camera analyzes the mouth movements captured by the camera using image processing technology and tracks the shape and movement of the mouth during speaking. The mouth movement detection means also precisely analyzes the mouth movements captured by the video image processing function using a mouth shape analysis function and a movement tracking function. These data are fused by the integrated analysis function to perform a comprehensive speech analysis that combines voice recognition and visual information. The AI ​​extracts features related to the speech content based on the vibration data and lip movement data, and compares them with a learning database using a matching means. The matching means utilizes a database containing a large amount of speech data and determines the speech content using a machine learning algorithm. The AI ​​also compares the features of the vibration data and lip movement data with an existing database through a database matching function, and identifies the speech content using a pattern matching function. In this process, a data fusion function with voice recognition data is used to improve the accuracy of speech recognition. The matching results are interpreted by the speech understanding function and used to provide appropriate responses and information. The voice output means by the speaker provides the speech content by synthetic voice to the hearing impaired. The voice output means by the speaker generates a natural voice that reflects the speaker's intention through a synthetic voice generation function, and converts this voice signal into a reproducible analog signal through a digital-to-analog conversion function. The voice synthesis means generates a voice having natural pronunciation and intonation according to the speech content, and outputs this from the speaker. The voice synthesis means uses text-to-speech (TTS) technology that converts text to speech, and has a built-in multilingual model that supports various languages ​​and dialects. The voice synthesis means generates a voice having pronunciation and intonation that supports different languages ​​and dialects through a multilingual text-to-speech function. An example of data collection that does not include sensors would be a user manually inputting speech via a keyboard or touch screen. This manual input method is particularly useful in noisy environments or when the speaker does not have access to a microphone. The manual user input method may also include a function where the user directly provides text information through an input interface, and the manually input information is subjected to analysis by the artificial intelligence via a text processing function. The manually input text data is similarly processed by the artificial intelligence and output as synthetic speech. The manually input text is also processed through a text analysis function and fed into a speech synthesis process for output as synthetic speech. For example, various functions can be added to each of the means of this embodiment. For example, the acquisition means can be added with a superdirectional microphone for instantaneous voice detection and analysis, thereby selectively acquiring only a specific person's voice and separating it from surrounding noise. The acquisition means can also incorporate a function for monitoring text-based communication from social media and messaging apps in real time and reproducing it as voice utterances. This allows the hearing impaired to directly receive online conversations as voice. The speech recognition means can utilize a multilingual model to handle multiple languages ​​and dialects, and can understand the speech of users with different cultural backgrounds. Furthermore, the speech recognition means can be equipped with an emotion analysis function to capture subtle changes in the speaker's voice in order to more accurately grasp the speaker's emotions and intentions. This makes it possible to reflect the speaker's emotions in the voice, rather than simply reading out the text, resulting in more human-like communication. The artificial intelligence means has the capability of continuous learning and self-improvement, and analyzes the speaker's speech characteristics and lip movement patterns more accurately over time. The artificial intelligence means can also be customized to the user's usage and preferences to provide the most natural and understandable speech analysis for each individual user. Such customizable AI can respond to the individual needs of each user, improving usability. Emotion recognition means can provide richer emotional information by adding a function to analyze not only the tone of a speaker's voice, but also subtle changes in facial expressions and body language. For example, it is possible to estimate the speaker's intention and the intensity of his or her emotions from the speaker's eye movements and hand gestures, achieving more accurate emotion recognition. Output means can be more diverse, using not only speakers, but also vibration and light signals to provide information to the hearing impaired. For example, smart devices can warn with vibration patterns or color changes depending on the speaker's emotional state and the urgency of what is being said. Information transmission systems can also be developed that use tactile information rather than synthetic voice to accommodate the visually impaired. These additional features can be implemented through hardware and software updates to existing smart devices, significantly improving the user experience. They can also be linked to cloud-based services to facilitate remote updates and customization, providing users with the latest technology at all times. Furthermore, these features can be widely applied as communication support tools in education, work, and daily life, and are expected to benefit a diverse range of users, including not only the hearing impaired, but also those with speech disabilities and non-native speakers. To further improve the acquisition method, an external noise canceling function can be added to collect clearer voice data. In addition, a real-time language conversion function can be introduced to facilitate communication with people who speak different languages. This will further strengthen the AI's ability to understand multiple languages ​​and expand its use in international communication. The AI ​​means will be further developed to have the ability to memorize what the user has said in the past and learn the individual's unique speaking style, which will enable the AI ​​to reproduce the user's speech more naturally and improve the quality of communication. It will also have an emotion-adaptive synthetic voice generation function that can read the user's emotions and respond with an appropriate tone. The speaker-based audio output means can be combined with a bone conduction headset or a vibrating wearable device so that the hearing impaired can experience sound more naturally. This allows the hearing impaired to intuitively grasp the rhythm and strength of the speaker's voice through the vibrations, enabling them to enjoy a richer communication experience. In addition, by adding customization features that allow users to adjust the volume and speed of audio output depending on the situation, it can be used to meet individual needs. It is also possible to incorporate a safety measure function that automatically increases the volume in an emergency. By integrating these functions, devices can be provided that are easier to use and more effective for the hearing impaired, as well as the elderly and other users who need to communicate in a variety of situations. Furthermore, these technologies are expected to be applied as educational support tools, remote communication, and multilingual dialogue systems, helping to promote social inclusion.

[0096] The process flow of each embodiment will be described below.

[0097] (Example 1) Step 1: Use your smartphone's microphone to pick up the other person's voice. Step 2: Use voice recognition technology to convert the other person's voice into text. Step 3: Mouth movements are detected using vibrations from the bone conduction microphone and the smartphone's built-in camera, and the content of what the other person is saying is understood by comparing this with data that the artificial intelligence has learned. Step 4: The speaker, which works in conjunction with a smartphone to play synthesized speech, provides the hearing-impaired person with audio of what the other person is saying. (Example 2) Step 1: Use the smartphone's built-in camera to detect the other person's mouth movements in real time. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: Using voice recognition technology, the other person's voice is converted into text and displayed on the smartphone screen. (Example 3) Step 1: The smartphone's bone conduction microphone is used to detect the vibrations of the other person's voice. Step 2: The AI ​​understands what the other person is saying by comparing it with the data it has learned. Step 3: The speaker, which works in conjunction with a smartphone to play synthetic speech, provides the hearing-impaired person with audio of what the other person is saying.

[0098] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0099] (Example 1) The embodiment of the present invention includes an acquisition means for acquiring the voice of the other party by a voice recognition means, an artificial intelligence for learning the movement of the mouth by the vibration of a bone conduction microphone and the built-in camera of a smartphone, a speaker for playing synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotions. Specifically, the other party's voice is acquired using the microphone of the smartphone and converted into text using voice recognition technology. In addition, the vibration of the bone conduction microphone and the built-in camera of the smartphone are detected, and the movement of the mouth is compared with the data learned by the artificial intelligence to understand the content of the other party's speech. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the speaker for playing synthetic voice in cooperation with the smartphone provides the other party's speech content and emotional state as voice to the hearing impaired person. The acquisition means is realized as a means for acquiring the voice of the other party using, for example, a microphone of a smartphone, and performing voice recognition and converting the voice into text by the specific processing unit 290 of the data processing device 12. In addition, artificial intelligence for learning the vibration of a bone conduction microphone and the movement of the mouth by the built-in camera of the smartphone is realized by the specific processing unit 290 of the data processing device 12. The speaker is realized as a means for playing a synthetic voice by the data processing device 12 or the control unit 46A of the robot 414. The emotion engine is realized as a means for analyzing the tone of the other party's voice and changes in facial expression by the specific processing unit 290 of the data processing device 12, for example, and estimating the emotional state. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The acquisition means has a function of accurately acquiring the voice of the other party as digital data by utilizing voice recognition technology. For example, the acquisition means acquires clear voice data while eliminating surrounding noise by using a high-sensitivity microphone equipped with a noise cancellation function. The acquisition means can also acquire the speech of the other party as digital voice data by using the voice recognition means via a microphone on the smart device. This voice data can clearly capture the human voice by the high-sensitivity microphone while reducing environmental noise by a noise canceling function. The voice recognition means performs processing to convert the spoken content into text data based on the acquired voice data, thereby achieving highly accurate text conversion. This process includes feature extraction from the voice signal, segmentation of utterance units, and recognition processing based on a voice model. The voice recognition means can also identify the speaker's voiceprint and speech pattern using a characteristic analysis function of the voice signal, and perform preprocessing to improve the accuracy of voice recognition. The acquired voice data can be subjected to voice-to-text conversion processing by extracting features of the voice data, performing segmentation, and converting the voice into text. The artificial intelligence means includes analyzing vibration patterns of a bone conduction microphone and capturing and analyzing mouth movements using a built-in camera. The means performs processing to infer the speaker's speech content from the acquired vibration data and video data. The artificial intelligence means can also capture the speaker's mouth movements and head vibrations using the bone conduction microphone and the built-in camera. The artificial intelligence means can analyze these data and understand the speaker's speech content. The bone conduction data analysis means analyzes the head vibrations captured by the bone conduction microphone and associates them with the voice to understand the speech content. The mouth movement analysis means can use video data captured by the smartphone camera to track the speaker's mouth movements and associate them with the speech content. These means are trained using machine learning techniques to learn the speaker's individual speech characteristics and achieve high-precision speech understanding in real time. The emotion recognition means has a function of estimating the emotional state of a speaker by using both voice data and video data. This means includes a voice emotion analysis means based on the analysis of the tone and intonation of the voice, and a facial expression analysis means that captures changes in facial expressions. The voice emotion analysis means analyzes the pitch, intensity, acoustic characteristics, etc. of the voice data to estimate the emotional state of the speaker. The emotion engine also has both a voice emotion analysis function and a facial expression analysis function, and can estimate the emotional state of the speaker. The facial expression analysis means detects the facial expressions of the speaker from the video data captured by the built-in camera, and estimates the emotional state by capturing subtle changes in facial expressions. These means are trained using an emotion recognition model, and by identifying emotional patterns, grasp the speaker's emotions in real time. The output means includes a speaker that cooperates with the smartphone to generate synthetic voice and provide it to the user. The means includes a text-to-speech (TTS) function that converts text to voice, and the tone and intonation of the synthesized voice are adjusted to reflect the speaker's spoken content and emotions. Furthermore, the output voice has clear pronunciation and an appropriate speed so that the hearing impaired can easily understand it. The output means through the speaker can also generate synthetic voice and provide the hearing impaired with the other party's spoken content and emotional state by voice. The voice flowing from the speaker is designed so that the user can easily grasp the other party's spoken content and emotions. This allows the user to grasp the other party's words not only by hearing but also by combining them with visual information. As an extension of the functions related to the acquisition means, it is possible to selectively pick up the voice of a specific speaker by using a directional microphone that is resistant to external noise. In addition, by utilizing a combination of a smart device and a wearable device, the user's body movements and gestures can be acquired to assist in understanding non-verbal communication. Furthermore, it is conceivable to utilize beamforming technology through the acquisition means to identify individual speakers even in situations where multiple people are talking and accurately grasp the content of each speaker's speech. In order to improve the accuracy of lip reading, the artificial intelligence means uses a high-resolution camera to capture more detailed mouth movements. In addition, by developing an algorithm that analyzes the intention of communication not only from the speaker's mouth movements but also from facial expressions and body language, a richer understanding of communication is realized. The artificial intelligence means can build a lip reading model that supports multiple languages ​​and can also handle international communication. Regarding emotion recognition, more accurate emotion recognition can be achieved by incorporating a biofeedback sensor that obtains physiological indicators such as heart rate and skin electrical conductivity and uses them to estimate the emotional state. In addition, a customized emotion recognition model can be built by learning an individual's emotional patterns, providing emotion recognition adapted to each individual user. The emotion recognition means can also capture subtle differences in emotions expressed by users in different contexts and social situations. As for the output means, a personalized audio system that adjusts the frequency band according to the user's hearing characteristics can be introduced, making it possible to provide audio tailored to each individual hearing-impaired person. In addition, by incorporating a function into a smartphone application that uses visual information to display the other person's speech in text in real time, it becomes possible to obtain information from both audio and vision. Furthermore, by combining synthetic voice with an expressive animated avatar, visual communication can be enhanced and more intuitive understanding can be supported. The acquisition method can be supplemented with an active noise canceling function that analyzes the sound environment around the user in real time and performs optimal noise reduction. In addition, by utilizing bone conduction earphone technology that allows users to hear sound without blocking their ears, it is possible to provide users with clear sound while still being aware of the surrounding environmental sounds. Furthermore, by linking with devices other than smartphones, such as smart watches and smart glasses, voice data can be acquired using multiple devices, creating a more comprehensive communication environment. Regarding artificial intelligence means, multitask learning will be introduced to simultaneously analyze not only lip reading but also non-verbal information such as the speaker's posture and hand movements to gain a deeper understanding of the context of communication. In addition, by using edge computing technology to perform part of the data processing on the smart device side, it is possible to reduce communication delays and improve real-time performance. Furthermore, a function will be developed that uses artificial intelligence to automatically identify and translate the language used by the speaker, and to support communication between different languages. The emotion recognition method combines machine vision technology to capture subtle changes in facial expressions and skin color, improving the accuracy of emotion analysis. In addition, bio-signals obtained from wearable devices are used for emotion recognition to more accurately evaluate the user's psychological state, such as stress level and fatigue. In addition, situation recognition technology is introduced to adjust the emotion recognition algorithm according to the user's environment and situation, allowing for more appropriate emotion estimation. As for output means, a visual assist function will be added that adjusts the size and contrast of text display to suit the user's visual acuity characteristics. In addition, an avatar-based sign language translation system will be incorporated that recognizes and generates sign language and gestures necessary for the hearing impaired to communicate with others. An emotional expression assist function will also be developed that allows users to convey their emotional state to others, realizing rich communication that includes emotions. Through these additional functions, the invention can provide a more comfortable and effective means of communication not only for the hearing impaired but also for users with various communication disabilities. In addition, by combining these functions, it is possible to reduce misunderstandings that occur during communication and realize smoother dialogue. (Example 2) The embodiment of the present invention includes an acquisition means for acquiring the other party's voice by a voice recognition means, an artificial intelligence for learning the movement of the mouth by vibration of a bone conduction microphone and the built-in camera of a smartphone, a speaker for playing a synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotion. Specifically, the smartphone's built-in camera is used to detect the other party's mouth movement in real time, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the other party's voice is converted into text using voice recognition technology and displayed on the smartphone screen. The acquisition means is realized, for example, as a means for detecting the movement of the other person's mouth in real time using a built-in camera of a smartphone, and for comparing the detected movement with data learned by the artificial intelligence by the specific processing unit 290 of the data processing device 12. The emotion engine is realized, for example, as a means for analyzing the tone of the other person's voice and changes in facial expression by the specific processing unit 290 of the data processing device 12, and for estimating the emotional state. The display means is realized, for example, by the specific processing unit 290 of the data processing device 12 as a means for converting the other person's voice into text and displaying it on the screen of the smartphone. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means includes a voice data acquisition function using advanced voice signal processing technology. This function uses a noise suppression algorithm to remove ambient noise and clearly capture the other party's voice. The noise suppression algorithm distinguishes between human voices and background noise through frequency analysis and subtracts noise components in real time. The voice recognition means also uses a machine learning voice recognition model to identify the speaker and the contents of the speech from the voice. This model has the ability to learn large amounts of voice data and identify the voice characteristics and speech patterns of a specific speaker. The voice data acquisition function, which uses voice recognition technology to acquire and analyze voice data obtained from the other party's voice, also utilizes an advanced signal processing algorithm to remove ambient noise and clearly identify the speaker's voice. This function improves the quality of communication by identifying and removing noise components in the voice data. In addition, as a speaker identification function, a model based on machine learning is used to learn the voice characteristics of each speaker and identify who the speaker is from the voice data. The acquisition means includes a mouth movement detection function using the camera of the smart device. This function combines image recognition technology and machine learning algorithms to infer the content of the speech from the shape and movement of the other person's mouth. The detected mouth movement data is sent to a data processing device in real time and analyzed by artificial intelligence. In this analysis process, the data is compared with a trained data set and the speech content is understood through lip reading. The artificial intelligence uses face recognition technology to identify a specific individual, learns the characteristics of the person's mouth movement, and builds a correspondence with the speech content. In addition, the mouth movement detection function uses the camera of the smart device to acquire an image of the other person's face, and uses lip reading technology to infer the speech content from the movement. This function detects the shape and movement of the mouth from the face image, and uses a learned lip reading model to analyze the words spoken by the speaker. In this process, the face recognition function is also utilized to recognize the mouth movement patterns of a specific individual and assist in understanding the speech content. The speaker means converts the text data into a voice having natural pronunciation and intonation using synthetic voice generation technology. This technology can reflect not only the accuracy of pronunciation but also the emotion and intention of the speaker in the text-to-voice conversion. The speaker may be built into the smart device, or may be an external wireless speaker that transmits an audio signal. In either case, an acoustic processing technology is employed to ensure a sound quality and volume that is easy for the user to hear. The speaker means for playing the synthetic voice includes a text-to-voice conversion function, which converts the text data into a voice having natural pronunciation and intonation using synthetic voice generation technology. This technology can reflect the emotion and intention of the speaker in the voice, and depending on the speaker used, a sound quality adjustment function and a volume adjustment function are also used in combination. The emotion recognition means has a function of estimating emotions from audio signals and video data. This function uses a voice emotion analysis algorithm that analyzes changes in voice tone, speech rate, and intonation, and a facial expression recognition algorithm that captures changes in facial expressions and eye movements. The emotion recognition means integrates these data to infer the emotional state of the other person. The emotion estimation result becomes important information in the context of communication and is provided as feedback to help the user understand the emotions of the other person. The emotion recognition means also estimates the emotional state of the speaker by analyzing the audio signals and video data. This means includes a voice emotion analysis algorithm that monitors changes in tone, speech rate, and intonation. It also includes a facial expression recognition algorithm that detects changes in facial expressions and eye movements from the video data. By integrating these pieces of information, the other person's emotional state is estimated in real time and the result is provided to the user. An example of data collection that does not involve sensors is when a user inputs a text message via a keyboard or voice input function of a smart device and transmits the message through a communication means. This method allows the user to communicate with the other party without relying on voice or video. Another example of data collection that does not involve sensors is text-based communication that is manually input by a user. The user inputs his / her own words as text using a keyboard or voice input function of a smart device and transmits the text to the other party. This method allows direct text communication without requiring a voice recognition means or a lip reading means. The voice recognition means, acquisition means, speaker means, and emotion engine in the embodiments of the present invention can further add other functions. For example, the voice recognition means can employ binaural recording technology and reproduce a more natural acoustic space using two microphones positioned at the position of human ears. This allows the user to easily sense the direction and distance of the other person's voice, allowing the user to enjoy a more realistic communication experience. Furthermore, the voice recognition means can be used to realize the operation of smart devices and the control of home automation systems by voice commands. The acquisition means may incorporate a feedback function to assist the user in speaking. The feedback function may provide guidance on mouth movements and pronunciation when the user speaks. Furthermore, the acquisition means may include a noise canceling function to effectively eliminate external noise and improve the accuracy of detecting mouth movements. The speaker means can have a function to automatically adjust the sound quality according to the user's hearing condition and hearing preferences. Advanced digital signal processing technology is utilized to provide the listener with an optimal hearing experience. The speaker means can also be added with a function to link with external smart home devices and in-car systems to expand the reach of the voice wherever the user is. The emotion engine can incorporate sensors to detect the user's physiological reactions and add a function to more accurately estimate the user's emotional state from changes in heart rate and electrical resistance of the skin. Such biofeedback functions can monitor the user's stress level and excitement in real time, deepening the understanding of emotions in communication. With the addition of these functions, the embodiment of the present invention becomes a more advanced communication support tool, enabling users to have richer and more emotionally rich conversations with others. These functions also provide effective support for users with disabilities or when communication is difficult in a particular environment. Furthermore, as smart devices evolve, these functions are expected to be expanded and optimized in the future to accommodate the needs of a variety of users. The voice recognition means of this embodiment can be equipped with a sensitivity enhancement function that accurately captures the contents of speech even when the user whispers or speaks at a very low volume. This function combines a special microphone array with an algorithm that analyzes ambient noise to make the speaker's voice stand out. In addition, a directional pickup function can be provided to selectively capture the voice of a specific speaker in a situation where multiple speakers are speaking at the same time. Acquisition methods can also be enhanced with real-time translation capabilities to instantly translate utterances spoken in different languages ​​into the user's native language. This functionality works with multilingual databases to reduce global communication barriers. Acquisition methods can also employ region-specific models to identify speaker accents and dialects for more accurate transcription and translation. The speaker means may be added with a personalized equalization function to analyze the user's hearing adaptation and provide an optimal hearing experience. This function realizes audio output that is tailored to the individual ear characteristics and hearing preferences of the user. The speaker means may also provide a virtual reality experience that enhances spatial awareness by adding 3D audio effects to the voice guidance. The emotion engine can be equipped with an emotion response function that tracks the user's emotional changes during the conversation and provides supportive and encouraging messages at the right time. The emotion engine can also estimate the user's psychological stress level from their behavior and speech patterns and suggest advice and relaxation techniques to reduce stress. These additional functions will provide users with a more detailed and personalized communication experience. Furthermore, these technologies are expected to be applied in fields such as education, business, and healthcare, contributing to improving the quality of communication and promoting accessibility. In addition, by providing interactive feedback according to the user's behavior and situation, more natural human-machine dialogue will be realized. The evolution of these technologies will promote a deeper level of user engagement in the future, laying the foundation for providing more valuable services to users. (Example 3) The embodiment of the present invention includes an acquisition means for acquiring the voice of the other party by a voice recognition means, an artificial intelligence for learning the vibration of a bone conduction microphone and the movement of the mouth by the built-in camera of a smartphone, a speaker for playing synthetic voice in cooperation with the smartphone, and an emotion engine for recognizing the user's emotions. Specifically, the bone conduction microphone of the smartphone is used to detect the vibration of the other party's voice, and the content of the other party's speech is understood by comparing it with the data learned by the artificial intelligence. Furthermore, the emotion engine analyzes the tone of the other party's voice and changes in facial expression to estimate the other party's emotional state. Then, the speaker for playing synthetic voice in cooperation with the smartphone provides the other party's speech content and emotional state as voice to the hearing impaired person. The acquisition means is realized, for example, as a means for detecting vibrations of the other person's voice using a bone conduction microphone of a smartphone, and for comparing the vibrations with data learned by the artificial intelligence by the specific processing unit 290 of the data processing device 12. The emotion engine is realized, for example, as a means for analyzing changes in the other person's tone of voice and facial expression by the specific processing unit 290 of the data processing device 12, and for estimating the emotional state. The speaker is realized as a means for playing a synthetic voice by the data processing device 12 or the control unit 46A of the robot 414. The correspondence between each means and the device or control unit is not limited to the above-mentioned example, and various changes are possible. The voice recognition means has a function of acquiring surrounding voice data through a microphone built into the smart device, and the acquired voice data is pre-processed by digital signal processing, such as noise removal and amplification, and converted into clear voice information. The voice recognition means also has a voice acquisition function for capturing voice from the surroundings, and applies a voice pre-processing function to the acquired voice data. This pre-processing includes the removal of environmental noise by a noise reduction function and the optimization of the volume by a gain adjustment function. This makes the voice signal clearer and converts it into a state suitable for subsequent processing. Next, based on this pre-processed voice data, a voice analysis algorithm extracts specific voice features and improves the accuracy of voice recognition. The extracted voice features include the frequency, amplitude, pitch, duration, etc. of the voice, which are important indicators for understanding the characteristics of the speaker's voice and the content of the words. The voice analysis algorithm also extracts features such as the frequency characteristics and temporal patterns of the voice through a feature extraction function, and grasps the meaning of the voice data using a speaker identification function and a speech content analysis function. The AI ​​learns the vibration patterns of speech detected by the bone conduction microphone as a data set and identifies the content of speech. This learning process uses multiple machine learning algorithms including deep learning technology. In particular, convolutional neural networks are suitable for extracting features from image data and are used to analyze mouth movements captured by the built-in camera of a smartphone. The AI ​​integrates both audio data and visual data and performs multimodal information processing, enabling accurate identification and understanding of the content of speech. The AI ​​also combines the analysis function of vibration data from the bone conduction microphone with the visual data analysis function to accurately identify the content of speech. The vibration data analysis function captures the physical characteristics of the voice obtained by bone conduction and compares it with the audio data. The visual data analysis function identifies the patterns of mouth movements obtained by the smartphone camera and integrates and processes this information to enable more advanced recognition from multimodal information. The speaker is built into the smart device and generates synthetic speech to provide audio information to the user in an easy-to-listen format. The synthetic speech is generated using text-to-speech technology and is designed to reproduce not only the content of the speech, but also changes in tone and intonation that express emotions. This enables the hearing impaired to understand not only the content of the speech of the other person, but also its emotional nuances. The speaker also converts text information into speech using a synthetic speech generation function, providing easy-to-listen audio information to the user. The synthetic speech adjustment function mimics the emotional elements of speech, including tone and intonation, to reproduce the richness of emotional expression during communication. The emotion recognition means performs a composite emotion analysis by integrating acoustic features such as tone, pitch, and speed of the voice with visual information such as changes in facial expressions captured by the smartphone camera. In this process, a machine learning model specialized in emotion recognition is used, and the user's emotions can be classified into multiple categories such as positive, negative, surprise, and sadness. As a result, the emotion recognition means estimates the user's emotional state and provides an emotional context in the communication. In addition, the emotion recognition means extracts emotion information from the voice data and visual data using an emotion analysis algorithm. This analysis includes a voice emotion analysis function and a facial expression analysis function, which are integrated to perform a multidimensional analysis of the user's emotional state, and the emotion estimation function understands the user's emotions. This makes it easier for the user to grasp the other person's emotions, improving the quality of communication. In implementing this embodiment, it is useful to consider further functional expansion. For example, it is possible to introduce beamforming technology into the acquisition means, which selectively acquires a specific voice from among surrounding sounds. This makes it possible to clearly capture the speaker's voice even in a noisy environment and improve the accuracy of voice recognition. In addition, a multilingual processing function for handling different languages ​​and dialects can be added to the artificial intelligence to accommodate a wide range of users. It is also possible to reduce communication barriers with foreign language speakers by incorporating a real-time translation function using artificial intelligence. As for the speakers, a function to adjust the frequency characteristics of the sound to suit the user's hearing characteristics will be added, providing even more personalized listening ease. The emotion engine will be able to convey emotional fluctuations to the user in line with the flow of the conversation by capturing emotional changes in real time. In addition, a context-aware function will be added to the emotion engine, which will estimate emotions based on the context of the conversation, achieving more accurate emotion recognition. Furthermore, it is conceivable to incorporat...

Claims

1. A voice recognition unit acquires voice information, which is a speaker's voice, by voice recognition and converts the voice information into text data; A learning method that integrates vibration data from a bone conduction microphone and video data from a smartphone's built-in camera to learn the correlation between a speaker's mouth movements and voice; A system for assisting communication for the hearing impaired, comprising: a speaker that cooperates with the smartphone to play synthetic voice corresponding to text data recognized by the voice recognition means.

2. 2. A system for assisting communication for the hearing impaired, as described in claim 1, further comprising a means for displaying the contents of the other person's speech on a smartphone screen based on learned mouth movement information.

3. A system for assisting communication for the hearing impaired, as described in claim 1, characterized in that the means for displaying the other person's speech on a smartphone screen based on learned mouth movement information corresponds to sign language movements.

4. 10. The system of claim 1, wherein the voice recognition means detects sounds other than speech and provides the detected sounds as a visual alert or a vibration signal.

5. 2. The system according to claim 1, wherein said learning means learns the correlation between the movement of the speaker's mouth and the voice by further learning the usage of words or phrases when the speaker speaks.

Images