System

Abstract

To provide a system that enables deaf and hard-of-hearing people to understand sound information in real time visually.SOLUTION: A system includes a sound collection section, a feature value extraction section, an image acquisition section, an analysis section, and a projection section. The sound collection section collects a sound from a sound source. The feature value extraction section extracts a feature value from a sound source whose sound is collected by the sound collection section. The image acquisition section acquires an image of the surroundings. The analysis section analyzes the feature value extracted by the feature value extraction section and image data acquired by the image acquisition section to generate sound information as characters. The projection section projects character information generated by the analysis section on a visual device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technology has the problem that it is difficult for people with hearing impairments or who are hearing-impaired to visually understand audio information in real time.

[0005] The system according to the embodiment aims to enable people with hearing loss or hearing impairments to visually understand audio information in real time. [Means for solving the problem]

[0006] The system according to the embodiment includes a sound collection unit, a feature value extraction unit, an image acquisition unit, an analysis unit, and a projection unit. The sound collection unit collects a sound source. The feature value extraction unit extracts feature values ​​from the sound source collected by the sound collection unit. The image acquisition unit acquires an image of the surroundings. The analysis unit analyzes the feature values ​​extracted by the feature value extraction unit and the image data acquired by the image acquisition unit, and generates voice information as characters. The projection unit projects the character information generated by the analysis unit onto a visual device. Effect of the Invention

[0007] The system according to the embodiment can enable the hearing impaired or deaf to visually comprehend audio information in real time. [Brief description of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., voice and / or text). Display 40A displays visible information such as text and images according to instructions from processor 46. Speaker 40B outputs voice according to instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Also, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, a home appliance, etc.) owned by a user. Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) The support system according to the embodiment of the present invention is a system that visually provides audio information to the hearing impaired or hearing-impaired in real time. This support system automatically extracts feature values ​​from a sound source collected in real time, multiplies the feature values ​​with information obtained from a 360° image, passes the result to a generation AI, and projects the output as text on a visual device. For example, the support system collects surrounding sounds in real time using a microphone. The collected sound source has feature values ​​automatically extracted using voice recognition technology. For example, the tone of the speaker's voice and the direction of the sound are extracted as feature values. Next, the support system acquires an image of the surroundings using a 360° camera. This image data is multiplied with the feature values ​​of the sound source and passed to the generation AI. The generation AI analyzes the feature values ​​of the sound source and the image data to generate audio information as text. The generated text information is projected on a visual device in real time. This allows the hearing impaired or hearing-impaired to visually check the audio information in real time. For example, the content of the conversation and information on the surrounding sounds are displayed as text, so the audio information can be visually understood. This system allows people with hearing loss or hearing impairments to visually confirm audio information, which is expected to facilitate smoother communication and improve their quality of life. For example, in situations such as meetings and lectures, the speaker's remarks can be confirmed as text in real time, preventing information from being missed. In addition, information about surrounding sounds is also displayed as text, which is useful for responding to emergencies and ensuring safety. This allows the support system to provide visual audio information in real time to people with hearing loss or hearing impairments.

[0029] The support system according to the embodiment includes a sound collection unit, a feature value extraction unit, an image acquisition unit, an analysis unit, and a projection unit. The sound collection unit collects surrounding sounds in real time. The sound collection unit collects surrounding sounds using, for example, a microphone. The sound collection unit can also collect sounds from a specific direction using a directional microphone. The sound collection unit can also collect sounds from all directions using multiple microphones. For example, the sound collection unit collects the voice of a speaker using a directional microphone. The sound collection unit can also collect ambient sounds using an omnidirectional microphone. The sound collection unit can also collect sounds while removing ambient noise using a noise canceling microphone. The feature value extraction unit extracts feature values ​​from the sound source collected by the sound collection unit. The feature value extraction unit extracts feature values ​​from the sound source using, for example, voice recognition technology. The feature value extraction unit can also analyze the frequency components of the sound source to extract feature values. The feature value extraction unit can also extract feature values ​​by analyzing the volume and duration of the sound source. For example, the feature value extraction unit analyzes the frequency spectrum of the sound source to extract feature values. The feature value extraction unit can also analyze changes in the volume of the sound source to extract feature values. The feature value extraction unit can also analyze the duration of the sound source to extract feature values. The image acquisition unit acquires images of the surroundings in real time. The image acquisition unit acquires images of the surroundings using, for example, a 360° camera. The image acquisition unit can also acquire detailed images using a high-resolution camera. The image acquisition unit can also acquire images even in dark places using an infrared camera. For example, the image acquisition unit acquires images in all directions using a 360° camera. The image acquisition unit can also acquire detailed images using a high-resolution camera. The image acquisition unit can also acquire images even in dark places using an infrared camera. The analysis unit uses a generation AI to analyze the feature values ​​extracted by the feature value extraction unit and the image data acquired by the image acquisition unit, and generates voice information as characters. The analysis unit analyzes the feature values ​​and image data using, for example, a deep learning model. The analysis unit can also generate voice information as text using natural language processing technology. The analysis unit can also generate voice information as text using generation AI.For example, the analysis unit analyzes the feature values ​​and image data using a deep learning model. The analysis unit can also generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using generation AI. The projection unit projects the character information generated by the analysis unit onto a visual device in real time. The projection unit projects the character information using, for example, a head-mounted display. The projection unit can also project the character information using smart glasses. The projection unit can also project the character information using a projector. For example, the projection unit projects the character information using a head-mounted display. The projection unit can also project the character information using smart glasses. The projection unit can also project the character information using a projector. As a result, the support system according to the embodiment can provide voice information visually in real time to people with hearing loss or hearing impairments.

[0030] The sound collection unit collects surrounding sounds in real time. The sound collection unit collects surrounding sounds using, for example, a microphone. Specifically, the sound collection unit uses a highly sensitive condenser microphone and can collect a wide range of sounds accurately. By using a directional microphone, sounds from a specific direction can be emphasized and collected, and for example, the voice of a speaker in a conference room can be clearly collected. In addition, by using an omnidirectional microphone, surrounding environmental sounds can be collected from all directions, and for example, the sense of realism at an outdoor event can be enhanced. Furthermore, by using a noise-canceling microphone, ambient noise can be effectively removed and clear voice can be collected. As a result, the sound collection unit can respond to various environments and applications and collect optimal voice data. The sound collection unit converts the collected voice data into a digital signal and transmits it to a database for processing in real time. As a result, the sound collection unit can achieve high-precision collection and rapid processing of voice data, thereby improving the performance of the entire system.

[0031] The feature value extraction unit extracts feature values ​​from the sound source collected by the sound collection unit. The feature value extraction unit extracts feature values ​​of the sound source using, for example, a voice recognition technology. Specifically, the feature value extraction unit performs a Fourier transform on the sound signal and analyzes the frequency components to extract a characteristic frequency spectrum of the sound source. In addition, the feature value extraction unit can analyze the energy distribution of the sound signal and extract volume changes and duration as feature values. For example, the feature value extraction unit analyzes the frequency spectrum of the sound source and identifies a specific voice pattern or noise. Furthermore, by analyzing the energy distribution of the sound signal, the feature value extraction unit can extract volume changes and duration of the sound source as feature values ​​and accurately grasp the strength and duration of the voice. As a result, the feature value extraction unit can extract various feature values ​​from the collected voice data and provide data necessary for advanced analysis in the analysis unit. The feature value extraction unit stores the extracted feature values ​​in a database and makes them accessible to the analysis unit. As a result, the feature value extraction unit can support detailed analysis of voice data and highly accurate voice recognition, thereby improving the performance of the entire system.

[0032] The image acquisition unit acquires images of the surroundings in real time. The image acquisition unit acquires images of the surroundings using, for example, a 360° camera. Specifically, the 360° camera can simultaneously capture images in all directions and collect visual information over a wide range. By using a high-resolution camera, detailed images can be acquired, and for example, facial expressions and gestures of speakers in a conference room can be clearly captured. In addition, by using an infrared camera, clear images can be acquired even in dark places or at night, which is useful for, for example, nighttime surveillance and work support in dark places. The image acquisition unit processes the acquired image data in real time and transmits it to a database. As a result, the image acquisition unit can collect visual information with high accuracy and quickly, and provide data necessary for advanced analysis in the analysis unit. The image acquisition unit converts the acquired image data into a digital signal and transmits it to a database for processing in real time. As a result, the image acquisition unit can realize high-accuracy collection and rapid processing of visual information, and improve the performance of the entire system.

[0033] The analysis unit uses the generation AI to analyze the feature values ​​extracted by the feature value extraction unit and the image data acquired by the image acquisition unit, and generates voice information as characters. The analysis unit analyzes the feature values ​​and image data, for example, using a deep learning model. Specifically, the deep learning model receives the feature values ​​of the voice data and image data as input, and learns patterns for generating voice information as characters. By using natural language processing technology, the voice information can be accurately converted into characters, and for example, minutes of meetings and speeches can be automatically transcribed. The generation AI comprehensively analyzes the voice data and image data, and generates voice information as characters with high accuracy. For example, it is possible to analyze the speaker's mouth movements and facial expressions to improve the accuracy of voice recognition. The analysis unit stores the generated character information in a database and makes it accessible to the projection unit. As a result, the analysis unit can realize high-precision analysis of voice information and character generation, and improve the performance of the entire system.

[0034] The projection unit projects the character information generated by the analysis unit onto a visual device in real time. The projection unit projects the character information using, for example, a head-mounted display. Specifically, the head-mounted display can display character information directly in the user's field of vision and visually provide audio information in real time. By using smart glasses, the user can naturally check character information in daily life, and for example, can check the content of remarks during a meeting in real time. In addition, by using a projector, character information can be displayed over a wide area, which is useful for information sharing in, for example, large-scale meetings and lectures. The projection unit updates the character information in real time and provides the latest information to the user. As a result, the projection unit can visually provide audio information and provide information in real time to people with hearing loss or hearing impairments. Furthermore, the projection unit can collect user feedback and continuously improve the accuracy and effectiveness of the display content. As a result, the projection unit can quickly and reliably provide information to the user and improve the performance of the entire system.

[0035] The feature value extraction unit can specify the direction of the sound source. The feature value extraction unit specifies the direction of the sound source, for example, by using triangulation. For example, the feature value extraction unit specifies the direction of the sound source by using multiple microphones. The feature value extraction unit can also specify the direction of the sound source by using beamforming technology. For example, the feature value extraction unit specifies the direction of the sound source by using triangulation. The feature value extraction unit can also specify the direction of the sound source by using multiple microphones. The feature value extraction unit can also specify the direction of the sound source by using beamforming technology. This allows more accurate audio information to be provided by specifying the direction of the sound source. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, by using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for specifying the direction of the sound source to the generation AI and cause the generation AI to specify the direction of the sound source.

[0036] The image acquisition unit can acquire images of the surroundings in real time. The image acquisition unit acquires images of the surroundings in real time, for example, using a 360° camera. For example, the image acquisition unit acquires detailed images in real time using a high-resolution camera. The image acquisition unit can also acquire images in real time even in dark places using an infrared camera. For example, the image acquisition unit acquires images in all directions in real time using a 360° camera. The image acquisition unit can also acquire detailed images in real time using a high-resolution camera. The image acquisition unit can also acquire images in real time even in dark places using an infrared camera. As a result, the latest visual information can be provided by acquiring images in real time. A part or all of the above-mentioned processing in the image acquisition unit may be performed, for example, using AI, or may be performed without using AI. For example, the image acquisition unit can input image data acquired in real time to the generation AI and cause the generation AI to execute analysis of the image data.

[0037] The analysis unit can analyze the feature values ​​and image data of the sound source using the generation AI and generate voice information as characters. The analysis unit can analyze the feature values ​​and image data of the sound source using, for example, a deep learning model. For example, the analysis unit can generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using the generation AI. For example, the analysis unit can analyze the feature values ​​and image data of the sound source using a deep learning model. The analysis unit can also generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using the generation AI. Thereby, by using the generation AI, the analysis accuracy of the voice information is improved. A part or all of the above-mentioned processing in the analysis unit may be performed using, for example, AI, or may be performed without using AI. For example, the analysis unit can input the feature values ​​and image data of the sound source to the generation AI and cause the generation AI to generate voice information.

[0038] The projection unit can project the generated character information onto a visual device in real time. The projection unit can, for example, use a head-mounted display to project the character information in real time. For example, the projection unit can use smart glasses to project the character information in real time. The projection unit can also use a projector to project the character information in real time. For example, the projection unit can use a head-mounted display to project the character information in real time. The projection unit can also use smart glasses to project the character information in real time. The projection unit can also use a projector to project the character information in real time. As a result, visual audio information can be provided instantly by projecting the character information in real time. A part or all of the above-mentioned processing in the projection unit can be performed, for example, using AI or can be performed without using AI. For example, the projection unit can input the generated character information to a generation AI and cause the generation AI to execute projection of the character information.

[0039] The analysis unit can analyze the tone of the speaker's voice and generate it as text information. The analysis unit, for example, analyzes the pitch of the voice to analyze the tone of the speaker's voice. For example, the analysis unit analyzes the change in the volume of the voice to analyze the tone of the speaker's voice. The analysis unit can also analyze the frequency components of the voice to analyze the tone of the speaker's voice. For example, the analysis unit analyzes the pitch of the voice to analyze the tone of the speaker's voice. The analysis unit can also analyze the change in the volume of the voice to analyze the tone of the speaker's voice. The analysis unit can also analyze the frequency components of the voice to analyze the tone of the speaker's voice. As a result, more detailed voice information can be provided by analyzing the tone of the speaker's voice. Part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing the tone of the speaker's voice to the generation AI and cause the generation AI to execute the analysis of the tone of the voice.

[0040] The sound collection unit can collect sound by emphasizing a specific frequency band when collecting sound. The sound collection unit collects sound by emphasizing, for example, the frequency band of human voice. For example, the sound collection unit collects sound by emphasizing the frequency band of environmental sound. The sound collection unit can also collect sound by emphasizing the frequency band of music. For example, the sound collection unit collects sound by emphasizing the frequency band of human voice. The sound collection unit can also collect sound by emphasizing the frequency band of environmental sound. The sound collection unit can also collect sound by emphasizing the frequency band of music. As a result, important audio information can be collected by emphasizing a specific frequency band. A part or all of the above-mentioned processing in the sound collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the sound collection unit can input data for emphasizing a specific frequency band to the generation AI and cause the generation AI to execute the emphasis of the frequency band.

[0041] The sound collection unit can add a filtering function that removes surrounding noise in real time when collecting sound. The sound collection unit, for example, removes surrounding wind sounds in real time. For example, the sound collection unit removes surrounding traffic sounds in real time. The sound collection unit can also remove surrounding people's talking voices in real time. For example, the sound collection unit removes surrounding wind sounds in real time. The sound collection unit can also remove surrounding traffic sounds in real time. The sound collection unit can also remove surrounding people's talking voices in real time. This makes it possible to provide clear audio information by removing surrounding noise. Part or all of the above-mentioned processing in the sound collection unit may be performed, for example, using AI or may be performed without using AI. For example, the sound collection unit can input data for removing surrounding noise to the generation AI and cause the generation AI to remove the noise.

[0042] The sound collection unit can automatically adjust the optimal microphone arrangement based on the user's position information when collecting sound. For example, when the user is in the center of the room, the sound collection unit arranges the microphones so as to collect sounds from all directions evenly. For example, when the user is facing a specific direction, the sound collection unit arranges the microphones so as to collect sounds from that direction with priority. When the user is moving, the sound collection unit can also adjust the arrangement of the microphones in real time. For example, when the user is in the center of the room, the sound collection unit arranges the microphones so as to collect sounds from all directions evenly. When the user is facing a specific direction, the sound collection unit can also arrange the microphones so as to collect sounds from that direction with priority. When the user is moving, the sound collection unit can also adjust the arrangement of the microphones in real time. This makes it possible to optimally collect sound by adjusting the microphone arrangement based on the user's position information. A part or all of the above-mentioned processing in the sound collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the sound collection unit can input data based on the user's position information to the generation AI and have the generation AI adjust the microphone placement.

[0043] The sound collection unit can automatically apply the optimal sound collection setting by referring to the user's past sound collection history at the time of sound collection. The sound collection unit automatically applies the optimal sound collection setting based on, for example, the settings of the sound source collected by the user in the past. For example, the sound collection unit automatically applies the sound collection setting suitable for a specific environment from the user's past sound collection history. The sound collection unit can also analyze the user's past sound collection history and automatically apply the most effective sound collection setting. For example, the sound collection unit automatically applies the optimal sound collection setting based on the settings of the sound source collected by the user in the past. The sound collection unit can also automatically apply the sound collection setting suitable for a specific environment from the user's past sound collection history. The sound collection unit can also analyze the user's past sound collection history and automatically apply the most effective sound collection setting. As a result, the optimal sound collection setting can be automatically applied by referring to the past sound collection history. A part or all of the above-mentioned processing in the sound collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the sound collection unit can input data based on the user's past sound collection history to the generation AI and have the generation AI apply the sound collection settings.

[0044] The feature value extraction unit can correct the feature value based on the echo or reverb of the sound source when extracting the feature value. The feature value extraction unit, for example, corrects the feature value by removing the echo of the sound source. For example, the feature value extraction unit corrects the feature value by removing the reverb of the sound source. The feature value extraction unit can also correct the feature value by removing the reverberation of the sound source. For example, the feature value extraction unit corrects the feature value by removing the echo of the sound source. The feature value extraction unit can also correct the feature value by removing the reverberation of the sound source. Thereby, by correcting the echo or reverb, a more accurate feature value can be extracted. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for correcting the echo or reverb to the generation AI and cause the generation AI to execute the correction.

[0045] The feature value extraction unit can generate feature values ​​by analyzing the frequency spectrum of the sound source in detail when extracting feature values. The feature value extraction unit, for example, analyzes the low frequency band of the sound source in detail to generate feature values. For example, the feature value extraction unit analyzes the mid frequency band of the sound source in detail to generate feature values. The feature value extraction unit can also analyze the high frequency band of the sound source in detail to generate feature values. For example, the feature value extraction unit analyzes the low frequency band of the sound source in detail to generate feature values. The feature value extraction unit can also analyze the mid frequency band of the sound source in detail to generate feature values. The feature value extraction unit can also analyze the high frequency band of the sound source in detail to generate feature values. This allows more detailed feature values ​​to be generated by analyzing the frequency spectrum in detail. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for analyzing the frequency spectrum to the generation AI and cause the generation AI to execute the analysis.

[0046] The feature value extraction unit can specify not only the direction of the sound source but also the distance when extracting the feature value. The feature value extraction unit, for example, simultaneously specifies the direction and distance of the sound source to extract the feature value. For example, the feature value extraction unit specifies the direction of the sound source, calculates the distance from that direction, and extracts the feature value. The feature value extraction unit can also specify the distance of the sound source, calculate the direction from that distance, and extract the feature value. For example, the feature value extraction unit simultaneously specifies the direction and distance of the sound source to extract the feature value. The feature value extraction unit can also specify the direction of the sound source, calculate the distance from that direction, and extract the feature value. The feature value extraction unit can also specify the distance of the sound source, calculate the direction from that distance, and extract the feature value. In this way, by specifying the direction and distance of the sound source, more detailed feature values ​​can be extracted. Part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for specifying the direction and distance of the sound source to the generation AI, and cause the generation AI to execute the specification.

[0047] The feature value extraction unit can analyze the environmental sound of the sound source and reflect the information in the feature value when extracting the feature value. The feature value extraction unit, for example, analyzes the environmental sound of the sound source and reflects the information in the feature value. For example, the feature value extraction unit can remove the environmental sound of the sound source and reflect the information in the feature value. The feature value extraction unit can also emphasize the environmental sound of the sound source and reflect the information in the feature value. For example, the feature value extraction unit can analyze the environmental sound of the sound source and reflect the information in the feature value. The feature value extraction unit can also remove the environmental sound of the sound source and reflect the information in the feature value. The feature value extraction unit can also emphasize the environmental sound of the sound source and reflect the information in the feature value. Thereby, by analyzing the environmental sound, more detailed feature values ​​can be generated. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for analyzing the environmental sound to the generation AI and cause the generation AI to execute the analysis.

[0048] The image acquisition unit can acquire an image by emphasizing a specific color or shape when acquiring the image. The image acquisition unit acquires an image by emphasizing, for example, a red object. For example, the image acquisition unit acquires an image by emphasizing a circular object. The image acquisition unit can also acquire an image by emphasizing a specific pattern. For example, the image acquisition unit acquires an image by emphasizing a red object. The image acquisition unit can also acquire an image by emphasizing a circular object. The image acquisition unit can also acquire an image by emphasizing a specific pattern. As a result, important visual information can be emphasized and acquired by emphasizing a specific color or shape. A part or all of the above-mentioned processing in the image acquisition unit may be performed, for example, using AI, or may be performed without using AI. For example, the image acquisition unit can input data for emphasizing a specific color or shape to the generation AI and cause the generation AI to execute the emphasis.

[0049] The image acquisition unit can automatically adjust the brightness and contrast of the surroundings when acquiring an image. The image acquisition unit automatically adjusts the brightness of the image according to the brightness of the surroundings, for example. For example, the image acquisition unit automatically adjusts the contrast of the image according to the contrast of the surroundings. The image acquisition unit can also automatically adjust the color temperature of the image according to the light source of the surroundings. For example, the image acquisition unit automatically adjusts the brightness of the image according to the brightness of the surroundings. The image acquisition unit can also automatically adjust the contrast of the image according to the contrast of the surroundings. The image acquisition unit can also automatically adjust the color temperature of the image according to the light source of the surroundings. As a result, an optimal image can be acquired by automatically adjusting the brightness and contrast of the surroundings. A part or all of the above-mentioned processing in the image acquisition unit may be performed, for example, using AI, or may be performed without using AI. For example, the image acquisition unit can input data for adjusting the brightness and contrast of the surroundings to the generation AI and cause the generation AI to execute the adjustment.

[0050] The image acquisition unit can track the user's line of sight direction when acquiring an image to acquire an optimal image. The image acquisition unit, for example, tracks the user's line of sight direction and acquires an image in that direction with priority. For example, when the user's line of sight moves, the image acquisition unit acquires an image according to the movement. The image acquisition unit can also track the user's line of sight direction in real time to acquire an optimal image. For example, the image acquisition unit tracks the user's line of sight direction and acquires an image in that direction with priority. When the user's line of sight moves, the image acquisition unit can also acquire an image according to the movement. The image acquisition unit can also track the user's line of sight direction in real time to acquire an optimal image. As a result, optimal visual information can be acquired by tracking the user's line of sight direction. A part or all of the above-mentioned processing in the image acquisition unit may be performed using, for example, AI, or may be performed without using AI. For example, the image acquisition unit can input data for tracking the user's line of sight direction to the generation AI and cause the generation AI to execute the tracking.

[0051] The image acquisition unit can automatically apply optimal settings by referring to the user's past image acquisition history when acquiring an image. The image acquisition unit automatically applies optimal settings based on, for example, settings of images acquired by the user in the past. For example, the image acquisition unit automatically applies settings suitable for a specific environment from the user's past image acquisition history. The image acquisition unit can also analyze the user's past image acquisition history and automatically apply the most effective settings. For example, the image acquisition unit automatically applies optimal settings based on settings of images acquired by the user in the past. The image acquisition unit can also automatically apply settings suitable for a specific environment from the user's past image acquisition history. The image acquisition unit can also analyze the user's past image acquisition history and automatically apply the most effective settings. As a result, the optimal settings can be automatically applied by referring to the past image acquisition history. A part or all of the above-mentioned processing in the image acquisition unit may be performed using, for example, AI, or may be performed without using AI. For example, the image acquisition unit can input data based on the user's past image acquisition history to the generation AI and cause the generation AI to apply the settings.

[0052] The analysis unit can generate character information based on context information of the sound source during analysis. The analysis unit generates character information, for example, taking into account the context of a conversation. For example, the analysis unit generates character information taking into account the context of music. The analysis unit can also generate character information taking into account the context of environmental sounds. For example, the analysis unit generates character information taking into account the context of a conversation. The analysis unit can also generate character information taking into account the context of music. The analysis unit can also generate character information taking into account the context of environmental sounds. This allows more accurate character information to be generated by taking into account the context information. A part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing context information to the generation AI and cause the generation AI to perform the analysis.

[0053] The analysis unit can generate text information based on the attribute information of the speaker of the sound source during analysis. The analysis unit generates text information, for example, taking into account the age of the speaker. For example, the analysis unit generates text information taking into account the gender of the speaker. The analysis unit can also generate text information taking into account the occupation of the speaker. For example, the analysis unit generates text information taking into account the age of the speaker. The analysis unit can also generate text information taking into account the gender of the speaker. The analysis unit can also generate text information taking into account the occupation of the speaker. This allows more detailed text information to be generated by taking into account the attribute information of the speaker. A part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing the attribute information of the speaker to the generation AI and cause the generation AI to perform the analysis.

[0054] The analysis unit can reflect the background sound of the sound source in the text information during the analysis. The analysis unit, for example, analyzes the background sound of the sound source and reflects the information in the text information. For example, the analysis unit removes the background sound of the sound source and reflects the information in the text information. The analysis unit can also emphasize the background sound of the sound source and reflect the information in the text information. For example, the analysis unit analyzes the background sound of the sound source and reflects the information in the text information. The analysis unit can also remove the background sound of the sound source and reflect the information in the text information. The analysis unit can also emphasize the background sound of the sound source and reflect the information in the text information. Thereby, by taking the background sound into consideration, more detailed text information can be generated. A part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing the background sound to the generation AI and cause the generation AI to execute the analysis.

[0055] The analysis unit can reflect the gesture information of the speaker of the sound source in the character information during the analysis. The analysis unit, for example, analyzes the speaker's hand movements and reflects the information in the character information. For example, the analysis unit can analyze the speaker's facial expression and reflect the information in the character information. The analysis unit can also analyze the speaker's body movements and reflect the information in the character information. For example, the analysis unit can analyze the speaker's hand movements and reflect the information in the character information. The analysis unit can also analyze the speaker's facial expression and reflect the information in the character information. The analysis unit can also analyze the speaker's body movements and reflect the information in the character information. This allows more detailed character information to be generated by taking the gesture information into consideration. A part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing gesture information to the generation AI and cause the generation AI to execute the analysis.

[0056] The projection unit can dynamically adjust the display position of the character information according to the user's line of sight during projection. For example, when the user's line of sight moves, the projection unit adjusts the display position of the character information according to the movement. For example, the projection unit tracks the user's line of sight in real time and provides an optimal display position. When the user's line of sight is fixed, the projection unit can also display the character information at that position. For example, when the user's line of sight moves, the projection unit adjusts the display position of the character information according to the movement. The projection unit can also track the user's line of sight in real time and provide an optimal display position. When the user's line of sight is fixed, the projection unit can also display the character information at that position. As a result, by adjusting the display position according to the line of sight, character information with high visibility can be provided. A part or all of the above-mentioned processing in the projection unit may be performed, for example, using AI, or may be performed without using AI. For example, the projection unit can input data for tracking the user's line of sight to the generation AI and cause the generation AI to execute line of sight tracking.

[0057] The projection unit can automatically adjust the color of the text information and the background color based on the surrounding environment during projection. The projection unit automatically adjusts the color of the text information according to, for example, the brightness of the surroundings. For example, the projection unit automatically adjusts the background color according to the surrounding hue. The projection unit can also automatically adjust the contrast of the text information according to the surrounding light source. For example, the projection unit automatically adjusts the color of the text information according to the brightness of the surroundings. The projection unit can also automatically adjust the background color according to the surrounding hue. The projection unit can also automatically adjust the contrast of the text information according to the surrounding light source. As a result, by adjusting the color and background color based on the surrounding environment, text information with high visibility can be provided. Part or all of the above-mentioned processing in the projection unit may be performed, for example, using AI, or may be performed without using AI. For example, the projection unit can input data for analyzing the surrounding environment to the generation AI and cause the generation AI to perform the analysis.

[0058] The projection unit can display character information in stages during projection to reduce the visual burden on the user. The projection unit, for example, displays long text information in stages. For example, the projection unit displays multiple pieces of information in stages. When the user's line of sight moves, the projection unit can also display the information in stages in accordance with the movement. For example, the projection unit displays long text information in stages. The projection unit can also display multiple pieces of information in stages. When the user's line of sight moves, the projection unit can also display the information in stages in accordance with the movement. This allows the visual burden to be reduced by displaying the information in stages. A part or all of the above-mentioned processing in the projection unit may be performed, for example, using AI, or may be performed without using AI. For example, the projection unit can input data for displaying in stages to the generation AI and cause the generation AI to execute the display.

[0059] The projection unit can automatically apply optimal settings by referring to the user's past projection history at the time of projection. The projection unit automatically applies optimal settings based on, for example, settings of character information projected by the user in the past. For example, the projection unit automatically applies settings suitable for a specific environment from the user's past projection history. The projection unit can also analyze the user's past projection history and automatically apply the most effective settings. For example, the projection unit automatically applies optimal settings based on settings of character information projected by the user in the past. The projection unit can also automatically apply settings suitable for a specific environment from the user's past projection history. The projection unit can also analyze the user's past projection history and automatically apply the most effective settings. As a result, the optimal settings can be automatically applied by referring to the past projection history. A part or all of the above-mentioned processing in the projection unit may be performed, for example, using AI, or may be performed without using AI. For example, the projection unit can input data based on the user's past projection history to the generation AI and cause the generation AI to apply the settings.

[0060] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0061] The feature value extraction unit can specify not only the direction of the sound source but also the distance of the sound source. For example, the feature value extraction unit specifies the direction and distance of the sound source at the same time to extract a feature value. The feature value extraction unit can also specify the direction of the sound source, calculate the distance from that direction, and extract a feature value. The feature value extraction unit can also specify the distance of the sound source, calculate the direction from that distance, and extract a feature value. In this way, by specifying the direction and distance of the sound source, more detailed feature values ​​can be extracted.

[0062] The image acquisition unit can track the user's line of sight and acquire images in that direction with priority. For example, when the user's line of sight moves, the image acquisition unit acquires images in accordance with the movement. The image acquisition unit can also track the user's line of sight in real time and acquire optimal images. This makes it possible to acquire optimal visual information by tracking the user's line of sight.

[0063] The analysis unit can generate text information based on context information of the sound source. For example, the analysis unit generates text information taking into account the context of a conversation. The analysis unit can also generate text information taking into account the context of music. The analysis unit can also generate text information taking into account the context of environmental sounds. In this way, more accurate text information can be generated by taking into account the context information.

[0064] When the user's line of sight moves, the projection unit can adjust the display position of the text information in accordance with the movement. For example, the projection unit tracks the user's line of sight in real time and provides the optimal display position. When the user's line of sight is fixed, the projection unit can also display the text information at that position. This makes it possible to provide highly visible text information by adjusting the display position in accordance with the line of sight.

[0065] The process flow of the first embodiment will be briefly described below.

[0066] Step 1: The sound collection unit collects ambient sounds in real time. The sound collection unit collects ambient sounds using, for example, a microphone. The sound collection unit can also collect sounds from a specific direction using a directional microphone. The sound collection unit can also collect sounds from all directions using multiple microphones. For example, the sound collection unit collects the voice of a speaker using a directional microphone. The sound collection unit can also collect ambient sounds using an omnidirectional microphone. The sound collection unit can also collect sounds while eliminating ambient noise using a noise-canceling microphone. Step 2: The feature value extraction unit extracts feature values ​​from the sound source collected by the sound collection unit. The feature value extraction unit extracts feature values ​​of the sound source using, for example, voice recognition technology. The feature value extraction unit can also extract feature values ​​by analyzing the frequency components of the sound source. The feature value extraction unit can also extract feature values ​​by analyzing the volume and duration of the sound source. For example, the feature value extraction unit extracts feature values ​​by analyzing the frequency spectrum of the sound source. The feature value extraction unit can also extract feature values ​​by analyzing changes in the volume of the sound source. The feature value extraction unit can also extract feature values ​​by analyzing the duration of the sound source. Step 3: The image acquisition unit acquires images of the surroundings in real time. The image acquisition unit acquires images of the surroundings, for example, using a 360° camera. The image acquisition unit can also acquire detailed images using a high-resolution camera. The image acquisition unit can also acquire images in dark places using an infrared camera. For example, the image acquisition unit acquires images in all directions using a 360° camera. The image acquisition unit can also acquire detailed images using a high-resolution camera. The image acquisition unit can also acquire images in dark places using an infrared camera. Step 4: The analysis unit uses the generation AI to analyze the feature values ​​extracted by the feature value extraction unit and the image data acquired by the image acquisition unit, and generates voice information as characters. The analysis unit analyzes the feature values ​​and image data, for example, using a deep learning model. The analysis unit can also generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using the generation AI. For example, the analysis unit analyzes the feature values ​​and image data using a deep learning model. The analysis unit can also generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using the generation AI. Step 5: The projection unit projects the character information generated by the analysis unit onto the visual device in real time. For example, the projection unit projects the character information using a head mounted display. The projection unit can also project the character information using smart glasses. The projection unit can also project the character information using a projector. For example, the projection unit projects the character information using a head mounted display. The projection unit can also project the character information using smart glasses. The projection unit can also project the character information using a projector.

[0067] (Example 2) The support system according to the embodiment of the present invention is a system that visually provides audio information to the hearing impaired or hearing-impaired in real time. This support system automatically extracts feature values ​​from a sound source collected in real time, multiplies the feature values ​​with information obtained from a 360° image, passes the result to a generation AI, and projects the output as text on a visual device. For example, the support system collects surrounding sounds in real time using a microphone. The collected sound source has feature values ​​automatically extracted using voice recognition technology. For example, the tone of the speaker's voice and the direction of the sound are extracted as feature values. Next, the support system acquires an image of the surroundings using a 360° camera. This image data is multiplied with the feature values ​​of the sound source and passed to the generation AI. The generation AI analyzes the feature values ​​of the sound source and the image data to generate audio information as text. The generated text information is projected on a visual device in real time. This allows the hearing impaired or hearing-impaired to visually check the audio information in real time. For example, the content of the conversation and information on the surrounding sounds are displayed as text, so the audio information can be visually understood. This system allows people with hearing loss or hearing impairments to visually confirm audio information, which is expected to facilitate smoother communication and improve their quality of life. For example, in situations such as meetings and lectures, the speaker's remarks can be confirmed as text in real time, preventing information from being missed. In addition, information about surrounding sounds is also displayed as text, which is useful for responding to emergencies and ensuring safety. This allows the support system to provide visual audio information in real time to people with hearing loss or hearing impairments.

[0068] The support system according to the embodiment includes a sound collection unit, a feature value extraction unit, an image acquisition unit, an analysis unit, and a projection unit. The sound collection unit collects surrounding sounds in real time. The sound collection unit collects surrounding sounds using, for example, a microphone. The sound collection unit can also collect sounds from a specific direction using a directional microphone. The sound collection unit can also collect sounds from all directions using multiple microphones. For example, the sound collection unit collects the voice of a speaker using a directional microphone. The sound collection unit can also collect ambient sounds using an omnidirectional microphone. The sound collection unit can also collect sounds while removing ambient noise using a noise canceling microphone. The feature value extraction unit extracts feature values ​​from the sound source collected by the sound collection unit. The feature value extraction unit extracts feature values ​​from the sound source using, for example, voice recognition technology. The feature value extraction unit can also analyze the frequency components of the sound source to extract feature values. The feature value extraction unit can also extract feature values ​​by analyzing the volume and duration of the sound source. For example, the feature value extraction unit analyzes the frequency spectrum of the sound source to extract feature values. The feature value extraction unit can also analyze changes in the volume of the sound source to extract feature values. The feature value extraction unit can also analyze the duration of the sound source to extract feature values. The image acquisition unit acquires images of the surroundings in real time. The image acquisition unit acquires images of the surroundings using, for example, a 360° camera. The image acquisition unit can also acquire detailed images using a high-resolution camera. The image acquisition unit can also acquire images even in dark places using an infrared camera. For example, the image acquisition unit acquires images in all directions using a 360° camera. The image acquisition unit can also acquire detailed images using a high-resolution camera. The image acquisition unit can also acquire images even in dark places using an infrared camera. The analysis unit uses a generation AI to analyze the feature values ​​extracted by the feature value extraction unit and the image data acquired by the image acquisition unit, and generates voice information as characters. The analysis unit analyzes the feature values ​​and image data using, for example, a deep learning model. The analysis unit can also generate voice information as text using natural language processing technology. The analysis unit can also generate voice information as text using generation AI.For example, the analysis unit analyzes the feature values ​​and image data using a deep learning model. The analysis unit can also generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using generation AI. The projection unit projects the character information generated by the analysis unit onto a visual device in real time. The projection unit projects the character information using, for example, a head-mounted display. The projection unit can also project the character information using smart glasses. The projection unit can also project the character information using a projector. For example, the projection unit projects the character information using a head-mounted display. The projection unit can also project the character information using smart glasses. The projection unit can also project the character information using a projector. As a result, the support system according to the embodiment can provide voice information visually in real time to people with hearing loss or hearing impairments.

[0069] The sound collection unit collects surrounding sounds in real time. The sound collection unit collects surrounding sounds using, for example, a microphone. Specifically, the sound collection unit uses a highly sensitive condenser microphone and can collect a wide range of sounds accurately. By using a directional microphone, sounds from a specific direction can be emphasized and collected, and for example, the voice of a speaker in a conference room can be clearly collected. In addition, by using an omnidirectional microphone, surrounding environmental sounds can be collected from all directions, and for example, the sense of realism at an outdoor event can be enhanced. Furthermore, by using a noise-canceling microphone, ambient noise can be effectively removed and clear voice can be collected. As a result, the sound collection unit can respond to various environments and applications and collect optimal voice data. The sound collection unit converts the collected voice data into a digital signal and transmits it to a database for processing in real time. As a result, the sound collection unit can achieve high-precision collection and rapid processing of voice data, thereby improving the performance of the entire system.

[0070] The feature value extraction unit extracts feature values ​​from the sound source collected by the sound collection unit. The feature value extraction unit extracts feature values ​​of the sound source using, for example, a voice recognition technology. Specifically, the feature value extraction unit performs a Fourier transform on the sound signal and analyzes the frequency components to extract a characteristic frequency spectrum of the sound source. In addition, the feature value extraction unit can analyze the energy distribution of the sound signal and extract volume changes and duration as feature values. For example, the feature value extraction unit analyzes the frequency spectrum of the sound source and identifies a specific voice pattern or noise. Furthermore, by analyzing the energy distribution of the sound signal, the feature value extraction unit can extract volume changes and duration of the sound source as feature values ​​and accurately grasp the strength and duration of the voice. As a result, the feature value extraction unit can extract various feature values ​​from the collected voice data and provide data necessary for advanced analysis in the analysis unit. The feature value extraction unit stores the extracted feature values ​​in a database and makes them accessible to the analysis unit. As a result, the feature value extraction unit can support detailed analysis of voice data and highly accurate voice recognition, thereby improving the performance of the entire system.

[0071] The image acquisition unit acquires images of the surroundings in real time. The image acquisition unit acquires images of the surroundings using, for example, a 360° camera. Specifically, the 360° camera can simultaneously capture images in all directions and collect visual information over a wide range. By using a high-resolution camera, detailed images can be acquired, and for example, facial expressions and gestures of speakers in a conference room can be clearly captured. In addition, by using an infrared camera, clear images can be acquired even in dark places or at night, which is useful for, for example, nighttime surveillance and work support in dark places. The image acquisition unit processes the acquired image data in real time and transmits it to a database. As a result, the image acquisition unit can collect visual information with high accuracy and quickly, and provide data necessary for advanced analysis in the analysis unit. The image acquisition unit converts the acquired image data into a digital signal and transmits it to a database for processing in real time. As a result, the image acquisition unit can realize high-accuracy collection and rapid processing of visual information, and improve the performance of the entire system.

[0072] The analysis unit uses the generation AI to analyze the feature values ​​extracted by the feature value extraction unit and the image data acquired by the image acquisition unit, and generates voice information as characters. The analysis unit analyzes the feature values ​​and image data, for example, using a deep learning model. Specifically, the deep learning model receives the feature values ​​of the voice data and image data as input, and learns patterns for generating voice information as characters. By using natural language processing technology, the voice information can be accurately converted into characters, and for example, minutes of meetings and speeches can be automatically transcribed. The generation AI comprehensively analyzes the voice data and image data, and generates voice information as characters with high accuracy. For example, it is possible to analyze the speaker's mouth movements and facial expressions to improve the accuracy of voice recognition. The analysis unit stores the generated character information in a database and makes it accessible to the projection unit. As a result, the analysis unit can realize high-precision analysis of voice information and character generation, and improve the performance of the entire system.

[0073] The projection unit projects the character information generated by the analysis unit onto a visual device in real time. The projection unit projects the character information using, for example, a head-mounted display. Specifically, the head-mounted display can display character information directly in the user's field of vision and visually provide audio information in real time. By using smart glasses, the user can naturally check character information in daily life, and for example, can check the content of remarks during a meeting in real time. In addition, by using a projector, character information can be displayed over a wide area, which is useful for information sharing in, for example, large-scale meetings and lectures. The projection unit updates the character information in real time and provides the latest information to the user. As a result, the projection unit can visually provide audio information and provide information in real time to people with hearing loss or hearing impairments. Furthermore, the projection unit can collect user feedback and continuously improve the accuracy and effectiveness of the display content. As a result, the projection unit can quickly and reliably provide information to the user and improve the performance of the entire system.

[0074] The feature value extraction unit can specify the direction of the sound source. The feature value extraction unit specifies the direction of the sound source, for example, by using triangulation. For example, the feature value extraction unit specifies the direction of the sound source by using multiple microphones. The feature value extraction unit can also specify the direction of the sound source by using beamforming technology. For example, the feature value extraction unit specifies the direction of the sound source by using triangulation. The feature value extraction unit can also specify the direction of the sound source by using multiple microphones. The feature value extraction unit can also specify the direction of the sound source by using beamforming technology. This allows more accurate audio information to be provided by specifying the direction of the sound source. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, by using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for specifying the direction of the sound source to the generation AI and cause the generation AI to specify the direction of the sound source.

[0075] The image acquisition unit can acquire images of the surroundings in real time. The image acquisition unit acquires images of the surroundings in real time, for example, using a 360° camera. For example, the image acquisition unit acquires detailed images in real time using a high-resolution camera. The image acquisition unit can also acquire images in real time even in dark places using an infrared camera. For example, the image acquisition unit acquires images in all directions in real time using a 360° camera. The image acquisition unit can also acquire detailed images in real time using a high-resolution camera. The image acquisition unit can also acquire images in real time even in dark places using an infrared camera. As a result, the latest visual information can be provided by acquiring images in real time. A part or all of the above-mentioned processing in the image acquisition unit may be performed, for example, using AI, or may be performed without using AI. For example, the image acquisition unit can input image data acquired in real time to the generation AI and cause the generation AI to execute analysis of the image data.

[0076] The analysis unit can analyze the feature values ​​and image data of the sound source using the generation AI and generate voice information as characters. The analysis unit can analyze the feature values ​​and image data of the sound source using, for example, a deep learning model. For example, the analysis unit can generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using the generation AI. For example, the analysis unit can analyze the feature values ​​and image data of the sound source using a deep learning model. The analysis unit can also generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using the generation AI. Thereby, by using the generation AI, the analysis accuracy of the voice information is improved. A part or all of the above-mentioned processing in the analysis unit may be performed using, for example, AI, or may be performed without using AI. For example, the analysis unit can input the feature values ​​and image data of the sound source to the generation AI and cause the generation AI to generate voice information.

[0077] The projection unit can project the generated character information onto a visual device in real time. The projection unit can, for example, use a head-mounted display to project the character information in real time. For example, the projection unit can use smart glasses to project the character information in real time. The projection unit can also use a projector to project the character information in real time. For example, the projection unit can use a head-mounted display to project the character information in real time. The projection unit can also use smart glasses to project the character information in real time. The projection unit can also use a projector to project the character information in real time. As a result, visual audio information can be provided instantly by projecting the character information in real time. A part or all of the above-mentioned processing in the projection unit can be performed, for example, using AI or can be performed without using AI. For example, the projection unit can input the generated character information to a generation AI and cause the generation AI to execute projection of the character information.

[0078] The analysis unit can analyze the tone of the speaker's voice and generate it as text information. The analysis unit, for example, analyzes the pitch of the voice to analyze the tone of the speaker's voice. For example, the analysis unit analyzes the change in the volume of the voice to analyze the tone of the speaker's voice. The analysis unit can also analyze the frequency components of the voice to analyze the tone of the speaker's voice. For example, the analysis unit analyzes the pitch of the voice to analyze the tone of the speaker's voice. The analysis unit can also analyze the change in the volume of the voice to analyze the tone of the speaker's voice. The analysis unit can also analyze the frequency components of the voice to analyze the tone of the speaker's voice. As a result, more detailed voice information can be provided by analyzing the tone of the speaker's voice. Part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing the tone of the speaker's voice to the generation AI and cause the generation AI to execute the analysis of the tone of the voice.

[0079] The sound collection unit can estimate the user's emotion and adjust the sound collection sensitivity based on the estimated user's emotion. For example, when the user is nervous, the sound collection unit increases the sound collection sensitivity to collect surrounding sounds in more detail. For example, when the user is relaxed, the sound collection unit decreases the sound collection sensitivity to reduce unnecessary noise. The sound collection unit can also optimize the sound collection sensitivity for a specific sound source when the user is concentrating. For example, when the user is nervous, the sound collection unit increases the sound collection sensitivity to collect surrounding sounds in more detail. When the user is relaxed, the sound collection unit can decrease the sound collection sensitivity to reduce unnecessary noise. When the user is concentrating, the sound collection unit can also optimize the sound collection sensitivity for a specific sound source. This allows more appropriate voice information to be provided by adjusting the sound collection sensitivity according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, but is not limited to, a text generation AI (for example, LLM) or a multimodal generation AI. A part or all of the above-mentioned processing in the sound collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the sound collection unit may input data for estimating the user's emotion to the generation AI, and cause the generation AI to estimate the emotion.

[0080] The sound collection unit can collect sound by emphasizing a specific frequency band when collecting sound. The sound collection unit collects sound by emphasizing, for example, the frequency band of human voice. For example, the sound collection unit collects sound by emphasizing the frequency band of environmental sound. The sound collection unit can also collect sound by emphasizing the frequency band of music. For example, the sound collection unit collects sound by emphasizing the frequency band of human voice. The sound collection unit can also collect sound by emphasizing the frequency band of environmental sound. The sound collection unit can also collect sound by emphasizing the frequency band of music. As a result, important audio information can be collected by emphasizing a specific frequency band. A part or all of the above-mentioned processing in the sound collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the sound collection unit can input data for emphasizing a specific frequency band to the generation AI and cause the generation AI to execute the emphasis of the frequency band.

[0081] The sound collection unit can add a filtering function that removes surrounding noise in real time when collecting sound. The sound collection unit, for example, removes surrounding wind sounds in real time. For example, the sound collection unit removes surrounding traffic sounds in real time. The sound collection unit can also remove surrounding people's talking voices in real time. For example, the sound collection unit removes surrounding wind sounds in real time. The sound collection unit can also remove surrounding traffic sounds in real time. The sound collection unit can also remove surrounding people's talking voices in real time. This makes it possible to provide clear audio information by removing surrounding noise. Part or all of the above-mentioned processing in the sound collection unit may be performed, for example, using AI or may be performed without using AI. For example, the sound collection unit can input data for removing surrounding noise to the generation AI and cause the generation AI to remove the noise.

[0082] The sound collection unit can estimate the user's emotion and determine the priority of the sound source to be collected based on the estimated user's emotion. For example, when the user is nervous, the sound collection unit collects the speaker's voice with priority. For example, when the user is relaxed, the sound collection unit collects the environmental sound with priority. The sound collection unit can also collect a specific sound source with priority when the user is concentrating. For example, when the user is nervous, the sound collection unit collects the speaker's voice with priority. The sound collection unit can also collect the environmental sound with priority when the user is relaxed. The sound collection unit can also collect a specific sound source with priority when the user is concentrating. This allows important voice information to be collected with priority by determining the priority of the sound source according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, but is not limited to, a text generation AI (for example, LLM) or a multimodal generation AI. A part or all of the above-mentioned processing in the sound collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the sound collection unit may input data for estimating the user's emotion to the generation AI, and cause the generation AI to estimate the emotion.

[0083] The sound collection unit can automatically adjust the optimal microphone arrangement based on the user's position information when collecting sound. For example, when the user is in the center of the room, the sound collection unit arranges the microphones so as to collect sounds from all directions evenly. For example, when the user is facing a specific direction, the sound collection unit arranges the microphones so as to collect sounds from that direction with priority. When the user is moving, the sound collection unit can also adjust the arrangement of the microphones in real time. For example, when the user is in the center of the room, the sound collection unit arranges the microphones so as to collect sounds from all directions evenly. When the user is facing a specific direction, the sound collection unit can also arrange the microphones so as to collect sounds from that direction with priority. When the user is moving, the sound collection unit can also adjust the arrangement of the microphones in real time. This makes it possible to optimally collect sound by adjusting the microphone arrangement based on the user's position information. A part or all of the above-mentioned processing in the sound collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the sound collection unit can input data based on the user's position information to the generation AI and have the generation AI adjust the microphone placement.

[0084] The sound collection unit can automatically apply the optimal sound collection setting by referring to the user's past sound collection history at the time of sound collection. The sound collection unit automatically applies the optimal sound collection setting based on, for example, the settings of the sound source collected by the user in the past. For example, the sound collection unit automatically applies the sound collection setting suitable for a specific environment from the user's past sound collection history. The sound collection unit can also analyze the user's past sound collection history and automatically apply the most effective sound collection setting. For example, the sound collection unit automatically applies the optimal sound collection setting based on the settings of the sound source collected by the user in the past. The sound collection unit can also automatically apply the sound collection setting suitable for a specific environment from the user's past sound collection history. The sound collection unit can also analyze the user's past sound collection history and automatically apply the most effective sound collection setting. As a result, the optimal sound collection setting can be automatically applied by referring to the past sound collection history. A part or all of the above-mentioned processing in the sound collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the sound collection unit can input data based on the user's past sound collection history to the generation AI and have the generation AI apply the sound collection settings.

[0085] The feature value extraction unit can estimate the user's emotion and adjust the feature value extraction method based on the estimated user's emotion. For example, when the user is nervous, the feature value extraction unit emphasizes the tone of the sound source to extract the feature value. For example, when the user is relaxed, the feature value extraction unit emphasizes the rhythm of the sound source to extract the feature value. When the user is concentrating, the feature value extraction unit can also extract the feature value by analyzing the frequency spectrum of the sound source in detail. For example, when the user is nervous, the feature value extraction unit emphasizes the tone of the sound source to extract the feature value. When the user is relaxed, the feature value extraction unit can also emphasize the rhythm of the sound source to extract the feature value. When the user is concentrating, the feature value extraction unit can also extract the feature value by analyzing the frequency spectrum of the sound source in detail. Thereby, by adjusting the feature value extraction method according to the user's emotion, a more appropriate feature value can be extracted. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit may input data for estimating the user's emotion to the generation AI and cause the generation AI to estimate the emotion.

[0086] The feature value extraction unit can correct the feature value based on the echo or reverb of the sound source when extracting the feature value. The feature value extraction unit, for example, corrects the feature value by removing the echo of the sound source. For example, the feature value extraction unit corrects the feature value by removing the reverb of the sound source. The feature value extraction unit can also correct the feature value by removing the reverberation of the sound source. For example, the feature value extraction unit corrects the feature value by removing the echo of the sound source. The feature value extraction unit can also correct the feature value by removing the reverberation of the sound source. Thereby, by correcting the echo or reverb, a more accurate feature value can be extracted. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for correcting the echo or reverb to the generation AI and cause the generation AI to execute the correction.

[0087] The feature value extraction unit can generate feature values ​​by analyzing the frequency spectrum of the sound source in detail when extracting feature values. The feature value extraction unit, for example, analyzes the low frequency band of the sound source in detail to generate feature values. For example, the feature value extraction unit analyzes the mid frequency band of the sound source in detail to generate feature values. The feature value extraction unit can also analyze the high frequency band of the sound source in detail to generate feature values. For example, the feature value extraction unit analyzes the low frequency band of the sound source in detail to generate feature values. The feature value extraction unit can also analyze the mid frequency band of the sound source in detail to generate feature values. The feature value extraction unit can also analyze the high frequency band of the sound source in detail to generate feature values. This allows more detailed feature values ​​to be generated by analyzing the frequency spectrum in detail. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for analyzing the frequency spectrum to the generation AI and cause the generation AI to execute the analysis.

[0088] The feature value extraction unit can estimate the user's emotion and determine the priority of the feature values ​​to be extracted based on the estimated user's emotion. For example, when the user is nervous, the feature value extraction unit extracts the feature values ​​by prioritizing the tone of the sound source. For example, when the user is relaxed, the feature value extraction unit extracts the feature values ​​by prioritizing the rhythm of the sound source. The feature value extraction unit can also extract the feature values ​​by prioritizing the frequency spectrum of the sound source when the user is concentrating. For example, when the user is nervous, the feature value extraction unit extracts the feature values ​​by prioritizing the tone of the sound source. When the user is relaxed, the feature value extraction unit can also extract the feature values ​​by prioritizing the rhythm of the sound source. When the user is concentrating, the feature value extraction unit can also extract the feature values ​​by prioritizing the frequency spectrum of the sound source. In this way, by determining the priority of the feature values ​​according to the user's emotion, important feature values ​​can be preferentially extracted. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit may input data for estimating the user's emotion to the generation AI and cause the generation AI to estimate the emotion.

[0089] The feature value extraction unit can specify not only the direction of the sound source but also the distance when extracting the feature value. The feature value extraction unit, for example, simultaneously specifies the direction and distance of the sound source to extract the feature value. For example, the feature value extraction unit specifies the direction of the sound source, calculates the distance from that direction, and extracts the feature value. The feature value extraction unit can also specify the distance of the sound source, calculate the direction from that distance, and extract the feature value. For example, the feature value extraction unit simultaneously specifies the direction and distance of the sound source to extract the feature value. The feature value extraction unit can also specify the direction of the sound source, calculate the distance from that direction, and extract the feature value. The feature value extraction unit can also specify the distance of the sound source, calculate the direction from that distance, and extract the feature value. In this way, by specifying the direction and distance of the sound source, more detailed feature values ​​can be extracted. Part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for specifying the direction and distance of the sound source to the generation AI, and cause the generation AI to execute the specification.

[0090] The feature value extraction unit can analyze the environmental sound of the sound source and reflect the information in the feature value when extracting the feature value. The feature value extraction unit, for example, analyzes the environmental sound of the sound source and reflects the information in the feature value. For example, the feature value extraction unit can remove the environmental sound of the sound source and reflect the information in the feature value. The feature value extraction unit can also emphasize the environmental sound of the sound source and reflect the information in the feature value. For example, the feature value extraction unit can analyze the environmental sound of the sound source and reflect the information in the feature value. The feature value extraction unit can also remove the environmental sound of the sound source and reflect the information in the feature value. The feature value extraction unit can also emphasize the environmental sound of the sound source and reflect the information in the feature value. Thereby, by analyzing the environmental sound, more detailed feature values ​​can be generated. A part or all of the above-mentioned processing in the feature value extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the feature value extraction unit can input data for analyzing the environmental sound to the generation AI and cause the generation AI to execute the analysis.

[0091] The image acquisition unit can estimate the user's emotion and adjust the frame rate of image acquisition based on the estimated user's emotion. For example, when the user is nervous, the image acquisition unit increases the frame rate to acquire detailed images. For example, when the user is relaxed, the image acquisition unit decreases the frame rate to reduce the load of image acquisition. The image acquisition unit can also optimize the frame rate to acquire images when the user is concentrating. For example, when the user is nervous, the image acquisition unit increases the frame rate to acquire detailed images. When the user is relaxed, the image acquisition unit can decrease the frame rate to reduce the load of image acquisition. When the user is concentrating, the image acquisition unit can also optimize the frame rate to acquire images. Thereby, by adjusting the frame rate according to the user's emotion, an optimal image can be acquired. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the image acquisition unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the image acquisition unit can input data for estimating a user's emotions to the generation AI and cause the generation AI to estimate the emotions.

[0092] The image acquisition unit can acquire an image by emphasizing a specific color or shape when acquiring the image. The image acquisition unit acquires an image by emphasizing, for example, a red object. For example, the image acquisition unit acquires an image by emphasizing a circular object. The image acquisition unit can also acquire an image by emphasizing a specific pattern. For example, the image acquisition unit acquires an image by emphasizing a red object. The image acquisition unit can also acquire an image by emphasizing a circular object. The image acquisition unit can also acquire an image by emphasizing a specific pattern. As a result, important visual information can be emphasized and acquired by emphasizing a specific color or shape. A part or all of the above-mentioned processing in the image acquisition unit may be performed, for example, using AI, or may be performed without using AI. For example, the image acquisition unit can input data for emphasizing a specific color or shape to the generation AI and cause the generation AI to execute the emphasis.

[0093] The image acquisition unit can automatically adjust the brightness and contrast of the surroundings when acquiring an image. The image acquisition unit automatically adjusts the brightness of the image according to the brightness of the surroundings, for example. For example, the image acquisition unit automatically adjusts the contrast of the image according to the contrast of the surroundings. The image acquisition unit can also automatically adjust the color temperature of the image according to the light source of the surroundings. For example, the image acquisition unit automatically adjusts the brightness of the image according to the brightness of the surroundings. The image acquisition unit can also automatically adjust the contrast of the image according to the contrast of the surroundings. The image acquisition unit can also automatically adjust the color temperature of the image according to the light source of the surroundings. As a result, an optimal image can be acquired by automatically adjusting the brightness and contrast of the surroundings. A part or all of the above-mentioned processing in the image acquisition unit may be performed, for example, using AI, or may be performed without using AI. For example, the image acquisition unit can input data for adjusting the brightness and contrast of the surroundings to the generation AI and cause the generation AI to execute the adjustment.

[0094] The image acquisition unit can estimate the user's emotion and determine the priority of images to be acquired based on the estimated user's emotion. For example, when the user is nervous, the image acquisition unit acquires images by prioritizing important objects. For example, when the user is relaxed, the image acquisition unit acquires images by prioritizing the overall scenery. The image acquisition unit can also acquire images by prioritizing a specific object when the user is concentrating. For example, when the user is nervous, the image acquisition unit acquires images by prioritizing important objects. When the user is relaxed, the image acquisition unit can also acquire images by prioritizing the overall scenery. When the user is concentrating, the image acquisition unit can also acquire images by prioritizing a specific object. In this way, important visual information can be acquired preferentially by determining the priority of images according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the image acquisition unit may be performed, for example, using an AI, or may be performed without using an AI. For example, the image acquisition unit can input data for estimating a user's emotions to the generation AI and cause the generation AI to estimate the emotions.

[0095] The image acquisition unit can track the user's line of sight direction when acquiring an image to acquire an optimal image. The image acquisition unit, for example, tracks the user's line of sight direction and acquires an image in that direction with priority. For example, when the user's line of sight moves, the image acquisition unit acquires an image according to the movement. The image acquisition unit can also track the user's line of sight direction in real time to acquire an optimal image. For example, the image acquisition unit tracks the user's line of sight direction and acquires an image in that direction with priority. When the user's line of sight moves, the image acquisition unit can also acquire an image according to the movement. The image acquisition unit can also track the user's line of sight direction in real time to acquire an optimal image. As a result, optimal visual information can be acquired by tracking the user's line of sight direction. A part or all of the above-mentioned processing in the image acquisition unit may be performed using, for example, AI, or may be performed without using AI. For example, the image acquisition unit can input data for tracking the user's line of sight direction to the generation AI and cause the generation AI to execute the tracking.

[0096] The image acquisition unit can automatically apply optimal settings by referring to the user's past image acquisition history when acquiring an image. The image acquisition unit automatically applies optimal settings based on, for example, settings of images acquired by the user in the past. For example, the image acquisition unit automatically applies settings suitable for a specific environment from the user's past image acquisition history. The image acquisition unit can also analyze the user's past image acquisition history and automatically apply the most effective settings. For example, the image acquisition unit automatically applies optimal settings based on settings of images acquired by the user in the past. The image acquisition unit can also automatically apply settings suitable for a specific environment from the user's past image acquisition history. The image acquisition unit can also analyze the user's past image acquisition history and automatically apply the most effective settings. As a result, the optimal settings can be automatically applied by referring to the past image acquisition history. A part or all of the above-mentioned processing in the image acquisition unit may be performed using, for example, AI, or may be performed without using AI. For example, the image acquisition unit can input data based on the user's past image acquisition history to the generation AI and cause the generation AI to apply the settings.

[0097] The analysis unit can estimate the user's emotion and adjust the analysis algorithm based on the estimated user's emotion. For example, when the user is nervous, the analysis unit speeds up the analysis algorithm to provide a result quickly. For example, when the user is relaxed, the analysis unit details the analysis algorithm to improve accuracy. When the user is concentrated, the analysis unit can also optimize the analysis algorithm to perform analysis efficiently. For example, when the user is nervous, the analysis unit speeds up the analysis algorithm to provide a result quickly. When the user is relaxed, the analysis unit can details the analysis algorithm to improve accuracy. When the user is concentrated, the analysis unit can optimize the analysis algorithm to perform analysis efficiently. This allows the analysis algorithm to be adjusted according to the user's emotion to provide an optimal analysis result. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the analysis unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the analysis unit can input data for estimating a user's emotions into the generation AI and cause the generation AI to estimate the emotions.

[0098] The analysis unit can generate character information based on context information of the sound source during analysis. The analysis unit generates character information, for example, taking into account the context of a conversation. For example, the analysis unit generates character information taking into account the context of music. The analysis unit can also generate character information taking into account the context of environmental sounds. For example, the analysis unit generates character information taking into account the context of a conversation. The analysis unit can also generate character information taking into account the context of music. The analysis unit can also generate character information taking into account the context of environmental sounds. This allows more accurate character information to be generated by taking into account the context information. A part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing context information to the generation AI and cause the generation AI to perform the analysis.

[0099] The analysis unit can generate text information based on the attribute information of the speaker of the sound source during analysis. The analysis unit generates text information, for example, taking into account the age of the speaker. For example, the analysis unit generates text information taking into account the gender of the speaker. The analysis unit can also generate text information taking into account the occupation of the speaker. For example, the analysis unit generates text information taking into account the age of the speaker. The analysis unit can also generate text information taking into account the gender of the speaker. The analysis unit can also generate text information taking into account the occupation of the speaker. This allows more detailed text information to be generated by taking into account the attribute information of the speaker. A part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing the attribute information of the speaker to the generation AI and cause the generation AI to perform the analysis.

[0100] The analysis unit can estimate the user's emotion and adjust the display method of the analysis result based on the estimated user's emotion. For example, when the user is nervous, the analysis unit provides a simple and highly visible display method. For example, when the user is relaxed, the analysis unit provides a display method including detailed information. When the user is in a hurry, the analysis unit can also provide a display method that focuses on the main points. For example, when the user is nervous, the analysis unit provides a simple and highly visible display method. When the user is relaxed, the analysis unit can also provide a display method including detailed information. When the user is in a hurry, the analysis unit can also provide a display method that focuses on the main points. This allows optimal display by adjusting the display method according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the analysis unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the analysis unit can input data for estimating a user's emotions into the generation AI and cause the generation AI to estimate the emotions.

[0101] The analysis unit can reflect the background sound of the sound source in the text information during the analysis. The analysis unit, for example, analyzes the background sound of the sound source and reflects the information in the text information. For example, the analysis unit removes the background sound of the sound source and reflects the information in the text information. The analysis unit can also emphasize the background sound of the sound source and reflect the information in the text information. For example, the analysis unit analyzes the background sound of the sound source and reflects the information in the text information. The analysis unit can also remove the background sound of the sound source and reflect the information in the text information. The analysis unit can also emphasize the background sound of the sound source and reflect the information in the text information. Thereby, by taking the background sound into consideration, more detailed text information can be generated. A part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing the background sound to the generation AI and cause the generation AI to execute the analysis.

[0102] The analysis unit can reflect the gesture information of the speaker of the sound source in the character information during the analysis. The analysis unit, for example, analyzes the speaker's hand movements and reflects the information in the character information. For example, the analysis unit can analyze the speaker's facial expression and reflect the information in the character information. The analysis unit can also analyze the speaker's body movements and reflect the information in the character information. For example, the analysis unit can analyze the speaker's hand movements and reflect the information in the character information. The analysis unit can also analyze the speaker's facial expression and reflect the information in the character information. The analysis unit can also analyze the speaker's body movements and reflect the information in the character information. This allows more detailed character information to be generated by taking the gesture information into consideration. A part or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI, or may be performed without using AI. For example, the analysis unit can input data for analyzing gesture information to the generation AI and cause the generation AI to execute the analysis.

[0103] The projection unit can estimate the user's emotion and adjust the font and size of the text information to be projected based on the estimated user's emotion. For example, when the user is nervous, the projection unit projects the text information in a font with high visibility and a large size. For example, when the user is relaxed, the projection unit projects the text information in a font with easy readability and an appropriate size. When the user is concentrating, the projection unit can also project the text information in a font with fine information and a small size. For example, when the user is nervous, the projection unit projects the text information in a font with high visibility and a large size. When the user is relaxed, the projection unit can also project the text information in a font with easy readability and an appropriate size. When the user is concentrating, the projection unit can also project the text information in a font with fine information and a small size. This makes it possible to provide text information with high visibility by adjusting the font and size according to the user's emotion. The estimation of the emotion is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the projection unit may be performed using, for example, AI, or may be performed without using AI. For example, the projection unit may input data for estimating the user's emotion to the generation AI and cause the generation AI to estimate the emotion.

[0104] The projection unit can dynamically adjust the display position of the character information according to the user's line of sight during projection. For example, when the user's line of sight moves, the projection unit adjusts the display position of the character information according to the movement. For example, the projection unit tracks the user's line of sight in real time and provides an optimal display position. When the user's line of sight is fixed, the projection unit can also display the character information at that position. For example, when the user's line of sight moves, the projection unit adjusts the display position of the character information according to the movement. The projection unit can also track the user's line of sight in real time and provide an optimal display position. When the user's line of sight is fixed, the projection unit can also display the character information at that position. As a result, by adjusting the display position according to the line of sight, character information with high visibility can be provided. A part or all of the above-mentioned processing in the projection unit may be performed, for example, using AI, or may be performed without using AI. For example, the projection unit can input data for tracking the user's line of sight to the generation AI and cause the generation AI to execute line of sight tracking.

[0105] The projection unit can automatically adjust the color of the text information and the background color based on the surrounding environment during projection. The projection unit automatically adjusts the color of the text information according to, for example, the brightness of the surroundings. For example, the projection unit automatically adjusts the background color according to the surrounding hue. The projection unit can also automatically adjust the contrast of the text information according to the surrounding light source. For example, the projection unit automatically adjusts the color of the text information according to the brightness of the surroundings. The projection unit can also automatically adjust the background color according to the surrounding hue. The projection unit can also automatically adjust the contrast of the text information according to the surrounding light source. As a result, by adjusting the color and background color based on the surrounding environment, text information with high visibility can be provided. Part or all of the above-mentioned processing in the projection unit may be performed, for example, using AI, or may be performed without using AI. For example, the projection unit can input data for analyzing the surrounding environment to the generation AI and cause the generation AI to perform the analysis.

[0106] The projection unit can estimate the user's emotion and determine the priority of the information to be projected based on the estimated user's emotion. For example, when the user is nervous, the projection unit prioritizes projecting important information. For example, when the user is relaxed, the projection unit prioritizes projecting the entire information. The projection unit can also prioritize projecting specific information when the user is concentrating. For example, when the user is nervous, the projection unit prioritizes projecting important information. When the user is relaxed, the projection unit can also prioritize projecting the entire information. When the user is concentrating, the projection unit can also prioritize projecting specific information. This allows important information to be provided preferentially by determining the priority of information according to the user's emotion. The estimation of emotion is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the projection unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the projection unit can input data for estimating a user's emotions into the generation AI and cause the generation AI to estimate the emotions.

[0107] The projection unit can display character information in stages during projection to reduce the visual burden on the user. The projection unit, for example, displays long text information in stages. For example, the projection unit displays multiple pieces of information in stages. When the user's line of sight moves, the projection unit can also display the information in stages in accordance with the movement. For example, the projection unit displays long text information in stages. The projection unit can also display multiple pieces of information in stages. When the user's line of sight moves, the projection unit can also display the information in stages in accordance with the movement. This allows the visual burden to be reduced by displaying the information in stages. A part or all of the above-mentioned processing in the projection unit may be performed, for example, using AI, or may be performed without using AI. For example, the projection unit can input data for displaying in stages to the generation AI and cause the generation AI to execute the display.

[0108] The projection unit can automatically apply optimal settings by referring to the user's past projection history at the time of projection. The projection unit automatically applies optimal settings based on, for example, settings of character information projected by the user in the past. For example, the projection unit automatically applies settings suitable for a specific environment from the user's past projection history. The projection unit can also analyze the user's past projection history and automatically apply the most effective settings. For example, the projection unit automatically applies optimal settings based on settings of character information projected by the user in the past. The projection unit can also automatically apply settings suitable for a specific environment from the user's past projection history. The projection unit can also analyze the user's past projection history and automatically apply the most effective settings. As a result, the optimal settings can be automatically applied by referring to the past projection history. A part or all of the above-mentioned processing in the projection unit may be performed, for example, using AI, or may be performed without using AI. For example, the projection unit can input data based on the user's past projection history to the generation AI and cause the generation AI to apply the settings.

[0109] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0110] The sound pickup unit can analyze changes in the tone and volume of the user's voice in real time and estimate the user's emotions. For example, when the user's voice gets louder, the sound pickup unit estimates that the user is nervous and adjusts the sound pickup sensitivity. When the user's voice gets lower, the sound pickup unit estimates that the user is relaxed and can lower the sound pickup sensitivity. In addition, when the volume of the user's voice changes suddenly, the sound pickup unit can estimate surprise or excitement and optimize the sound pickup sensitivity for a specific sound source. This makes it possible to provide more appropriate audio information by adjusting the sound pickup sensitivity according to the user's emotions.

[0111] The feature value extraction unit can specify not only the direction of the sound source but also the distance of the sound source. For example, the feature value extraction unit specifies the direction and distance of the sound source at the same time to extract a feature value. The feature value extraction unit can also specify the direction of the sound source, calculate the distance from that direction, and extract a feature value. The feature value extraction unit can also specify the distance of the sound source, calculate the direction from that distance, and extract a feature value. In this way, by specifying the direction and distance of the sound source, more detailed feature values ​​can be extracted.

[0112] The image acquisition unit can track the user's line of sight and acquire images in that direction with priority. For example, when the user's line of sight moves, the image acquisition unit acquires images in accordance with the movement. The image acquisition unit can also track the user's line of sight in real time and acquire optimal images. This makes it possible to acquire optimal visual information by tracking the user's line of sight.

[0113] The analysis unit can generate text information based on context information of the sound source. For example, the analysis unit generates text information taking into account the context of a conversation. The analysis unit can also generate text information taking into account the context of music. The analysis unit can also generate text information taking into account the context of environmental sounds. In this way, more accurate text information can be generated by taking into account the context information.

[0114] When the user's line of sight moves, the projection unit can adjust the display position of the text information in accordance with the movement. For example, the projection unit tracks the user's line of sight in real time and provides the optimal display position. When the user's line of sight is fixed, the projection unit can also display the text information at that position. This makes it possible to provide highly visible text information by adjusting the display position in accordance with the line of sight.

[0115] The sound collection unit can estimate the user's emotion and determine the priority of the sound source to be collected based on the estimated user's emotion. For example, when the user is nervous, the sound collection unit prioritizes collecting the speaker's voice. When the user is relaxed, the sound collection unit can also prioritize collecting environmental sounds. When the user is concentrating, the sound collection unit can also prioritize collecting a specific sound source. In this way, important audio information can be collected preferentially by determining the priority of the sound source according to the user's emotion.

[0116] The feature value extraction unit can estimate the user's emotion and adjust the feature value extraction method based on the estimated user's emotion. For example, when the user is nervous, the feature value extraction unit emphasizes the tone of the sound source to extract the feature value. When the user is relaxed, the feature value extraction unit can also emphasize the rhythm of the sound source to extract the feature value. When the user is concentrating, the feature value extraction unit can also extract the feature value by analyzing the frequency spectrum of the sound source in detail. In this way, by adjusting the feature value extraction method according to the user's emotion, more appropriate feature values ​​can be extracted.

[0117] The image acquisition unit can estimate the user's emotion and adjust the frame rate of image acquisition based on the estimated user's emotion. For example, when the user is nervous, the image acquisition unit increases the frame rate to acquire detailed images. When the user is relaxed, the image acquisition unit can also reduce the load of image acquisition by lowering the frame rate. When the user is concentrating, the image acquisition unit can also optimize the frame rate to acquire images. In this way, the frame rate can be adjusted according to the user's emotion to acquire optimal images.

[0118] The analysis unit can estimate the user's emotions and adjust the analysis algorithm based on the estimated user's emotions. For example, when the user is nervous, the analysis unit can speed up the analysis algorithm to provide results quickly. When the user is relaxed, the analysis unit can also refine the analysis algorithm to improve accuracy. When the user is concentrating, the analysis unit can also optimize the analysis algorithm to perform analysis efficiently. In this way, the analysis algorithm can be adjusted according to the user's emotions to provide optimal analysis results.

[0119] The projection unit can estimate the user's emotion and adjust the font and size of the text information to be projected based on the estimated user's emotion. For example, when the user is nervous, the projection unit projects text information in a highly visible font and large size. When the user is relaxed, the projection unit can also project text information in an easy-to-read font and appropriate size. When the user is concentrating, the projection unit can also project text information in a font containing detailed information and small size. In this way, highly visible text information can be provided by adjusting the font and size according to the user's emotion.

[0120] The process flow of the second embodiment will be briefly described below.

[0121] Step 1: The sound collection unit collects ambient sounds in real time. The sound collection unit collects ambient sounds using, for example, a microphone. The sound collection unit can also collect sounds from a specific direction using a directional microphone. The sound collection unit can also collect sounds from all directions using multiple microphones. For example, the sound collection unit collects the voice of a speaker using a directional microphone. The sound collection unit can also collect ambient sounds using an omnidirectional microphone. The sound collection unit can also collect sounds while eliminating ambient noise using a noise-canceling microphone. Step 2: The feature value extraction unit extracts feature values ​​from the sound source collected by the sound collection unit. The feature value extraction unit extracts feature values ​​of the sound source using, for example, voice recognition technology. The feature value extraction unit can also extract feature values ​​by analyzing the frequency components of the sound source. The feature value extraction unit can also extract feature values ​​by analyzing the volume and duration of the sound source. For example, the feature value extraction unit extracts feature values ​​by analyzing the frequency spectrum of the sound source. The feature value extraction unit can also extract feature values ​​by analyzing changes in the volume of the sound source. The feature value extraction unit can also extract feature values ​​by analyzing the duration of the sound source. Step 3: The image acquisition unit acquires images of the surroundings in real time. The image acquisition unit acquires images of the surroundings, for example, using a 360° camera. The image acquisition unit can also acquire detailed images using a high-resolution camera. The image acquisition unit can also acquire images in dark places using an infrared camera. For example, the image acquisition unit acquires images in all directions using a 360° camera. The image acquisition unit can also acquire detailed images using a high-resolution camera. The image acquisition unit can also acquire images in dark places using an infrared camera. Step 4: The analysis unit uses the generation AI to analyze the feature values ​​extracted by the feature value extraction unit and the image data acquired by the image acquisition unit, and generates voice information as characters. The analysis unit analyzes the feature values ​​and image data, for example, using a deep learning model. The analysis unit can also generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using the generation AI. For example, the analysis unit analyzes the feature values ​​and image data using a deep learning model. The analysis unit can also generate voice information as characters using natural language processing technology. The analysis unit can also generate voice information as characters using the generation AI. Step 5: The projection unit projects the character information generated by the analysis unit onto the visual device in real time. For example, the projection unit projects the character information using a head mounted display. The projection unit can also project the character information using smart glasses. The projection unit can also project the character information using a projector. For example, the projection unit projects the character information using a head mounted display. The projection unit can also project the character information using smart glasses. The projection unit can also project the character information using a projector.

[0122] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0123] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) are input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may also be an AI agent. In addition, when the processes of each of the above-mentioned parts are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples.In addition, processing performed by AI including the generation AI may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI including the generation AI.

[0124] Furthermore, the above-described processing by the data processing system 10 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or an external device.

[0125] Each of the multiple elements including the above-mentioned sound collection unit, feature value extraction unit, image acquisition unit, analysis unit, and projection unit is realized, for example, by at least one of the smart device 14 and the data processing device 12. For example, the sound collection unit collects surrounding sounds using the microphone of the smart device 14. The feature value extraction unit is realized by the specific processing unit 290 of the data processing device 12 and extracts feature values ​​from the collected sound source. The image acquisition unit acquires surrounding images using the camera of the smart device 14. The analysis unit is realized by the specific processing unit 290 of the data processing device 12 and analyzes the feature values ​​and image data to generate voice information as characters. The projection unit projects character information in real time using the display of the smart device 14. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0126] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0127] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0128] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0129] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0130] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0131] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0132] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0133] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0134] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0135] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0136] In the smart glasses 214, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart glasses 214 have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0137] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0138] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0139] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0140] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device.

[0141] Each of the multiple elements including the above-mentioned sound collection unit, feature value extraction unit, image acquisition unit, analysis unit, and projection unit is realized, for example, by at least one of the smart glasses 214 and the data processing device 12. For example, the sound collection unit collects surrounding sounds using a microphone of the smart glasses 214. The feature value extraction unit is realized by the specific processing unit 290 of the data processing device 12 and extracts feature values ​​from the collected sound source. The image acquisition unit acquires surrounding images using the camera of the smart glasses 214. The analysis unit is realized by the specific processing unit 290 of the data processing device 12 and analyzes the feature values ​​and image data to generate voice information as characters. The projection unit projects character information in real time using the display of the smart glasses 214. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0142] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0143] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0144] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0145] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0146] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0147] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0148] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0149] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0150] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0151] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0152] In the headset type terminal 314, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50, and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0153] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0154] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0155] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0156] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device.

[0157] Each of the multiple elements including the above-mentioned sound collection unit, feature value extraction unit, image acquisition unit, analysis unit, and projection unit is realized, for example, by at least one of the headset type terminal 314 and the data processing device 12. For example, the sound collection unit collects surrounding sounds using a microphone of the headset type terminal 314. The feature value extraction unit is realized by the specific processing unit 290 of the data processing device 12, and extracts feature values ​​from the collected sound source. The image acquisition unit acquires surrounding images using the camera of the headset type terminal 314. The analysis unit is realized by the specific processing unit 290 of the data processing device 12, and analyzes the feature values ​​and image data to generate voice information as characters. The projection unit projects character information in real time using the display of the headset type terminal 314. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0158] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0159] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0160] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0161] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0162] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0163] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person).

[0164] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0165] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and legs. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, and legs. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0166] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0167] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0168] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0169] In the robot 414, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads out the specific program 60 from the storage 50, and executes the specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. The robot 414 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0170] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0171] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0172] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0173] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0174] Each of the multiple elements including the above-mentioned sound collection unit, feature value extraction unit, image acquisition unit, analysis unit, and projection unit is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the sound collection unit collects surrounding sounds using a microphone of the robot 414. The feature value extraction unit is realized by the specific processing unit 290 of the data processing device 12, and extracts feature values ​​from the collected sound source. The image acquisition unit acquires surrounding images using a camera of the robot 414. The analysis unit is realized by the specific processing unit 290 of the data processing device 12, and analyzes the feature values ​​and image data to generate voice information as characters. The projection unit projects character information in real time using the display of the robot 414. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0175] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[0176] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0177] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0178] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0179] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0180] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0181] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[0182] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0183] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0184] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0185] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0186] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0187] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0188] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0189] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0190] In the above example, the first to fourth embodiments are described separately, but some or all of these embodiments may be combined. The smart device 14, the smart glasses 214, the headset terminal 314, and the robot 414 are only examples, and they may be combined with each other, or may be other devices. In the above example, the first and second embodiments are described separately, but they may be combined.

[0191] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replaced with respect to the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the description and illustrations shown above omit explanations of technical common sense that do not require particular explanation in enabling the implementation of the technology of the present disclosure.

[0192] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0193] (Appendix 1) A sound pickup unit that picks up a sound source; a feature value extraction unit that extracts feature values ​​from a sound source collected by the sound collection unit; An image acquisition unit that acquires an image of the surroundings; an analysis unit that analyzes the feature values ​​extracted by the feature value extraction unit and the image data acquired by the image acquisition unit, and generates voice information as characters; A projection unit that projects the character information generated by the analysis unit onto a visual device. A system characterized by: (Appendix 2) The feature value extraction unit Includes a method for determining the direction of a sound source 2. The system of claim 1, (Appendix 3) The image acquisition unit includes: Includes a method for acquiring images of the surroundings in real time 2. The system of claim 1, (Appendix 4) The analysis part is This includes a method of using generative AI to analyze the feature values ​​of the sound source and image data, and generate voice information as text. 2. The system of claim 1, (Appendix 5) The projection section is A method for projecting the generated textual information onto a visual device in real time is also included. 2. The system of claim 1, (Appendix 6) The analysis part is Includes a method to analyze the tone of a speaker's voice and generate text information. 2. The system of claim 1, (Appendix 7) The sound pickup section is A method for estimating a user's emotion and adjusting the sensitivity of the sound pickup based on the estimated user's emotion is included. 2. The system of claim 1, (Appendix 8) The sound pickup section is This includes a method of emphasizing a specific frequency band when recording sound. 2. The system of claim 1, (Appendix 9) The sound pickup section is Includes a method to add filtering functionality to remove ambient noise in real time during recording. 2. The system of claim 1, (Appendix 10) The sound pickup section is A method for estimating a user's emotion and prioritizing sound sources to be picked up based on the estimated user's emotion is also included. 2. The system of claim 1, (Appendix 11) The sound pickup section is When recording, the system automatically adjusts the optimal microphone placement based on the user's position information. 2. The system of claim 1, (Appendix 12) The sound pickup section is When recording, the optimal recording settings are automatically applied based on the user's past recording history. 2. The system of claim 1, (Appendix 13) The feature value extraction unit A method for estimating a user's emotion and adjusting a feature value extraction method based on the estimated user's emotion is included. 2. The system of claim 1, (Appendix 14) The feature value extraction unit When extracting feature values, the feature values ​​are corrected based on the echo and reverb of the sound source. 2. The system of claim 1, (Appendix 15) The feature value extraction unit When extracting feature values, a method is included for generating feature values ​​by analyzing the frequency spectrum of the sound source in detail. 2. The system of claim 1, (Appendix 16) The feature value extraction unit A method for estimating a user's emotion and determining a priority order for feature values ​​to be extracted based on the estimated user's emotion is also included. 2. The system of claim 1, (Appendix 17) The feature value extraction unit This includes a method to add a function to identify not only the direction but also the distance of the sound source when extracting feature values. 2. The system of claim 1, (Appendix 18) The feature value extraction unit When extracting feature values, a method is included in which the environmental sound of the sound source is analyzed and reflected in the feature values. 2. The system of claim 1, (Appendix 19) The image acquisition unit includes: and a method for estimating a user's emotion and adjusting a frame rate of image capture based on the estimated user's emotion. 2. The system of claim 1, (Appendix 20) The image acquisition unit includes: Includes a method to add a function to capture images by emphasizing specific colors or shapes. 2. The system of claim 1, (Appendix 21) The image acquisition unit includes: Includes a method to add a function to automatically adjust the surrounding brightness and contrast when acquiring images. 2. The system of claim 1, (Appendix 22) The image acquisition unit includes: A method for estimating a user's emotion and prioritizing images to be acquired based on the estimated user's emotion is also included. 2. The system of claim 1, (Appendix 23) The image acquisition unit includes: This includes a method for adding a function to track the user's gaze direction when acquiring an image and acquire the optimal image. 2. The system of claim 1, (Appendix 24) The image acquisition unit includes: When acquiring images, the optimal settings are automatically applied based on the user's previous image acquisition history. 2. The system of claim 1, (Appendix 25) The analysis part is A method for estimating a user's emotion and adjusting an analysis algorithm based on the estimated user's emotion is also included. 2. The system of claim 1, (Appendix 26) The analysis part is During analysis, text information is generated based on the context of the audio source. 2. The system of claim 1, (Appendix 27) The analysis part is During analysis, text information is generated based on the speaker's attributes. 2. The system of claim 1, (Appendix 28) The analysis part is A method for estimating a user's emotion and adjusting the display method of the analysis result based on the estimated user's emotion is included. 2. The system of claim 1, (Appendix 29) The analysis part is During analysis, the text information is updated based on the background sound of the sound source. 2. The system of claim 1, (Appendix 30) The analysis part is During analysis, the text information is updated based on the gestures of the speaker of the audio source. 2. The system of claim 1, (Appendix 31) The projection section is A method for estimating a user's emotion and adjusting the font and size of the text information to be projected based on the estimated user's emotion is also included. 2. The system of claim 1, (Appendix 32) The projection section is A method for dynamically adjusting the display position of text information according to the user's line of sight during projection is also included. 2. The system of claim 1, (Appendix 33) The projection section is A method for automatically adjusting the color of text information and background color based on the surrounding environment during projection is included. 2. The system of claim 1, (Appendix 34) The projection section is A method for estimating a user's emotion and determining a priority of information to be projected based on the estimated user's emotion is included. 2. The system of claim 1, (Appendix 35) The projection section is This includes a method for adding a function to display text information in stages during projection in order to reduce the user's visual burden. 2. The system of claim 1, (Appendix 36) The projection section is When projecting, the optimal settings are automatically applied based on the user's past projection history 2. The system of claim 1, [Explanation of symbols]

[0194] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. A sound pickup unit that picks up a sound source; a feature value extraction unit that extracts feature values ​​from the sound source collected by the sound collection unit; An image acquisition unit that acquires an image of the surroundings; an analysis unit that analyzes the feature values ​​extracted by the feature value extraction unit and the image data acquired by the image acquisition unit, and generates voice information as characters; A projection unit that projects the character information generated by the analysis unit onto a visual device. A system characterized by:

2. The feature value extraction unit Includes a method for determining the direction of a sound source 2. The system of claim 1 .

3. The image acquisition unit includes: Includes a method for acquiring images of the surroundings in real time 2. The system of claim 1 .

4. The analysis unit is This includes a method of using AI to analyze the feature values ​​of the sound source and image data, and generate voice information as text.

2. The system of claim 1 .

5. The projection unit is A method for projecting the generated textual information onto a visual device in real time is also included.

2. The system of claim 1 .

6. The analysis unit is Includes a method to analyze the tone of a speaker's voice and generate text information.

2. The system of claim 1 .

7. The sound collection unit includes: A method for estimating a user's emotion and adjusting the sensitivity of the sound pickup based on the estimated user's emotion is included.

2. The system of claim 1 .

8. The sound collection unit includes: This includes a method of emphasizing a specific frequency band when recording sound.

2. The system of claim 1 .

Images