System

Abstract

To provide a system for achieving both comfortable environment and privacy protection.SOLUTION: The system includes means for capturing ambient sound, means for analyzing the captured sound data, means for generating an optimal masking sound based on the analysis, and means for playing back the generated masking sound.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, the increase in remote meetings, noise issues at home, and noise in shared spaces can cause stress, resulting in a lack of space for comfortable meetings and phone conversations. Such environments reduce user satisfaction and, where privacy is a priority, can lead to issues with conversations being leaked. Facilities such as offices, cafes, libraries, hospitals, and financial institutions require both a comfortable environment and privacy protection. This invention aims to solve these issues and provide a comfortable and efficient user environment. [Means for solving the problem]

[0005] The present invention provides a system including a means for capturing ambient sounds, a means for analyzing the captured sound data, a means for generating an optimal masking sound based on the analysis results, and a means for playing the generated masking sound. The system further includes a means for transmitting the captured sound data to a server, a means for the server to analyze the sound data and generate an optimal masking sound, and a means for transmitting the generated masking sound to a terminal. Furthermore, since the masking sound is generated and played in real time and the captured sound data includes volume and frequency spectrum, it is possible to effectively cancel out noise and provide a comfortable space for the user.

[0006] "Ambient sounds" refers to the noise and background sounds that occur within the environment in which the system is operating.

[0007] The term "capturing means" refers to a mechanism for acquiring surrounding sounds using a device such as a microphone or audio sensor.

[0008] "Sound data" refers to data that quantifies captured sound, including volume, frequency spectrum, and temporal fluctuation.

[0009] "Means for analysis" refers to algorithms and software for processing sound data and extracting and analyzing its characteristics.

[0010] "Optimal masking sound" refers to a sound that is generated based on the analyzed characteristics of the noise and is designed to effectively cancel out the noise.

[0011] "Means for generating" refers to AI models and algorithms for automatically generating optimal masking sounds based on the analysis results.

[0012] "Means for playing" refers to a mechanism for actually playing the generated masking sound through a speaker or audio output device.

[0013] "Server" refers to a central processing system for analyzing captured sound data, generating optimal masking sounds, and transmitting them to the terminal.

[0014] "Terminal" refers to a device equipped with a microphone and speaker that captures ambient sounds, receives data from the server, and plays masking sounds.

[0015] "Real-time" refers to the ability of the entire system to operate without delay, instantly process captured sound data, and generate and play masking sounds.

[0016] "Volume" is a physical quantity that indicates the intensity of sound and is measured in decibels (dB).

[0017] A "frequency spectrum" indicates the distribution of frequency components of a sound, and indicates how much energy is contained in each frequency.

[0018] In data communication, a "packet" is a unit into which transmission data is divided and transmitted over a network. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The system for implementing this invention provides a comfortable space by inputting noise and playing the optimal masking sound created by the AI ​​in real time. This system operates in cooperation with the server, terminals, and users.

[0041] System configuration

[0042] 1. Terminal: The terminal has a built-in microphone and speaker, which allows it to capture ambient sounds and play back the generated masking sound. The terminal also has a network connection function for communicating with the server.

[0043] 2. Server: The server has a means of receiving and analyzing noise data sent from the device. Furthermore, the server has a generative AI model that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time.

[0044] 3. User: The user can start the system and use the terminal to maintain a comfortable environment. The user can enjoy the automatic noise reduction of the system without any special operation.

[0045] Explaining the program's processing

[0046] The program of this system is executed in the following procedure, with the terminal, server, and user each taking on a different role.

[0047] 1. Booting the system

[0048] When a user activates the system on their device, the device's microphone begins capturing ambient sound. The captured sound data is converted into volume and frequency spectrum data and sent to the server.

[0049] 2. Analysis of noise data

[0050] The server receives noise data sent from the device and analyzes it using a proprietary algorithm. Once the characteristics of the noise have been clarified, the server generates the optimal masking sound based on the analysis results.

[0051] 3. Masking sound generation

[0052] The server uses a generative AI model to generate the optimal masking sound, which can be white noise or soft background music based on the analyzed noise characteristics.

[0053] 4. Transmitting and playing masking sounds

[0054] The generated masking sound is sent from the server to the device, which then decodes it and plays it back through the speaker in real time. This process effectively cancels out the noise around the user, providing a comfortable environment.

[0055] Specific examples

[0056] Example 1: Use in the office

[0057] A user in a meeting in a shared office space starts the system. The device's microphone captures ambient noise and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker, preventing the content of the conversation from leaking to the outside world. The user also enjoys a quiet environment that makes it easier to concentrate.

[0058] Example 2: Use at a cafe

[0059] A user working remotely in a cafe starts the system. Noise captured by the device's microphone is sent to the server, which analyzes it and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the noise in the cafe, allowing the user to continue working in a quiet environment.

[0060] Through these processes, the system can reduce noise in real time and provide a comfortable space for users.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user starts the system on the device. When the user launches an application and presses the system start button, the device's microphone starts up and is ready to capture ambient sounds. The device sends a connection request to the server to notify that the system has started.

[0064] Step 2:

[0065] The device's microphone captures audio data in real time, and the captured audio is converted into digital data, which includes information such as volume (dB) and frequency spectrum.

[0066] Step 3:

[0067] The device then collects the captured audio data and sends it to the server, where it receives a confirmation response confirming that the transmission was successful.

[0068] Step 4:

[0069] The server analyzes the received noise data, including parameters such as volume, frequency distribution, and duration, and determines which type of masking sound is most suitable based on the analysis results.

[0070] Step 5:

[0071] The server uses a generative AI model to generate the optimal masking sound, which can be white noise or soft background music based on the analyzed noise characteristics.

[0072] Step 6:

[0073] The server assembles the generated masking sounds into packets and prepares to send them to the terminal. The server sends the generated masking sounds to the terminal and receives a confirmation response from the terminal.

[0074] Step 7:

[0075] The device decodes the received masking sound data and prepares it for playback. The masking sound is played in real time through the speaker to reduce ambient noise.

[0076] Step 8:

[0077] The generated masking sound allows the user to continue working in a comfortable, noise-reduced environment. When the user presses the system shutdown button, all processes are stopped and the device's microphone and speaker are turned off.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] In modern society, noise has become a major problem in daily life and work environments. Noise can reduce concentration and cause stress and fatigue. Conventional technologies have not provided an efficient method for reducing noise. In particular, there has been no system that can detect noise in real time and generate and play appropriate masking sounds based on that detection. Therefore, there is a need for a system that can effectively cancel noise and provide a comfortable space for users.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server includes means for capturing ambient sounds, means for converting the captured sound data into volume and frequency spectrum, means for transmitting the converted sound data to the server, means for the server to analyze the sound data and extract noise characteristics, means for generating an optimal masking sound using a generative AI model, means for transmitting the generated masking sound to the terminal, means for the terminal to decode the masking sound and play it through a speaker, and means for playing the generated masking sound. This makes it possible to detect noise in real time and generate and play an adaptive masking sound.

[0083] A "means for capturing ambient sound" is a method of converting sound waves in the environment into electrical signals using a sensor device (e.g., a microphone).

[0084] "Means for converting captured sound data into volume and frequency spectrum" refers to algorithms or software that analyze audio signals and express their sound pressure (volume) and frequency components as digital data.

[0085] The "means for transmitting the converted sound data to the server" is a process for communicating the sound data from the terminal to the server using the network connection function.

[0086] "Means for the server to analyze sound data and extract noise characteristics" refers to technology that receives audio data and analyzes specific frequency bands and volume levels to identify noise patterns and characteristics.

[0087] "Means for generating optimal masking sounds using generative AI models" refers to a process that utilizes machine learning algorithms to design and generate optimal masking sounds based on the characteristics of noise.

[0088] The "means for transmitting the generated masking sound to the terminal" is a method for communicating the generated audio data from the server to the terminal.

[0089] The "means for the terminal to decode the masking sound and play it through the speaker" refers to the process of converting the audio data received by the terminal into an audio signal and playing it through the speaker device.

[0090] The "means for reproducing the generated masking sound" refers to a technique for converting the generated audio data into physical sounds and reproducing them in the environment.

[0091] A "prompt" is a text-based instruction that allows a generative AI model to input specific conditions or data.

[0092] The system for implementing this invention provides a comfortable space by inputting noise and playing the optimal masking sound created by the AI ​​in real time. This system operates in cooperation with the server, terminals, and users.

[0093] System configuration

[0094] 1. Terminal

[0095] The device has a built-in microphone and speaker, and has a means for capturing ambient sounds and a means for playing back the generated masking sounds. The device also has a network connection function for communicating with a server, and can transmit the captured sound data to the server.

[0096] For example, the microphone captures ambient sounds in real time and converts them into volume and frequency spectrum using built-in audio processing libraries such as Librosa.

[0097] 2. Server

[0098] The server has a generative AI model that analyzes the received sound data and generates the optimal masking sound. The server uses Python (registered trademark)-based scripts and machine learning libraries such as TENSORFLOW (registered trademark) and PyTorch to analyze the sound data. After analysis, the server sends the generated masking sound to the device. When the server uses a generative AI model (e.g., OpenAI (registered trademark)'s GPT-3 (registered trademark) or GPT-4 (registered trademark)) to generate the optimal masking sound, it uses the following prompt statement:

[0099] Prompt statement:

[0100] "The frequency spectrum of the current ambient sound is as follows:\n[frequency data]. Please generate the optimal masking sound to cancel out this ambient sound."

[0101] 3. Users

[0102] Users can activate the system and use their devices to maintain a comfortable environment, and the system automatically reduces noise without any special operations.

[0103] Specific examples

[0104] Example 1: Use in the office

[0105] A user in a meeting in a shared office space starts the system. The device's microphone captures ambient noise and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker, preventing the content of the conversation from being leaked to the outside world and providing the user with a quiet environment where they can concentrate easily.

[0106] Example 2: Use at a cafe

[0107] A user working remotely in a cafe starts the system. Noise captured by the device's microphone is sent to the server, which analyzes it and generates the optimal masking sound. The generated masking sound is played in real time through the device's speaker, effectively canceling out the cafe's noise, allowing the user to continue working in a quiet environment.

[0108] In this way, the system of the present invention can reduce noise in real time and provide a comfortable space for the user.

[0109] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0110] Step 1:

[0111] The user starts an application on the terminal. The user presses the "Start" button, and the system enters the startup state. The input is the user's operation. The output is the system initialization completion state.

[0112] Step 2:

[0113] The device's microphone captures the surrounding sound in real time. The input is the surrounding sound, and the output is the captured audio data. Specifically, the device's microphone converts analog audio into a digital signal.

[0114] Step 3:

[0115] The device converts captured audio data into volume and frequency spectrum. The input is a digital audio signal, and the output is data containing volume and frequency spectrum. The device uses the Librosa library to do this. For example, Librosa can be used to convert audio data into a spectrogram using techniques such as fft and stft.

[0116] Step 4:

[0117] The device sends the converted sound data to the server. The input is data including volume and frequency spectrum, and the output is an HTTP POST request to the server. The device transfers this data to the server using its network connection function.

[0118] Step 5:

[0119] The server receives the transmitted sound data and begins analysis. The input is volume and frequency spectrum data, and the output is data extracting the noise characteristics. The server performs analysis using algorithms implemented in Python, as well as TensorFlow and PyTorch.

[0120] Step 6:

[0121] The server uses a generative AI model based on the characteristics of the noise to generate the optimal masking sound. The input is the analyzed noise characteristic data, and the output is the generated masking sound. Models such as GPT-3 and GPT-4 are used as generative AI models. The specific generation procedure uses the following prompt sentence:

[0122] Prompt statement:

[0123] "The frequency spectrum of the current ambient sound is as follows:\n[frequency data]. Please generate the optimal masking sound to cancel out this ambient sound."

[0124] Step 7:

[0125] The server sends the generated masking sound to the device. The input is the generated masking sound, and the output is the audio data as an HTTP POST request to the device.

[0126] Step 8:

[0127] The device decodes the received masking sound and plays it through the speaker. The input is the audio data sent from the server, and the output is the actual masking sound. The device uses FFmpeg or Librosa to decode the audio data into a playable format and play it through the speaker.

[0128] (Application example 1)

[0129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] In modern society, activities in virtual environments have become commonplace, but noise and other disturbances can significantly disrupt the user experience. Conventional sound environment control systems primarily control sound in physical spaces and are not suitable for noise reduction in virtual stores and other virtual environments. Therefore, an efficient system for providing a comfortable sound environment in virtual environments is needed.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0132] In this invention, the server includes means for capturing ambient sounds, means for analyzing the captured sound data, means for generating an optimal masking sound based on the analysis results, means for playing back the generated masking sound, and means for analyzing noise in the virtual environment and providing an optimal sound environment for a display device worn by the user. This makes it possible to effectively reduce noise and other disturbances in the virtual environment and provide a comfortable sound environment for the user.

[0133] The "means for capturing ambient sound" is a function for acquiring ambient environmental sound using a device such as a microphone.

[0134] "Means for analyzing captured sound data" refers to algorithms or software that analyze the characteristics of the acquired sound data.

[0135] The "means for generating the optimum masking sound based on the analysis results" is a function that performs a process to create the optimum masking sound for the user according to the characteristics of the analyzed sound.

[0136] The "means for reproducing the generated masking sound" is a function for allowing the user to hear the generated masking sound using an audio device such as a speaker or earphones.

[0137] The "means for transmitting captured sound data to a server" is a function for transmitting the acquired sound data to a remote server via the Internet or a network.

[0138] "The means by which the server analyzes the sound data and generates the optimal masking sound" refers to the process of analyzing the transmitted sound data using an analysis algorithm or generation AI model on the server and generating the optimal masking sound.

[0139] The "means for transmitting the generated masking sound to the terminal" is a function for transmitting the masking sound generated by the server to the user's terminal via the network.

[0140] "Means for connecting to a display device worn by the user to provide an optimal sound environment" refers to a function that communicates with devices such as smart glasses and head-mounted displays (HMDs) to provide an optimal sound environment.

[0141] "Means for responding to dynamic changes in noise in a virtual environment" refers to a function that responds in real time to changes in noise and the sound environment within a virtual space and generates appropriate masking sounds.

[0142] In this embodiment, a comfortable sound environment is provided by inputting noise and playing optimal masking sounds generated by a generative AI model in real time. This system operates in cooperation with a server, terminals, and users.

[0143] System configuration

[0144] 1. Device:

[0145] The device has a built-in microphone and speaker, and has a means for capturing ambient sounds and a means for playing back the generated masking sounds. The device also has a network connection function for communicating with a server. In particular, it includes a means for connecting to devices such as smart glasses and head-mounted displays (HMDs) in a virtual environment to provide an optimal sound environment.

[0146] 2. Server:

[0147] The server has a means of receiving and analyzing noise data sent from the device. Furthermore, the server has a generative AI model that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time. The Python Librosa library is used to analyze the noise data, and TensorFlow and PyTorch are used for the generative AI model.

[0148] 3. User:

[0149] Users can activate the system and use their devices to maintain a comfortable environment, and the system automatically reduces noise without any special operations.

[0150] Program processing

[0151] The programs in this system are executed by dividing the roles of the terminal, server, and user.

[0152] When a user puts on smart glasses or an HMD and starts the system, the device's microphone begins capturing ambient sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server.

[0153] The server receives noise data sent from the device and analyzes it using a proprietary algorithm. Once the characteristics of the noise have been clarified, the server generates the optimal masking sound based on the analysis results. During this generation process, white noise, soft background music, etc. are created based on the analyzed noise characteristics.

[0154] The generated masking sound is sent from the server to the device and played in real time through the device's speaker, effectively canceling out the user's surrounding noise and providing a comfortable sound environment.

[0155] Specific examples

[0156] Suppose a user is distracted by other users' chats and background music in a virtual store. The user puts on smart glasses and launches an application. The device's microphone captures the noise in the virtual store and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time from the smart glasses' speakers, allowing the user to enjoy shopping in a comfortable acoustic environment.

[0157] Prompt Sentence Examples

[0158] The generative AI model is given a prompt like this:

[0159] Noise data: {sound data}

[0160] Output conditions: Soft background music or white noise

[0161] Response format: data

[0162] With this configuration and means, the system can effectively reduce noise and disturbances in the virtual environment, providing the user with a comfortable sound environment.

[0163] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0164] Step 1:

[0165] A user puts on smart glasses or a head-mounted display (HMD) and starts the system. The user's operation causes the device's microphone to start capturing ambient sounds. At this point, the input is the ambient sound, and the output is the captured sound data.

[0166] Step 2:

[0167] The device converts the captured sound data into volume and frequency spectrum data using a library for audio signal processing (e.g., Python's Librosa library). The input is the captured sound data, and the output is the converted volume and frequency spectrum data.

[0168] Step 3:

[0169] The terminal transmits the converted volume and frequency spectrum data to the server. The data is transmitted over the network. The input is the converted volume and frequency spectrum data, and the output is the data transmitted to the server.

[0170] Step 4:

[0171] The server analyzes the received volume and frequency spectrum data. A proprietary algorithm is used for the analysis, and a generative AI model (e.g., TensorFlow or PyTorch) is used for this process. The input is the volume and frequency spectrum data sent to the server, and the output is the analysis result.

[0172] Step 5:

[0173] The server generates the optimal masking sound based on the analysis results. During this generation process, white noise or soft background music is selected depending on the noise characteristics. An example prompt sentence is input to the generation AI model. The input is the analysis results and the prompt sentence, and the output is the generated masking sound data. An example prompt sentence is as follows:

[0174] Noise data: {sound data}

[0175] Output conditions: Soft background music or white noise

[0176] Response format: data

[0177] Step 6:

[0178] The server sends the generated masking sound data to the terminal. The data is sent via the network and arrives at the user's terminal in real time. The input is the generated masking sound data, and the output is the data sent to the terminal.

[0179] Step 7:

[0180] The device decodes the received masking sound data and plays it through its built-in speaker. This process effectively cancels out the user's surrounding noise in real time. The input is the masking sound data received from the server, and the output is the masking sound being played.

[0181] Step 8:

[0182] The user enjoys the comfortable sound environment obtained by the operation of the system. In this step, the user does not need to perform any special operations; the system automatically reduces noise. The input is the reproduced masking sound, and the output is a comfortable sound environment.

[0183] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0184] The system for implementing this invention combines a conventional noise control system with an emotion engine that recognizes the user's emotions, providing a more comfortable space for the user. This system operates in cooperation with a server, terminals, and users.

[0185] System configuration

[0186] 1. Device: The device is equipped with a microphone, speaker, camera, and other emotion sensors to capture ambient sounds and the user's emotions, and to play back the generated masking sounds. The device also has a network connection function to communicate with the server.

[0187] 2. Server: The server has a means of receiving and analyzing noise data and emotion data sent from the device. Furthermore, the server has a generative AI model and emotion engine that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time.

[0188] 3. Emotion Engine: The emotion engine analyzes the user's stress level and emotional state from facial expressions and vocal tone, and helps adjust the masking sound accordingly.

[0189] 4. User: The user can start the system and use the device to maintain a comfortable environment. The user can enjoy the system automatically analyzing and reducing noise and emotional state without any special operation.

[0190] Explaining the program's processing

[0191] The program of this system is executed in the following procedure, with each of the roles of terminal, server, and emotion engine being assigned.

[0192] 1. Booting the system

[0193] When a user activates the system on their device, the device's microphone begins capturing ambient sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server. At the same time, the device uses a camera and emotion sensors to collect the user's emotion data, which is also sent to the server.

[0194] 2. Data Analysis

[0195] The server receives noise and emotion data sent from the device and analyzes the data using a proprietary algorithm. Once the characteristics of the noise and the user's emotional state are clarified, the server generates the optimal masking sound based on the analysis results.

[0196] 3. Masking sound generation

[0197] The server uses a generative AI model to generate optimal masking sounds, such as white noise or soft background music, based on the analyzed noise characteristics and the user's emotional state as determined by the emotion engine.

[0198] 4. Transmitting and playing masking sounds

[0199] The generated masking sound is sent from the server to the device, which then decodes it and plays it back through the speaker in real time, reducing ambient noise and providing an optimal environment for the user's emotional state.

[0200] Specific examples

[0201] Example 1: Use in the office

[0202] A user in a meeting in a shared office space activates the system. The device's microphone captures ambient noise, while the camera and emotion sensor recognize the user's emotional state. The server analyzes the captured data and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker to prevent the content of the conversation from leaking to the outside. The system also adjusts according to the user's emotional state to reduce stress levels.

[0203] Example 2: Use at a cafe

[0204] A user working remotely in a cafe starts the system. Noise captured by the device's microphone and emotional state analyzed by the emotion engine are sent to the server, which then analyzes the data and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the cafe's noise and providing an environment suited to the user's emotional state.

[0205] Through these processes, the system can reduce noise in real time and provide an optimal space according to the user's emotional state.

[0206] The processing flow will be explained below.

[0207] Step 1:

[0208] The user starts the system on the device. When the user launches the application and presses the system's start button, the device's microphone, camera, and emotion sensor are activated and ready to capture ambient sounds and the user's emotional state. The device sends a connection request to the server to notify that the system has started.

[0209] Step 2:

[0210] The device's microphone captures voice data in real time, which is then converted into digital signals containing data such as volume (dB) and frequency spectrum. At the same time, the device's camera and emotion sensor analyze the user's facial expressions and voice tone to collect emotional data.

[0211] Step 3:

[0212] The device then collects the captured sound data and emotion data into packets and sends them to the server. The device receives a confirmation response from the server to confirm that the data transmission was successful.

[0213] Step 4:

[0214] The server analyzes the received noise data and emotion data, including parameters such as volume, frequency distribution, and duration. Based on the analysis results, the server determines which type of masking sound is optimal.

[0215] Step 5:

[0216] The server uses a generative AI model to generate optimal masking sounds, such as white noise or soft background music, based on the analyzed noise characteristics and the user's emotional state as determined by the emotion engine.

[0217] Step 6:

[0218] The generated masking sound is sent from the server to the terminal. The server sends packetized data and receives an acknowledgment from the terminal. The terminal decodes the received masking sound data and prepares for playback.

[0219] Step 7:

[0220] The device plays masking sounds through the speaker in real time, reducing ambient noise. The emotion engine also monitors the user's emotional state in real time and adjusts the masking sounds as needed.

[0221] Step 8:

[0222] Users can work and talk in a comfortable environment thanks to the generated masking sounds and optimization by the emotion engine. If the user feels stressed, the masking sounds are automatically adjusted to maintain comfort. When the user presses the system's shutdown button, all processes are stopped and the device's microphone, camera, emotion sensor, and speaker are turned off.

[0223] Example 2

[0224] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0225] In modern urban environments, noise levels are constantly increasing, causing stress for many people. Existing noise control systems focus on capturing sounds and generating masking sounds, but they do not take into account the user's emotional state, making it difficult to provide an optimal sound environment.

[0226] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing ambient sounds, means for capturing user emotional data, means for transmitting the captured sound data and emotional data to the server, means for the server to analyze the sound data and emotional data, means for generating an optimal masking sound using a generative AI model based on the analysis results, means for transmitting the generated masking sound to the terminal, and means for playing the generated masking sound. This makes it possible to provide an optimal sound environment according to the user's emotional state while effectively reducing noise.

[0227] The "means for capturing ambient sound" refers to a device or system for collecting sounds in the environment, and specifically refers to a microphone or the like.

[0228] "Means for capturing user emotional data" refers to a device or system for collecting the user's emotional state, and specifically includes a camera or a voice analysis sensor.

[0229] "Means for transmitting captured sound data and emotional data to a server" refers to a network connection device or system for communicating the collected data to a server, specifically Wi-Fi, Bluetooth, the Internet, etc.

[0230] "Means for the server to analyze sound data and emotional data" refers to software or algorithms for processing and analyzing sound data and emotional data received by the server.

[0231] "Means for generating optimal masking sound using an AI model based on the analysis results" refers to software that includes an artificial intelligence model or algorithm for generating optimal masking sound based on the analyzed data.

[0232] The "means for transmitting the generated masking sound to the terminal" refers to a network connection device or system for communicating the generated masking sound to the terminal, and specifically includes the Internet, Wi-Fi, Bluetooth, etc.

[0233] The "means for reproducing the generated masking sound" refers to a device or system for reproducing the received masking sound, and specifically refers to a speaker, a headphone, or the like.

[0234] MODE FOR CARRYING OUT THE INVENTION

[0235] The system of the present invention provides optimal masking sound based on the ambient noise and the user's emotional state by linking a server, a terminal, and an emotion engine. The system uses the following hardware and software:

[0236] Hardware and software used

[0237] 1. Device:

[0238] Microphone: A device for capturing ambient sounds.

[0239] Camera: A device for capturing a user's facial expressions.

[0240] Emotion sensors: Sensors for capturing the user's tone of voice and other vital signs.

[0241] Network connectivity: A means of communicating with the server (e.g., Wi-Fi, Bluetooth).

[0242] Speaker: A device for playing back the masking sound sent from the server.

[0243] 2. Server:

[0244] Data analysis module: An algorithm that analyzes noise data and emotion data sent from the device.

[0245] Generative AI model: A program that generates optimal masking sounds based on analysis results (e.g., OpenAI's GPT-3 and DALL-E).

[0246] Network connection function: A means for communicating with terminals.

[0247] 3. Emotion Engine:

[0248] Emotion analysis algorithm: Software for analyzing the emotional state of a user.

[0249] Operational flow and concrete examples

[0250] When the system is started, the device's microphone first captures the surrounding sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server. At the same time, the device's camera and emotion sensor collect the user's emotion data and send it to the server.

[0251] The server receives the noise and emotion data sent from the device and analyzes each data. The analysis uses a data analysis module and emotion analysis algorithm. Based on the analysis results, the server uses a generative AI model to generate optimal masking sounds. For example, if the noise is intense and the user is feeling stressed, it generates relaxing white noise or soft background music.

[0252] The generated masking sound is sent from the server to the device, which then receives it and plays it in real time through the speaker, effectively reducing ambient noise and providing the optimal sound environment according to the user's emotional state.

[0253] Specific examples

[0254] Example 1: Use in the office

[0255] A user in a meeting in a shared office space activates the system. The device's microphone captures ambient noise, while the camera and emotion sensor recognize the user's emotional state. The server analyzes the captured data and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker to prevent the content of the conversation from leaking to the outside. The system also adjusts according to the user's emotional state to reduce stress levels.

[0256] Example prompt:

[0257] "Generate masking sounds suitable for situations where people are having a meeting in a shared office space. Users are stressed and want a relaxing environment."

[0258] Example 2: Use at a cafe

[0259] A user working remotely in a cafe starts the system. Noise captured by the device's microphone and emotional state analyzed by the emotion engine are sent to the server, which then analyzes the data and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the cafe's noise and providing an environment suited to the user's emotional state.

[0260] Example prompt:

[0261] "For users working remotely in a cafe, please generate masking sounds that cancel out surrounding noise and make it easier for them to concentrate."

[0262] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0263] Step 1:

[0264] The user starts the system on the device. The device first initializes and enables the built-in microphone, camera, and emotion sensor, ensuring that the hardware is operational and ready to begin capturing data.

[0265] Specific behavior:

[0266] The user clicks the "System Start" button.

[0267] Input: User action (clicking the launch button)

[0268] Output: Device hardware initialization complete

[0269] Step 2:

[0270] The device's microphone begins capturing ambient sound, which is then converted into volume and frequency spectrum data, which is then sent to the server.

[0271] Specific behavior:

[0272] The device's microphone records the surrounding sounds and samples the waveform data in real time, converts the data into a digital format (e.g., PCM format), and sends it to the server.

[0273] Input: Ambient sound

[0274] Output: Volume and frequency spectrum converted data

[0275] Step 3:

[0276] The device's camera and emotion sensors capture the user's facial expressions and voice tone to collect emotion data, which is also sent to the server.

[0277] Specific behavior:

[0278] The device's camera captures the user's face and analyzes the movements of each part of the face (eyes, mouth, eyebrows, etc.). The audio sensor also analyzes the user's tone of voice and speaking style. This data is combined to determine the user's emotional state and is sent to the server.

[0279] Input: User facial expressions and voice tone

[0280] Output: Emotion data

[0281] Step 4:

[0282] The server receives the noise and emotion data sent from the device. The received data is processed by the analysis module. The server uses a proprietary algorithm to analyze the characteristics of the noise and the user's emotional state.

[0283] Specific behavior:

[0284] The server analyzes the characteristics of the received noise data and identifies the sound volume, frequency components, source, etc. At the same time, it detects the user's current emotional state (stress level, relaxation level) based on the received emotional data.

[0285] Input: Noise data and emotion data

[0286] Output: Noise analysis results and emotion analysis results

[0287] Step 5:

[0288] The server uses a generative AI model to generate optimal masking sounds based on the analysis results. The generated masking sounds are tailored to the user's emotional state, such as white noise or relaxing music.

[0289] Specific behavior:

[0290] Based on the analysis results, the server inputs prompts to the generative AI model, which then designs appropriate masking sounds and returns the sound data.

[0291] Input: Noise analysis results and emotion analysis results

[0292] Output: Generated masking sound

[0293] Example prompt:

[0294] "For users working remotely in a cafe, please generate masking sounds that cancel out surrounding noise and make it easier for them to concentrate."

[0295] Step 6:

[0296] The generated masking sound is sent from the server to the device, which decodes the received sound data and plays it back in real time through the speaker.

[0297] Specific behavior:

[0298] The server sends the generated masking sound to the device, which then decodes the received sound data and plays it back using the speaker, allowing the user to work in a comfortable environment.

[0299] Input: Generated masking sound

[0300] Output: Played masking sound

[0301] (Application example 2)

[0302] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0303] In the food delivery industry, delivery staff often experience mental stress due to traffic noise and other disturbances at delivery destinations. This reduces the efficiency of delivery work and staff comfort. The present invention aims to provide an environment in which delivery staff can perform their work comfortably and efficiently.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing ambient sounds, means for analyzing the captured sound data and the user's emotional state, and means for generating an optimal masking sound based on the analysis results and the user's emotional state. This effectively masks the noise around the delivery staff, enabling them to perform their work while reducing mental stress.

[0305] The "means for capturing ambient sound" is a device including a microphone and a sensor for collecting environmental sounds around the delivery staff.

[0306] "Means for analyzing captured sound data and the user's emotional state" refers to algorithms or engines that analyze the collected sound data and the emotional state obtained from the delivery staff's facial expressions, vocal tone, etc.

[0307] The "means for generating optimal masking sounds based on the analysis results and the user's emotional state" refers to a generative AI model or sound engine that generates optimal sounds to reduce the mental stress of delivery staff based on noise data and emotional data.

[0308] The "means for reproducing the generated masking sound" refers to a speaker or audio reproduction device for reproducing the generated masking sound in real time.

[0309] The "means for transmitting captured sound data and emotional data to a server" refers to a communication module or protocol for transmitting the collected sound data and emotional data to a server via a network.

[0310] "The means by which the server analyzes sound data and emotional data and generates optimal masking sounds" refers to a function that uses an analysis algorithm or generation AI model running on the server to analyze collected data and generate masking sounds.

[0311] The "means for transmitting the generated masking sound to the terminal" is a communication function for transmitting the masking sound generated by the server to the terminal of the delivery staff via the network.

[0312] A "generative AI model" is a type of artificial intelligence model used for data analysis and sound generation, and is an algorithm that is particularly suited to generating voice and music.

[0313] A "prompt" is a set of instructions or questions input to a generative AI model to obtain a desired output result.

[0314] The food delivery system that embodies this invention operates through the cooperation of a server, terminals, and users in order to improve the comfort and work efficiency of delivery staff. The program of this system is configured as follows.

[0315] Hardware and software used

[0316] Hardware

[0317] 1. Smartphone

[0318] microphone

[0319] speaker

[0320] camera

[0321] Network connectivity features

[0322] software

[0323] 1. Smartphone App

[0324] Environmental sound and emotion data capture function using microphone and camera

[0325] Communication with the server

[0326] Masking sound decoding and playback function

[0327] 2. Server

[0328] Receiving and analyzing noise and emotion data

[0329] Generative AI models (e.g., DeepMind WaveNet, OpenAI GPT-4)

[0330] Emotion engine (e.g., Microsoft® Azure® Emotion API)

[0331] Program processing details

[0332] The server receives and analyzes the sound data and emotional data sent from the device. The analysis involves checking the volume and frequency spectrum of the sound data, and using algorithms to determine the user's stress level and emotional state for the emotional data. Based on the analysis results, the server uses a generative AI model to generate optimal masking sounds. These masking sounds help reduce the user's mental stress and provide a comfortable environment.

[0333] The masking sound is transmitted from the server to the device and played in real time through the smartphone speaker, effectively canceling out surrounding noise for delivery staff, allowing them to continue working in a relaxed state.

[0334] Examples of concrete examples and prompts

[0335] Specific examples

[0336] When a delivery person launches the smartphone app, the smartphone's microphone begins capturing surrounding traffic and construction sounds. At the same time, a camera analyzes the delivery person's facial expressions to detect temporary highs and lows of stress. The server receives this data and instructs a generative AI model to generate soothing music with a subdued bass rhythm. This sound is then transmitted to the smartphone and played through the speaker, effectively canceling out the noise around the delivery person.

[0337] Prompt Sentence Examples

[0338] The following prompt sentence is input into a generative AI model (e.g., OpenAI GPT-4) to generate the optimal masking sound based on noise analysis and emotion data.

[0339] Consider the surrounding environmental noise (60dB, low frequency) and emotional state (tension, high temporary stress level), and generate music that effectively masks these noises and promotes relaxation. For example, soothing music with a low bass rhythm or a blend of nature sounds would be suitable.

[0340] Output sound requirements

[0341] Frequency range: 20Hz - 20kHz

[0342] Volume: 45dB or less

[0343] Sound quality: Clear, relaxing

[0344] In this way, the present invention aims to reduce the noise around delivery staff in real time and relieve their mental stress.

[0345] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0346] Step 1:

[0347] When a user activates the system on their smartphone, the device's microphone starts capturing ambient sounds, while the camera and emotion sensor simultaneously capture the user's facial expressions, voice tone, and other emotional data.

[0348] Input: ambient sound data, user emotion data

[0349] Output: Sound data and emotion data collected on the device

[0350] Step 2:

[0351] The device transmits the collected sound data and emotion data to the server via the network, converts the data into an appropriate format, and connects to the server using a communication module.

[0352] Input: Sound data and emotion data collected on the device

[0353] Output: Sound data and emotion data sent to the server

[0354] Step 3:

[0355] The server analyzes the received sound data and emotional data. The sound data is analyzed to check the volume and frequency spectrum, and the emotional data is processed through algorithms to determine the user's stress level and emotional state.

[0356] Input: Sound data and emotion data sent to the server

[0357] Output: Analysis results (sound data characteristics, emotional data status)

[0358] Step 4:

[0359] The server inputs a prompt sentence into the generative AI model based on the analysis results to generate the optimal masking sound. The generative AI model follows the process to create the optimal masking sound that can reduce the user's mental stress.

[0360] Input: Analysis results (sound data characteristics, emotional data state), prompt

[0361] Output: Generated masking sound

[0362] Step 5:

[0363] The server transmits the generated masking sound to the terminal via the network, and the server appropriately formats the transmission data and transmits it to the terminal using a communication protocol.

[0364] Input: Generated masking sound

[0365] Output: Masking sound sent to the device

[0366] Step 6:

[0367] The device decodes the masking sound received from the server and plays it in real time through the speaker, effectively masking surrounding noise and allowing the user to work in a more comfortable environment.

[0368] Input: Masking sound sent to the device

[0369] Output: Masking sound played through the device speaker

[0370] Examples of concrete examples and prompts

[0371] Specific examples

[0372] When a delivery person launches the smartphone app, the smartphone's microphone begins capturing surrounding traffic and construction sounds. At the same time, a camera analyzes the delivery person's facial expressions to detect temporary highs and lows of stress. The server receives this data and instructs a generative AI model to generate soothing music with a subdued bass rhythm. This sound is then transmitted to the smartphone and played through the speaker, effectively canceling out the noise around the delivery person.

[0373] Prompt Sentence Examples

[0374] The following prompt sentence is input into a generative AI model (e.g., OpenAI GPT-4) to generate the optimal masking sound based on noise analysis and emotion data.

[0375] Consider the surrounding environmental noise (60dB, low frequency) and emotional state (tension, high temporary stress level), and generate music that effectively masks these noises and promotes relaxation. For example, soothing music with a low bass rhythm or a blend of nature sounds would be suitable.

[0376] Output sound requirements

[0377] Frequency range: 20Hz - 20kHz

[0378] Volume: 45dB or less

[0379] Sound quality: Clear, relaxing

[0380] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0381] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0382] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0383] [Second embodiment]

[0384] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0385] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0386] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0387] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0388] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0389] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0390] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0391] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0392] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0393] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0394] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0395] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0396] The system for implementing this invention provides a comfortable space by inputting noise and playing the optimal masking sound created by the AI ​​in real time. This system operates in cooperation with the server, terminals, and users.

[0397] System configuration

[0398] 1. Terminal: The terminal has a built-in microphone and speaker, which allows it to capture ambient sounds and play back the generated masking sound. The terminal also has a network connection function for communicating with the server.

[0399] 2. Server: The server has a means of receiving and analyzing noise data sent from the device. Furthermore, the server has a generative AI model that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time.

[0400] 3. User: The user can start the system and use the terminal to maintain a comfortable environment. The user can enjoy the automatic noise reduction of the system without any special operation.

[0401] Explaining the program's processing

[0402] The program of this system is executed in the following procedure, with the terminal, server, and user each taking on a different role.

[0403] 1. Booting the system

[0404] When a user activates the system on their device, the device's microphone begins capturing ambient sound. The captured sound data is converted into volume and frequency spectrum data and sent to the server.

[0405] 2. Analysis of noise data

[0406] The server receives noise data sent from the device and analyzes it using a proprietary algorithm. Once the characteristics of the noise have been clarified, the server generates the optimal masking sound based on the analysis results.

[0407] 3. Masking sound generation

[0408] The server uses a generative AI model to generate the optimal masking sound, which can be white noise or soft background music based on the analyzed noise characteristics.

[0409] 4. Transmitting and playing masking sounds

[0410] The generated masking sound is sent from the server to the device, which then decodes it and plays it back through the speaker in real time. This process effectively cancels out the noise around the user, providing a comfortable environment.

[0411] Specific examples

[0412] Example 1: Use in the office

[0413] A user in a meeting in a shared office space starts the system. The device's microphone captures ambient noise and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker, preventing the content of the conversation from leaking to the outside world. The user also enjoys a quiet environment that makes it easier to concentrate.

[0414] Example 2: Use at a cafe

[0415] A user working remotely in a cafe starts the system. Noise captured by the device's microphone is sent to the server, which analyzes it and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the noise in the cafe, allowing the user to continue working in a quiet environment.

[0416] Through these processes, the system can reduce noise in real time and provide a comfortable space for users.

[0417] The processing flow will be explained below.

[0418] Step 1:

[0419] The user starts the system on the device. When the user launches an application and presses the system start button, the device's microphone starts up and is ready to capture ambient sounds. The device sends a connection request to the server to notify that the system has started.

[0420] Step 2:

[0421] The device's microphone captures audio data in real time, and the captured audio is converted into digital data, which includes information such as volume (dB) and frequency spectrum.

[0422] Step 3:

[0423] The device then collects the captured audio data and sends it to the server, where it receives a confirmation response confirming that the transmission was successful.

[0424] Step 4:

[0425] The server analyzes the received noise data, including parameters such as volume, frequency distribution, and duration, and determines which type of masking sound is most suitable based on the analysis results.

[0426] Step 5:

[0427] The server uses a generative AI model to generate the optimal masking sound, which can be white noise or soft background music based on the analyzed noise characteristics.

[0428] Step 6:

[0429] The server assembles the generated masking sounds into packets and prepares to send them to the terminal. The server sends the generated masking sounds to the terminal and receives a confirmation response from the terminal.

[0430] Step 7:

[0431] The device decodes the received masking sound data and prepares it for playback. The masking sound is played in real time through the speaker to reduce ambient noise.

[0432] Step 8:

[0433] The generated masking sound allows the user to continue working in a comfortable, noise-reduced environment. When the user presses the system shutdown button, all processes are stopped and the device's microphone and speaker are turned off.

[0434] Example 1

[0435] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0436] In modern society, noise has become a major problem in daily life and work environments. Noise can reduce concentration and cause stress and fatigue. Conventional technologies have not provided an efficient method for reducing noise. In particular, there has been no system that can detect noise in real time and generate and play appropriate masking sounds based on that detection. Therefore, there is a need for a system that can effectively cancel noise and provide a comfortable space for users.

[0437] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0438] In this invention, the server includes means for capturing ambient sounds, means for converting the captured sound data into volume and frequency spectrum, means for transmitting the converted sound data to the server, means for the server to analyze the sound data and extract noise characteristics, means for generating an optimal masking sound using a generative AI model, means for transmitting the generated masking sound to the terminal, means for the terminal to decode the masking sound and play it through a speaker, and means for playing the generated masking sound. This makes it possible to detect noise in real time and generate and play an adaptive masking sound.

[0439] A "means for capturing ambient sound" is a method of converting sound waves in the environment into electrical signals using a sensor device (e.g., a microphone).

[0440] "Means for converting captured sound data into volume and frequency spectrum" refers to algorithms or software that analyze audio signals and express their sound pressure (volume) and frequency components as digital data.

[0441] The "means for transmitting the converted sound data to the server" is a process for communicating the sound data from the terminal to the server using the network connection function.

[0442] "Means for the server to analyze sound data and extract noise characteristics" refers to technology that receives audio data and analyzes specific frequency bands and volume levels to identify noise patterns and characteristics.

[0443] "Means for generating optimal masking sounds using generative AI models" refers to a process that utilizes machine learning algorithms to design and generate optimal masking sounds based on the characteristics of noise.

[0444] The "means for transmitting the generated masking sound to the terminal" is a method for communicating the generated audio data from the server to the terminal.

[0445] The "means for the terminal to decode the masking sound and play it through the speaker" refers to the process of converting the audio data received by the terminal into an audio signal and playing it through the speaker device.

[0446] The "means for reproducing the generated masking sound" refers to a technique for converting the generated audio data into physical sounds and reproducing them in the environment.

[0447] A "prompt" is a text-based instruction that allows a generative AI model to input specific conditions or data.

[0448] The system for implementing this invention provides a comfortable space by inputting noise and playing the optimal masking sound created by the AI ​​in real time. This system operates in cooperation with the server, terminals, and users.

[0449] System configuration

[0450] 1. Terminal

[0451] The device has a built-in microphone and speaker, and has a means for capturing ambient sounds and a means for playing back the generated masking sounds. The device also has a network connection function for communicating with a server, and can transmit the captured sound data to the server.

[0452] For example, the microphone captures ambient sounds in real time and converts them into volume and frequency spectrum using built-in audio processing libraries such as Librosa.

[0453] 2. Server

[0454] The server has a generative AI model that analyzes the received sound data and generates the optimal masking sound. The server uses Python-based scripts and machine learning libraries such as TensorFlow and PyTorch to analyze the sound data. After analysis, the generated masking sound is sent to the device. When the server uses a generative AI model (for example, OpenAI's GPT-3 or GPT-4) to generate the optimal masking sound, it uses the following prompt statement:

[0455] Prompt statement:

[0456] "The frequency spectrum of the current ambient sound is as follows:\n[frequency data]. Please generate the optimal masking sound to cancel out this ambient sound."

[0457] 3. Users

[0458] Users can activate the system and use their devices to maintain a comfortable environment, and the system automatically reduces noise without any special operations.

[0459] Specific examples

[0460] Example 1: Use in the office

[0461] A user in a meeting in a shared office space starts the system. The device's microphone captures ambient noise and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker, preventing the content of the conversation from being leaked to the outside world and providing the user with a quiet environment where they can concentrate easily.

[0462] Example 2: Use at a cafe

[0463] A user working remotely in a cafe starts the system. Noise captured by the device's microphone is sent to the server, which analyzes it and generates the optimal masking sound. The generated masking sound is played in real time through the device's speaker, effectively canceling out the cafe's noise, allowing the user to continue working in a quiet environment.

[0464] In this way, the system of the present invention can reduce noise in real time and provide a comfortable space for the user.

[0465] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0466] Step 1:

[0467] The user starts an application on the terminal. The user presses the "Start" button, and the system enters the startup state. The input is the user's operation. The output is the system initialization completion state.

[0468] Step 2:

[0469] The device's microphone captures the surrounding sound in real time. The input is the surrounding sound, and the output is the captured audio data. Specifically, the device's microphone converts analog audio into a digital signal.

[0470] Step 3:

[0471] The device converts captured audio data into volume and frequency spectrum. The input is a digital audio signal, and the output is data containing volume and frequency spectrum. The device uses the Librosa library to do this. For example, Librosa can be used to convert audio data into a spectrogram using techniques such as fft and stft.

[0472] Step 4:

[0473] The device sends the converted sound data to the server. The input is data including volume and frequency spectrum, and the output is an HTTP POST request to the server. The device transfers this data to the server using its network connection function.

[0474] Step 5:

[0475] The server receives the transmitted sound data and begins analysis. The input is volume and frequency spectrum data, and the output is data extracting the noise characteristics. The server performs analysis using algorithms implemented in Python, as well as TensorFlow and PyTorch.

[0476] Step 6:

[0477] The server uses a generative AI model based on the characteristics of the noise to generate the optimal masking sound. The input is the analyzed noise characteristic data, and the output is the generated masking sound. Models such as GPT-3 and GPT-4 are used as generative AI models. The specific generation procedure uses the following prompt sentence:

[0478] Prompt statement:

[0479] "The frequency spectrum of the current ambient sound is as follows:\n[frequency data]. Please generate the optimal masking sound to cancel out this ambient sound."

[0480] Step 7:

[0481] The server sends the generated masking sound to the device. The input is the generated masking sound, and the output is the audio data as an HTTP POST request to the device.

[0482] Step 8:

[0483] The device decodes the received masking sound and plays it through the speaker. The input is the audio data sent from the server, and the output is the actual masking sound. The device uses FFmpeg or Librosa to decode the audio data into a playable format and play it through the speaker.

[0484] (Application example 1)

[0485] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0486] In modern society, activities in virtual environments have become commonplace, but noise and other disturbances can significantly disrupt the user experience. Conventional sound environment control systems primarily control sound in physical spaces and are not suitable for noise reduction in virtual stores and other virtual environments. Therefore, an efficient system for providing a comfortable sound environment in virtual environments is needed.

[0487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0488] In this invention, the server includes means for capturing ambient sounds, means for analyzing the captured sound data, means for generating an optimal masking sound based on the analysis results, means for playing back the generated masking sound, and means for analyzing noise in the virtual environment and providing an optimal sound environment for a display device worn by the user. This makes it possible to effectively reduce noise and other disturbances in the virtual environment and provide a comfortable sound environment for the user.

[0489] The "means for capturing ambient sound" is a function for acquiring ambient environmental sound using a device such as a microphone.

[0490] "Means for analyzing captured sound data" refers to algorithms or software that analyze the characteristics of the acquired sound data.

[0491] The "means for generating the optimum masking sound based on the analysis results" is a function that performs a process to create the optimum masking sound for the user according to the characteristics of the analyzed sound.

[0492] The "means for reproducing the generated masking sound" is a function for allowing the user to hear the generated masking sound using an audio device such as a speaker or earphones.

[0493] The "means for transmitting captured sound data to a server" is a function for transmitting the acquired sound data to a remote server via the Internet or a network.

[0494] "The means by which the server analyzes the sound data and generates the optimal masking sound" refers to the process of analyzing the transmitted sound data using an analysis algorithm or generation AI model on the server and generating the optimal masking sound.

[0495] The "means for transmitting the generated masking sound to the terminal" is a function for transmitting the masking sound generated by the server to the user's terminal via the network.

[0496] "Means for connecting to a display device worn by the user to provide an optimal sound environment" refers to a function that communicates with devices such as smart glasses and head-mounted displays (HMDs) to provide an optimal sound environment.

[0497] "Means for responding to dynamic changes in noise in a virtual environment" refers to a function that responds in real time to changes in noise and the sound environment within a virtual space and generates appropriate masking sounds.

[0498] In this embodiment, a comfortable sound environment is provided by inputting noise and playing optimal masking sounds generated by a generative AI model in real time. This system operates in cooperation with a server, terminals, and users.

[0499] System configuration

[0500] 1. Device:

[0501] The device has a built-in microphone and speaker, and has a means for capturing ambient sounds and a means for playing back the generated masking sounds. The device also has a network connection function for communicating with a server. In particular, it includes a means for connecting to devices such as smart glasses and head-mounted displays (HMDs) in a virtual environment to provide an optimal sound environment.

[0502] 2. Server:

[0503] The server has a means of receiving and analyzing noise data sent from the device. Furthermore, the server has a generative AI model that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time. The Python Librosa library is used to analyze the noise data, and TensorFlow and PyTorch are used for the generative AI model.

[0504] 3. User:

[0505] Users can activate the system and use their devices to maintain a comfortable environment, and the system automatically reduces noise without any special operations.

[0506] Program processing

[0507] The programs in this system are executed by dividing the roles of the terminal, server, and user.

[0508] When a user puts on smart glasses or an HMD and starts the system, the device's microphone begins capturing ambient sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server.

[0509] The server receives noise data sent from the device and analyzes it using a proprietary algorithm. Once the characteristics of the noise have been clarified, the server generates the optimal masking sound based on the analysis results. During this generation process, white noise, soft background music, etc. are created based on the analyzed noise characteristics.

[0510] The generated masking sound is sent from the server to the device and played in real time through the device's speaker, effectively canceling out the user's surrounding noise and providing a comfortable sound environment.

[0511] Specific examples

[0512] Suppose a user is distracted by other users' chats and background music in a virtual store. The user puts on smart glasses and launches an application. The device's microphone captures the noise in the virtual store and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time from the smart glasses' speakers, allowing the user to enjoy shopping in a comfortable acoustic environment.

[0513] Prompt Sentence Examples

[0514] The generative AI model is given a prompt like this:

[0515] Noise data: {sound data}

[0516] Output conditions: Soft background music or white noise

[0517] Response format: data

[0518] With this configuration and means, the system can effectively reduce noise and disturbances in the virtual environment, providing the user with a comfortable sound environment.

[0519] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0520] Step 1:

[0521] A user puts on smart glasses or a head-mounted display (HMD) and starts the system. The user's operation causes the device's microphone to start capturing ambient sounds. At this point, the input is the ambient sound, and the output is the captured sound data.

[0522] Step 2:

[0523] The device converts the captured sound data into volume and frequency spectrum data using a library for audio signal processing (e.g., Python's Librosa library). The input is the captured sound data, and the output is the converted volume and frequency spectrum data.

[0524] Step 3:

[0525] The terminal transmits the converted volume and frequency spectrum data to the server. The data is transmitted over the network. The input is the converted volume and frequency spectrum data, and the output is the data transmitted to the server.

[0526] Step 4:

[0527] The server analyzes the received volume and frequency spectrum data. A proprietary algorithm is used for the analysis, and a generative AI model (e.g., TensorFlow or PyTorch) is used for this process. The input is the volume and frequency spectrum data sent to the server, and the output is the analysis result.

[0528] Step 5:

[0529] The server generates the optimal masking sound based on the analysis results. During this generation process, white noise or soft background music is selected depending on the noise characteristics. An example prompt sentence is input to the generation AI model. The input is the analysis results and the prompt sentence, and the output is the generated masking sound data. An example prompt sentence is as follows:

[0530] Noise data: {sound data}

[0531] Output conditions: Soft background music or white noise

[0532] Response format: data

[0533] Step 6:

[0534] The server sends the generated masking sound data to the terminal. The data is sent via the network and arrives at the user's terminal in real time. The input is the generated masking sound data, and the output is the data sent to the terminal.

[0535] Step 7:

[0536] The device decodes the received masking sound data and plays it through its built-in speaker. This process effectively cancels out the user's surrounding noise in real time. The input is the masking sound data received from the server, and the output is the masking sound being played.

[0537] Step 8:

[0538] The user enjoys the comfortable sound environment obtained by the operation of the system. In this step, the user does not need to perform any special operations; the system automatically reduces noise. The input is the reproduced masking sound, and the output is a comfortable sound environment.

[0539] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0540] The system for implementing this invention combines a conventional noise control system with an emotion engine that recognizes the user's emotions, providing a more comfortable space for the user. This system operates in cooperation with a server, terminals, and users.

[0541] System configuration

[0542] 1. Device: The device is equipped with a microphone, speaker, camera, and other emotion sensors to capture ambient sounds and the user's emotions, and to play back the generated masking sounds. The device also has a network connection function to communicate with the server.

[0543] 2. Server: The server has a means of receiving and analyzing noise data and emotion data sent from the device. Furthermore, the server has a generative AI model and emotion engine that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time.

[0544] 3. Emotion Engine: The emotion engine analyzes the user's stress level and emotional state from facial expressions and vocal tone, and helps adjust the masking sound accordingly.

[0545] 4. User: The user can start the system and use the device to maintain a comfortable environment. The user can enjoy the system automatically analyzing and reducing noise and emotional state without any special operation.

[0546] Explaining the program's processing

[0547] The program of this system is executed in the following procedure, with each of the roles of terminal, server, and emotion engine being assigned.

[0548] 1. Booting the system

[0549] When a user activates the system on their device, the device's microphone begins capturing ambient sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server. At the same time, the device uses a camera and emotion sensors to collect the user's emotion data, which is also sent to the server.

[0550] 2. Data Analysis

[0551] The server receives noise and emotion data sent from the device and analyzes the data using a proprietary algorithm. Once the characteristics of the noise and the user's emotional state are clarified, the server generates the optimal masking sound based on the analysis results.

[0552] 3. Masking sound generation

[0553] The server uses a generative AI model to generate optimal masking sounds, such as white noise or soft background music, based on the analyzed noise characteristics and the user's emotional state as determined by the emotion engine.

[0554] 4. Transmitting and playing masking sounds

[0555] The generated masking sound is sent from the server to the device, which then decodes it and plays it back through the speaker in real time, reducing ambient noise and providing an optimal environment for the user's emotional state.

[0556] Specific examples

[0557] Example 1: Use in the office

[0558] A user in a meeting in a shared office space activates the system. The device's microphone captures ambient noise, while the camera and emotion sensor recognize the user's emotional state. The server analyzes the captured data and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker to prevent the content of the conversation from leaking to the outside. The system also adjusts according to the user's emotional state to reduce stress levels.

[0559] Example 2: Use at a cafe

[0560] A user working remotely in a cafe starts the system. Noise captured by the device's microphone and emotional state analyzed by the emotion engine are sent to the server, which then analyzes the data and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the cafe's noise and providing an environment suited to the user's emotional state.

[0561] Through these processes, the system can reduce noise in real time and provide an optimal space according to the user's emotional state.

[0562] The processing flow will be explained below.

[0563] Step 1:

[0564] The user starts the system on the device. When the user launches the application and presses the system's start button, the device's microphone, camera, and emotion sensor are activated and ready to capture ambient sounds and the user's emotional state. The device sends a connection request to the server to notify that the system has started.

[0565] Step 2:

[0566] The device's microphone captures voice data in real time, which is then converted into digital signals containing data such as volume (dB) and frequency spectrum. At the same time, the device's camera and emotion sensor analyze the user's facial expressions and voice tone to collect emotional data.

[0567] Step 3:

[0568] The device then collects the captured sound data and emotion data into packets and sends them to the server. The device receives a confirmation response from the server to confirm that the data transmission was successful.

[0569] Step 4:

[0570] The server analyzes the received noise data and emotion data, including parameters such as volume, frequency distribution, and duration. Based on the analysis results, the server determines which type of masking sound is optimal.

[0571] Step 5:

[0572] The server uses a generative AI model to generate optimal masking sounds, such as white noise or soft background music, based on the analyzed noise characteristics and the user's emotional state as determined by the emotion engine.

[0573] Step 6:

[0574] The generated masking sound is sent from the server to the terminal. The server sends packetized data and receives an acknowledgment from the terminal. The terminal decodes the received masking sound data and prepares for playback.

[0575] Step 7:

[0576] The device plays masking sounds through the speaker in real time, reducing ambient noise. The emotion engine also monitors the user's emotional state in real time and adjusts the masking sounds as needed.

[0577] Step 8:

[0578] Users can work and talk in a comfortable environment thanks to the generated masking sounds and optimization by the emotion engine. If the user feels stressed, the masking sounds are automatically adjusted to maintain comfort. When the user presses the system's shutdown button, all processes are stopped and the device's microphone, camera, emotion sensor, and speaker are turned off.

[0579] Example 2

[0580] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0581] In modern urban environments, noise levels are constantly increasing, causing stress for many people. Existing noise control systems focus on capturing sounds and generating masking sounds, but they do not take into account the user's emotional state, making it difficult to provide an optimal sound environment.

[0582] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing ambient sounds, means for capturing user emotional data, means for transmitting the captured sound data and emotional data to the server, means for the server to analyze the sound data and emotional data, means for generating an optimal masking sound using a generative AI model based on the analysis results, means for transmitting the generated masking sound to the terminal, and means for playing the generated masking sound. This makes it possible to provide an optimal sound environment according to the user's emotional state while effectively reducing noise.

[0583] The "means for capturing ambient sound" refers to a device or system for collecting sounds in the environment, and specifically refers to a microphone or the like.

[0584] "Means for capturing user emotional data" refers to a device or system for collecting the user's emotional state, and specifically includes a camera or a voice analysis sensor.

[0585] "Means for transmitting captured sound data and emotional data to a server" refers to a network connection device or system for communicating the collected data to a server, specifically Wi-Fi, Bluetooth, the Internet, etc.

[0586] "Means for the server to analyze sound data and emotional data" refers to software or algorithms for processing and analyzing sound data and emotional data received by the server.

[0587] "Means for generating optimal masking sound using an AI model based on the analysis results" refers to software that includes an artificial intelligence model or algorithm for generating optimal masking sound based on the analyzed data.

[0588] The "means for transmitting the generated masking sound to the terminal" refers to a network connection device or system for communicating the generated masking sound to the terminal, and specifically includes the Internet, Wi-Fi, Bluetooth, etc.

[0589] The "means for reproducing the generated masking sound" refers to a device or system for reproducing the received masking sound, and specifically refers to a speaker, a headphone, or the like.

[0590] MODE FOR CARRYING OUT THE INVENTION

[0591] The system of the present invention provides optimal masking sound based on the ambient noise and the user's emotional state by linking a server, a terminal, and an emotion engine. The system uses the following hardware and software:

[0592] Hardware and software used

[0593] 1. Device:

[0594] Microphone: A device for capturing ambient sounds.

[0595] Camera: A device for capturing a user's facial expressions.

[0596] Emotion sensors: Sensors for capturing the user's tone of voice and other vital signs.

[0597] Network connectivity: A means of communicating with the server (e.g., Wi-Fi, Bluetooth).

[0598] Speaker: A device for playing back the masking sound sent from the server.

[0599] 2. Server:

[0600] Data analysis module: An algorithm that analyzes noise data and emotion data sent from the device.

[0601] Generative AI model: A program that generates optimal masking sounds based on analysis results (e.g., OpenAI's GPT-3 and DALL-E).

[0602] Network connection function: A means for communicating with terminals.

[0603] 3. Emotion Engine:

[0604] Emotion analysis algorithm: Software for analyzing the emotional state of a user.

[0605] Operational flow and concrete examples

[0606] When the system is started, the device's microphone first captures the surrounding sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server. At the same time, the device's camera and emotion sensor collect the user's emotion data and send it to the server.

[0607] The server receives the noise and emotion data sent from the device and analyzes each data. The analysis uses a data analysis module and emotion analysis algorithm. Based on the analysis results, the server uses a generative AI model to generate optimal masking sounds. For example, if the noise is intense and the user is feeling stressed, it generates relaxing white noise or soft background music.

[0608] The generated masking sound is sent from the server to the device, which then receives it and plays it in real time through the speaker, effectively reducing ambient noise and providing the optimal sound environment according to the user's emotional state.

[0609] Specific examples

[0610] Example 1: Use in the office

[0611] A user in a meeting in a shared office space activates the system. The device's microphone captures ambient noise, while the camera and emotion sensor recognize the user's emotional state. The server analyzes the captured data and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker to prevent the content of the conversation from leaking to the outside. The system also adjusts according to the user's emotional state to reduce stress levels.

[0612] Example prompt:

[0613] "Generate masking sounds suitable for situations where people are having a meeting in a shared office space. Users are stressed and want a relaxing environment."

[0614] Example 2: Use at a cafe

[0615] A user working remotely in a cafe starts the system. Noise captured by the device's microphone and emotional state analyzed by the emotion engine are sent to the server, which then analyzes the data and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the cafe's noise and providing an environment suited to the user's emotional state.

[0616] Example prompt:

[0617] "For users working remotely in a cafe, please generate masking sounds that cancel out surrounding noise and make it easier for them to concentrate."

[0618] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0619] Step 1:

[0620] The user starts the system on the device. The device first initializes and enables the built-in microphone, camera, and emotion sensor, ensuring that the hardware is operational and ready to begin capturing data.

[0621] Specific behavior:

[0622] The user clicks the "System Start" button.

[0623] Input: User action (clicking the launch button)

[0624] Output: Device hardware initialization complete

[0625] Step 2:

[0626] The device's microphone begins capturing ambient sound, which is then converted into volume and frequency spectrum data, which is then sent to the server.

[0627] Specific behavior:

[0628] The device's microphone records the surrounding sounds and samples the waveform data in real time, converts the data into a digital format (e.g., PCM format), and sends it to the server.

[0629] Input: Ambient sound

[0630] Output: Volume and frequency spectrum converted data

[0631] Step 3:

[0632] The device's camera and emotion sensors capture the user's facial expressions and voice tone to collect emotion data, which is also sent to the server.

[0633] Specific behavior:

[0634] The device's camera captures the user's face and analyzes the movements of each part of the face (eyes, mouth, eyebrows, etc.). The audio sensor also analyzes the user's tone of voice and speaking style. This data is combined to determine the user's emotional state and is sent to the server.

[0635] Input: User facial expressions and voice tone

[0636] Output: Emotion data

[0637] Step 4:

[0638] The server receives the noise and emotion data sent from the device. The received data is processed by the analysis module. The server uses a proprietary algorithm to analyze the characteristics of the noise and the user's emotional state.

[0639] Specific behavior:

[0640] The server analyzes the characteristics of the received noise data and identifies the sound volume, frequency components, source, etc. At the same time, it detects the user's current emotional state (stress level, relaxation level) based on the received emotional data.

[0641] Input: Noise data and emotion data

[0642] Output: Noise analysis results and emotion analysis results

[0643] Step 5:

[0644] The server uses a generative AI model to generate optimal masking sounds based on the analysis results. The generated masking sounds are tailored to the user's emotional state, such as white noise or relaxing music.

[0645] Specific behavior:

[0646] Based on the analysis results, the server inputs prompts to the generative AI model, which then designs appropriate masking sounds and returns the sound data.

[0647] Input: Noise analysis results and emotion analysis results

[0648] Output: Generated masking sound

[0649] Example prompt:

[0650] "For users working remotely in a cafe, please generate masking sounds that cancel out surrounding noise and make it easier for them to concentrate."

[0651] Step 6:

[0652] The generated masking sound is sent from the server to the device, which decodes the received sound data and plays it back in real time through the speaker.

[0653] Specific behavior:

[0654] The server sends the generated masking sound to the device, which then decodes the received sound data and plays it back using the speaker, allowing the user to work in a comfortable environment.

[0655] Input: Generated masking sound

[0656] Output: Played masking sound

[0657] (Application example 2)

[0658] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0659] In the food delivery industry, delivery staff often experience mental stress due to traffic noise and other disturbances at delivery destinations. This reduces the efficiency of delivery work and staff comfort. The present invention aims to provide an environment in which delivery staff can perform their work comfortably and efficiently.

[0660] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing ambient sounds, means for analyzing the captured sound data and the user's emotional state, and means for generating an optimal masking sound based on the analysis results and the user's emotional state. This effectively masks the noise around the delivery staff, enabling them to perform their work while reducing mental stress.

[0661] The "means for capturing ambient sound" is a device including a microphone and a sensor for collecting environmental sounds around the delivery staff.

[0662] "Means for analyzing captured sound data and the user's emotional state" refers to algorithms or engines that analyze the collected sound data and the emotional state obtained from the delivery staff's facial expressions, vocal tone, etc.

[0663] The "means for generating optimal masking sounds based on the analysis results and the user's emotional state" refers to a generative AI model or sound engine that generates optimal sounds to reduce the mental stress of delivery staff based on noise data and emotional data.

[0664] The "means for reproducing the generated masking sound" refers to a speaker or audio reproduction device for reproducing the generated masking sound in real time.

[0665] The "means for transmitting captured sound data and emotional data to a server" refers to a communication module or protocol for transmitting the collected sound data and emotional data to a server via a network.

[0666] "The means by which the server analyzes sound data and emotional data and generates optimal masking sounds" refers to a function that uses an analysis algorithm or generation AI model running on the server to analyze collected data and generate masking sounds.

[0667] The "means for transmitting the generated masking sound to the terminal" is a communication function for transmitting the masking sound generated by the server to the terminal of the delivery staff via the network.

[0668] A "generative AI model" is a type of artificial intelligence model used for data analysis and sound generation, and is an algorithm that is particularly suited to generating voice and music.

[0669] A "prompt" is a set of instructions or questions input to a generative AI model to obtain a desired output result.

[0670] The food delivery system that embodies this invention operates through the cooperation of a server, terminals, and users in order to improve the comfort and work efficiency of delivery staff. The program of this system is configured as follows.

[0671] Hardware and software used

[0672] Hardware

[0673] 1. Smartphone

[0674] microphone

[0675] speaker

[0676] camera

[0677] Network connectivity features

[0678] software

[0679] 1. Smartphone App

[0680] Environmental sound and emotion data capture function using microphone and camera

[0681] Communication with the server

[0682] Masking sound decoding and playback function

[0683] 2. Server

[0684] Receiving and analyzing noise and emotion data

[0685] Generative AI models (e.g., DeepMind WaveNet, OpenAI GPT-4)

[0686] Emotion engine (e.g. Microsoft Azure Emotion API)

[0687] Program processing details

[0688] The server receives and analyzes the sound data and emotional data sent from the device. The analysis involves checking the volume and frequency spectrum of the sound data, and using algorithms to determine the user's stress level and emotional state for the emotional data. Based on the analysis results, the server uses a generative AI model to generate optimal masking sounds. These masking sounds help reduce the user's mental stress and provide a comfortable environment.

[0689] The masking sound is transmitted from the server to the device and played in real time through the smartphone speaker, effectively canceling out surrounding noise for delivery staff, allowing them to continue working in a relaxed state.

[0690] Examples of concrete examples and prompts

[0691] Specific examples

[0692] When a delivery person launches the smartphone app, the smartphone's microphone begins capturing surrounding traffic and construction sounds. At the same time, a camera analyzes the delivery person's facial expressions to detect temporary highs and lows of stress. The server receives this data and instructs a generative AI model to generate soothing music with a subdued bass rhythm. This sound is then transmitted to the smartphone and played through the speaker, effectively canceling out the noise around the delivery person.

[0693] Prompt Sentence Examples

[0694] The following prompt sentence is input into a generative AI model (e.g., OpenAI GPT-4) to generate the optimal masking sound based on noise analysis and emotion data.

[0695] Consider the surrounding environmental noise (60dB, low frequency) and emotional state (tension, high temporary stress level), and generate music that effectively masks these noises and promotes relaxation. For example, soothing music with a low bass rhythm or a blend of nature sounds would be suitable.

[0696] Output sound requirements

[0697] Frequency range: 20Hz - 20kHz

[0698] Volume: 45dB or less

[0699] Sound quality: Clear, relaxing

[0700] In this way, the present invention aims to reduce the noise around delivery staff in real time and relieve their mental stress.

[0701] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0702] Step 1:

[0703] When a user activates the system on their smartphone, the device's microphone starts capturing ambient sounds, while the camera and emotion sensor simultaneously capture the user's facial expressions, voice tone, and other emotional data.

[0704] Input: ambient sound data, user emotion data

[0705] Output: Sound data and emotion data collected on the device

[0706] Step 2:

[0707] The device transmits the collected sound data and emotion data to the server via the network, converts the data into an appropriate format, and connects to the server using a communication module.

[0708] Input: Sound data and emotion data collected on the device

[0709] Output: Sound data and emotion data sent to the server

[0710] Step 3:

[0711] The server analyzes the received sound data and emotional data. The sound data is analyzed to check the volume and frequency spectrum, and the emotional data is processed through algorithms to determine the user's stress level and emotional state.

[0712] Input: Sound data and emotion data sent to the server

[0713] Output: Analysis results (sound data characteristics, emotional data status)

[0714] Step 4:

[0715] The server inputs a prompt sentence into the generative AI model based on the analysis results to generate the optimal masking sound. The generative AI model follows the process to create the optimal masking sound that can reduce the user's mental stress.

[0716] Input: Analysis results (sound data characteristics, emotional data state), prompt

[0717] Output: Generated masking sound

[0718] Step 5:

[0719] The server transmits the generated masking sound to the terminal via the network, and the server appropriately formats the transmission data and transmits it to the terminal using a communication protocol.

[0720] Input: Generated masking sound

[0721] Output: Masking sound sent to the device

[0722] Step 6:

[0723] The device decodes the masking sound received from the server and plays it in real time through the speaker, effectively masking surrounding noise and allowing the user to work in a more comfortable environment.

[0724] Input: Masking sound sent to the device

[0725] Output: Masking sound played through the device speaker

[0726] Examples of concrete examples and prompts

[0727] Specific examples

[0728] When a delivery person launches the smartphone app, the smartphone's microphone begins capturing surrounding traffic and construction sounds. At the same time, a camera analyzes the delivery person's facial expressions to detect temporary highs and lows of stress. The server receives this data and instructs a generative AI model to generate soothing music with a subdued bass rhythm. This sound is then transmitted to the smartphone and played through the speaker, effectively canceling out the noise around the delivery person.

[0729] Prompt Sentence Examples

[0730] The following prompt sentence is input into a generative AI model (e.g., OpenAI GPT-4) to generate the optimal masking sound based on noise analysis and emotion data.

[0731] Consider the surrounding environmental noise (60dB, low frequency) and emotional state (tension, high temporary stress level), and generate music that effectively masks these noises and promotes relaxation. For example, soothing music with a low bass rhythm or a blend of nature sounds would be suitable.

[0732] Output sound requirements

[0733] Frequency range: 20Hz - 20kHz

[0734] Volume: 45dB or less

[0735] Sound quality: Clear, relaxing

[0736] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0737] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0738] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0739] [Third embodiment]

[0740] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0741] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0742] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0743] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0744] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0745] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0746] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0747] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0748] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0749] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0750] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0751] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0752] The system for implementing this invention provides a comfortable space by inputting noise and playing the optimal masking sound created by the AI ​​in real time. This system operates in cooperation with the server, terminals, and users.

[0753] System configuration

[0754] 1. Terminal: The terminal has a built-in microphone and speaker, which allows it to capture ambient sounds and play back the generated masking sound. The terminal also has a network connection function for communicating with the server.

[0755] 2. Server: The server has a means of receiving and analyzing noise data sent from the device. Furthermore, the server has a generative AI model that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time.

[0756] 3. User: The user can start the system and use the terminal to maintain a comfortable environment. The user can enjoy the automatic noise reduction of the system without any special operation.

[0757] Explaining the program's processing

[0758] The program of this system is executed in the following procedure, with the terminal, server, and user each taking on a different role.

[0759] 1. Booting the system

[0760] When a user activates the system on their device, the device's microphone begins capturing ambient sound. The captured sound data is converted into volume and frequency spectrum data and sent to the server.

[0761] 2. Analysis of noise data

[0762] The server receives noise data sent from the device and analyzes it using a proprietary algorithm. Once the characteristics of the noise have been clarified, the server generates the optimal masking sound based on the analysis results.

[0763] 3. Masking sound generation

[0764] The server uses a generative AI model to generate the optimal masking sound, which can be white noise or soft background music based on the analyzed noise characteristics.

[0765] 4. Transmitting and playing masking sounds

[0766] The generated masking sound is sent from the server to the device, which then decodes it and plays it back through the speaker in real time. This process effectively cancels out the noise around the user, providing a comfortable environment.

[0767] Specific examples

[0768] Example 1: Use in the office

[0769] A user in a meeting in a shared office space starts the system. The device's microphone captures ambient noise and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker, preventing the content of the conversation from leaking to the outside world. The user also enjoys a quiet environment that makes it easier to concentrate.

[0770] Example 2: Use at a cafe

[0771] A user working remotely in a cafe starts the system. Noise captured by the device's microphone is sent to the server, which analyzes it and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the noise in the cafe, allowing the user to continue working in a quiet environment.

[0772] Through these processes, the system can reduce noise in real time and provide a comfortable space for users.

[0773] The processing flow will be explained below.

[0774] Step 1:

[0775] The user starts the system on the device. When the user launches an application and presses the system start button, the device's microphone starts up and is ready to capture ambient sounds. The device sends a connection request to the server to notify that the system has started.

[0776] Step 2:

[0777] The device's microphone captures audio data in real time, and the captured audio is converted into digital data, which includes information such as volume (dB) and frequency spectrum.

[0778] Step 3:

[0779] The device then collects the captured audio data and sends it to the server, where it receives a confirmation response confirming that the transmission was successful.

[0780] Step 4:

[0781] The server analyzes the received noise data, including parameters such as volume, frequency distribution, and duration, and determines which type of masking sound is most suitable based on the analysis results.

[0782] Step 5:

[0783] The server uses a generative AI model to generate the optimal masking sound, which can be white noise or soft background music based on the analyzed noise characteristics.

[0784] Step 6:

[0785] The server assembles the generated masking sounds into packets and prepares to send them to the terminal. The server sends the generated masking sounds to the terminal and receives a confirmation response from the terminal.

[0786] Step 7:

[0787] The device decodes the received masking sound data and prepares it for playback. The masking sound is played in real time through the speaker to reduce ambient noise.

[0788] Step 8:

[0789] The generated masking sound allows the user to continue working in a comfortable, noise-reduced environment. When the user presses the system shutdown button, all processes are stopped and the device's microphone and speaker are turned off.

[0790] Example 1

[0791] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0792] In modern society, noise has become a major problem in daily life and work environments. Noise can reduce concentration and cause stress and fatigue. Conventional technologies have not provided an efficient method for reducing noise. In particular, there has been no system that can detect noise in real time and generate and play appropriate masking sounds based on that detection. Therefore, there is a need for a system that can effectively cancel noise and provide a comfortable space for users.

[0793] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0794] In this invention, the server includes means for capturing ambient sounds, means for converting the captured sound data into volume and frequency spectrum, means for transmitting the converted sound data to the server, means for the server to analyze the sound data and extract noise characteristics, means for generating an optimal masking sound using a generative AI model, means for transmitting the generated masking sound to the terminal, means for the terminal to decode the masking sound and play it through a speaker, and means for playing the generated masking sound. This makes it possible to detect noise in real time and generate and play an adaptive masking sound.

[0795] A "means for capturing ambient sound" is a method of converting sound waves in the environment into electrical signals using a sensor device (e.g., a microphone).

[0796] "Means for converting captured sound data into volume and frequency spectrum" refers to algorithms or software that analyze audio signals and express their sound pressure (volume) and frequency components as digital data.

[0797] The "means for transmitting the converted sound data to the server" is a process for communicating the sound data from the terminal to the server using the network connection function.

[0798] "Means for the server to analyze sound data and extract noise characteristics" refers to technology that receives audio data and analyzes specific frequency bands and volume levels to identify noise patterns and characteristics.

[0799] "Means for generating optimal masking sounds using generative AI models" refers to a process that utilizes machine learning algorithms to design and generate optimal masking sounds based on the characteristics of noise.

[0800] The "means for transmitting the generated masking sound to the terminal" is a method for communicating the generated audio data from the server to the terminal.

[0801] The "means for the terminal to decode the masking sound and play it through the speaker" refers to the process of converting the audio data received by the terminal into an audio signal and playing it through the speaker device.

[0802] The "means for reproducing the generated masking sound" refers to a technique for converting the generated audio data into physical sounds and reproducing them in the environment.

[0803] A "prompt" is a text-based instruction that allows a generative AI model to input specific conditions or data.

[0804] The system for implementing this invention provides a comfortable space by inputting noise and playing the optimal masking sound created by the AI ​​in real time. This system operates in cooperation with the server, terminals, and users.

[0805] System configuration

[0806] 1. Terminal

[0807] The device has a built-in microphone and speaker, and has a means for capturing ambient sounds and a means for playing back the generated masking sounds. The device also has a network connection function for communicating with a server, and can transmit the captured sound data to the server.

[0808] For example, the microphone captures ambient sounds in real time and converts them into volume and frequency spectrum using built-in audio processing libraries such as Librosa.

[0809] 2. Server

[0810] The server has a generative AI model that analyzes the received sound data and generates the optimal masking sound. The server uses Python-based scripts and machine learning libraries such as TensorFlow and PyTorch to analyze the sound data. After analysis, the generated masking sound is sent to the device. When the server uses a generative AI model (for example, OpenAI's GPT-3 or GPT-4) to generate the optimal masking sound, it uses the following prompt statement:

[0811] Prompt statement:

[0812] "The frequency spectrum of the current ambient sound is as follows:\n[frequency data]. Please generate the optimal masking sound to cancel out this ambient sound."

[0813] 3. Users

[0814] Users can activate the system and use their devices to maintain a comfortable environment, and the system automatically reduces noise without any special operations.

[0815] Specific examples

[0816] Example 1: Use in the office

[0817] A user in a meeting in a shared office space starts the system. The device's microphone captures ambient noise and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker, preventing the content of the conversation from being leaked to the outside world and providing the user with a quiet environment where they can concentrate easily.

[0818] Example 2: Use at a cafe

[0819] A user working remotely in a cafe starts the system. Noise captured by the device's microphone is sent to the server, which analyzes it and generates the optimal masking sound. The generated masking sound is played in real time through the device's speaker, effectively canceling out the cafe's noise, allowing the user to continue working in a quiet environment.

[0820] In this way, the system of the present invention can reduce noise in real time and provide a comfortable space for the user.

[0821] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0822] Step 1:

[0823] The user starts an application on the terminal. The user presses the "Start" button, and the system enters the startup state. The input is the user's operation. The output is the system initialization completion state.

[0824] Step 2:

[0825] The device's microphone captures the surrounding sound in real time. The input is the surrounding sound, and the output is the captured audio data. Specifically, the device's microphone converts analog audio into a digital signal.

[0826] Step 3:

[0827] The device converts captured audio data into volume and frequency spectrum. The input is a digital audio signal, and the output is data containing volume and frequency spectrum. The device uses the Librosa library to do this. For example, Librosa can be used to convert audio data into a spectrogram using techniques such as fft and stft.

[0828] Step 4:

[0829] The device sends the converted sound data to the server. The input is data including volume and frequency spectrum, and the output is an HTTP POST request to the server. The device transfers this data to the server using its network connection function.

[0830] Step 5:

[0831] The server receives the transmitted sound data and begins analysis. The input is volume and frequency spectrum data, and the output is data extracting the noise characteristics. The server performs analysis using algorithms implemented in Python, as well as TensorFlow and PyTorch.

[0832] Step 6:

[0833] The server uses a generative AI model based on the characteristics of the noise to generate the optimal masking sound. The input is the analyzed noise characteristic data, and the output is the generated masking sound. Models such as GPT-3 and GPT-4 are used as generative AI models. The specific generation procedure uses the following prompt sentence:

[0834] Prompt statement:

[0835] "The frequency spectrum of the current ambient sound is as follows:\n[frequency data]. Please generate the optimal masking sound to cancel out this ambient sound."

[0836] Step 7:

[0837] The server sends the generated masking sound to the device. The input is the generated masking sound, and the output is the audio data as an HTTP POST request to the device.

[0838] Step 8:

[0839] The device decodes the received masking sound and plays it through the speaker. The input is the audio data sent from the server, and the output is the actual masking sound. The device uses FFmpeg or Librosa to decode the audio data into a playable format and play it through the speaker.

[0840] (Application example 1)

[0841] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0842] In modern society, activities in virtual environments have become commonplace, but noise and other disturbances can significantly disrupt the user experience. Conventional sound environment control systems primarily control sound in physical spaces and are not suitable for noise reduction in virtual stores and other virtual environments. Therefore, an efficient system for providing a comfortable sound environment in virtual environments is needed.

[0843] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0844] In this invention, the server includes means for capturing ambient sounds, means for analyzing the captured sound data, means for generating an optimal masking sound based on the analysis results, means for playing back the generated masking sound, and means for analyzing noise in the virtual environment and providing an optimal sound environment for a display device worn by the user. This makes it possible to effectively reduce noise and other disturbances in the virtual environment and provide a comfortable sound environment for the user.

[0845] The "means for capturing ambient sound" is a function for acquiring ambient environmental sound using a device such as a microphone.

[0846] "Means for analyzing captured sound data" refers to algorithms or software that analyze the characteristics of the acquired sound data.

[0847] The "means for generating the optimum masking sound based on the analysis results" is a function that performs a process to create the optimum masking sound for the user according to the characteristics of the analyzed sound.

[0848] The "means for reproducing the generated masking sound" is a function for allowing the user to hear the generated masking sound using an audio device such as a speaker or earphones.

[0849] The "means for transmitting captured sound data to a server" is a function for transmitting the acquired sound data to a remote server via the Internet or a network.

[0850] "The means by which the server analyzes the sound data and generates the optimal masking sound" refers to the process of analyzing the transmitted sound data using an analysis algorithm or generation AI model on the server and generating the optimal masking sound.

[0851] The "means for transmitting the generated masking sound to the terminal" is a function for transmitting the masking sound generated by the server to the user's terminal via the network.

[0852] "Means for connecting to a display device worn by the user to provide an optimal sound environment" refers to a function that communicates with devices such as smart glasses and head-mounted displays (HMDs) to provide an optimal sound environment.

[0853] "Means for responding to dynamic changes in noise in a virtual environment" refers to a function that responds in real time to changes in noise and the sound environment within a virtual space and generates appropriate masking sounds.

[0854] In this embodiment, a comfortable sound environment is provided by inputting noise and playing optimal masking sounds generated by a generative AI model in real time. This system operates in cooperation with a server, terminals, and users.

[0855] System configuration

[0856] 1. Device:

[0857] The device has a built-in microphone and speaker, and has a means for capturing ambient sounds and a means for playing back the generated masking sounds. The device also has a network connection function for communicating with a server. In particular, it includes a means for connecting to devices such as smart glasses and head-mounted displays (HMDs) in a virtual environment to provide an optimal sound environment.

[0858] 2. Server:

[0859] The server has a means of receiving and analyzing noise data sent from the device. Furthermore, the server has a generative AI model that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time. The Python Librosa library is used to analyze the noise data, and TensorFlow and PyTorch are used for the generative AI model.

[0860] 3. User:

[0861] Users can activate the system and use their devices to maintain a comfortable environment, and the system automatically reduces noise without any special operations.

[0862] Program processing

[0863] The programs in this system are executed by dividing the roles of the terminal, server, and user.

[0864] When a user puts on smart glasses or an HMD and starts the system, the device's microphone begins capturing ambient sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server.

[0865] The server receives noise data sent from the device and analyzes it using a proprietary algorithm. Once the characteristics of the noise have been clarified, the server generates the optimal masking sound based on the analysis results. During this generation process, white noise, soft background music, etc. are created based on the analyzed noise characteristics.

[0866] The generated masking sound is sent from the server to the device and played in real time through the device's speaker, effectively canceling out the user's surrounding noise and providing a comfortable sound environment.

[0867] Specific examples

[0868] Suppose a user is distracted by other users' chats and background music in a virtual store. The user puts on smart glasses and launches an application. The device's microphone captures the noise in the virtual store and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time from the smart glasses' speakers, allowing the user to enjoy shopping in a comfortable acoustic environment.

[0869] Prompt Sentence Examples

[0870] The generative AI model is given a prompt like this:

[0871] Noise data: {sound data}

[0872] Output conditions: Soft background music or white noise

[0873] Response format: data

[0874] With this configuration and means, the system can effectively reduce noise and disturbances in the virtual environment, providing the user with a comfortable sound environment.

[0875] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0876] Step 1:

[0877] A user puts on smart glasses or a head-mounted display (HMD) and starts the system. The user's operation causes the device's microphone to start capturing ambient sounds. At this point, the input is the ambient sound, and the output is the captured sound data.

[0878] Step 2:

[0879] The device converts the captured sound data into volume and frequency spectrum data using a library for audio signal processing (e.g., Python's Librosa library). The input is the captured sound data, and the output is the converted volume and frequency spectrum data.

[0880] Step 3:

[0881] The terminal transmits the converted volume and frequency spectrum data to the server. The data is transmitted over the network. The input is the converted volume and frequency spectrum data, and the output is the data transmitted to the server.

[0882] Step 4:

[0883] The server analyzes the received volume and frequency spectrum data. A proprietary algorithm is used for the analysis, and a generative AI model (e.g., TensorFlow or PyTorch) is used for this process. The input is the volume and frequency spectrum data sent to the server, and the output is the analysis result.

[0884] Step 5:

[0885] The server generates the optimal masking sound based on the analysis results. During this generation process, white noise or soft background music is selected depending on the noise characteristics. An example prompt sentence is input to the generation AI model. The input is the analysis results and the prompt sentence, and the output is the generated masking sound data. An example prompt sentence is as follows:

[0886] Noise data: {sound data}

[0887] Output conditions: Soft background music or white noise

[0888] Response format: data

[0889] Step 6:

[0890] The server sends the generated masking sound data to the terminal. The data is sent via the network and arrives at the user's terminal in real time. The input is the generated masking sound data, and the output is the data sent to the terminal.

[0891] Step 7:

[0892] The device decodes the received masking sound data and plays it through its built-in speaker. This process effectively cancels out the user's surrounding noise in real time. The input is the masking sound data received from the server, and the output is the masking sound being played.

[0893] Step 8:

[0894] The user enjoys the comfortable sound environment obtained by the operation of the system. In this step, the user does not need to perform any special operations; the system automatically reduces noise. The input is the reproduced masking sound, and the output is a comfortable sound environment.

[0895] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0896] The system for implementing this invention combines a conventional noise control system with an emotion engine that recognizes the user's emotions, providing a more comfortable space for the user. This system operates in cooperation with a server, terminals, and users.

[0897] System configuration

[0898] 1. Device: The device is equipped with a microphone, speaker, camera, and other emotion sensors to capture ambient sounds and the user's emotions, and to play back the generated masking sounds. The device also has a network connection function to communicate with the server.

[0899] 2. Server: The server has a means of receiving and analyzing noise data and emotion data sent from the device. Furthermore, the server has a generative AI model and emotion engine that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time.

[0900] 3. Emotion Engine: The emotion engine analyzes the user's stress level and emotional state from facial expressions and vocal tone, and helps adjust the masking sound accordingly.

[0901] 4. User: The user can start the system and use the device to maintain a comfortable environment. The user can enjoy the system automatically analyzing and reducing noise and emotional state without any special operation.

[0902] Explaining the program's processing

[0903] The program of this system is executed in the following procedure, with each of the roles of terminal, server, and emotion engine being assigned.

[0904] 1. Booting the system

[0905] When a user activates the system on their device, the device's microphone begins capturing ambient sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server. At the same time, the device uses a camera and emotion sensors to collect the user's emotion data, which is also sent to the server.

[0906] 2. Data Analysis

[0907] The server receives noise and emotion data sent from the device and analyzes the data using a proprietary algorithm. Once the characteristics of the noise and the user's emotional state are clarified, the server generates the optimal masking sound based on the analysis results.

[0908] 3. Masking sound generation

[0909] The server uses a generative AI model to generate optimal masking sounds, such as white noise or soft background music, based on the analyzed noise characteristics and the user's emotional state as determined by the emotion engine.

[0910] 4. Transmitting and playing masking sounds

[0911] The generated masking sound is sent from the server to the device, which then decodes it and plays it back through the speaker in real time, reducing ambient noise and providing an optimal environment for the user's emotional state.

[0912] Specific examples

[0913] Example 1: Use in the office

[0914] A user in a meeting in a shared office space activates the system. The device's microphone captures ambient noise, while the camera and emotion sensor recognize the user's emotional state. The server analyzes the captured data and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker to prevent the content of the conversation from leaking to the outside. The system also adjusts according to the user's emotional state to reduce stress levels.

[0915] Example 2: Use at a cafe

[0916] A user working remotely in a cafe starts the system. Noise captured by the device's microphone and emotional state analyzed by the emotion engine are sent to the server, which then analyzes the data and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the cafe's noise and providing an environment suited to the user's emotional state.

[0917] Through these processes, the system can reduce noise in real time and provide an optimal space according to the user's emotional state.

[0918] The processing flow will be explained below.

[0919] Step 1:

[0920] The user starts the system on the device. When the user launches the application and presses the system's start button, the device's microphone, camera, and emotion sensor are activated and ready to capture ambient sounds and the user's emotional state. The device sends a connection request to the server to notify that the system has started.

[0921] Step 2:

[0922] The device's microphone captures voice data in real time, which is then converted into digital signals containing data such as volume (dB) and frequency spectrum. At the same time, the device's camera and emotion sensor analyze the user's facial expressions and voice tone to collect emotional data.

[0923] Step 3:

[0924] The device then collects the captured sound data and emotion data into packets and sends them to the server. The device receives a confirmation response from the server to confirm that the data transmission was successful.

[0925] Step 4:

[0926] The server analyzes the received noise data and emotion data, including parameters such as volume, frequency distribution, and duration. Based on the analysis results, the server determines which type of masking sound is optimal.

[0927] Step 5:

[0928] The server uses a generative AI model to generate optimal masking sounds, such as white noise or soft background music, based on the analyzed noise characteristics and the user's emotional state as determined by the emotion engine.

[0929] Step 6:

[0930] The generated masking sound is sent from the server to the terminal. The server sends packetized data and receives an acknowledgment from the terminal. The terminal decodes the received masking sound data and prepares for playback.

[0931] Step 7:

[0932] The device plays masking sounds through the speaker in real time, reducing ambient noise. The emotion engine also monitors the user's emotional state in real time and adjusts the masking sounds as needed.

[0933] Step 8:

[0934] Users can work and talk in a comfortable environment thanks to the generated masking sounds and optimization by the emotion engine. If the user feels stressed, the masking sounds are automatically adjusted to maintain comfort. When the user presses the system's shutdown button, all processes are stopped and the device's microphone, camera, emotion sensor, and speaker are turned off.

[0935] Example 2

[0936] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0937] In modern urban environments, noise levels are constantly increasing, causing stress for many people. Existing noise control systems focus on capturing sounds and generating masking sounds, but they do not take into account the user's emotional state, making it difficult to provide an optimal sound environment.

[0938] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing ambient sounds, means for capturing user emotional data, means for transmitting the captured sound data and emotional data to the server, means for the server to analyze the sound data and emotional data, means for generating an optimal masking sound using a generative AI model based on the analysis results, means for transmitting the generated masking sound to the terminal, and means for playing the generated masking sound. This makes it possible to provide an optimal sound environment according to the user's emotional state while effectively reducing noise.

[0939] The "means for capturing ambient sound" refers to a device or system for collecting sounds in the environment, and specifically refers to a microphone or the like.

[0940] "Means for capturing user emotional data" refers to a device or system for collecting the user's emotional state, and specifically includes a camera or a voice analysis sensor.

[0941] "Means for transmitting captured sound data and emotional data to a server" refers to a network connection device or system for communicating the collected data to a server, specifically Wi-Fi, Bluetooth, the Internet, etc.

[0942] "Means for the server to analyze sound data and emotional data" refers to software or algorithms for processing and analyzing sound data and emotional data received by the server.

[0943] "Means for generating optimal masking sound using an AI model based on the analysis results" refers to software that includes an artificial intelligence model or algorithm for generating optimal masking sound based on the analyzed data.

[0944] The "means for transmitting the generated masking sound to the terminal" refers to a network connection device or system for communicating the generated masking sound to the terminal, and specifically includes the Internet, Wi-Fi, Bluetooth, etc.

[0945] The "means for reproducing the generated masking sound" refers to a device or system for reproducing the received masking sound, and specifically refers to a speaker, a headphone, or the like.

[0946] MODE FOR CARRYING OUT THE INVENTION

[0947] The system of the present invention provides optimal masking sound based on the ambient noise and the user's emotional state by linking a server, a terminal, and an emotion engine. The system uses the following hardware and software:

[0948] Hardware and software used

[0949] 1. Device:

[0950] Microphone: A device for capturing ambient sounds.

[0951] Camera: A device for capturing a user's facial expressions.

[0952] Emotion sensors: Sensors for capturing the user's tone of voice and other vital signs.

[0953] Network connectivity: A means of communicating with the server (e.g., Wi-Fi, Bluetooth).

[0954] Speaker: A device for playing back the masking sound sent from the server.

[0955] 2. Server:

[0956] Data analysis module: An algorithm that analyzes noise data and emotion data sent from the device.

[0957] Generative AI model: A program that generates optimal masking sounds based on analysis results (e.g., OpenAI's GPT-3 and DALL-E).

[0958] Network connection function: A means for communicating with terminals.

[0959] 3. Emotion Engine:

[0960] Emotion analysis algorithm: Software for analyzing the emotional state of a user.

[0961] Operational flow and concrete examples

[0962] When the system is started, the device's microphone first captures the surrounding sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server. At the same time, the device's camera and emotion sensor collect the user's emotion data and send it to the server.

[0963] The server receives the noise and emotion data sent from the device and analyzes each data. The analysis uses a data analysis module and emotion analysis algorithm. Based on the analysis results, the server uses a generative AI model to generate optimal masking sounds. For example, if the noise is intense and the user is feeling stressed, it generates relaxing white noise or soft background music.

[0964] The generated masking sound is sent from the server to the device, which then receives it and plays it in real time through the speaker, effectively reducing ambient noise and providing the optimal sound environment according to the user's emotional state.

[0965] Specific examples

[0966] Example 1: Use in the office

[0967] A user in a meeting in a shared office space activates the system. The device's microphone captures ambient noise, while the camera and emotion sensor recognize the user's emotional state. The server analyzes the captured data and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker to prevent the content of the conversation from leaking to the outside. The system also adjusts according to the user's emotional state to reduce stress levels.

[0968] Example prompt:

[0969] "Generate masking sounds suitable for situations where people are having a meeting in a shared office space. Users are stressed and want a relaxing environment."

[0970] Example 2: Use at a cafe

[0971] A user working remotely in a cafe starts the system. Noise captured by the device's microphone and emotional state analyzed by the emotion engine are sent to the server, which then analyzes the data and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the cafe's noise and providing an environment suited to the user's emotional state.

[0972] Example prompt:

[0973] "For users working remotely in a cafe, please generate masking sounds that cancel out surrounding noise and make it easier for them to concentrate."

[0974] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0975] Step 1:

[0976] The user starts the system on the device. The device first initializes and enables the built-in microphone, camera, and emotion sensor, ensuring that the hardware is operational and ready to begin capturing data.

[0977] Specific behavior:

[0978] The user clicks the "System Start" button.

[0979] Input: User action (clicking the launch button)

[0980] Output: Device hardware initialization complete

[0981] Step 2:

[0982] The device's microphone begins capturing ambient sound, which is then converted into volume and frequency spectrum data, which is then sent to the server.

[0983] Specific behavior:

[0984] The device's microphone records the surrounding sounds and samples the waveform data in real time, converts the data into a digital format (e.g., PCM format), and sends it to the server.

[0985] Input: Ambient sound

[0986] Output: Volume and frequency spectrum converted data

[0987] Step 3:

[0988] The device's camera and emotion sensors capture the user's facial expressions and voice tone to collect emotion data, which is also sent to the server.

[0989] Specific behavior:

[0990] The device's camera captures the user's face and analyzes the movements of each part of the face (eyes, mouth, eyebrows, etc.). The audio sensor also analyzes the user's tone of voice and speaking style. This data is combined to determine the user's emotional state and is sent to the server.

[0991] Input: User facial expressions and voice tone

[0992] Output: Emotion data

[0993] Step 4:

[0994] The server receives the noise and emotion data sent from the device. The received data is processed by the analysis module. The server uses a proprietary algorithm to analyze the characteristics of the noise and the user's emotional state.

[0995] Specific behavior:

[0996] The server analyzes the characteristics of the received noise data and identifies the sound volume, frequency components, source, etc. At the same time, it detects the user's current emotional state (stress level, relaxation level) based on the received emotional data.

[0997] Input: Noise data and emotion data

[0998] Output: Noise analysis results and emotion analysis results

[0999] Step 5:

[1000] The server uses a generative AI model to generate optimal masking sounds based on the analysis results. The generated masking sounds are tailored to the user's emotional state, such as white noise or relaxing music.

[1001] Specific behavior:

[1002] Based on the analysis results, the server inputs prompts to the generative AI model, which then designs appropriate masking sounds and returns the sound data.

[1003] Input: Noise analysis results and emotion analysis results

[1004] Output: Generated masking sound

[1005] Example prompt:

[1006] "For users working remotely in a cafe, please generate masking sounds that cancel out surrounding noise and make it easier for them to concentrate."

[1007] Step 6:

[1008] The generated masking sound is sent from the server to the device, which decodes the received sound data and plays it back in real time through the speaker.

[1009] Specific behavior:

[1010] The server sends the generated masking sound to the device, which then decodes the received sound data and plays it back using the speaker, allowing the user to work in a comfortable environment.

[1011] Input: Generated masking sound

[1012] Output: Played masking sound

[1013] (Application example 2)

[1014] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1015] In the food delivery industry, delivery staff often experience mental stress due to traffic noise and other disturbances at delivery destinations. This reduces the efficiency of delivery work and staff comfort. The present invention aims to provide an environment in which delivery staff can perform their work comfortably and efficiently.

[1016] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing ambient sounds, means for analyzing the captured sound data and the user's emotional state, and means for generating an optimal masking sound based on the analysis results and the user's emotional state. This effectively masks the noise around the delivery staff, enabling them to perform their work while reducing mental stress.

[1017] The "means for capturing ambient sound" is a device including a microphone and a sensor for collecting environmental sounds around the delivery staff.

[1018] "Means for analyzing captured sound data and the user's emotional state" refers to algorithms or engines that analyze the collected sound data and the emotional state obtained from the delivery staff's facial expressions, vocal tone, etc.

[1019] The "means for generating optimal masking sounds based on the analysis results and the user's emotional state" refers to a generative AI model or sound engine that generates optimal sounds to reduce the mental stress of delivery staff based on noise data and emotional data.

[1020] The "means for reproducing the generated masking sound" refers to a speaker or audio reproduction device for reproducing the generated masking sound in real time.

[1021] The "means for transmitting captured sound data and emotional data to a server" refers to a communication module or protocol for transmitting the collected sound data and emotional data to a server via a network.

[1022] "The means by which the server analyzes sound data and emotional data and generates optimal masking sounds" refers to a function that uses an analysis algorithm or generation AI model running on the server to analyze collected data and generate masking sounds.

[1023] The "means for transmitting the generated masking sound to the terminal" is a communication function for transmitting the masking sound generated by the server to the terminal of the delivery staff via the network.

[1024] A "generative AI model" is a type of artificial intelligence model used for data analysis and sound generation, and is an algorithm that is particularly suited to generating voice and music.

[1025] A "prompt" is a set of instructions or questions input to a generative AI model to obtain a desired output result.

[1026] The food delivery system that embodies this invention operates through the cooperation of a server, terminals, and users in order to improve the comfort and work efficiency of delivery staff. The program of this system is configured as follows.

[1027] Hardware and software used

[1028] Hardware

[1029] 1. Smartphone

[1030] microphone

[1031] speaker

[1032] camera

[1033] Network connectivity features

[1034] software

[1035] 1. Smartphone App

[1036] Environmental sound and emotion data capture function using microphone and camera

[1037] Communication with the server

[1038] Masking sound decoding and playback function

[1039] 2. Server

[1040] Receiving and analyzing noise and emotion data

[1041] Generative AI models (e.g., DeepMind WaveNet, OpenAI GPT-4)

[1042] Emotion engine (e.g. Microsoft Azure Emotion API)

[1043] Program processing details

[1044] The server receives and analyzes the sound data and emotional data sent from the device. The analysis involves checking the volume and frequency spectrum of the sound data, and using algorithms to determine the user's stress level and emotional state for the emotional data. Based on the analysis results, the server uses a generative AI model to generate optimal masking sounds. These masking sounds help reduce the user's mental stress and provide a comfortable environment.

[1045] The masking sound is transmitted from the server to the device and played in real time through the smartphone speaker, effectively canceling out surrounding noise for delivery staff, allowing them to continue working in a relaxed state.

[1046] Examples of concrete examples and prompts

[1047] Specific examples

[1048] When a delivery person launches the smartphone app, the smartphone's microphone begins capturing surrounding traffic and construction sounds. At the same time, a camera analyzes the delivery person's facial expressions to detect temporary highs and lows of stress. The server receives this data and instructs a generative AI model to generate soothing music with a subdued bass rhythm. This sound is then transmitted to the smartphone and played through the speaker, effectively canceling out the noise around the delivery person.

[1049] Prompt Sentence Examples

[1050] The following prompt sentence is input into a generative AI model (e.g., OpenAI GPT-4) to generate the optimal masking sound based on noise analysis and emotion data.

[1051] Consider the surrounding environmental noise (60dB, low frequency) and emotional state (tension, high temporary stress level), and generate music that effectively masks these noises and promotes relaxation. For example, soothing music with a low bass rhythm or a blend of nature sounds would be suitable.

[1052] Output sound requirements

[1053] Frequency range: 20Hz - 20kHz

[1054] Volume: 45dB or less

[1055] Sound quality: Clear, relaxing

[1056] In this way, the present invention aims to reduce the noise around delivery staff in real time and relieve their mental stress.

[1057] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1058] Step 1:

[1059] When a user activates the system on their smartphone, the device's microphone starts capturing ambient sounds, while the camera and emotion sensor simultaneously capture the user's facial expressions, voice tone, and other emotional data.

[1060] Input: ambient sound data, user emotion data

[1061] Output: Sound data and emotion data collected on the device

[1062] Step 2:

[1063] The device transmits the collected sound data and emotion data to the server via the network, converts the data into an appropriate format, and connects to the server using a communication module.

[1064] Input: Sound data and emotion data collected on the device

[1065] Output: Sound data and emotion data sent to the server

[1066] Step 3:

[1067] The server analyzes the received sound data and emotional data. The sound data is analyzed to check the volume and frequency spectrum, and the emotional data is processed through algorithms to determine the user's stress level and emotional state.

[1068] Input: Sound data and emotion data sent to the server

[1069] Output: Analysis results (sound data characteristics, emotional data status)

[1070] Step 4:

[1071] The server inputs a prompt sentence into the generative AI model based on the analysis results to generate the optimal masking sound. The generative AI model follows the process to create the optimal masking sound that can reduce the user's mental stress.

[1072] Input: Analysis results (sound data characteristics, emotional data state), prompt

[1073] Output: Generated masking sound

[1074] Step 5:

[1075] The server transmits the generated masking sound to the terminal via the network, and the server appropriately formats the transmission data and transmits it to the terminal using a communication protocol.

[1076] Input: Generated masking sound

[1077] Output: Masking sound sent to the device

[1078] Step 6:

[1079] The device decodes the masking sound received from the server and plays it in real time through the speaker, effectively masking surrounding noise and allowing the user to work in a more comfortable environment.

[1080] Input: Masking sound sent to the device

[1081] Output: Masking sound played through the device speaker

[1082] Examples of concrete examples and prompts

[1083] Specific examples

[1084] When a delivery person launches the smartphone app, the smartphone's microphone begins capturing surrounding traffic and construction sounds. At the same time, a camera analyzes the delivery person's facial expressions to detect temporary highs and lows of stress. The server receives this data and instructs a generative AI model to generate soothing music with a subdued bass rhythm. This sound is then transmitted to the smartphone and played through the speaker, effectively canceling out the noise around the delivery person.

[1085] Prompt Sentence Examples

[1086] The following prompt sentence is input into a generative AI model (e.g., OpenAI GPT-4) to generate the optimal masking sound based on noise analysis and emotion data.

[1087] Consider the surrounding environmental noise (60dB, low frequency) and emotional state (tension, high temporary stress level), and generate music that effectively masks these noises and promotes relaxation. For example, soothing music with a low bass rhythm or a blend of nature sounds would be suitable.

[1088] Output sound requirements

[1089] Frequency range: 20Hz - 20kHz

[1090] Volume: 45dB or less

[1091] Sound quality: Clear, relaxing

[1092] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1093] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1094] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1095] [Fourth embodiment]

[1096] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1097] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1098] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1099] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1100] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1101] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1102] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1103] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1104] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1105] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1106] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1107] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1108] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1109] The system for implementing this invention provides a comfortable space by inputting noise and playing the optimal masking sound created by the AI ​​in real time. This system operates in cooperation with the server, terminals, and users.

[1110] System configuration

[1111] 1. Terminal: The terminal has a built-in microphone and speaker, which allows it to capture ambient sounds and play back the generated masking sound. The terminal also has a network connection function for communicating with the server.

[1112] 2. Server: The server has a means of receiving and analyzing noise data sent from the device. Furthermore, the server has a generative AI model that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time.

[1113] 3. User: The user can start the system and use the terminal to maintain a comfortable environment. The user can enjoy the automatic noise reduction of the system without any special operation.

[1114] Explaining the program's processing

[1115] The program of this system is executed in the following procedure, with the terminal, server, and user each taking on a different role.

[1116] 1. Booting the system

[1117] When a user activates the system on their device, the device's microphone begins capturing ambient sound. The captured sound data is converted into volume and frequency spectrum data and sent to the server.

[1118] 2. Analysis of noise data

[1119] The server receives noise data sent from the device and analyzes it using a proprietary algorithm. Once the characteristics of the noise have been clarified, the server generates the optimal masking sound based on the analysis results.

[1120] 3. Masking sound generation

[1121] The server uses a generative AI model to generate the optimal masking sound, which can be white noise or soft background music based on the analyzed noise characteristics.

[1122] 4. Transmitting and playing masking sounds

[1123] The generated masking sound is sent from the server to the device, which then decodes it and plays it back through the speaker in real time. This process effectively cancels out the noise around the user, providing a comfortable environment.

[1124] Specific examples

[1125] Example 1: Use in the office

[1126] A user in a meeting in a shared office space starts the system. The device's microphone captures ambient noise and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker, preventing the content of the conversation from leaking to the outside world. The user also enjoys a quiet environment that makes it easier to concentrate.

[1127] Example 2: Use at a cafe

[1128] A user working remotely in a cafe starts the system. Noise captured by the device's microphone is sent to the server, which analyzes it and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the noise in the cafe, allowing the user to continue working in a quiet environment.

[1129] Through these processes, the system can reduce noise in real time and provide a comfortable space for users.

[1130] The processing flow will be explained below.

[1131] Step 1:

[1132] The user starts the system on the device. When the user launches an application and presses the system start button, the device's microphone starts up and is ready to capture ambient sounds. The device sends a connection request to the server to notify that the system has started.

[1133] Step 2:

[1134] The device's microphone captures audio data in real time, and the captured audio is converted into digital data, which includes information such as volume (dB) and frequency spectrum.

[1135] Step 3:

[1136] The device then collects the captured audio data and sends it to the server, where it receives a confirmation response confirming that the transmission was successful.

[1137] Step 4:

[1138] The server analyzes the received noise data, including parameters such as volume, frequency distribution, and duration, and determines which type of masking sound is most suitable based on the analysis results.

[1139] Step 5:

[1140] The server uses a generative AI model to generate the optimal masking sound, which can be white noise or soft background music based on the analyzed noise characteristics.

[1141] Step 6:

[1142] The server assembles the generated masking sounds into packets and prepares to send them to the terminal. The server sends the generated masking sounds to the terminal and receives a confirmation response from the terminal.

[1143] Step 7:

[1144] The device decodes the received masking sound data and prepares it for playback. The masking sound is played in real time through the speaker to reduce ambient noise.

[1145] Step 8:

[1146] The generated masking sound allows the user to continue working in a comfortable, noise-reduced environment. When the user presses the system shutdown button, all processes are stopped and the device's microphone and speaker are turned off.

[1147] Example 1

[1148] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1149] In modern society, noise has become a major problem in daily life and work environments. Noise can reduce concentration and cause stress and fatigue. Conventional technologies have not provided an efficient method for reducing noise. In particular, there has been no system that can detect noise in real time and generate and play appropriate masking sounds based on that detection. Therefore, there is a need for a system that can effectively cancel noise and provide a comfortable space for users.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1151] In this invention, the server includes means for capturing ambient sounds, means for converting the captured sound data into volume and frequency spectrum, means for transmitting the converted sound data to the server, means for the server to analyze the sound data and extract noise characteristics, means for generating an optimal masking sound using a generative AI model, means for transmitting the generated masking sound to the terminal, means for the terminal to decode the masking sound and play it through a speaker, and means for playing the generated masking sound. This makes it possible to detect noise in real time and generate and play an adaptive masking sound.

[1152] A "means for capturing ambient sound" is a method of converting sound waves in the environment into electrical signals using a sensor device (e.g., a microphone).

[1153] "Means for converting captured sound data into volume and frequency spectrum" refers to algorithms or software that analyze audio signals and express their sound pressure (volume) and frequency components as digital data.

[1154] The "means for transmitting the converted sound data to the server" is a process for communicating the sound data from the terminal to the server using the network connection function.

[1155] "Means for the server to analyze sound data and extract noise characteristics" refers to technology that receives audio data and analyzes specific frequency bands and volume levels to identify noise patterns and characteristics.

[1156] "Means for generating optimal masking sounds using generative AI models" refers to a process that utilizes machine learning algorithms to design and generate optimal masking sounds based on the characteristics of noise.

[1157] The "means for transmitting the generated masking sound to the terminal" is a method for communicating the generated audio data from the server to the terminal.

[1158] The "means for the terminal to decode the masking sound and play it through the speaker" refers to the process of converting the audio data received by the terminal into an audio signal and playing it through the speaker device.

[1159] The "means for reproducing the generated masking sound" refers to a technique for converting the generated audio data into physical sounds and reproducing them in the environment.

[1160] A "prompt" is a text-based instruction that allows a generative AI model to input specific conditions or data.

[1161] The system for implementing this invention provides a comfortable space by inputting noise and playing the optimal masking sound created by the AI ​​in real time. This system operates in cooperation with the server, terminals, and users.

[1162] System configuration

[1163] 1. Terminal

[1164] The device has a built-in microphone and speaker, and has a means for capturing ambient sounds and a means for playing back the generated masking sounds. The device also has a network connection function for communicating with a server, and can transmit the captured sound data to the server.

[1165] For example, the microphone captures ambient sounds in real time and converts them into volume and frequency spectrum using built-in audio processing libraries such as Librosa.

[1166] 2. Server

[1167] The server has a generative AI model that analyzes the received sound data and generates the optimal masking sound. The server uses Python-based scripts and machine learning libraries such as TensorFlow and PyTorch to analyze the sound data. After analysis, the generated masking sound is sent to the device. When the server uses a generative AI model (for example, OpenAI's GPT-3 or GPT-4) to generate the optimal masking sound, it uses the following prompt statement:

[1168] Prompt statement:

[1169] "The frequency spectrum of the current ambient sound is as follows:\n[frequency data]. Please generate the optimal masking sound to cancel out this ambient sound."

[1170] 3. Users

[1171] Users can activate the system and use their devices to maintain a comfortable environment, and the system automatically reduces noise without any special operations.

[1172] Specific examples

[1173] Example 1: Use in the office

[1174] A user in a meeting in a shared office space starts the system. The device's microphone captures ambient noise and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker, preventing the content of the conversation from being leaked to the outside world and providing the user with a quiet environment where they can concentrate easily.

[1175] Example 2: Use at a cafe

[1176] A user working remotely in a cafe starts the system. Noise captured by the device's microphone is sent to the server, which analyzes it and generates the optimal masking sound. The generated masking sound is played in real time through the device's speaker, effectively canceling out the cafe's noise, allowing the user to continue working in a quiet environment.

[1177] In this way, the system of the present invention can reduce noise in real time and provide a comfortable space for the user.

[1178] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1179] Step 1:

[1180] The user starts an application on the terminal. The user presses the "Start" button, and the system enters the startup state. The input is the user's operation. The output is the system initialization completion state.

[1181] Step 2:

[1182] The device's microphone captures the surrounding sound in real time. The input is the surrounding sound, and the output is the captured audio data. Specifically, the device's microphone converts analog audio into a digital signal.

[1183] Step 3:

[1184] The device converts captured audio data into volume and frequency spectrum. The input is a digital audio signal, and the output is data containing volume and frequency spectrum. The device uses the Librosa library to do this. For example, Librosa can be used to convert audio data into a spectrogram using techniques such as fft and stft.

[1185] Step 4:

[1186] The device sends the converted sound data to the server. The input is data including volume and frequency spectrum, and the output is an HTTP POST request to the server. The device transfers this data to the server using its network connection function.

[1187] Step 5:

[1188] The server receives the transmitted sound data and begins analysis. The input is volume and frequency spectrum data, and the output is data extracting the noise characteristics. The server performs analysis using algorithms implemented in Python, as well as TensorFlow and PyTorch.

[1189] Step 6:

[1190] The server uses a generative AI model based on the characteristics of the noise to generate the optimal masking sound. The input is the analyzed noise characteristic data, and the output is the generated masking sound. Models such as GPT-3 and GPT-4 are used as generative AI models. The specific generation procedure uses the following prompt sentence:

[1191] Prompt statement:

[1192] "The frequency spectrum of the current ambient sound is as follows:\n[frequency data]. Please generate the optimal masking sound to cancel out this ambient sound."

[1193] Step 7:

[1194] The server sends the generated masking sound to the device. The input is the generated masking sound, and the output is the audio data as an HTTP POST request to the device.

[1195] Step 8:

[1196] The device decodes the received masking sound and plays it through the speaker. The input is the audio data sent from the server, and the output is the actual masking sound. The device uses FFmpeg or Librosa to decode the audio data into a playable format and play it through the speaker.

[1197] (Application example 1)

[1198] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1199] In modern society, activities in virtual environments have become commonplace, but noise and other disturbances can significantly disrupt the user experience. Conventional sound environment control systems primarily control sound in physical spaces and are not suitable for noise reduction in virtual stores and other virtual environments. Therefore, an efficient system for providing a comfortable sound environment in virtual environments is needed.

[1200] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1201] In this invention, the server includes means for capturing ambient sounds, means for analyzing the captured sound data, means for generating an optimal masking sound based on the analysis results, means for playing back the generated masking sound, and means for analyzing noise in the virtual environment and providing an optimal sound environment for a display device worn by the user. This makes it possible to effectively reduce noise and other disturbances in the virtual environment and provide a comfortable sound environment for the user.

[1202] The "means for capturing ambient sound" is a function for acquiring ambient environmental sound using a device such as a microphone.

[1203] "Means for analyzing captured sound data" refers to algorithms or software that analyze the characteristics of the acquired sound data.

[1204] The "means for generating the optimum masking sound based on the analysis results" is a function that performs a process to create the optimum masking sound for the user according to the characteristics of the analyzed sound.

[1205] The "means for reproducing the generated masking sound" is a function for allowing the user to hear the generated masking sound using an audio device such as a speaker or earphones.

[1206] The "means for transmitting captured sound data to a server" is a function for transmitting the acquired sound data to a remote server via the Internet or a network.

[1207] "The means by which the server analyzes the sound data and generates the optimal masking sound" refers to the process of analyzing the transmitted sound data using an analysis algorithm or generation AI model on the server and generating the optimal masking sound.

[1208] The "means for transmitting the generated masking sound to the terminal" is a function for transmitting the masking sound generated by the server to the user's terminal via the network.

[1209] "Means for connecting to a display device worn by the user to provide an optimal sound environment" refers to a function that communicates with devices such as smart glasses and head-mounted displays (HMDs) to provide an optimal sound environment.

[1210] "Means for responding to dynamic changes in noise in a virtual environment" refers to a function that responds in real time to changes in noise and the sound environment within a virtual space and generates appropriate masking sounds.

[1211] In this embodiment, a comfortable sound environment is provided by inputting noise and playing optimal masking sounds generated by a generative AI model in real time. This system operates in cooperation with a server, terminals, and users.

[1212] System configuration

[1213] 1. Device:

[1214] The device has a built-in microphone and speaker, and has a means for capturing ambient sounds and a means for playing back the generated masking sounds. The device also has a network connection function for communicating with a server. In particular, it includes a means for connecting to devices such as smart glasses and head-mounted displays (HMDs) in a virtual environment to provide an optimal sound environment.

[1215] 2. Server:

[1216] The server has a means of receiving and analyzing noise data sent from the device. Furthermore, the server has a generative AI model that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time. The Python Librosa library is used to analyze the noise data, and TensorFlow and PyTorch are used for the generative AI model.

[1217] 3. User:

[1218] Users can activate the system and use their devices to maintain a comfortable environment, and the system automatically reduces noise without any special operations.

[1219] Program processing

[1220] The programs in this system are executed by dividing the roles of the terminal, server, and user.

[1221] When a user puts on smart glasses or an HMD and starts the system, the device's microphone begins capturing ambient sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server.

[1222] The server receives noise data sent from the device and analyzes it using a proprietary algorithm. Once the characteristics of the noise have been clarified, the server generates the optimal masking sound based on the analysis results. During this generation process, white noise, soft background music, etc. are created based on the analyzed noise characteristics.

[1223] The generated masking sound is sent from the server to the device and played in real time through the device's speaker, effectively canceling out the user's surrounding noise and providing a comfortable sound environment.

[1224] Specific examples

[1225] Suppose a user is distracted by other users' chats and background music in a virtual store. The user puts on smart glasses and launches an application. The device's microphone captures the noise in the virtual store and sends it to the server. The server analyzes the captured noise and generates an optimal masking sound. The generated masking sound is played in real time from the smart glasses' speakers, allowing the user to enjoy shopping in a comfortable acoustic environment.

[1226] Prompt Sentence Examples

[1227] The generative AI model is given a prompt like this:

[1228] Noise data: {sound data}

[1229] Output conditions: Soft background music or white noise

[1230] Response format: data

[1231] With this configuration and means, the system can effectively reduce noise and disturbances in the virtual environment, providing the user with a comfortable sound environment.

[1232] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1233] Step 1:

[1234] A user puts on smart glasses or a head-mounted display (HMD) and starts the system. The user's operation causes the device's microphone to start capturing ambient sounds. At this point, the input is the ambient sound, and the output is the captured sound data.

[1235] Step 2:

[1236] The device converts the captured sound data into volume and frequency spectrum data using a library for audio signal processing (e.g., Python's Librosa library). The input is the captured sound data, and the output is the converted volume and frequency spectrum data.

[1237] Step 3:

[1238] The terminal transmits the converted volume and frequency spectrum data to the server. The data is transmitted over the network. The input is the converted volume and frequency spectrum data, and the output is the data transmitted to the server.

[1239] Step 4:

[1240] The server analyzes the received volume and frequency spectrum data. A proprietary algorithm is used for the analysis, and a generative AI model (e.g., TensorFlow or PyTorch) is used for this process. The input is the volume and frequency spectrum data sent to the server, and the output is the analysis result.

[1241] Step 5:

[1242] The server generates the optimal masking sound based on the analysis results. During this generation process, white noise or soft background music is selected depending on the noise characteristics. An example prompt sentence is input to the generation AI model. The input is the analysis results and the prompt sentence, and the output is the generated masking sound data. An example prompt sentence is as follows:

[1243] Noise data: {sound data}

[1244] Output conditions: Soft background music or white noise

[1245] Response format: data

[1246] Step 6:

[1247] The server sends the generated masking sound data to the terminal. The data is sent via the network and arrives at the user's terminal in real time. The input is the generated masking sound data, and the output is the data sent to the terminal.

[1248] Step 7:

[1249] The device decodes the received masking sound data and plays it through its built-in speaker. This process effectively cancels out the user's surrounding noise in real time. The input is the masking sound data received from the server, and the output is the masking sound being played.

[1250] Step 8:

[1251] The user enjoys the comfortable sound environment obtained by the operation of the system. In this step, the user does not need to perform any special operations; the system automatically reduces noise. The input is the reproduced masking sound, and the output is a comfortable sound environment.

[1252] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1253] The system for implementing this invention combines a conventional noise control system with an emotion engine that recognizes the user's emotions, providing a more comfortable space for the user. This system operates in cooperation with a server, terminals, and users.

[1254] System configuration

[1255] 1. Device: The device is equipped with a microphone, speaker, camera, and other emotion sensors to capture ambient sounds and the user's emotions, and to play back the generated masking sounds. The device also has a network connection function to communicate with the server.

[1256] 2. Server: The server has a means of receiving and analyzing noise data and emotion data sent from the device. Furthermore, the server has a generative AI model and emotion engine that generates optimal masking sounds based on the analysis results. The generated masking sounds are sent to the device and played in real time.

[1257] 3. Emotion Engine: The emotion engine analyzes the user's stress level and emotional state from facial expressions and vocal tone, and helps adjust the masking sound accordingly.

[1258] 4. User: The user can start the system and use the device to maintain a comfortable environment. The user can enjoy the system automatically analyzing and reducing noise and emotional state without any special operation.

[1259] Explaining the program's processing

[1260] The program of this system is executed in the following procedure, with each of the roles of terminal, server, and emotion engine being assigned.

[1261] 1. Booting the system

[1262] When a user activates the system on their device, the device's microphone begins capturing ambient sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server. At the same time, the device uses a camera and emotion sensors to collect the user's emotion data, which is also sent to the server.

[1263] 2. Data Analysis

[1264] The server receives noise and emotion data sent from the device and analyzes the data using a proprietary algorithm. Once the characteristics of the noise and the user's emotional state are clarified, the server generates the optimal masking sound based on the analysis results.

[1265] 3. Masking sound generation

[1266] The server uses a generative AI model to generate optimal masking sounds, such as white noise or soft background music, based on the analyzed noise characteristics and the user's emotional state as determined by the emotion engine.

[1267] 4. Transmitting and playing masking sounds

[1268] The generated masking sound is sent from the server to the device, which then decodes it and plays it back through the speaker in real time, reducing ambient noise and providing an optimal environment for the user's emotional state.

[1269] Specific examples

[1270] Example 1: Use in the office

[1271] A user in a meeting in a shared office space activates the system. The device's microphone captures ambient noise, while the camera and emotion sensor recognize the user's emotional state. The server analyzes the captured data and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker to prevent the content of the conversation from leaking to the outside. The system also adjusts according to the user's emotional state to reduce stress levels.

[1272] Example 2: Use at a cafe

[1273] A user working remotely in a cafe starts the system. Noise captured by the device's microphone and emotional state analyzed by the emotion engine are sent to the server, which then analyzes the data and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the cafe's noise and providing an environment suited to the user's emotional state.

[1274] Through these processes, the system can reduce noise in real time and provide an optimal space according to the user's emotional state.

[1275] The processing flow will be explained below.

[1276] Step 1:

[1277] The user starts the system on the device. When the user launches the application and presses the system's start button, the device's microphone, camera, and emotion sensor are activated and ready to capture ambient sounds and the user's emotional state. The device sends a connection request to the server to notify that the system has started.

[1278] Step 2:

[1279] The device's microphone captures voice data in real time, which is then converted into digital signals containing data such as volume (dB) and frequency spectrum. At the same time, the device's camera and emotion sensor analyze the user's facial expressions and voice tone to collect emotional data.

[1280] Step 3:

[1281] The device then collects the captured sound data and emotion data into packets and sends them to the server. The device receives a confirmation response from the server to confirm that the data transmission was successful.

[1282] Step 4:

[1283] The server analyzes the received noise data and emotion data, including parameters such as volume, frequency distribution, and duration. Based on the analysis results, the server determines which type of masking sound is optimal.

[1284] Step 5:

[1285] The server uses a generative AI model to generate optimal masking sounds, such as white noise or soft background music, based on the analyzed noise characteristics and the user's emotional state as determined by the emotion engine.

[1286] Step 6:

[1287] The generated masking sound is sent from the server to the terminal. The server sends packetized data and receives an acknowledgment from the terminal. The terminal decodes the received masking sound data and prepares for playback.

[1288] Step 7:

[1289] The device plays masking sounds through the speaker in real time, reducing ambient noise. The emotion engine also monitors the user's emotional state in real time and adjusts the masking sounds as needed.

[1290] Step 8:

[1291] Users can work and talk in a comfortable environment thanks to the generated masking sounds and optimization by the emotion engine. If the user feels stressed, the masking sounds are automatically adjusted to maintain comfort. When the user presses the system's shutdown button, all processes are stopped and the device's microphone, camera, emotion sensor, and speaker are turned off.

[1292] Example 2

[1293] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1294] In modern urban environments, noise levels are constantly increasing, causing stress for many people. Existing noise control systems focus on capturing sounds and generating masking sounds, but they do not take into account the user's emotional state, making it difficult to provide an optimal sound environment.

[1295] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing ambient sounds, means for capturing user emotional data, means for transmitting the captured sound data and emotional data to the server, means for the server to analyze the sound data and emotional data, means for generating an optimal masking sound using a generative AI model based on the analysis results, means for transmitting the generated masking sound to the terminal, and means for playing the generated masking sound. This makes it possible to provide an optimal sound environment according to the user's emotional state while effectively reducing noise.

[1296] The "means for capturing ambient sound" refers to a device or system for collecting sounds in the environment, and specifically refers to a microphone or the like.

[1297] "Means for capturing user emotional data" refers to a device or system for collecting the user's emotional state, and specifically includes a camera or a voice analysis sensor.

[1298] "Means for transmitting captured sound data and emotional data to a server" refers to a network connection device or system for communicating the collected data to a server, specifically Wi-Fi, Bluetooth, the Internet, etc.

[1299] "Means for the server to analyze sound data and emotional data" refers to software or algorithms for processing and analyzing sound data and emotional data received by the server.

[1300] "Means for generating optimal masking sound using an AI model based on the analysis results" refers to software that includes an artificial intelligence model or algorithm for generating optimal masking sound based on the analyzed data.

[1301] The "means for transmitting the generated masking sound to the terminal" refers to a network connection device or system for communicating the generated masking sound to the terminal, and specifically includes the Internet, Wi-Fi, Bluetooth, etc.

[1302] The "means for reproducing the generated masking sound" refers to a device or system for reproducing the received masking sound, and specifically refers to a speaker, a headphone, or the like.

[1303] MODE FOR CARRYING OUT THE INVENTION

[1304] The system of the present invention provides optimal masking sound based on the ambient noise and the user's emotional state by linking a server, a terminal, and an emotion engine. The system uses the following hardware and software:

[1305] Hardware and software used

[1306] 1. Device:

[1307] Microphone: A device for capturing ambient sounds.

[1308] Camera: A device for capturing a user's facial expressions.

[1309] Emotion sensors: Sensors for capturing the user's tone of voice and other vital signs.

[1310] Network connectivity: A means of communicating with the server (e.g., Wi-Fi, Bluetooth).

[1311] Speaker: A device for playing back the masking sound sent from the server.

[1312] 2. Server:

[1313] Data analysis module: An algorithm that analyzes noise data and emotion data sent from the device.

[1314] Generative AI model: A program that generates optimal masking sounds based on analysis results (e.g., OpenAI's GPT-3 and DALL-E).

[1315] Network connection function: A means for communicating with terminals.

[1316] 3. Emotion Engine:

[1317] Emotion analysis algorithm: Software for analyzing the emotional state of a user.

[1318] Operational flow and concrete examples

[1319] When the system is started, the device's microphone first captures the surrounding sounds. The captured sound data is converted into volume and frequency spectrum data and sent to the server. At the same time, the device's camera and emotion sensor collect the user's emotion data and send it to the server.

[1320] The server receives the noise and emotion data sent from the device and analyzes each data. The analysis uses a data analysis module and emotion analysis algorithm. Based on the analysis results, the server uses a generative AI model to generate optimal masking sounds. For example, if the noise is intense and the user is feeling stressed, it generates relaxing white noise or soft background music.

[1321] The generated masking sound is sent from the server to the device, which then receives it and plays it in real time through the speaker, effectively reducing ambient noise and providing the optimal sound environment according to the user's emotional state.

[1322] Specific examples

[1323] Example 1: Use in the office

[1324] A user in a meeting in a shared office space activates the system. The device's microphone captures ambient noise, while the camera and emotion sensor recognize the user's emotional state. The server analyzes the captured data and generates an optimal masking sound. The generated masking sound is played in real time through the device's speaker to prevent the content of the conversation from leaking to the outside. The system also adjusts according to the user's emotional state to reduce stress levels.

[1325] Example prompt:

[1326] "Generate masking sounds suitable for situations where people are having a meeting in a shared office space. Users are stressed and want a relaxing environment."

[1327] Example 2: Use at a cafe

[1328] A user working remotely in a cafe starts the system. Noise captured by the device's microphone and emotional state analyzed by the emotion engine are sent to the server, which then analyzes the data and generates an optimal masking sound. The generated masking sound is played back through the device's speaker, effectively canceling out the cafe's noise and providing an environment suited to the user's emotional state.

[1329] Example prompt:

[1330] "For users working remotely in a cafe, please generate masking sounds that cancel out surrounding noise and make it easier for them to concentrate."

[1331] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1332] Step 1:

[1333] The user starts the system on the device. The device first initializes and enables the built-in microphone, camera, and emotion sensor, ensuring that the hardware is operational and ready to begin capturing data.

[1334] Specific behavior:

[1335] The user clicks the "System Start" button.

[1336] Input: User action (clicking the launch button)

[1337] Output: Device hardware initialization complete

[1338] Step 2:

[1339] The device's microphone begins capturing ambient sound, which is then converted into volume and frequency spectrum data, which is then sent to the server.

[1340] Specific behavior:

[1341] The device's microphone records the surrounding sounds and samples the waveform data in real time, converts the data into a digital format (e.g., PCM format), and sends it to the server.

[1342] Input: Ambient sound

[1343] Output: Volume and frequency spectrum converted data

[1344] Step 3:

[1345] The device's camera and emotion sensors capture the user's facial expressions and voice tone to collect emotion data, which is also sent to the server.

[1346] Specific behavior:

[1347] The device's camera captures the user's face and analyzes the movements of each part of the face (eyes, mouth, eyebrows, etc.). The audio sensor also analyzes the user's tone of voice and speaking style. This data is combined to determine the user's emotional state and is sent to the server.

[1348] Input: User facial expressions and voice tone

[1349] Output: Emotion data

[1350] Step 4:

[1351] The server receives the noise and emotion data sent from the device. The received data is processed by the analysis module. The server uses a proprietary algorithm to analyze the characteristics of the noise and the user's emotional state.

[1352] Specific behavior:

[1353] The server analyzes the characteristics of the received noise data and identifies the sound volume, frequency components, source, etc. At the same time, it detects the user's current emotional state (stress level, relaxation level) based on the received emotional data.

[1354] Input: Noise data and emotion data

[1355] Output: Noise analysis results and emotion analysis results

[1356] Step 5:

[1357] The server uses a generative AI model to generate optimal masking sounds based on the analysis results. The generated masking sounds are tailored to the user's emotional state, such as white noise or relaxing music.

[1358] Specific behavior:

[1359] Based on the analysis results, the server inputs prompts to the generative AI model, which then designs appropriate masking sounds and returns the sound data.

[1360] Input: Noise analysis results and emotion analysis results

[1361] Output: Generated masking sound

[1362] Example prompt:

[1363] "For users working remotely in a cafe, please generate masking sounds that cancel out surrounding noise and make it easier for them to concentrate."

[1364] Step 6:

[1365] The generated masking sound is sent from the server to the device, which decodes the received sound data and plays it back in real time through the speaker.

[1366] Specific behavior:

[1367] The server sends the generated masking sound to the device, which then decodes the received sound data and plays it back using the speaker, allowing the user to work in a comfortable environment.

[1368] Input: Generated masking sound

[1369] Output: Played masking sound

[1370] (Application example 2)

[1371] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1372] In the food delivery industry, delivery staff often experience mental stress due to traffic noise and other disturbances at delivery destinations. This reduces the efficiency of delivery work and staff comfort. The present invention aims to provide an environment in which delivery staff can perform their work comfortably and efficiently.

[1373] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing ambient sounds, means for analyzing the captured sound data and the user's emotional state, and means for generating an optimal masking sound based on the analysis results and the user's emotional state. This effectively masks the noise around the delivery staff, enabling them to perform their work while reducing mental stress.

[1374] The "means for capturing ambient sound" is a device including a microphone and a sensor for collecting environmental sounds around the delivery staff.

[1375] "Means for analyzing captured sound data and the user's emotional state" refers to algorithms or engines that analyze the collected sound data and the emotional state obtained from the delivery staff's facial expressions, vocal tone, etc.

[1376] The "means for generating optimal masking sounds based on the analysis results and the user's emotional state" refers to a generative AI model or sound engine that generates optimal sounds to reduce the mental stress of delivery staff based on noise data and emotional data.

[1377] The "means for reproducing the generated masking sound" refers to a speaker or audio reproduction device for reproducing the generated masking sound in real time.

[1378] The "means for transmitting captured sound data and emotional data to a server" refers to a communication module or protocol for transmitting the collected sound data and emotional data to a server via a network.

[1379] "The means by which the server analyzes sound data and emotional data and generates optimal masking sounds" refers to a function that uses an analysis algorithm or generation AI model running on the server to analyze collected data and generate masking sounds.

[1380] The "means for transmitting the generated masking sound to the terminal" is a communication function for transmitting the masking sound generated by the server to the terminal of the delivery staff via the network.

[1381] A "generative AI model" is a type of artificial intelligence model used for data analysis and sound generation, and is an algorithm that is particularly suited to generating voice and music.

[1382] A "prompt" is a set of instructions or questions input to a generative AI model to obtain a desired output result.

[1383] The food delivery system that embodies this invention operates through the cooperation of a server, terminals, and users in order to improve the comfort and work efficiency of delivery staff. The program of this system is configured as follows.

[1384] Hardware and software used

[1385] Hardware

[1386] 1. Smartphone

[1387] microphone

[1388] speaker

[1389] camera

[1390] Network connectivity features

[1391] software

[1392] 1. Smartphone App

[1393] Environmental sound and emotion data capture function using microphone and camera

[1394] Communication with the server

[1395] Masking sound decoding and playback function

[1396] 2. Server

[1397] Receiving and analyzing noise and emotion data

[1398] Generative AI models (e.g., DeepMind WaveNet, OpenAI GPT-4)

[1399] Emotion engine (e.g. Microsoft Azure Emotion API)

[1400] Program processing details

[1401] The server receives and analyzes the sound data and emotional data sent from the device. The analysis involves checking the volume and frequency spectrum of the sound data, and using algorithms to determine the user's stress level and emotional state for the emotional data. Based on the analysis results, the server uses a generative AI model to generate optimal masking sounds. These masking sounds help reduce the user's mental stress and provide a comfortable environment.

[1402] The masking sound is transmitted from the server to the device and played in real time through the smartphone speaker, effectively canceling out surrounding noise for delivery staff, allowing them to continue working in a relaxed state.

[1403] Examples of concrete examples and prompts

[1404] Specific examples

[1405] When a delivery person launches the smartphone app, the smartphone's microphone begins capturing surrounding traffic and construction sounds. At the same time, a camera analyzes the delivery person's facial expressions to detect temporary highs and lows of stress. The server receives this data and instructs a generative AI model to generate soothing music with a subdued bass rhythm. This sound is then transmitted to the smartphone and played through the speaker, effectively canceling out the noise around the delivery person.

[1406] Prompt Sentence Examples

[1407] The following prompt sentence is input into a generative AI model (e.g., OpenAI GPT-4) to generate the optimal masking sound based on noise analysis and emotion data.

[1408] Consider the surrounding environmental noise (60dB, low frequency) and emotional state (tension, high temporary stress level), and generate music that effectively masks these noises and promotes relaxation. For example, soothing music with a low bass rhythm or a blend of nature sounds would be suitable.

[1409] Output sound requirements

[1410] Frequency range: 20Hz - 20kHz

[1411] Volume: 45dB or less

[1412] Sound quality: Clear, relaxing

[1413] In this way, the present invention aims to reduce the noise around delivery staff in real time and relieve their mental stress.

[1414] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1415] Step 1:

[1416] When a user activates the system on their smartphone, the device's microphone starts capturing ambient sounds, while the camera and emotion sensor simultaneously capture the user's facial expressions, voice tone, and other emotional data.

[1417] Input: ambient sound data, user emotion data

[1418] Output: Sound data and emotion data collected on the device

[1419] Step 2:

[1420] The device transmits the collected sound data and emotion data to the server via the network, converts the data into an appropriate format, and connects to the server using a communication module.

[1421] Input: Sound data and emotion data collected on the device

[1422] Output: Sound data and emotion data sent to the server

[1423] Step 3:

[1424] The server analyzes the received sound data and emotional data. The sound data is analyzed to check the volume and frequency spectrum, and the emotional data is processed through algorithms to determine the user's stress level and emotional state.

[1425] Input: Sound data and emotion data sent to the server

[1426] Output: Analysis results (sound data characteristics, emotional data status)

[1427] Step 4:

[1428] The server inputs a prompt sentence into the generative AI model based on the analysis results to generate the optimal masking sound. The generative AI model follows the process to create the optimal masking sound that can reduce the user's mental stress.

[1429] Input: Analysis results (sound data characteristics, emotional data state), prompt

[1430] Output: Generated masking sound

[1431] Step 5:

[1432] The server transmits the generated masking sound to the terminal via the network, and the server appropriately formats the transmission data and transmits it to the terminal using a communication protocol.

[1433] Input: Generated masking sound

[1434] Output: Masking sound sent to the device

[1435] Step 6:

[1436] The device decodes the masking sound received from the server and plays it in real time through the speaker, effectively masking surrounding noise and allowing the user to work in a more comfortable environment.

[1437] Input: Masking sound sent to the device

[1438] Output: Masking sound played through the device speaker

[1439] Examples of concrete examples and prompts

[1440] Specific examples

[1441] When a delivery person launches the smartphone app, the smartphone's microphone begins capturing surrounding traffic and construction sounds. At the same time, a camera analyzes the delivery person's facial expressions to detect temporary highs and lows of stress. The server receives this data and instructs a generative AI model to generate soothing music with a subdued bass rhythm. This sound is then transmitted to the smartphone and played through the speaker, effectively canceling out the noise around the delivery person.

[1442] Prompt Sentence Examples

[1443] The following prompt sentence is input into a generative AI model (e.g., OpenAI GPT-4) to generate the optimal masking sound based on noise analysis and emotion data.

[1444] Consider the surrounding environmental noise (60dB, low frequency) and emotional state (tension, high temporary stress level), and generate music that effectively masks these noises and promotes relaxation. For example, soothing music with a low bass rhythm or a blend of nature sounds would be suitable.

[1445] Output sound requirements

[1446] Frequency range: 20Hz - 20kHz

[1447] Volume: 45dB or less

[1448] Sound quality: Clear, relaxing

[1449] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1450] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1451] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1452] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1453] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1454] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1455] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1456] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1457] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1458] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1459] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1460] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1461] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1462] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1463] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1464] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1465] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1466] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1467] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1468] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1469] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1470] The following is further disclosed regarding the above embodiment.

[1471] (Claim 1)

[1472] a means for capturing ambient sounds;

[1473] A means for analyzing the captured sound data;

[1474] means for generating an optimal masking sound based on the analysis results;

[1475] means for playing the generated masking sound;

[1476] A system including:

[1477] (Claim 2)

[1478] means for transmitting the captured sound data to a server;

[1479] A means for the server to analyze sound data and generate an optimal masking sound;

[1480] means for transmitting the generated masking sound to a terminal;

[1481] The system of claim 1 further comprising:

[1482] (Claim 3)

[1483] means for generating and playing the masking sound in real time;

[1484] a means for capturing sound data including volume and frequency spectrum;

[1485] The system of claim 1 further comprising:

[1486] "Example 1"

[1487] (Claim 1)

[1488] a means for capturing ambient sounds;

[1489] A means of converting the captured sound data into volume and frequency spectrum;

[1490] means for transmitting the converted sound data to a server;

[1491] A means for the server to analyze the sound data and extract the characteristics of the noise;

[1492] A means for generating an optimal masking sound using a generative AI model; and

[1493] means for transmitting the generated masking sound to a terminal;

[1494] means for the device to decode the masking sound and play it through a speaker;

[1495] means for playing the generated masking sound;

[1496] A system including:

[1497] (Claim 2)

[1498] means for the generative AI model to generate an optimal masking sound using a prompt sentence;

[1499] The system of claim 1 further comprising:

[1500] (Claim 3)

[1501] means for generating and playing the masking sound in real time;

[1502] a means for capturing sound data including volume and frequency spectrum;

[1503] The system of claim 1 further comprising:

[1504] "Application Example 1"

[1505] (Claim 1)

[1506] a means for capturing ambient sounds;

[1507] A means for analyzing the captured sound data;

[1508] means for generating an optimal masking sound based on the analysis results;

[1509] means for playing the generated masking sound;

[1510] a means for analyzing noise in the virtual environment and providing an optimal sound environment to a display device worn by a user;

[1511] A system including:

[1512] (Claim 2)

[1513] means for transmitting the captured sound data to a server;

[1514] A means for the server to analyze sound data and generate an optimal masking sound;

[1515] means for transmitting the generated masking sound to a terminal;

[1516] a means for connecting to a display device worn by a user to provide an optimal sound environment;

[1517] The system of claim 1 further comprising:

[1518] (Claim 3)

[1519] means for generating and playing the masking sound in real time;

[1520] a means for capturing sound data including volume and frequency spectrum;

[1521] a means for responding to dynamic changes in noise in the virtual environment;

[1522] The system of claim 1 further comprising:

[1523] "Example 2: Combining Emotion Engines"

[1524] (Claim 1)

[1525] a means for capturing ambient sounds;

[1526] means for capturing user emotion data;

[1527] means for transmitting the captured sound data and emotion data to a server;

[1528] means for the server to analyze sound data and emotion data;

[1529] A means for generating optimal masking sounds using an AI model based on the analysis results;

[1530] means for transmitting the generated masking sound to a terminal;

[1531] means for playing the generated masking sound;

[1532] A system including:

[1533] (Claim 2)

[1534] The system of claim 1 , wherein the masking sound is generated and played in real time.

[1535] (Claim 3)

[1536] 10. The system of claim 1, wherein the captured sound data includes volume and frequency spectrum.

[1537] "Application example 2 when combining emotion engines"

[1538] (Claim 1)

[1539] a means for capturing ambient sounds;

[1540] means for analyzing the captured sound data and the user's emotional state;

[1541] means for generating an optimal masking sound based on the analysis result and the emotional state of the user;

[1542] means for playing the generated masking sound;

[1543] A system including:

[1544] (Claim 2)

[1545] means for transmitting the captured sound data and emotion data to a server;

[1546] a means for the server to analyze sound data and emotion data and generate an optimal masking sound;

[1547] means for transmitting the generated masking sound to a terminal;

[1548] The system of claim 1 further comprising:

[1549] (Claim 3)

[1550] means for generating and playing the masking sound in real time;

[1551] a means for capturing sound data including volume and frequency spectrum;

[1552] a means for generating a masking sound using a generative AI model;

[1553] The system of claim 1 further comprising: [Explanation of symbols]

[1554] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for capturing ambient sounds; A means for analyzing the captured sound data; means for generating an optimal masking sound based on the analysis results; means for playing the generated masking sound; A system including:

2. means for transmitting the captured sound data to a server; A means for the server to analyze sound data and generate an optimal masking sound; means for transmitting the generated masking sound to a terminal; The system of claim 1 further comprising:

3. means for generating and playing the masking sound in real time; a means for capturing sound data including volume and frequency spectrum; The system of claim 1 further comprising:

Images