System

Abstract

To solve the problem that a conventional noise countermeasure reproduces a fixed masking sound and cannot exhibit a sufficient effect to noise varying in real time in many cases.SOLUTION: The system includes a means for detecting noise, a means for analyzing noise data, and a means for reproducing the generated masking sound. Specifically, the surrounding noise is monitored in real time using a means for detecting the noise, and the type and characteristics of the noise are analyzed using a means for analyzing the detected noise data. Then, an optimal masking sound is generated by using the generated AI, and the generated masking sound is provided to the user through a means for reproducing the generated masking sound. As a result, it is possible to provide an optimum and comfortable sound environment for the user in response to a noise environment that varies in real time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, noise problems have been increasing in urban areas, offices, and homes, making it difficult to maintain comfortable living and working environments. Conventional noise countermeasures involve the reproduction of fixed masking sounds, which are often ineffective against noise that fluctuates in real time. Furthermore, there are limited means of providing masking sounds that are optimal for individual environments. This poses a challenge, making it difficult to provide a truly comfortable sound environment for users. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means. The system includes a noise detection means, a noise data analysis means, and a generated masking sound playback means. Specifically, the noise detection means is used to monitor ambient noise in real time, and the detected noise data analysis means is used to analyze the type and characteristics of the noise. Then, an optimal masking sound is generated using a generation AI, and the generated masking sound is provided to the user through a playback means. This makes it possible to respond to noise environments that change in real time and provide the user with an optimal and comfortable sound environment.

[0006] "Noise detection means" means a device, technology, or program that monitors ambient sounds in real time and detects a certain noise level.

[0007] "Means for analyzing noise data" refers to technology, programs, or devices for identifying and analyzing the characteristics and types of sound based on detected noise data.

[0008] "Means for playing the generated masking sound" refers to a device, technology, or program that plays the sound to provide the user with the masking sound generated based on the analysis results.

[0009] The "server" is a computer system for storing and analyzing data and generating masking sounds using AI.

[0010] A "terminal" is a device that detects noise and communicates with a server to send and receive data.

[0011] "User" refers to an individual or group who uses the system to seek a comfortable sound environment.

[0012] "Generative AI" is a program or technology that uses artificial intelligence technology to analyze noise data and generate appropriate masking sounds. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention relates to a system that detects and analyzes noise, and generates and reproduces an optimal masking sound based on the noise. Specific embodiments of this system will be described below.

[0035] System configuration

[0036] The system consists of the following main components:

[0037] 1. Terminal: A device that detects noise and communicates with the server.

[0038] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0039] 3. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0040] Program processing

[0041] The processing of the program by each component of the system will be described in detail below.

[0042] Noise detection and data transmission

[0043] The user places the device in a location where noise is a problem (e.g., a conference room or office). The device uses a built-in microphone to monitor the surrounding sounds in real time. When noise exceeding a certain threshold is detected, the device records the sound. The noise data includes the sound's waveform information, frequency components, and sound intensity. The recorded noise data is then sent from the device to a server.

[0044] Noise data analysis

[0045] The server analyzes the received noise data, specifically analyzing the type of sound and frequency components to identify the type of noise. This analysis uses machine learning algorithms and acoustic processing technology.

[0046] Masking sound generation

[0047] The server's AI generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, it may determine that natural sounds (such as a flowing river or white noise) are optimal. The AI ​​selects the optimal masking sound from a large sound database and generates new masking sounds as needed.

[0048] Sending and playing masking sounds

[0049] The generated masking sound is sent from the server to the device, which then plays it back through its built-in speaker. By mixing this masking sound with the noise, the noise is effectively camouflaged, providing a comfortable sound environment for the user.

[0050] Specific examples

[0051] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. The generated waterfall sound data is sent to the device, and the waterfall sound is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly.

[0052] In this way, by closely coordinating each step, it is possible to respond to noise environments that change in real time and provide users with an optimal and comfortable sound environment.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user places the terminal in a location where noise is a concern, such as a conference room or office.

[0056] Step 2:

[0057] The device detects noise. The device's built-in microphone monitors the surrounding sounds in real time and detects noise that exceeds a certain threshold.

[0058] Step 3:

[0059] The device records the noise data. The device records the detected noise data, which includes sound waveform information, frequency components, volume, etc.

[0060] Step 4:

[0061] The device transmits the noise data to the server. The recorded noise data is transmitted from the device to the server.

[0062] Step 5:

[0063] The server receives the noise data. The server receives the noise data transmitted from the terminal.

[0064] Step 6:

[0065] The server analyzes the noise data and identifies the type and characteristics of the noise (e.g., construction noise, vehicle noise, talking, etc.).

[0066] Step 7:

[0067] The server generates the optimal masking sound. Based on the analysis results, the server's AI generates the optimal masking sound (e.g., white noise, natural sounds).

[0068] Step 8:

[0069] The server sends the masking sound to the terminal. The server then sends the generated masking sound to the terminal.

[0070] Step 9:

[0071] The terminal receives the masking sound. The terminal receives the masking sound transmitted from the server.

[0072] Step 10:

[0073] The device will play the masking sound, which will then be played through the built-in speaker to effectively camouflage the noise.

[0074] Step 11:

[0075] The user enjoys a comfortable sound environment. The user enjoys a comfortable environment in which noise is reduced by the masking sound.

[0076] Example 1

[0077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0078] Conventional noise masking systems have the problem of being unable to generate and play appropriate masking sounds in real time even when they detect noise. High analytical accuracy is required to select and generate appropriate masking sounds, and adjustments are also necessary to ensure that the masking sounds are comfortable for the user, but current technology has difficulty meeting these requirements.

[0079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0080] In this invention, the server includes means for detecting noise, means for recording noise data and transmitting it to the server, means for analyzing the noise data received by the server, means for generating an optimal masking sound based on the analysis results using a generative AI model, means for transmitting the generated masking sound to the terminal, and means for playing the masking sound received by the terminal. This makes it possible to accurately analyze noise in real time and provide the user with a comfortable masking sound.

[0081] The "noise detection means" is a function that uses a built-in microphone to monitor surrounding sounds in real time and detect noise that exceeds a certain threshold.

[0082] The "means for recording noise data and transmitting it to a server" is a function that records the waveform information, frequency components, sound intensity, etc. of the detected noise and transmits the data to a server using a secure communication protocol.

[0083] The "means for analyzing noise data received by the server" refers to a function that analyzes the received noise data using an acoustic processing library or machine learning algorithm to identify the type and characteristics of the noise.

[0084] "Means for generating optimal masking sounds based on analysis results using a generative AI model" refers to a function that uses a generative AI model based on analysis results to select or generate optimal masking sounds.

[0085] The "means for transmitting the generated masking sound to the terminal" is a communication function for transferring the generated masking sound data to the terminal.

[0086] The "means for reproducing the masking sound received by the terminal" is a function for reproducing the masking sound data received by the terminal through the built-in speaker.

[0087] A "generative AI model" is an artificial intelligence algorithm that generates or selects the optimal sound in response to a specific prompt.

[0088] A "prompt sentence" is an input sentence used to generate a specific output for a generative AI model.

[0089] This invention relates to a system that detects and analyzes noise, and generates and reproduces an optimal masking sound based on the noise. Specific embodiments of this system will be described below.

[0090] System Configuration

[0091] The system consists of the following main components:

[0092] 1. Terminal: A device that detects noise and communicates with the server.

[0093] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0094] 3. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0095] Noise Monitoring

[0096] The user places the device in a location where noise is a problem. The device uses a built-in, highly sensitive microphone to monitor the surrounding sounds in real time. For monitoring, an acoustic library (e.g., Librosa) is used to acquire sound waveform information, frequency components, and sound intensity.

[0097] Noise recording and transmission

[0098] When the device detects noise above a certain threshold, it records the data. The recording includes high-resolution waveform data, frequency content, and sound intensity. The recorded noise data is then sent from the device to a server via a secure communication protocol (e.g., HTTPS).

[0099] Noise data analysis

[0100] The server analyzes the received noise data. Machine learning algorithms (e.g., CNN, RNN) are used to identify the type of sound and its frequency components. The server then provides a prompt to the generative AI model to ask it for the optimal type of masking sound.

[0101] Masking sound generation

[0102] The generative AI model generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, the AI ​​model will select an appropriate natural sound (e.g., a waterfall or a flowing river) or generate a new one. The final masking sound data is stored on a server.

[0103] Sending and playing masking sounds

[0104] The server transmits the generated masking sound to the device, which then plays it back using its built-in speaker. This blends the noise and masking sound, providing a comfortable sound environment for the user.

[0105] Specific examples

[0106] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. The generated waterfall sound data is sent to the device, and the waterfall sound is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly.

[0107] Prompt Sentence Examples

[0108] Examples of prompts you can give to a generative AI model include:

[0109] "Construction noise detected. Please generate suitable masking sound."

[0110] "Based on the frequency content and waveform data, please suggest a sound that will effectively camouflage this noise."

[0111] By using such prompt sentences to have the generative AI model perform advanced sound generation, it is possible to provide optimal masking sounds in real time.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1: Monitoring the noise

[0114] The user places the device in a location where noise is a problem. The device uses a built-in, highly sensitive microphone to monitor the surrounding sounds in real time. For monitoring, an acoustic library (e.g., Librosa) is used to acquire sound waveform information, frequency components, and sound intensity.

[0115] Input: Real-time sound environment data

[0116] Output: Sound waveform information, frequency components, sound intensity data

[0117] Specific operation: The device's microphone captures surrounding sounds, and the acoustic library analyzes the captured sound data to obtain waveform information and frequency components in real time.

[0118] Step 2: Record and transmit the noise

[0119] When the device detects noise above a certain threshold, it records the data. The recording includes high-resolution waveform data, frequency content, and sound intensity. The recorded noise data is then sent to a server using a secure communication protocol (e.g., HTTPS).

[0120] Input: Monitored sound data

[0121] Output: Noise data sent to the server

[0122] Specific operation: When the device exceeds the noise threshold, it buffers the sound data and then transmits the saved noise data to the server through a secure communication protocol.

[0123] Step 3: Analyze the noise data

[0124] The server analyzes the received noise data. Machine learning algorithms (e.g., CNN or RNN) and acoustic processing libraries are used to identify the type and characteristics of the sound. Based on the analysis results, the server provides a prompt to the generative AI model.

[0125] Input: Transmitted noise data

[0126] Output: type and characteristics of noise, prompt

[0127] Specific operation: The server preprocesses the received sound data using an acoustic processing library, then inputs it into a machine learning model to identify the type and characteristics of the noise. Based on the identification results, it generates and sends a prompt to the generative AI model.

[0128] Step 4: Generate masking sounds

[0129] The generative AI model generates the optimal masking sound based on the prompt. For example, if the sound of construction work is detected, the AI ​​model will select an appropriate natural sound (e.g., a waterfall or a flowing river) or generate a new one. The generated masking sound data is stored on a server.

[0130] Input: prompt sentence, sound database

[0131] Output: Optimal masking sound data

[0132] Specific operation: The generative AI model searches for appropriate masking sounds from a sound database according to the prompt sentence, and generates new sounds as needed. The generated masking sound data is stored on the server.

[0133] Step 5: Send and play the masking sound

[0134] The server transmits the generated masking sound to the device, which then plays it back using its built-in speaker. This blends the noise and the masking sound, providing a comfortable sound environment for the user.

[0135] Input: Generated masking sound data

[0136] Output: Played masking sound

[0137] Specific operation: The server sends the generated masking sound data to the device via a secure communication protocol, and the device decodes the received masking sound and plays it on the built-in speaker. By playing the masking sound, the noise is effectively camouflaged.

[0138] (Application example 1)

[0139] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0140] Modern brick-and-mortar stores are required to provide a comfortable shopping environment for customers, but noise in the store can cause stress and reduce comfort. Noise can also make it difficult to accurately convey product explanations and guidance, reducing the quality of the customer experience. Therefore, there is a need for a system that can suppress noise in real time and provide a comfortable sound environment for customers.

[0141] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0142] In this invention, the server includes means for analyzing noise data and generating an optimal masking sound, means for generating the masking sound using a generation AI model, and means for transmitting the generated masking sound to a terminal or a wearable device. This makes it possible to generate an optimal masking sound adapted to a noisy environment in real time and play it on a terminal or a device held by the customer.

[0143] - "Means for detecting noise" means a device for recognizing sounds occurring in the environment and measuring and recording such sounds.

[0144] "Means for analyzing noise data" refers to algorithms or systems that process and analyze collected noise data and identify the type and characteristics of noise.

[0145] The "means for reproducing the generated masking sound" refers to an output device such as a speaker or earphones for providing the masking sound sent from the server to the user.

[0146] "Means for playing masking sounds generated to adapt to a noisy environment through an audio device worn by the user" refers to a mechanism by which the user plays the optimal masking sound through a mobile terminal or wearable device, effectively camouflaging the noise.

[0147] "Means for monitoring noise using mobile or wearable devices" refers to the ability to detect and record noise in real time using portable devices such as smartphones or smart glasses.

[0148] "Means for generating masking sounds using generative AI models" refers to technology that uses machine learning algorithms to generate optimal masking sounds based on the results of noise analysis.

[0149] The "prompt sentence output by the generative AI model" is an instruction sentence that the AI ​​uses to generate masking sounds based on the noise analysis results.

[0150] System configuration

[0151] A system embodying this invention mainly comprises the following components: means for detecting noise, means for analyzing noise data, means for playing back generated masking sounds, means for playing back masking sounds generated to adapt to the noisy environment through an acoustic device worn by the user, and means for monitoring noise using a mobile terminal or a wearable device.

[0152] Hardware and Software

[0153] Devices: Mobile and wearable devices such as smartphones and smart glasses that monitor and record noise in real time.

[0154] Server: A high-performance computer for analyzing noise data and generating masking sounds. It runs generative AI models using machine learning frameworks such as TENSORFLOW (registered trademark) and PyTorch.

[0155] An acoustic device, such as earphones, headphones, or the built-in speakers of smart glasses, that plays the generated masking sound.

[0156] Noise detection and data transmission

[0157] 1. The device detects noise and uses the built-in microphone to monitor the surrounding sound in real time. When noise exceeding a certain threshold is detected, the device records the sound. This noise data includes the sound's waveform information, frequency components, and sound intensity.

[0158] 2. The device sends the recorded noise data to the server.

[0159] Noise data analysis and masking sound generation

[0160] 1. The server analyzes the received noise data using a generative AI model built with TensorFlow and PyTorch. The analysis identifies the type of noise and its frequency components.

[0161] 2. Based on the analysis results, the generative AI model generates the optimal masking sound using a prompt sentence. For example, if the noise is a person speaking, the sound of light rain or wind will be determined to be an appropriate masking sound.

[0162] Example prompt sentence:

[0163] "Perform frequency analysis of human speech and generate non-disturbing natural sounds (e.g., rain or rustling leaves)."

[0164] Playback of masking sound

[0165] 1. The server generates masking sounds and sends them to the user's terminal or wearable device.

[0166] 2. The terminal or wearable device plays the received masking sound through earphones, headphones, or speakers in smart glasses.

[0167] Specific examples

[0168] Consider a specific example of how to make a customer's time in a store more comfortable. For example, a customer is selecting a product in a noisy physical store when noise occurs. A smartphone detects the noise and sends the data to a server. The server analyzes the noise as talking and uses a generative AI model to generate the sound of rain. This rain sound is sent to the customer's smartphone and played through earphones. This allows the customer to enjoy shopping in a comfortable acoustic environment.

[0169] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0170] Step 1:

[0171] The device detects noise and uses the built-in microphone to monitor surrounding sounds in real time. When the noise level exceeds a certain threshold, the device records the sound. At this time, the sound's waveform information, frequency components, and sound intensity are acquired as recorded data. The input is the surrounding ambient sound, and the output is noise data obtained from the microphone.

[0172] Step 2:

[0173] The device sends the recorded noise data to the server. The data includes sound waveform information, frequency components, and sound intensity. The input is the noise data acquired in step 1, and the output is a data packet sent to the server.

[0174] Step 3:

[0175] The server receives the noise data and begins analysis based on the received data. This analysis uses machine learning frameworks such as TensorFlow and PyTorch. The input is the data sent from the device, and the output is the analysis results that identify the type and characteristics of the noise.

[0176] Step 4:

[0177] The server uses a generative AI model to generate optimal masking sounds based on the analysis results. In this process, masking sounds are generated using predefined prompts. The inputs are the analysis results and prompts, and the output is the generated masking sound data. Specifically, the prompt used is, "Perform frequency analysis of human speech and generate non-interfering natural sounds (e.g., the sound of rain or rustling leaves)."

[0178] Step 5:

[0179] The server transmits the generated masking sound to a terminal or wearable device. The input is the generated masking sound data, and the output is a data packet for transmission.

[0180] Step 6:

[0181] The terminal or wearable device receives the masking sound data sent from the server and plays the received data on an acoustic device (such as earphones, headphones, or speakers in smart glasses). The input is the transmitted masking sound data, and the output is the played sound.

[0182] Step 7:

[0183] The user enjoys a comfortable sound environment through the masking sound that is played back. Specifically, the user listens to the masking sound created by the generative AI model as sound that reaches the ears through earphones or headphones.

[0184] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0185] This invention combines a system that detects and analyzes noise, and generates and plays optimal masking sounds based on the noise, with the ability to recognize the user's emotions and optimize the sound environment. Specific embodiments of this system are described below.

[0186] System configuration

[0187] The system consists of the following main components:

[0188] 1. Terminal: A device that detects noise and communicates with the server.

[0189] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0190] 3. Emotion engine: An engine that recognizes the user's emotions.

[0191] 4. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0192] Program processing

[0193] The processing of the program by each component of the system will be described in detail below.

[0194] Noise detection and data transmission

[0195] The user places the device in a location where noise is a problem (e.g., a conference room or office). The device uses a built-in microphone to monitor the surrounding sounds in real time. When noise exceeding a certain threshold is detected, the device records the sound and transmits the noise data to a server. This data includes the sound's waveform information, frequency components, and sound intensity.

[0196] Noise data analysis

[0197] The server analyzes the received noise data, specifically analyzing the type of sound and frequency components to identify the characteristics of the noise. This analysis uses machine learning algorithms and acoustic processing technology.

[0198] Masking sound generation

[0199] The server's AI generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, it may determine that natural sounds (such as a flowing river or white noise) are optimal. The AI ​​selects the optimal masking sound from a large sound database and, in some cases, generates a new masking sound.

[0200] Acquiring and analyzing emotion data

[0201] To further optimize the user's emotional state, the emotion engine captures and analyzes the user's facial expression, tone of voice, or biometric information in real time. For example, if the user's facial expression is recognized as tired or stressed, the emotion engine will send this information to the server.

[0202] Adjusting the masking sound

[0203] Based on the user's emotional data obtained from the emotion engine, the server further fine-tunes the generated masking sound. For example, if the user is feeling stressed, it may consider changing the sound to a more relaxing one (e.g., the sound of calming waves).

[0204] Sending and playing masking sounds

[0205] The generated masking sound (sound optimized for the user's emotions) is sent from the server to the device, which then plays it back through its built-in speaker. By mixing this masking sound with the noise, the noise is effectively camouflaged, providing the user with a comfortable sound environment that takes into account their emotional state.

[0206] Specific examples

[0207] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. At the same time, if the server determines that a meeting participant is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a quieter, gentler waterfall. The generated waterfall sound data is sent to the device, and the quiet, gentle sound of a waterfall is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly and participants to meet in a comfortable and relaxed environment.

[0208] In this way, by closely coordinating each step, it is possible to respond to changing noise environments in real time and provide users with an optimal, comfortable sound environment that also takes into account their emotional state.

[0209] The processing flow will be explained below.

[0210] Step 1:

[0211] The user places the terminal. The user places the terminal in a place where noise is a concern, such as a conference room or office.

[0212] Step 2:

[0213] The device detects noise. The device's built-in microphone monitors the surrounding sounds in real time and detects noise that exceeds a certain threshold.

[0214] Step 3:

[0215] The device records the noise data. The device records the detected noise data, which includes sound waveform information, frequency components, volume, etc.

[0216] Step 4:

[0217] The device transmits the noise data to the server. The recorded noise data is transmitted from the device to the server.

[0218] Step 5:

[0219] The server receives the noise data. The server receives the noise data transmitted from the terminal.

[0220] Step 6:

[0221] The server analyzes the noise data and performs analysis to identify the type and characteristics of the sound (e.g., construction noise, vehicle noise, voices, etc.) from the received data.

[0222] Step 7:

[0223] The server generates the optimal masking sound. Based on the analysis results, the server's AI automatically generates the optimal masking sound (e.g., white noise, natural sounds).

[0224] Step 8:

[0225] To recognize the user's emotions, the emotion engine captures the user's facial expressions and tone of voice in real time. The emotion engine collects the user's emotion data using a camera and microphone.

[0226] Step 9:

[0227] The emotion engine analyzes the user's emotional data, determines the user's emotional state from their facial expressions and tone of voice, and sends that data to the server.

[0228] Step 10:

[0229] The server receives and analyzes the emotional data and adjusts the masking sound. Based on the data from the emotion engine, the server fine-tunes the type and volume of the masking sound according to the user's emotional state.

[0230] Step 11:

[0231] The server sends the adjusted masking sound to the terminal. The optimized masking sound data is sent from the server to the terminal.

[0232] Step 12:

[0233] The device receives and plays the masking sound. The device plays the adjusted masking sound through the built-in speaker, effectively camouflaging the noise.

[0234] Step 13:

[0235] Users can enjoy a comfortable sound environment. By reducing noise and providing a sound environment that takes emotional state into consideration, users can enjoy a comfortable sound environment.

[0236] Example 2

[0237] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0238] Existing noise control systems simply detect noise and generate and play masking sounds, making it difficult to provide a flexible sound environment that responds to the user's emotional state. Furthermore, optimizing a noisy environment in real time requires integrated processing of noise detection, masking sound generation, and user emotion recognition, but no system currently exists that can do this efficiently.

[0239] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0240] In this invention, the server includes means for detecting noise, means for analyzing noise data, means for playing back the generated masking sound, means for recognizing and analyzing the emotional state of the user, and means for adjusting the masking sound based on the emotional state, thereby making it possible to provide a comfortable sound environment that takes the emotional state of the user into consideration.

[0241] "Means for detecting noise" refers to a device that monitors sounds in the environment in real time and detects sounds that exceed a threshold.

[0242] "Means for analyzing noise data" refers to technology that uses received sound data to analyze its frequency components, volume, etc., and identify specific noises.

[0243] "Means for reproducing the generated masking sound" refers to a device that selects an appropriate masking sound based on the analysis results and outputs it.

[0244] "Means for recognizing and analyzing the user's emotional state" refers to technology that acquires and analyzes the user's facial expressions, tone of voice, biometric information, etc. to identify the user's emotions.

[0245] The "means for adjusting the masking sound based on the emotional state" refers to a technique for adjusting the type and volume of the generated masking sound depending on the emotional state of the user.

[0246] "Means for transmitting noise data to a server" refers to a technology for transmitting noise data collected by a terminal to a server via a network.

[0247] The "means by which the server analyzes the noise data and generates an optimal masking sound" refers to a function by which the server analyzes the received noise data and generates an optimal masking sound.

[0248] The "means for transmitting the generated masking sound to the terminal" refers to a technique for transmitting the masking sound generated by the server to the terminal via a network.

[0249] "Means for acquiring a user's emotional state using their facial expressions, tone of voice, and biometric information" refers to devices such as cameras, microphones, and biometric sensors for acquiring a user's emotional state in real time, as well as the analysis technology for those devices.

[0250] MODE FOR CARRYING OUT THE INVENTION

[0251] This invention is a system that combines a system that detects and analyzes noise, and generates and plays optimal masking sounds based on the noise, with a function that recognizes the user's emotions and optimizes the sound environment. Specific embodiments of this system are described in detail below.

[0252] System configuration

[0253] The system consists of the following main components:

[0254] 1. Terminal: A device that detects noise and communicates with the server.

[0255] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0256] 3. Emotion engine: An engine that recognizes the user's emotions.

[0257] 4. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0258] Hardware and Software

[0259] Terminal

[0260] The device has a built-in microphone and speaker, a camera to recognize the user's facial expressions, a microphone to analyze the tone of voice, and sensors to capture biometric information. These devices are used to detect noise and capture emotional data.

[0261] server

[0262] The server requires powerful computing resources to analyze the noise data and generate masking sounds, using machine learning frameworks such as TensorFlow and PyTorch, as well as the Librosa acoustic processing library.

[0263] Emotion Engine

[0264] The emotion engine uses OpenFace to analyze the user's facial expressions and PyAudio to analyze audio data, making it possible to analyze the user's emotional state in real time.

[0265] Data processing and calculation

[0266] The system processes the following data:

[0267] 1. Noise detection and data transmission

[0268] The user places the device in a location where there is a noise problem. The device's built-in microphone monitors the surrounding sounds in real time, and when noise exceeding a certain threshold is detected, the device records the sound and transmits the noise data to the server. This data includes the sound's waveform information, frequency components, and sound intensity.

[0269] 2. Analysis of noise data

[0270] The server analyzes the received noise data, specifically by analyzing the frequency components of the sound using FFT (Fast Fourier Transform) and classifying the type of sound using a machine learning model.

[0271] 3. Masking sound generation

[0272] Based on the analysis results, the server's AI generates the optimal masking sound. The AI ​​selects the optimal masking sound from a large sound database (e.g., FreeSound) and, in some cases, generates a new masking sound.

[0273] 4. Acquisition and Analysis of Emotion Data

[0274] To further optimize the user's emotional state, the emotion engine captures and analyzes the user's facial expressions, tone of voice, or biometric information in real time through cameras, microphones, and biometric sensors.

[0275] 5. Adjusting the masking sound

[0276] Based on the data from the emotion engine, the server further adjusts the generated masking sounds, for example, changing them to more relaxing sounds if the user is feeling stressed.

[0277] 6. Sending and playing masking sounds

[0278] The generated masking sound is sent from the server to the device, which then plays it back through its built-in speaker. This sound blends with the noise, providing a comfortable sound environment for the user.

[0279] Specific examples

[0280] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. At the same time, if the server determines that a meeting participant is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a quieter, gentler waterfall. The generated waterfall sound data is sent to the device, and the quiet, gentle sound of a waterfall is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly and participants to meet in a comfortable and relaxed environment.

[0281] Prompt Sentence Examples

[0282] "There's construction noise during a meeting. Generate optimal sounds to mask the noise and adjust the sound based on the emotional data of the meeting participants."

[0283] In this way, this system can respond to changing noise environments in real time and provide users with an optimal, comfortable sound environment that also takes into account their emotional state.

[0284] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0285] Step 1: Noise detection and data transmission

[0286] The user places the device in a location where there is a noise problem. The device's built-in microphone monitors the surrounding sounds in real time, and if noise exceeding a certain threshold is detected, the device records the sound. The recorded data includes the sound's waveform information, frequency components, and sound intensity. After recording, the device compresses the data and sends it to the server.

[0287] Input: Ambient sound

[0288] Output: Compressed noise data

[0289] Specific operation: The device samples the sound waveform data and compares it with a threshold. If the threshold is exceeded, the device starts recording, then compresses the data and sends it to the server.

[0290] Step 2: Analyze the noise data

[0291] The server receives the noise data sent from the device, analyzes the frequency components using FFT (Fast Fourier Transform), and classifies the type of sound using a machine learning model (e.g., random forest or neural network).The analysis results are stored in a database.

[0292] Input: Compressed noise data

[0293] Output: Analysis results (sound frequency components and types)

[0294] Specific operation: The server reads the received data, applies FFT to extract frequency components, and classifies the sound type using a machine learning model. The analysis results are saved in a database.

[0295] Step 3: Generate the masking sound

[0296] The server's generation AI generates the optimal masking sound based on the analysis results. The generation AI selects the optimal masking sound from many sound databases (e.g., FreeSound) and generates new masking sounds as needed. The generated sound data is saved as a temporary file.

[0297] Input: Analysis results (sound frequency components and types)

[0298] Output: Generated masking sound data

[0299] Specific operation: The server's generation AI selects the optimal masking sound from the database based on the analysis results. If necessary, it generates new sounds and saves them in a temporary file.

[0300] Step 4: Acquire and analyze emotion data

[0301] The emotion engine installed in the device captures and analyzes the user's facial expressions, tone of voice, and biometric information in real time. Specifically, the emotion engine analyzes the data using tools such as OpenFace and PyAudio, and transmits the obtained emotional state to the server.

[0302] Input: User's facial expression, tone of voice, biometric information

[0303] Output: Parsed emotion data

[0304] Specific operation: The device captures the user's facial expressions and voice using the camera and microphone, analyzes them using OpenFace and PyAudio, and sends the analysis results to the server.

[0305] Step 5: Adjust the masking sound

[0306] The server receives the emotion data and adjusts the generated masking sound. If the user is feeling stressed, it changes the sound to a relaxing one. The adjusted sound data is also saved in a temporary file.

[0307] Input: Analyzed emotion data, generated masking sound data

[0308] Output: Adjusted masking sound data

[0309] Specific operation: The server analyzes the received emotional data and adjusts the generated masking sound based on the user's emotional state. The adjusted sound is saved in a temporary file.

[0310] Step 6: Send and play masking sounds

[0311] The server sends the adjusted masking sound data to the device, which then decodes the received masking sound and plays it through its built-in speaker, effectively camouflaging the noise.

[0312] Input: Adjusted masking sound data

[0313] Output: Masking sound played through the speaker

[0314] Specific operation: The server sends the generated sound data to the device, which decodes it and plays it through the speaker, optimizing the sound environment.

[0315] Through these steps, it becomes possible to provide a comfortable sound environment for users and optimize the noise environment in real time.

[0316] (Application example 2)

[0317] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] In modern brick-and-mortar retail environments, noise is a major problem that disrupts customers' comfortable shopping experiences. Furthermore, there is a need to provide a sound environment that not only reduces noise but also takes into account the emotional state of the customer. However, conventional noise masking systems only detect noise data and generate masking sounds, but do not support optimizing the sound environment based on the user's emotions. This poses a challenge in improving customer satisfaction in stores.

[0319] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0320] In this invention, the server includes means for analyzing noise data, means for generating the generated masking sound, means for recognizing the user's emotion and optimizing the sound environment, and means for transmitting the user's emotion data to the server. This makes it possible to centrally manage everything from noise detection and analysis to the generation and playback of masking sound according to the emotion, and to provide the optimal sound environment for customers in a physical store.

[0321] The "means for detecting noise" is a function that monitors sounds in the environment in real time and detects noise that exceeds a certain threshold.

[0322] The "means for analyzing noise data" is a function for analyzing detected noise data based on the type of sound, frequency components, intensity, etc.

[0323] The "means for reproducing the generated masking sound" is a function for reproducing the masking sound generated based on the analysis results through an audio output device such as a speaker.

[0324] "Means to recognize the user's emotions and optimize the sound environment" refers to a function that analyzes the user's facial expressions, tone of voice, biometric information, etc. to recognize emotions in real time, and selects and adjusts the optimal masking sound based on that.

[0325] "Means for transmitting noise data to a server" refers to a function for transmitting detected and recorded noise data to a server via a communication means such as the Internet.

[0326] The "means for the server to analyze noise data and generate the most suitable masking sound" is a function that analyzes the noise data received by the server and generates the most suitable masking sound.

[0327] The "means for transmitting the generated masking sound to the terminal" is a function for transmitting the optimal masking sound generated by the server to the terminal via a communication means such as the Internet.

[0328] The "means for transmitting user emotion data to the server" is a function for transmitting data obtained by analyzing the user emotion to the server.

[0329] "Means for adjusting to provide a comfortable sound environment" refers to a function that further fine-tunes the masking sounds generated and selected based on the user's emotions and noise characteristics to further enhance user comfort.

[0330] The present invention provides a system that effectively masks noise in environments such as brick-and-mortar stores, while providing a sound environment that takes into consideration the emotional state of the user. Specific embodiments for carrying out the present invention are described below.

[0331] System Configuration

[0332] This system consists of a terminal that detects noise, a server that performs analysis and generation, an emotion engine that recognizes the user's emotions, and a user.

[0333] Noise detection

[0334] The device has a built-in microphone for real-time noise monitoring. It constantly monitors the sounds in the environment, and when it detects noise exceeding a certain threshold, it records the sound's waveform information, frequency components, sound intensity, etc., and recognizes it as noise data.

[0335] Noise data transmission and analysis

[0336] The device sends the detected noise data to a server, which then analyzes the received noise data to identify the type and characteristics of the sound. This analysis is performed using machine learning algorithms and acoustic processing technologies (e.g., TensorFlow, PyTorch, etc.).

[0337] Masking sound generation

[0338] The server generates optimal masking sounds based on the analysis results. For example, natural sounds, especially the sound of a waterfall or white noise, may be suitable for noise caused by construction work. Using a generative AI model, optimal masking sounds are selected from an existing database or newly generated.

[0339] Emotion Recognition and Data Transmission

[0340] The user's emotional data (e.g., facial expressions, tone of voice, biometric information, etc.) recognized on the device is analyzed through the emotion engine. The analysis is performed using a camera and microphone, and machine learning models (e.g., OpenCV, TensorFlow) are used for emotion recognition. The recognized emotional data is then sent to the server.

[0341] Adjusting and playing masking sounds

[0342] The server receives the emotional data and further adjusts the masking sound based on the user's emotional state (for example, if the user is stressed, the sound may be changed to a more relaxing sound). The adjusted masking sound is then sent to the device and played through the device's speaker.

[0343] Specific examples

[0344] For example, imagine a cafe in a shopping mall with many customers and different types of noises occurring in the background. The device detects the noise and sends the data to the server. The server analyzes the data and generates the sound of a flowing river as the optimal masking sound. At the same time, if the server determines that the customer is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a more gentle sound of a flowing river. The adjusted sound is sent to the device and played in the cafe, allowing the customer to enjoy a comfortable acoustic environment.

[0345] Prompt Sentence Examples

[0346] "Analyze current noise data and generate optimal masking sound."

[0347] "Recognize the emotions from the user's facial expressions and select sounds that have a relaxing effect."

[0348] This makes it possible to realize a system that can solve noise problems in brick-and-mortar stores and provide a comfortable sound environment for customers. This invention is an important technology for providing a relaxing environment for store users.

[0349] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0350] Step 1:

[0351] The device monitors the noise in the environment in real time and detects noise exceeding a certain threshold. The input is the environmental sound picked up by the device's microphone. The output is the detected noise data, which includes sound waveform information, frequency components, sound intensity, etc.

[0352] Step 2:

[0353] The terminal transmits the detected noise data to a server via the Internet. This process uses a transmission protocol (e.g., HTTP). The input is the detected noise data, and the output is a confirmation of transmission to the server.

[0354] Step 3:

[0355] The server analyzes the received noise data. This analysis uses machine learning algorithms (e.g., TensorFlow, PyTorch) to identify the type of sound and its frequency components. The input is the transmitted noise data, and the output is the analysis results.

[0356] Step 4:

[0357] The server generates the optimal masking sound based on the analysis results. A generative AI model is used for generation. The input is the analysis results, and the output is the generated masking sound. The generated sound is selected from natural sounds (such as the sound of a waterfall or white noise).

[0358] Step 5:

[0359] The user's emotional data (e.g., facial expressions, tone of voice, biometric information, etc.) recognized by the device is analyzed through the emotion engine. Real-time data from the camera and microphone is input for this analysis. The analysis results are sent to the server. The input is the user's real-time data, and the output is emotional data.

[0360] Step 6:

[0361] The server receives the emotional data and adjusts the masking sound based on the user's emotional state. For example, if the user is feeling stressed, the sound is changed to one with a relaxing effect. The input is the emotional data and the generated masking sound, and the output is the adjusted masking sound.

[0362] Step 7:

[0363] The adjusted masking sound is sent to the device and played back from the device's speaker. The input is the adjusted masking sound, and the output is the sound to be played back. This sound is mixed with the noise to create a comfortable sound environment for the user.

[0364] Through this series of steps, noise detection, analysis, generation of masking sounds, adjustment based on the user's emotions, and playback are all seamlessly performed, providing a comfortable sound environment for users in physical stores. This system not only effectively solves noise problems, but also provides a new experience that takes into account the user's emotional state.

[0365] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0366] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0367] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0368] [Second embodiment]

[0369] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0370] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0371] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0372] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0373] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0374] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0375] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0376] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0377] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0378] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0379] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0380] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0381] This invention relates to a system that detects and analyzes noise, and generates and reproduces an optimal masking sound based on the noise. Specific embodiments of this system will be described below.

[0382] System configuration

[0383] The system consists of the following main components:

[0384] 1. Terminal: A device that detects noise and communicates with the server.

[0385] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0386] 3. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0387] Program processing

[0388] The processing of the program by each component of the system will be described in detail below.

[0389] Noise detection and data transmission

[0390] The user places the device in a location where noise is a problem (e.g., a conference room or office). The device uses a built-in microphone to monitor the surrounding sounds in real time. When noise exceeding a certain threshold is detected, the device records the sound. The noise data includes the sound's waveform information, frequency components, and sound intensity. The recorded noise data is then sent from the device to a server.

[0391] Noise data analysis

[0392] The server analyzes the received noise data, specifically analyzing the type of sound and frequency components to identify the type of noise. This analysis uses machine learning algorithms and acoustic processing technology.

[0393] Masking sound generation

[0394] The server's AI generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, it may determine that natural sounds (such as a flowing river or white noise) are optimal. The AI ​​selects the optimal masking sound from a large sound database and generates new masking sounds as needed.

[0395] Sending and playing masking sounds

[0396] The generated masking sound is sent from the server to the device, which then plays it back through its built-in speaker. By mixing this masking sound with the noise, the noise is effectively camouflaged, providing a comfortable sound environment for the user.

[0397] Specific examples

[0398] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. The generated waterfall sound data is sent to the device, and the waterfall sound is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly.

[0399] In this way, by closely coordinating each step, it is possible to respond to noise environments that change in real time and provide users with an optimal and comfortable sound environment.

[0400] The processing flow will be explained below.

[0401] Step 1:

[0402] The user places the terminal in a location where noise is a concern, such as a conference room or office.

[0403] Step 2:

[0404] The device detects noise. The device's built-in microphone monitors the surrounding sounds in real time and detects noise that exceeds a certain threshold.

[0405] Step 3:

[0406] The device records the noise data. The device records the detected noise data, which includes sound waveform information, frequency components, volume, etc.

[0407] Step 4:

[0408] The device transmits the noise data to the server. The recorded noise data is transmitted from the device to the server.

[0409] Step 5:

[0410] The server receives the noise data. The server receives the noise data transmitted from the terminal.

[0411] Step 6:

[0412] The server analyzes the noise data and identifies the type and characteristics of the noise (e.g., construction noise, vehicle noise, talking, etc.).

[0413] Step 7:

[0414] The server generates the optimal masking sound. Based on the analysis results, the server's AI generates the optimal masking sound (e.g., white noise, natural sounds).

[0415] Step 8:

[0416] The server sends the masking sound to the terminal. The server then sends the generated masking sound to the terminal.

[0417] Step 9:

[0418] The terminal receives the masking sound. The terminal receives the masking sound transmitted from the server.

[0419] Step 10:

[0420] The device will play the masking sound, which will then be played through the built-in speaker to effectively camouflage the noise.

[0421] Step 11:

[0422] The user enjoys a comfortable sound environment. The user enjoys a comfortable environment in which noise is reduced by the masking sound.

[0423] Example 1

[0424] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0425] Conventional noise masking systems have the problem of being unable to generate and play appropriate masking sounds in real time even when they detect noise. High analytical accuracy is required to select and generate appropriate masking sounds, and adjustments are also necessary to ensure that the masking sounds are comfortable for the user, but current technology has difficulty meeting these requirements.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0427] In this invention, the server includes means for detecting noise, means for recording noise data and transmitting it to the server, means for analyzing the noise data received by the server, means for generating an optimal masking sound based on the analysis results using a generative AI model, means for transmitting the generated masking sound to the terminal, and means for playing the masking sound received by the terminal. This makes it possible to accurately analyze noise in real time and provide the user with a comfortable masking sound.

[0428] The "noise detection means" is a function that uses a built-in microphone to monitor surrounding sounds in real time and detect noise that exceeds a certain threshold.

[0429] The "means for recording noise data and transmitting it to a server" is a function that records the waveform information, frequency components, sound intensity, etc. of the detected noise and transmits the data to a server using a secure communication protocol.

[0430] The "means for analyzing noise data received by the server" refers to a function that analyzes the received noise data using an acoustic processing library or machine learning algorithm to identify the type and characteristics of the noise.

[0431] "Means for generating optimal masking sounds based on analysis results using a generative AI model" refers to a function that uses a generative AI model based on analysis results to select or generate optimal masking sounds.

[0432] The "means for transmitting the generated masking sound to the terminal" is a communication function for transferring the generated masking sound data to the terminal.

[0433] The "means for reproducing the masking sound received by the terminal" is a function for reproducing the masking sound data received by the terminal through the built-in speaker.

[0434] A "generative AI model" is an artificial intelligence algorithm that generates or selects the optimal sound in response to a specific prompt.

[0435] A "prompt sentence" is an input sentence used to generate a specific output for a generative AI model.

[0436] This invention relates to a system that detects and analyzes noise, and generates and reproduces an optimal masking sound based on the noise. Specific embodiments of this system will be described below.

[0437] System Configuration

[0438] The system consists of the following main components:

[0439] 1. Terminal: A device that detects noise and communicates with the server.

[0440] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0441] 3. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0442] Noise Monitoring

[0443] The user places the device in a location where noise is a problem. The device uses a built-in, highly sensitive microphone to monitor the surrounding sounds in real time. For monitoring, an acoustic library (e.g., Librosa) is used to acquire sound waveform information, frequency components, and sound intensity.

[0444] Noise recording and transmission

[0445] When the device detects noise above a certain threshold, it records the data. The recording includes high-resolution waveform data, frequency content, and sound intensity. The recorded noise data is then sent from the device to a server via a secure communication protocol (e.g., HTTPS).

[0446] Noise data analysis

[0447] The server analyzes the received noise data. Machine learning algorithms (e.g., CNN, RNN) are used to identify the type of sound and its frequency components. The server then provides a prompt to the generative AI model to ask it for the optimal type of masking sound.

[0448] Masking sound generation

[0449] The generative AI model generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, the AI ​​model will select an appropriate natural sound (e.g., a waterfall or a flowing river) or generate a new one. The final masking sound data is stored on a server.

[0450] Sending and playing masking sounds

[0451] The server transmits the generated masking sound to the device, which then plays it back using its built-in speaker. This blends the noise and masking sound, providing a comfortable sound environment for the user.

[0452] Specific examples

[0453] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. The generated waterfall sound data is sent to the device, and the waterfall sound is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly.

[0454] Prompt Sentence Examples

[0455] Examples of prompts you can give to a generative AI model include:

[0456] "Construction noise detected. Please generate suitable masking sound."

[0457] "Based on the frequency content and waveform data, please suggest a sound that will effectively camouflage this noise."

[0458] By using such prompt sentences to have the generative AI model perform advanced sound generation, it is possible to provide optimal masking sounds in real time.

[0459] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0460] Step 1: Monitoring the noise

[0461] The user places the device in a location where noise is a problem. The device uses a built-in, highly sensitive microphone to monitor the surrounding sounds in real time. For monitoring, an acoustic library (e.g., Librosa) is used to acquire sound waveform information, frequency components, and sound intensity.

[0462] Input: Real-time sound environment data

[0463] Output: Sound waveform information, frequency components, sound intensity data

[0464] Specific operation: The device's microphone captures surrounding sounds, and the acoustic library analyzes the captured sound data to obtain waveform information and frequency components in real time.

[0465] Step 2: Record and transmit the noise

[0466] When the device detects noise above a certain threshold, it records the data. The recording includes high-resolution waveform data, frequency content, and sound intensity. The recorded noise data is then sent to a server using a secure communication protocol (e.g., HTTPS).

[0467] Input: Monitored sound data

[0468] Output: Noise data sent to the server

[0469] Specific operation: When the device exceeds the noise threshold, it buffers the sound data and then transmits the saved noise data to the server through a secure communication protocol.

[0470] Step 3: Analyze the noise data

[0471] The server analyzes the received noise data. Machine learning algorithms (e.g., CNN or RNN) and acoustic processing libraries are used to identify the type and characteristics of the sound. Based on the analysis results, the server provides a prompt to the generative AI model.

[0472] Input: Transmitted noise data

[0473] Output: type and characteristics of noise, prompt

[0474] Specific operation: The server preprocesses the received sound data using an acoustic processing library, then inputs it into a machine learning model to identify the type and characteristics of the noise. Based on the identification results, it generates and sends a prompt to the generative AI model.

[0475] Step 4: Generate masking sounds

[0476] The generative AI model generates the optimal masking sound based on the prompt. For example, if the sound of construction work is detected, the AI ​​model will select an appropriate natural sound (e.g., a waterfall or a flowing river) or generate a new one. The generated masking sound data is stored on a server.

[0477] Input: prompt sentence, sound database

[0478] Output: Optimal masking sound data

[0479] Specific operation: The generative AI model searches for appropriate masking sounds from a sound database according to the prompt sentence, and generates new sounds as needed. The generated masking sound data is stored on the server.

[0480] Step 5: Send and play the masking sound

[0481] The server transmits the generated masking sound to the device, which then plays it back using its built-in speaker. This blends the noise and the masking sound, providing a comfortable sound environment for the user.

[0482] Input: Generated masking sound data

[0483] Output: Played masking sound

[0484] Specific operation: The server sends the generated masking sound data to the device via a secure communication protocol, and the device decodes the received masking sound and plays it on the built-in speaker. By playing the masking sound, the noise is effectively camouflaged.

[0485] (Application example 1)

[0486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0487] Modern brick-and-mortar stores are required to provide a comfortable shopping environment for customers, but noise in the store can cause stress and reduce comfort. Noise can also make it difficult to accurately convey product explanations and guidance, reducing the quality of the customer experience. Therefore, there is a need for a system that can suppress noise in real time and provide a comfortable sound environment for customers.

[0488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0489] In this invention, the server includes means for analyzing noise data and generating an optimal masking sound, means for generating the masking sound using a generation AI model, and means for transmitting the generated masking sound to a terminal or a wearable device. This makes it possible to generate an optimal masking sound adapted to a noisy environment in real time and play it on a terminal or a device held by the customer.

[0490] - "Means for detecting noise" means a device for recognizing sounds occurring in the environment and measuring and recording such sounds.

[0491] "Means for analyzing noise data" refers to algorithms or systems that process and analyze collected noise data and identify the type and characteristics of noise.

[0492] The "means for reproducing the generated masking sound" refers to an output device such as a speaker or earphones for providing the masking sound sent from the server to the user.

[0493] "Means for playing masking sounds generated to adapt to a noisy environment through an audio device worn by the user" refers to a mechanism by which the user plays the optimal masking sound through a mobile terminal or wearable device, effectively camouflaging the noise.

[0494] "Means for monitoring noise using mobile or wearable devices" refers to the ability to detect and record noise in real time using portable devices such as smartphones or smart glasses.

[0495] "Means for generating masking sounds using generative AI models" refers to technology that uses machine learning algorithms to generate optimal masking sounds based on the results of noise analysis.

[0496] The "prompt sentence output by the generative AI model" is an instruction sentence that the AI ​​uses to generate masking sounds based on the noise analysis results.

[0497] System configuration

[0498] A system embodying this invention mainly comprises the following components: means for detecting noise, means for analyzing noise data, means for playing back generated masking sounds, means for playing back masking sounds generated to adapt to the noisy environment through an acoustic device worn by the user, and means for monitoring noise using a mobile terminal or a wearable device.

[0499] Hardware and Software

[0500] Devices: Mobile and wearable devices such as smartphones and smart glasses that monitor and record noise in real time.

[0501] Server: A high-performance computer for analyzing noise data and generating masking sounds. It runs generative AI models using machine learning frameworks such as TensorFlow and PyTorch.

[0502] An acoustic device, such as earphones, headphones, or the built-in speakers of smart glasses, that plays the generated masking sound.

[0503] Noise detection and data transmission

[0504] 1. The device detects noise and uses the built-in microphone to monitor the surrounding sound in real time. When noise exceeding a certain threshold is detected, the device records the sound. This noise data includes the sound's waveform information, frequency components, and sound intensity.

[0505] 2. The device sends the recorded noise data to the server.

[0506] Noise data analysis and masking sound generation

[0507] 1. The server analyzes the received noise data using a generative AI model built with TensorFlow and PyTorch. The analysis identifies the type of noise and its frequency components.

[0508] 2. Based on the analysis results, the generative AI model generates the optimal masking sound using a prompt sentence. For example, if the noise is a person speaking, the sound of light rain or wind will be determined to be an appropriate masking sound.

[0509] Example prompt sentence:

[0510] "Perform frequency analysis of human speech and generate non-disturbing natural sounds (e.g., rain or rustling leaves)."

[0511] Playback of masking sound

[0512] 1. The server generates masking sounds and sends them to the user's terminal or wearable device.

[0513] 2. The terminal or wearable device plays the received masking sound through earphones, headphones, or speakers in smart glasses.

[0514] Specific examples

[0515] Consider a specific example of how to make a customer's time in a store more comfortable. For example, a customer is selecting a product in a noisy physical store when noise occurs. A smartphone detects the noise and sends the data to a server. The server analyzes the noise as talking and uses a generative AI model to generate the sound of rain. This rain sound is sent to the customer's smartphone and played through earphones. This allows the customer to enjoy shopping in a comfortable acoustic environment.

[0516] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0517] Step 1:

[0518] The device detects noise and uses the built-in microphone to monitor surrounding sounds in real time. When the noise level exceeds a certain threshold, the device records the sound. At this time, the sound's waveform information, frequency components, and sound intensity are acquired as recorded data. The input is the surrounding ambient sound, and the output is noise data obtained from the microphone.

[0519] Step 2:

[0520] The device sends the recorded noise data to the server. The data includes sound waveform information, frequency components, and sound intensity. The input is the noise data acquired in step 1, and the output is a data packet sent to the server.

[0521] Step 3:

[0522] The server receives the noise data and begins analysis based on the received data. This analysis uses machine learning frameworks such as TensorFlow and PyTorch. The input is the data sent from the device, and the output is the analysis results that identify the type and characteristics of the noise.

[0523] Step 4:

[0524] The server uses a generative AI model to generate optimal masking sounds based on the analysis results. In this process, masking sounds are generated using predefined prompts. The inputs are the analysis results and prompts, and the output is the generated masking sound data. Specifically, the prompt used is, "Perform frequency analysis of human speech and generate non-interfering natural sounds (e.g., the sound of rain or rustling leaves)."

[0525] Step 5:

[0526] The server transmits the generated masking sound to a terminal or wearable device. The input is the generated masking sound data, and the output is a data packet for transmission.

[0527] Step 6:

[0528] The terminal or wearable device receives the masking sound data sent from the server and plays the received data on an acoustic device (such as earphones, headphones, or speakers in smart glasses). The input is the transmitted masking sound data, and the output is the played sound.

[0529] Step 7:

[0530] The user enjoys a comfortable sound environment through the masking sound that is played back. Specifically, the user listens to the masking sound created by the generative AI model as sound that reaches the ears through earphones or headphones.

[0531] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0532] This invention combines a system that detects and analyzes noise, and generates and plays optimal masking sounds based on the noise, with the ability to recognize the user's emotions and optimize the sound environment. Specific embodiments of this system are described below.

[0533] System configuration

[0534] The system consists of the following main components:

[0535] 1. Terminal: A device that detects noise and communicates with the server.

[0536] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0537] 3. Emotion engine: An engine that recognizes the user's emotions.

[0538] 4. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0539] Program processing

[0540] The processing of the program by each component of the system will be described in detail below.

[0541] Noise detection and data transmission

[0542] The user places the device in a location where noise is a problem (e.g., a conference room or office). The device uses a built-in microphone to monitor the surrounding sounds in real time. When noise exceeding a certain threshold is detected, the device records the sound and transmits the noise data to a server. This data includes the sound's waveform information, frequency components, and sound intensity.

[0543] Noise data analysis

[0544] The server analyzes the received noise data, specifically analyzing the type of sound and frequency components to identify the characteristics of the noise. This analysis uses machine learning algorithms and acoustic processing technology.

[0545] Masking sound generation

[0546] The server's AI generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, it may determine that natural sounds (such as a flowing river or white noise) are optimal. The AI ​​selects the optimal masking sound from a large sound database and, in some cases, generates a new masking sound.

[0547] Acquiring and analyzing emotion data

[0548] To further optimize the user's emotional state, the emotion engine captures and analyzes the user's facial expression, tone of voice, or biometric information in real time. For example, if the user's facial expression is recognized as tired or stressed, the emotion engine will send this information to the server.

[0549] Adjusting the masking sound

[0550] Based on the user's emotional data obtained from the emotion engine, the server further fine-tunes the generated masking sound. For example, if the user is feeling stressed, it may consider changing the sound to a more relaxing one (e.g., the sound of calming waves).

[0551] Sending and playing masking sounds

[0552] The generated masking sound (sound optimized for the user's emotions) is sent from the server to the device, which then plays it back through its built-in speaker. By mixing this masking sound with the noise, the noise is effectively camouflaged, providing the user with a comfortable sound environment that takes into account their emotional state.

[0553] Specific examples

[0554] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. At the same time, if the server determines that a meeting participant is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a quieter, gentler waterfall. The generated waterfall sound data is sent to the device, and the quiet, gentle sound of a waterfall is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly and participants to meet in a comfortable and relaxed environment.

[0555] In this way, by closely coordinating each step, it is possible to respond to changing noise environments in real time and provide users with an optimal, comfortable sound environment that also takes into account their emotional state.

[0556] The processing flow will be explained below.

[0557] Step 1:

[0558] The user places the terminal. The user places the terminal in a place where noise is a concern, such as a conference room or office.

[0559] Step 2:

[0560] The device detects noise. The device's built-in microphone monitors the surrounding sounds in real time and detects noise that exceeds a certain threshold.

[0561] Step 3:

[0562] The device records the noise data. The device records the detected noise data, which includes sound waveform information, frequency components, volume, etc.

[0563] Step 4:

[0564] The device transmits the noise data to the server. The recorded noise data is transmitted from the device to the server.

[0565] Step 5:

[0566] The server receives the noise data. The server receives the noise data transmitted from the terminal.

[0567] Step 6:

[0568] The server analyzes the noise data and performs analysis to identify the type and characteristics of the sound (e.g., construction noise, vehicle noise, voices, etc.) from the received data.

[0569] Step 7:

[0570] The server generates the optimal masking sound. Based on the analysis results, the server's AI automatically generates the optimal masking sound (e.g., white noise, natural sounds).

[0571] Step 8:

[0572] To recognize the user's emotions, the emotion engine captures the user's facial expressions and tone of voice in real time. The emotion engine collects the user's emotion data using a camera and microphone.

[0573] Step 9:

[0574] The emotion engine analyzes the user's emotional data, determines the user's emotional state from their facial expressions and tone of voice, and sends that data to the server.

[0575] Step 10:

[0576] The server receives and analyzes the emotional data and adjusts the masking sound. Based on the data from the emotion engine, the server fine-tunes the type and volume of the masking sound according to the user's emotional state.

[0577] Step 11:

[0578] The server sends the adjusted masking sound to the terminal. The optimized masking sound data is sent from the server to the terminal.

[0579] Step 12:

[0580] The device receives and plays the masking sound. The device plays the adjusted masking sound through the built-in speaker, effectively camouflaging the noise.

[0581] Step 13:

[0582] Users can enjoy a comfortable sound environment. By reducing noise and providing a sound environment that takes emotional state into consideration, users can enjoy a comfortable sound environment.

[0583] Example 2

[0584] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0585] Existing noise control systems simply detect noise and generate and play masking sounds, making it difficult to provide a flexible sound environment that responds to the user's emotional state. Furthermore, optimizing a noisy environment in real time requires integrated processing of noise detection, masking sound generation, and user emotion recognition, but no system currently exists that can do this efficiently.

[0586] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0587] In this invention, the server includes means for detecting noise, means for analyzing noise data, means for playing back the generated masking sound, means for recognizing and analyzing the emotional state of the user, and means for adjusting the masking sound based on the emotional state, thereby making it possible to provide a comfortable sound environment that takes the emotional state of the user into consideration.

[0588] "Means for detecting noise" refers to a device that monitors sounds in the environment in real time and detects sounds that exceed a threshold.

[0589] "Means for analyzing noise data" refers to technology that uses received sound data to analyze its frequency components, volume, etc., and identify specific noises.

[0590] "Means for reproducing the generated masking sound" refers to a device that selects an appropriate masking sound based on the analysis results and outputs it.

[0591] "Means for recognizing and analyzing the user's emotional state" refers to technology that acquires and analyzes the user's facial expressions, tone of voice, biometric information, etc. to identify the user's emotions.

[0592] The "means for adjusting the masking sound based on the emotional state" refers to a technique for adjusting the type and volume of the generated masking sound depending on the emotional state of the user.

[0593] "Means for transmitting noise data to a server" refers to a technology for transmitting noise data collected by a terminal to a server via a network.

[0594] The "means by which the server analyzes the noise data and generates an optimal masking sound" refers to a function by which the server analyzes the received noise data and generates an optimal masking sound.

[0595] The "means for transmitting the generated masking sound to the terminal" refers to a technique for transmitting the masking sound generated by the server to the terminal via a network.

[0596] "Means for acquiring a user's emotional state using their facial expressions, tone of voice, and biometric information" refers to devices such as cameras, microphones, and biometric sensors for acquiring a user's emotional state in real time, as well as the analysis technology for those devices.

[0597] MODE FOR CARRYING OUT THE INVENTION

[0598] This invention is a system that combines a system that detects and analyzes noise, and generates and plays optimal masking sounds based on the noise, with a function that recognizes the user's emotions and optimizes the sound environment. Specific embodiments of this system are described in detail below.

[0599] System configuration

[0600] The system consists of the following main components:

[0601] 1. Terminal: A device that detects noise and communicates with the server.

[0602] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0603] 3. Emotion engine: An engine that recognizes the user's emotions.

[0604] 4. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0605] Hardware and Software

[0606] Terminal

[0607] The device has a built-in microphone and speaker, a camera to recognize the user's facial expressions, a microphone to analyze the tone of voice, and sensors to capture biometric information. These devices are used to detect noise and capture emotional data.

[0608] server

[0609] The server requires powerful computing resources to analyze the noise data and generate masking sounds, using machine learning frameworks such as TensorFlow and PyTorch, as well as the Librosa acoustic processing library.

[0610] Emotion Engine

[0611] The emotion engine uses OpenFace to analyze the user's facial expressions and PyAudio to analyze audio data, making it possible to analyze the user's emotional state in real time.

[0612] Data processing and calculation

[0613] The system processes the following data:

[0614] 1. Noise detection and data transmission

[0615] The user places the device in a location where there is a noise problem. The device's built-in microphone monitors the surrounding sounds in real time, and when noise exceeding a certain threshold is detected, the device records the sound and transmits the noise data to the server. This data includes the sound's waveform information, frequency components, and sound intensity.

[0616] 2. Analysis of noise data

[0617] The server analyzes the received noise data, specifically by analyzing the frequency components of the sound using FFT (Fast Fourier Transform) and classifying the type of sound using a machine learning model.

[0618] 3. Masking sound generation

[0619] Based on the analysis results, the server's AI generates the optimal masking sound. The AI ​​selects the optimal masking sound from a large sound database (e.g., FreeSound) and, in some cases, generates a new masking sound.

[0620] 4. Acquisition and Analysis of Emotion Data

[0621] To further optimize the user's emotional state, the emotion engine captures and analyzes the user's facial expressions, tone of voice, or biometric information in real time through cameras, microphones, and biometric sensors.

[0622] 5. Adjusting the masking sound

[0623] Based on the data from the emotion engine, the server further adjusts the generated masking sounds, for example, changing them to more relaxing sounds if the user is feeling stressed.

[0624] 6. Sending and playing masking sounds

[0625] The generated masking sound is sent from the server to the device, which then plays it back through its built-in speaker. This sound blends with the noise, providing a comfortable sound environment for the user.

[0626] Specific examples

[0627] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. At the same time, if the server determines that a meeting participant is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a quieter, gentler waterfall. The generated waterfall sound data is sent to the device, and the quiet, gentle sound of a waterfall is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly and participants to meet in a comfortable and relaxed environment.

[0628] Prompt Sentence Examples

[0629] "There's construction noise during a meeting. Generate optimal sounds to mask the noise and adjust the sound based on the emotional data of the meeting participants."

[0630] In this way, this system can respond to changing noise environments in real time and provide users with an optimal, comfortable sound environment that also takes into account their emotional state.

[0631] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0632] Step 1: Noise detection and data transmission

[0633] The user places the device in a location where there is a noise problem. The device's built-in microphone monitors the surrounding sounds in real time, and if noise exceeding a certain threshold is detected, the device records the sound. The recorded data includes the sound's waveform information, frequency components, and sound intensity. After recording, the device compresses the data and sends it to the server.

[0634] Input: Ambient sound

[0635] Output: Compressed noise data

[0636] Specific operation: The device samples the sound waveform data and compares it with a threshold. If the threshold is exceeded, the device starts recording, then compresses the data and sends it to the server.

[0637] Step 2: Analyze the noise data

[0638] The server receives the noise data sent from the device, analyzes the frequency components using FFT (Fast Fourier Transform), and classifies the type of sound using a machine learning model (e.g., random forest or neural network).The analysis results are stored in a database.

[0639] Input: Compressed noise data

[0640] Output: Analysis results (sound frequency components and types)

[0641] Specific operation: The server reads the received data, applies FFT to extract frequency components, and classifies the sound type using a machine learning model. The analysis results are saved in a database.

[0642] Step 3: Generate the masking sound

[0643] The server's generation AI generates the optimal masking sound based on the analysis results. The generation AI selects the optimal masking sound from many sound databases (e.g., FreeSound) and generates new masking sounds as needed. The generated sound data is saved as a temporary file.

[0644] Input: Analysis results (sound frequency components and types)

[0645] Output: Generated masking sound data

[0646] Specific operation: The server's generation AI selects the optimal masking sound from the database based on the analysis results. If necessary, it generates new sounds and saves them in a temporary file.

[0647] Step 4: Acquire and analyze emotion data

[0648] The emotion engine installed in the device captures and analyzes the user's facial expressions, tone of voice, and biometric information in real time. Specifically, the emotion engine analyzes the data using tools such as OpenFace and PyAudio, and transmits the obtained emotional state to the server.

[0649] Input: User's facial expression, tone of voice, biometric information

[0650] Output: Parsed emotion data

[0651] Specific operation: The device captures the user's facial expressions and voice using the camera and microphone, analyzes them using OpenFace and PyAudio, and sends the analysis results to the server.

[0652] Step 5: Adjust the masking sound

[0653] The server receives the emotion data and adjusts the generated masking sound. If the user is feeling stressed, it changes the sound to a relaxing one. The adjusted sound data is also saved in a temporary file.

[0654] Input: Analyzed emotion data, generated masking sound data

[0655] Output: Adjusted masking sound data

[0656] Specific operation: The server analyzes the received emotional data and adjusts the generated masking sound based on the user's emotional state. The adjusted sound is saved in a temporary file.

[0657] Step 6: Send and play masking sounds

[0658] The server sends the adjusted masking sound data to the device, which then decodes the received masking sound and plays it through its built-in speaker, effectively camouflaging the noise.

[0659] Input: Adjusted masking sound data

[0660] Output: Masking sound played through the speaker

[0661] Specific operation: The server sends the generated sound data to the device, which decodes it and plays it through the speaker, optimizing the sound environment.

[0662] Through these steps, it becomes possible to provide a comfortable sound environment for users and optimize the noise environment in real time.

[0663] (Application example 2)

[0664] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0665] In modern brick-and-mortar retail environments, noise is a major problem that disrupts customers' comfortable shopping experiences. Furthermore, there is a need to provide a sound environment that not only reduces noise but also takes into account the emotional state of the customer. However, conventional noise masking systems only detect noise data and generate masking sounds, but do not support optimizing the sound environment based on the user's emotions. This poses a challenge in improving customer satisfaction in stores.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0667] In this invention, the server includes means for analyzing noise data, means for generating the generated masking sound, means for recognizing the user's emotion and optimizing the sound environment, and means for transmitting the user's emotion data to the server. This makes it possible to centrally manage everything from noise detection and analysis to the generation and playback of masking sound according to the emotion, and to provide the optimal sound environment for customers in a physical store.

[0668] The "means for detecting noise" is a function that monitors sounds in the environment in real time and detects noise that exceeds a certain threshold.

[0669] The "means for analyzing noise data" is a function for analyzing detected noise data based on the type of sound, frequency components, intensity, etc.

[0670] The "means for reproducing the generated masking sound" is a function for reproducing the masking sound generated based on the analysis results through an audio output device such as a speaker.

[0671] "Means to recognize the user's emotions and optimize the sound environment" refers to a function that analyzes the user's facial expressions, tone of voice, biometric information, etc. to recognize emotions in real time, and selects and adjusts the optimal masking sound based on that.

[0672] "Means for transmitting noise data to a server" refers to a function for transmitting detected and recorded noise data to a server via a communication means such as the Internet.

[0673] The "means for the server to analyze noise data and generate the most suitable masking sound" is a function that analyzes the noise data received by the server and generates the most suitable masking sound.

[0674] The "means for transmitting the generated masking sound to the terminal" is a function for transmitting the optimal masking sound generated by the server to the terminal via a communication means such as the Internet.

[0675] The "means for transmitting user emotion data to the server" is a function for transmitting data obtained by analyzing the user emotion to the server.

[0676] "Means for adjusting to provide a comfortable sound environment" refers to a function that further fine-tunes the masking sounds generated and selected based on the user's emotions and noise characteristics to further enhance user comfort.

[0677] The present invention provides a system that effectively masks noise in environments such as brick-and-mortar stores, while providing a sound environment that takes into consideration the emotional state of the user. Specific embodiments for carrying out the present invention are described below.

[0678] System Configuration

[0679] This system consists of a terminal that detects noise, a server that performs analysis and generation, an emotion engine that recognizes the user's emotions, and a user.

[0680] Noise detection

[0681] The device has a built-in microphone for real-time noise monitoring. It constantly monitors the sounds in the environment, and when it detects noise exceeding a certain threshold, it records the sound's waveform information, frequency components, sound intensity, etc., and recognizes it as noise data.

[0682] Noise data transmission and analysis

[0683] The device sends the detected noise data to a server, which then analyzes the received noise data to identify the type and characteristics of the sound. This analysis is performed using machine learning algorithms and acoustic processing technologies (e.g., TensorFlow, PyTorch, etc.).

[0684] Masking sound generation

[0685] The server generates optimal masking sounds based on the analysis results. For example, natural sounds, especially the sound of a waterfall or white noise, may be suitable for noise caused by construction work. Using a generative AI model, optimal masking sounds are selected from an existing database or newly generated.

[0686] Emotion Recognition and Data Transmission

[0687] The user's emotional data (e.g., facial expressions, tone of voice, biometric information, etc.) recognized on the device is analyzed through the emotion engine. The analysis is performed using a camera and microphone, and machine learning models (e.g., OpenCV, TensorFlow) are used for emotion recognition. The recognized emotional data is then sent to the server.

[0688] Adjusting and playing masking sounds

[0689] The server receives the emotional data and further adjusts the masking sound based on the user's emotional state (for example, if the user is stressed, the sound may be changed to a more relaxing sound). The adjusted masking sound is then sent to the device and played through the device's speaker.

[0690] Specific examples

[0691] For example, imagine a cafe in a shopping mall with many customers and different types of noises occurring in the background. The device detects the noise and sends the data to the server. The server analyzes the data and generates the sound of a flowing river as the optimal masking sound. At the same time, if the server determines that the customer is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a more gentle sound of a flowing river. The adjusted sound is sent to the device and played in the cafe, allowing the customer to enjoy a comfortable acoustic environment.

[0692] Prompt Sentence Examples

[0693] "Analyze current noise data and generate optimal masking sound."

[0694] "Recognize the emotions from the user's facial expressions and select sounds that have a relaxing effect."

[0695] This makes it possible to realize a system that can solve noise problems in brick-and-mortar stores and provide a comfortable sound environment for customers. This invention is an important technology for providing a relaxing environment for store users.

[0696] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0697] Step 1:

[0698] The device monitors the noise in the environment in real time and detects noise exceeding a certain threshold. The input is the environmental sound picked up by the device's microphone. The output is the detected noise data, which includes sound waveform information, frequency components, sound intensity, etc.

[0699] Step 2:

[0700] The terminal transmits the detected noise data to a server via the Internet. This process uses a transmission protocol (e.g., HTTP). The input is the detected noise data, and the output is a confirmation of transmission to the server.

[0701] Step 3:

[0702] The server analyzes the received noise data. This analysis uses machine learning algorithms (e.g., TensorFlow, PyTorch) to identify the type of sound and its frequency components. The input is the transmitted noise data, and the output is the analysis results.

[0703] Step 4:

[0704] The server generates the optimal masking sound based on the analysis results. A generative AI model is used for generation. The input is the analysis results, and the output is the generated masking sound. The generated sound is selected from natural sounds (such as the sound of a waterfall or white noise).

[0705] Step 5:

[0706] The user's emotional data (e.g., facial expressions, tone of voice, biometric information, etc.) recognized by the device is analyzed through the emotion engine. Real-time data from the camera and microphone is input for this analysis. The analysis results are sent to the server. The input is the user's real-time data, and the output is emotional data.

[0707] Step 6:

[0708] The server receives the emotional data and adjusts the masking sound based on the user's emotional state. For example, if the user is feeling stressed, the sound is changed to one with a relaxing effect. The input is the emotional data and the generated masking sound, and the output is the adjusted masking sound.

[0709] Step 7:

[0710] The adjusted masking sound is sent to the device and played back from the device's speaker. The input is the adjusted masking sound, and the output is the sound to be played back. This sound is mixed with the noise to create a comfortable sound environment for the user.

[0711] Through this series of steps, noise detection, analysis, generation of masking sounds, adjustment based on the user's emotions, and playback are all seamlessly performed, providing a comfortable sound environment for users in physical stores. This system not only effectively solves noise problems, but also provides a new experience that takes into account the user's emotional state.

[0712] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0713] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0714] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0715] [Third embodiment]

[0716] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0717] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0718] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0719] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0720] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0721] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0722] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0723] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0724] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0725] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0726] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0727] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0728] This invention relates to a system that detects and analyzes noise, and generates and reproduces an optimal masking sound based on the noise. Specific embodiments of this system will be described below.

[0729] System configuration

[0730] The system consists of the following main components:

[0731] 1. Terminal: A device that detects noise and communicates with the server.

[0732] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0733] 3. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0734] Program processing

[0735] The processing of the program by each component of the system will be described in detail below.

[0736] Noise detection and data transmission

[0737] The user places the device in a location where noise is a problem (e.g., a conference room or office). The device uses a built-in microphone to monitor the surrounding sounds in real time. When noise exceeding a certain threshold is detected, the device records the sound. The noise data includes the sound's waveform information, frequency components, and sound intensity. The recorded noise data is then sent from the device to a server.

[0738] Noise data analysis

[0739] The server analyzes the received noise data, specifically analyzing the type of sound and frequency components to identify the type of noise. This analysis uses machine learning algorithms and acoustic processing technology.

[0740] Masking sound generation

[0741] The server's AI generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, it may determine that natural sounds (such as a flowing river or white noise) are optimal. The AI ​​selects the optimal masking sound from a large sound database and generates new masking sounds as needed.

[0742] Sending and playing masking sounds

[0743] The generated masking sound is sent from the server to the device, which then plays it back through its built-in speaker. By mixing this masking sound with the noise, the noise is effectively camouflaged, providing a comfortable sound environment for the user.

[0744] Specific examples

[0745] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. The generated waterfall sound data is sent to the device, and the waterfall sound is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly.

[0746] In this way, by closely coordinating each step, it is possible to respond to noise environments that change in real time and provide users with an optimal and comfortable sound environment.

[0747] The processing flow will be explained below.

[0748] Step 1:

[0749] The user places the terminal in a location where noise is a concern, such as a conference room or office.

[0750] Step 2:

[0751] The device detects noise. The device's built-in microphone monitors the surrounding sounds in real time and detects noise that exceeds a certain threshold.

[0752] Step 3:

[0753] The device records the noise data. The device records the detected noise data, which includes sound waveform information, frequency components, volume, etc.

[0754] Step 4:

[0755] The device transmits the noise data to the server. The recorded noise data is transmitted from the device to the server.

[0756] Step 5:

[0757] The server receives the noise data. The server receives the noise data transmitted from the terminal.

[0758] Step 6:

[0759] The server analyzes the noise data and identifies the type and characteristics of the noise (e.g., construction noise, vehicle noise, talking, etc.).

[0760] Step 7:

[0761] The server generates the optimal masking sound. Based on the analysis results, the server's AI generates the optimal masking sound (e.g., white noise, natural sounds).

[0762] Step 8:

[0763] The server sends the masking sound to the terminal. The server then sends the generated masking sound to the terminal.

[0764] Step 9:

[0765] The terminal receives the masking sound. The terminal receives the masking sound transmitted from the server.

[0766] Step 10:

[0767] The device will play the masking sound, which will then be played through the built-in speaker to effectively camouflage the noise.

[0768] Step 11:

[0769] The user enjoys a comfortable sound environment. The user enjoys a comfortable environment in which noise is reduced by the masking sound.

[0770] Example 1

[0771] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0772] Conventional noise masking systems have the problem of being unable to generate and play appropriate masking sounds in real time even when they detect noise. High analytical accuracy is required to select and generate appropriate masking sounds, and adjustments are also necessary to ensure that the masking sounds are comfortable for the user, but current technology has difficulty meeting these requirements.

[0773] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0774] In this invention, the server includes means for detecting noise, means for recording noise data and transmitting it to the server, means for analyzing the noise data received by the server, means for generating an optimal masking sound based on the analysis results using a generative AI model, means for transmitting the generated masking sound to the terminal, and means for playing the masking sound received by the terminal. This makes it possible to accurately analyze noise in real time and provide the user with a comfortable masking sound.

[0775] The "noise detection means" is a function that uses a built-in microphone to monitor surrounding sounds in real time and detect noise that exceeds a certain threshold.

[0776] The "means for recording noise data and transmitting it to a server" is a function that records the waveform information, frequency components, sound intensity, etc. of the detected noise and transmits the data to a server using a secure communication protocol.

[0777] The "means for analyzing noise data received by the server" refers to a function that analyzes the received noise data using an acoustic processing library or machine learning algorithm to identify the type and characteristics of the noise.

[0778] "Means for generating optimal masking sounds based on analysis results using a generative AI model" refers to a function that uses a generative AI model based on analysis results to select or generate optimal masking sounds.

[0779] The "means for transmitting the generated masking sound to the terminal" is a communication function for transferring the generated masking sound data to the terminal.

[0780] The "means for reproducing the masking sound received by the terminal" is a function for reproducing the masking sound data received by the terminal through the built-in speaker.

[0781] A "generative AI model" is an artificial intelligence algorithm that generates or selects the optimal sound in response to a specific prompt.

[0782] A "prompt sentence" is an input sentence used to generate a specific output for a generative AI model.

[0783] This invention relates to a system that detects and analyzes noise, and generates and reproduces an optimal masking sound based on the noise. Specific embodiments of this system will be described below.

[0784] System Configuration

[0785] The system consists of the following main components:

[0786] 1. Terminal: A device that detects noise and communicates with the server.

[0787] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0788] 3. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0789] Noise Monitoring

[0790] The user places the device in a location where noise is a problem. The device uses a built-in, highly sensitive microphone to monitor the surrounding sounds in real time. For monitoring, an acoustic library (e.g., Librosa) is used to acquire sound waveform information, frequency components, and sound intensity.

[0791] Noise recording and transmission

[0792] When the device detects noise above a certain threshold, it records the data. The recording includes high-resolution waveform data, frequency content, and sound intensity. The recorded noise data is then sent from the device to a server via a secure communication protocol (e.g., HTTPS).

[0793] Noise data analysis

[0794] The server analyzes the received noise data. Machine learning algorithms (e.g., CNN, RNN) are used to identify the type of sound and its frequency components. The server then provides a prompt to the generative AI model to ask it for the optimal type of masking sound.

[0795] Masking sound generation

[0796] The generative AI model generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, the AI ​​model will select an appropriate natural sound (e.g., a waterfall or a flowing river) or generate a new one. The final masking sound data is stored on a server.

[0797] Sending and playing masking sounds

[0798] The server transmits the generated masking sound to the device, which then plays it back using its built-in speaker. This blends the noise and masking sound, providing a comfortable sound environment for the user.

[0799] Specific examples

[0800] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. The generated waterfall sound data is sent to the device, and the waterfall sound is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly.

[0801] Prompt Sentence Examples

[0802] Examples of prompts you can give to a generative AI model include:

[0803] "Construction noise detected. Please generate suitable masking sound."

[0804] "Based on the frequency content and waveform data, please suggest a sound that will effectively camouflage this noise."

[0805] By using such prompt sentences to have the generative AI model perform advanced sound generation, it is possible to provide optimal masking sounds in real time.

[0806] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0807] Step 1: Monitoring the noise

[0808] The user places the device in a location where noise is a problem. The device uses a built-in, highly sensitive microphone to monitor the surrounding sounds in real time. For monitoring, an acoustic library (e.g., Librosa) is used to acquire sound waveform information, frequency components, and sound intensity.

[0809] Input: Real-time sound environment data

[0810] Output: Sound waveform information, frequency components, sound intensity data

[0811] Specific operation: The device's microphone captures surrounding sounds, and the acoustic library analyzes the captured sound data to obtain waveform information and frequency components in real time.

[0812] Step 2: Record and transmit the noise

[0813] When the device detects noise above a certain threshold, it records the data. The recording includes high-resolution waveform data, frequency content, and sound intensity. The recorded noise data is then sent to a server using a secure communication protocol (e.g., HTTPS).

[0814] Input: Monitored sound data

[0815] Output: Noise data sent to the server

[0816] Specific operation: When the device exceeds the noise threshold, it buffers the sound data and then transmits the saved noise data to the server through a secure communication protocol.

[0817] Step 3: Analyze the noise data

[0818] The server analyzes the received noise data. Machine learning algorithms (e.g., CNN or RNN) and acoustic processing libraries are used to identify the type and characteristics of the sound. Based on the analysis results, the server provides a prompt to the generative AI model.

[0819] Input: Transmitted noise data

[0820] Output: type and characteristics of noise, prompt

[0821] Specific operation: The server preprocesses the received sound data using an acoustic processing library, then inputs it into a machine learning model to identify the type and characteristics of the noise. Based on the identification results, it generates and sends a prompt to the generative AI model.

[0822] Step 4: Generate masking sounds

[0823] The generative AI model generates the optimal masking sound based on the prompt. For example, if the sound of construction work is detected, the AI ​​model will select an appropriate natural sound (e.g., a waterfall or a flowing river) or generate a new one. The generated masking sound data is stored on a server.

[0824] Input: prompt sentence, sound database

[0825] Output: Optimal masking sound data

[0826] Specific operation: The generative AI model searches for appropriate masking sounds from a sound database according to the prompt sentence, and generates new sounds as needed. The generated masking sound data is stored on the server.

[0827] Step 5: Send and play the masking sound

[0828] The server transmits the generated masking sound to the device, which then plays it back using its built-in speaker. This blends the noise and the masking sound, providing a comfortable sound environment for the user.

[0829] Input: Generated masking sound data

[0830] Output: Played masking sound

[0831] Specific operation: The server sends the generated masking sound data to the device via a secure communication protocol, and the device decodes the received masking sound and plays it on the built-in speaker. By playing the masking sound, the noise is effectively camouflaged.

[0832] (Application example 1)

[0833] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0834] Modern brick-and-mortar stores are required to provide a comfortable shopping environment for customers, but noise in the store can cause stress and reduce comfort. Noise can also make it difficult to accurately convey product explanations and guidance, reducing the quality of the customer experience. Therefore, there is a need for a system that can suppress noise in real time and provide a comfortable sound environment for customers.

[0835] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0836] In this invention, the server includes means for analyzing noise data and generating an optimal masking sound, means for generating the masking sound using a generation AI model, and means for transmitting the generated masking sound to a terminal or a wearable device. This makes it possible to generate an optimal masking sound adapted to a noisy environment in real time and play it on a terminal or a device held by the customer.

[0837] - "Means for detecting noise" means a device for recognizing sounds occurring in the environment and measuring and recording such sounds.

[0838] "Means for analyzing noise data" refers to algorithms or systems that process and analyze collected noise data and identify the type and characteristics of noise.

[0839] The "means for reproducing the generated masking sound" refers to an output device such as a speaker or earphones for providing the masking sound sent from the server to the user.

[0840] "Means for playing masking sounds generated to adapt to a noisy environment through an audio device worn by the user" refers to a mechanism by which the user plays the optimal masking sound through a mobile terminal or wearable device, effectively camouflaging the noise.

[0841] "Means for monitoring noise using mobile or wearable devices" refers to the ability to detect and record noise in real time using portable devices such as smartphones or smart glasses.

[0842] "Means for generating masking sounds using generative AI models" refers to technology that uses machine learning algorithms to generate optimal masking sounds based on the results of noise analysis.

[0843] The "prompt sentence output by the generative AI model" is an instruction sentence that the AI ​​uses to generate masking sounds based on the noise analysis results.

[0844] System configuration

[0845] A system embodying this invention mainly comprises the following components: means for detecting noise, means for analyzing noise data, means for playing back generated masking sounds, means for playing back masking sounds generated to adapt to the noisy environment through an acoustic device worn by the user, and means for monitoring noise using a mobile terminal or a wearable device.

[0846] Hardware and Software

[0847] Devices: Mobile and wearable devices such as smartphones and smart glasses that monitor and record noise in real time.

[0848] Server: A high-performance computer for analyzing noise data and generating masking sounds. It runs generative AI models using machine learning frameworks such as TensorFlow and PyTorch.

[0849] An acoustic device, such as earphones, headphones, or the built-in speakers of smart glasses, that plays the generated masking sound.

[0850] Noise detection and data transmission

[0851] 1. The device detects noise and uses the built-in microphone to monitor the surrounding sound in real time. When noise exceeding a certain threshold is detected, the device records the sound. This noise data includes the sound's waveform information, frequency components, and sound intensity.

[0852] 2. The device sends the recorded noise data to the server.

[0853] Noise data analysis and masking sound generation

[0854] 1. The server analyzes the received noise data using a generative AI model built with TensorFlow and PyTorch. The analysis identifies the type of noise and its frequency components.

[0855] 2. Based on the analysis results, the generative AI model generates the optimal masking sound using a prompt sentence. For example, if the noise is a person speaking, the sound of light rain or wind will be determined to be an appropriate masking sound.

[0856] Example prompt sentence:

[0857] "Perform frequency analysis of human speech and generate non-disturbing natural sounds (e.g., rain or rustling leaves)."

[0858] Playback of masking sound

[0859] 1. The server generates masking sounds and sends them to the user's terminal or wearable device.

[0860] 2. The terminal or wearable device plays the received masking sound through earphones, headphones, or speakers in smart glasses.

[0861] Specific examples

[0862] Consider a specific example of how to make a customer's time in a store more comfortable. For example, a customer is selecting a product in a noisy physical store when noise occurs. A smartphone detects the noise and sends the data to a server. The server analyzes the noise as talking and uses a generative AI model to generate the sound of rain. This rain sound is sent to the customer's smartphone and played through earphones. This allows the customer to enjoy shopping in a comfortable acoustic environment.

[0863] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0864] Step 1:

[0865] The device detects noise and uses the built-in microphone to monitor surrounding sounds in real time. When the noise level exceeds a certain threshold, the device records the sound. At this time, the sound's waveform information, frequency components, and sound intensity are acquired as recorded data. The input is the surrounding ambient sound, and the output is noise data obtained from the microphone.

[0866] Step 2:

[0867] The device sends the recorded noise data to the server. The data includes sound waveform information, frequency components, and sound intensity. The input is the noise data acquired in step 1, and the output is a data packet sent to the server.

[0868] Step 3:

[0869] The server receives the noise data and begins analysis based on the received data. This analysis uses machine learning frameworks such as TensorFlow and PyTorch. The input is the data sent from the device, and the output is the analysis results that identify the type and characteristics of the noise.

[0870] Step 4:

[0871] The server uses a generative AI model to generate optimal masking sounds based on the analysis results. In this process, masking sounds are generated using predefined prompts. The inputs are the analysis results and prompts, and the output is the generated masking sound data. Specifically, the prompt used is, "Perform frequency analysis of human speech and generate non-interfering natural sounds (e.g., the sound of rain or rustling leaves)."

[0872] Step 5:

[0873] The server transmits the generated masking sound to a terminal or wearable device. The input is the generated masking sound data, and the output is a data packet for transmission.

[0874] Step 6:

[0875] The terminal or wearable device receives the masking sound data sent from the server and plays the received data on an acoustic device (such as earphones, headphones, or speakers in smart glasses). The input is the transmitted masking sound data, and the output is the played sound.

[0876] Step 7:

[0877] The user enjoys a comfortable sound environment through the masking sound that is played back. Specifically, the user listens to the masking sound created by the generative AI model as sound that reaches the ears through earphones or headphones.

[0878] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0879] This invention combines a system that detects and analyzes noise, and generates and plays optimal masking sounds based on the noise, with the ability to recognize the user's emotions and optimize the sound environment. Specific embodiments of this system are described below.

[0880] System configuration

[0881] The system consists of the following main components:

[0882] 1. Terminal: A device that detects noise and communicates with the server.

[0883] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0884] 3. Emotion engine: An engine that recognizes the user's emotions.

[0885] 4. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0886] Program processing

[0887] The processing of the program by each component of the system will be described in detail below.

[0888] Noise detection and data transmission

[0889] The user places the device in a location where noise is a problem (e.g., a conference room or office). The device uses a built-in microphone to monitor the surrounding sounds in real time. When noise exceeding a certain threshold is detected, the device records the sound and transmits the noise data to a server. This data includes the sound's waveform information, frequency components, and sound intensity.

[0890] Noise data analysis

[0891] The server analyzes the received noise data, specifically analyzing the type of sound and frequency components to identify the characteristics of the noise. This analysis uses machine learning algorithms and acoustic processing technology.

[0892] Masking sound generation

[0893] The server's AI generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, it may determine that natural sounds (such as a flowing river or white noise) are optimal. The AI ​​selects the optimal masking sound from a large sound database and, in some cases, generates a new masking sound.

[0894] Acquiring and analyzing emotion data

[0895] To further optimize the user's emotional state, the emotion engine captures and analyzes the user's facial expression, tone of voice, or biometric information in real time. For example, if the user's facial expression is recognized as tired or stressed, the emotion engine will send this information to the server.

[0896] Adjusting the masking sound

[0897] Based on the user's emotional data obtained from the emotion engine, the server further fine-tunes the generated masking sound. For example, if the user is feeling stressed, it may consider changing the sound to a more relaxing one (e.g., the sound of calming waves).

[0898] Sending and playing masking sounds

[0899] The generated masking sound (sound optimized for the user's emotions) is sent from the server to the device, which then plays it back through its built-in speaker. By mixing this masking sound with the noise, the noise is effectively camouflaged, providing the user with a comfortable sound environment that takes into account their emotional state.

[0900] Specific examples

[0901] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. At the same time, if the server determines that a meeting participant is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a quieter, gentler waterfall. The generated waterfall sound data is sent to the device, and the quiet, gentle sound of a waterfall is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly and participants to meet in a comfortable and relaxed environment.

[0902] In this way, by closely coordinating each step, it is possible to respond to changing noise environments in real time and provide users with an optimal, comfortable sound environment that also takes into account their emotional state.

[0903] The processing flow will be explained below.

[0904] Step 1:

[0905] The user places the terminal. The user places the terminal in a place where noise is a concern, such as a conference room or office.

[0906] Step 2:

[0907] The device detects noise. The device's built-in microphone monitors the surrounding sounds in real time and detects noise that exceeds a certain threshold.

[0908] Step 3:

[0909] The device records the noise data. The device records the detected noise data, which includes sound waveform information, frequency components, volume, etc.

[0910] Step 4:

[0911] The device transmits the noise data to the server. The recorded noise data is transmitted from the device to the server.

[0912] Step 5:

[0913] The server receives the noise data. The server receives the noise data transmitted from the terminal.

[0914] Step 6:

[0915] The server analyzes the noise data and performs analysis to identify the type and characteristics of the sound (e.g., construction noise, vehicle noise, voices, etc.) from the received data.

[0916] Step 7:

[0917] The server generates the optimal masking sound. Based on the analysis results, the server's AI automatically generates the optimal masking sound (e.g., white noise, natural sounds).

[0918] Step 8:

[0919] To recognize the user's emotions, the emotion engine captures the user's facial expressions and tone of voice in real time. The emotion engine collects the user's emotion data using a camera and microphone.

[0920] Step 9:

[0921] The emotion engine analyzes the user's emotional data, determines the user's emotional state from their facial expressions and tone of voice, and sends that data to the server.

[0922] Step 10:

[0923] The server receives and analyzes the emotional data and adjusts the masking sound. Based on the data from the emotion engine, the server fine-tunes the type and volume of the masking sound according to the user's emotional state.

[0924] Step 11:

[0925] The server sends the adjusted masking sound to the terminal. The optimized masking sound data is sent from the server to the terminal.

[0926] Step 12:

[0927] The device receives and plays the masking sound. The device plays the adjusted masking sound through the built-in speaker, effectively camouflaging the noise.

[0928] Step 13:

[0929] Users can enjoy a comfortable sound environment. By reducing noise and providing a sound environment that takes emotional state into consideration, users can enjoy a comfortable sound environment.

[0930] Example 2

[0931] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0932] Existing noise control systems simply detect noise and generate and play masking sounds, making it difficult to provide a flexible sound environment that responds to the user's emotional state. Furthermore, optimizing a noisy environment in real time requires integrated processing of noise detection, masking sound generation, and user emotion recognition, but no system currently exists that can do this efficiently.

[0933] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0934] In this invention, the server includes means for detecting noise, means for analyzing noise data, means for playing back the generated masking sound, means for recognizing and analyzing the emotional state of the user, and means for adjusting the masking sound based on the emotional state, thereby making it possible to provide a comfortable sound environment that takes the emotional state of the user into consideration.

[0935] "Means for detecting noise" refers to a device that monitors sounds in the environment in real time and detects sounds that exceed a threshold.

[0936] "Means for analyzing noise data" refers to technology that uses received sound data to analyze its frequency components, volume, etc., and identify specific noises.

[0937] "Means for reproducing the generated masking sound" refers to a device that selects an appropriate masking sound based on the analysis results and outputs it.

[0938] "Means for recognizing and analyzing the user's emotional state" refers to technology that acquires and analyzes the user's facial expressions, tone of voice, biometric information, etc. to identify the user's emotions.

[0939] The "means for adjusting the masking sound based on the emotional state" refers to a technique for adjusting the type and volume of the generated masking sound depending on the emotional state of the user.

[0940] "Means for transmitting noise data to a server" refers to a technology for transmitting noise data collected by a terminal to a server via a network.

[0941] The "means by which the server analyzes the noise data and generates an optimal masking sound" refers to a function by which the server analyzes the received noise data and generates an optimal masking sound.

[0942] The "means for transmitting the generated masking sound to the terminal" refers to a technique for transmitting the masking sound generated by the server to the terminal via a network.

[0943] "Means for acquiring a user's emotional state using their facial expressions, tone of voice, and biometric information" refers to devices such as cameras, microphones, and biometric sensors for acquiring a user's emotional state in real time, as well as the analysis technology for those devices.

[0944] MODE FOR CARRYING OUT THE INVENTION

[0945] This invention is a system that combines a system that detects and analyzes noise, and generates and plays optimal masking sounds based on the noise, with a function that recognizes the user's emotions and optimizes the sound environment. Specific embodiments of this system are described in detail below.

[0946] System configuration

[0947] The system consists of the following main components:

[0948] 1. Terminal: A device that detects noise and communicates with the server.

[0949] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[0950] 3. Emotion engine: An engine that recognizes the user's emotions.

[0951] 4. User: An individual or group that uses the system and seeks a comfortable sound environment.

[0952] Hardware and Software

[0953] Terminal

[0954] The device has a built-in microphone and speaker, a camera to recognize the user's facial expressions, a microphone to analyze the tone of voice, and sensors to capture biometric information. These devices are used to detect noise and capture emotional data.

[0955] server

[0956] The server requires powerful computing resources to analyze the noise data and generate masking sounds, using machine learning frameworks such as TensorFlow and PyTorch, as well as the Librosa acoustic processing library.

[0957] Emotion Engine

[0958] The emotion engine uses OpenFace to analyze the user's facial expressions and PyAudio to analyze audio data, making it possible to analyze the user's emotional state in real time.

[0959] Data processing and calculation

[0960] The system processes the following data:

[0961] 1. Noise detection and data transmission

[0962] The user places the device in a location where there is a noise problem. The device's built-in microphone monitors the surrounding sounds in real time, and when noise exceeding a certain threshold is detected, the device records the sound and transmits the noise data to the server. This data includes the sound's waveform information, frequency components, and sound intensity.

[0963] 2. Analysis of noise data

[0964] The server analyzes the received noise data, specifically by analyzing the frequency components of the sound using FFT (Fast Fourier Transform) and classifying the type of sound using a machine learning model.

[0965] 3. Masking sound generation

[0966] Based on the analysis results, the server's AI generates the optimal masking sound. The AI ​​selects the optimal masking sound from a large sound database (e.g., FreeSound) and, in some cases, generates a new masking sound.

[0967] 4. Acquisition and Analysis of Emotion Data

[0968] To further optimize the user's emotional state, the emotion engine captures and analyzes the user's facial expressions, tone of voice, or biometric information in real time through cameras, microphones, and biometric sensors.

[0969] 5. Adjusting the masking sound

[0970] Based on the data from the emotion engine, the server further adjusts the generated masking sounds, for example, changing them to more relaxing sounds if the user is feeling stressed.

[0971] 6. Sending and playing masking sounds

[0972] The generated masking sound is sent from the server to the device, which then plays it back through its built-in speaker. This sound blends with the noise, providing a comfortable sound environment for the user.

[0973] Specific examples

[0974] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. At the same time, if the server determines that a meeting participant is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a quieter, gentler waterfall. The generated waterfall sound data is sent to the device, and the quiet, gentle sound of a waterfall is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly and participants to meet in a comfortable and relaxed environment.

[0975] Prompt Sentence Examples

[0976] "There's construction noise during a meeting. Generate optimal sounds to mask the noise and adjust the sound based on the emotional data of the meeting participants."

[0977] In this way, this system can respond to changing noise environments in real time and provide users with an optimal, comfortable sound environment that also takes into account their emotional state.

[0978] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0979] Step 1: Noise detection and data transmission

[0980] The user places the device in a location where there is a noise problem. The device's built-in microphone monitors the surrounding sounds in real time, and if noise exceeding a certain threshold is detected, the device records the sound. The recorded data includes the sound's waveform information, frequency components, and sound intensity. After recording, the device compresses the data and sends it to the server.

[0981] Input: Ambient sound

[0982] Output: Compressed noise data

[0983] Specific operation: The device samples the sound waveform data and compares it with a threshold. If the threshold is exceeded, the device starts recording, then compresses the data and sends it to the server.

[0984] Step 2: Analyze the noise data

[0985] The server receives the noise data sent from the device, analyzes the frequency components using FFT (Fast Fourier Transform), and classifies the type of sound using a machine learning model (e.g., random forest or neural network).The analysis results are stored in a database.

[0986] Input: Compressed noise data

[0987] Output: Analysis results (sound frequency components and types)

[0988] Specific operation: The server reads the received data, applies FFT to extract frequency components, and classifies the sound type using a machine learning model. The analysis results are saved in a database.

[0989] Step 3: Generate the masking sound

[0990] The server's generation AI generates the optimal masking sound based on the analysis results. The generation AI selects the optimal masking sound from many sound databases (e.g., FreeSound) and generates new masking sounds as needed. The generated sound data is saved as a temporary file.

[0991] Input: Analysis results (sound frequency components and types)

[0992] Output: Generated masking sound data

[0993] Specific operation: The server's generation AI selects the optimal masking sound from the database based on the analysis results. If necessary, it generates new sounds and saves them in a temporary file.

[0994] Step 4: Acquire and analyze emotion data

[0995] The emotion engine installed in the device captures and analyzes the user's facial expressions, tone of voice, and biometric information in real time. Specifically, the emotion engine analyzes the data using tools such as OpenFace and PyAudio, and transmits the obtained emotional state to the server.

[0996] Input: User's facial expression, tone of voice, biometric information

[0997] Output: Parsed emotion data

[0998] Specific operation: The device captures the user's facial expressions and voice using the camera and microphone, analyzes them using OpenFace and PyAudio, and sends the analysis results to the server.

[0999] Step 5: Adjust the masking sound

[1000] The server receives the emotion data and adjusts the generated masking sound. If the user is feeling stressed, it changes the sound to a relaxing one. The adjusted sound data is also saved in a temporary file.

[1001] Input: Analyzed emotion data, generated masking sound data

[1002] Output: Adjusted masking sound data

[1003] Specific operation: The server analyzes the received emotional data and adjusts the generated masking sound based on the user's emotional state. The adjusted sound is saved in a temporary file.

[1004] Step 6: Send and play masking sounds

[1005] The server sends the adjusted masking sound data to the device, which then decodes the received masking sound and plays it through its built-in speaker, effectively camouflaging the noise.

[1006] Input: Adjusted masking sound data

[1007] Output: Masking sound played through the speaker

[1008] Specific operation: The server sends the generated sound data to the device, which decodes it and plays it through the speaker, optimizing the sound environment.

[1009] Through these steps, it becomes possible to provide a comfortable sound environment for users and optimize the noise environment in real time.

[1010] (Application example 2)

[1011] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1012] In modern brick-and-mortar retail environments, noise is a major problem that disrupts customers' comfortable shopping experiences. Furthermore, there is a need to provide a sound environment that not only reduces noise but also takes into account the emotional state of the customer. However, conventional noise masking systems only detect noise data and generate masking sounds, but do not support optimizing the sound environment based on the user's emotions. This poses a challenge in improving customer satisfaction in stores.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1014] In this invention, the server includes means for analyzing noise data, means for generating the generated masking sound, means for recognizing the user's emotion and optimizing the sound environment, and means for transmitting the user's emotion data to the server. This makes it possible to centrally manage everything from noise detection and analysis to the generation and playback of masking sound according to the emotion, and to provide the optimal sound environment for customers in a physical store.

[1015] The "means for detecting noise" is a function that monitors sounds in the environment in real time and detects noise that exceeds a certain threshold.

[1016] The "means for analyzing noise data" is a function for analyzing detected noise data based on the type of sound, frequency components, intensity, etc.

[1017] The "means for reproducing the generated masking sound" is a function for reproducing the masking sound generated based on the analysis results through an audio output device such as a speaker.

[1018] "Means to recognize the user's emotions and optimize the sound environment" refers to a function that analyzes the user's facial expressions, tone of voice, biometric information, etc. to recognize emotions in real time, and selects and adjusts the optimal masking sound based on that.

[1019] "Means for transmitting noise data to a server" refers to a function for transmitting detected and recorded noise data to a server via a communication means such as the Internet.

[1020] The "means for the server to analyze noise data and generate the most suitable masking sound" is a function that analyzes the noise data received by the server and generates the most suitable masking sound.

[1021] The "means for transmitting the generated masking sound to the terminal" is a function for transmitting the optimal masking sound generated by the server to the terminal via a communication means such as the Internet.

[1022] The "means for transmitting user emotion data to the server" is a function for transmitting data obtained by analyzing the user emotion to the server.

[1023] "Means for adjusting to provide a comfortable sound environment" refers to a function that further fine-tunes the masking sounds generated and selected based on the user's emotions and noise characteristics to further enhance user comfort.

[1024] The present invention provides a system that effectively masks noise in environments such as brick-and-mortar stores, while providing a sound environment that takes into consideration the emotional state of the user. Specific embodiments for carrying out the present invention are described below.

[1025] System Configuration

[1026] This system consists of a terminal that detects noise, a server that performs analysis and generation, an emotion engine that recognizes the user's emotions, and a user.

[1027] Noise detection

[1028] The device has a built-in microphone for real-time noise monitoring. It constantly monitors the sounds in the environment, and when it detects noise exceeding a certain threshold, it records the sound's waveform information, frequency components, sound intensity, etc., and recognizes it as noise data.

[1029] Noise data transmission and analysis

[1030] The device sends the detected noise data to a server, which then analyzes the received noise data to identify the type and characteristics of the sound. This analysis is performed using machine learning algorithms and acoustic processing technologies (e.g., TensorFlow, PyTorch, etc.).

[1031] Masking sound generation

[1032] The server generates optimal masking sounds based on the analysis results. For example, natural sounds, especially the sound of a waterfall or white noise, may be suitable for noise caused by construction work. Using a generative AI model, optimal masking sounds are selected from an existing database or newly generated.

[1033] Emotion Recognition and Data Transmission

[1034] The user's emotional data (e.g., facial expressions, tone of voice, biometric information, etc.) recognized on the device is analyzed through the emotion engine. The analysis is performed using a camera and microphone, and machine learning models (e.g., OpenCV, TensorFlow) are used for emotion recognition. The recognized emotional data is then sent to the server.

[1035] Adjusting and playing masking sounds

[1036] The server receives the emotional data and further adjusts the masking sound based on the user's emotional state (for example, if the user is stressed, the sound may be changed to a more relaxing sound). The adjusted masking sound is then sent to the device and played through the device's speaker.

[1037] Specific examples

[1038] For example, imagine a cafe in a shopping mall with many customers and different types of noises occurring in the background. The device detects the noise and sends the data to the server. The server analyzes the data and generates the sound of a flowing river as the optimal masking sound. At the same time, if the server determines that the customer is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a more gentle sound of a flowing river. The adjusted sound is sent to the device and played in the cafe, allowing the customer to enjoy a comfortable acoustic environment.

[1039] Prompt Sentence Examples

[1040] "Analyze current noise data and generate optimal masking sound."

[1041] "Recognize the emotions from the user's facial expressions and select sounds that have a relaxing effect."

[1042] This makes it possible to realize a system that can solve noise problems in brick-and-mortar stores and provide a comfortable sound environment for customers. This invention is an important technology for providing a relaxing environment for store users.

[1043] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1044] Step 1:

[1045] The device monitors the noise in the environment in real time and detects noise exceeding a certain threshold. The input is the environmental sound picked up by the device's microphone. The output is the detected noise data, which includes sound waveform information, frequency components, sound intensity, etc.

[1046] Step 2:

[1047] The terminal transmits the detected noise data to a server via the Internet. This process uses a transmission protocol (e.g., HTTP). The input is the detected noise data, and the output is a confirmation of transmission to the server.

[1048] Step 3:

[1049] The server analyzes the received noise data. This analysis uses machine learning algorithms (e.g., TensorFlow, PyTorch) to identify the type of sound and its frequency components. The input is the transmitted noise data, and the output is the analysis results.

[1050] Step 4:

[1051] The server generates the optimal masking sound based on the analysis results. A generative AI model is used for generation. The input is the analysis results, and the output is the generated masking sound. The generated sound is selected from natural sounds (such as the sound of a waterfall or white noise).

[1052] Step 5:

[1053] The user's emotional data (e.g., facial expressions, tone of voice, biometric information, etc.) recognized by the device is analyzed through the emotion engine. Real-time data from the camera and microphone is input for this analysis. The analysis results are sent to the server. The input is the user's real-time data, and the output is emotional data.

[1054] Step 6:

[1055] The server receives the emotional data and adjusts the masking sound based on the user's emotional state. For example, if the user is feeling stressed, the sound is changed to one with a relaxing effect. The input is the emotional data and the generated masking sound, and the output is the adjusted masking sound.

[1056] Step 7:

[1057] The adjusted masking sound is sent to the device and played back from the device's speaker. The input is the adjusted masking sound, and the output is the sound to be played back. This sound is mixed with the noise to create a comfortable sound environment for the user.

[1058] Through this series of steps, noise detection, analysis, generation of masking sounds, adjustment based on the user's emotions, and playback are all seamlessly performed, providing a comfortable sound environment for users in physical stores. This system not only effectively solves noise problems, but also provides a new experience that takes into account the user's emotional state.

[1059] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1060] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1061] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1062] [Fourth embodiment]

[1063] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1064] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1065] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1066] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1067] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1068] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1069] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1070] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1071] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1072] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1073] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1074] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1075] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1076] This invention relates to a system that detects and analyzes noise, and generates and reproduces an optimal masking sound based on the noise. Specific embodiments of this system will be described below.

[1077] System configuration

[1078] The system consists of the following main components:

[1079] 1. Terminal: A device that detects noise and communicates with the server.

[1080] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[1081] 3. User: An individual or group that uses the system and seeks a comfortable sound environment.

[1082] Program processing

[1083] The processing of the program by each component of the system will be described in detail below.

[1084] Noise detection and data transmission

[1085] The user places the device in a location where noise is a problem (e.g., a conference room or office). The device uses a built-in microphone to monitor the surrounding sounds in real time. When noise exceeding a certain threshold is detected, the device records the sound. The noise data includes the sound's waveform information, frequency components, and sound intensity. The recorded noise data is then sent from the device to a server.

[1086] Noise data analysis

[1087] The server analyzes the received noise data, specifically analyzing the type of sound and frequency components to identify the type of noise. This analysis uses machine learning algorithms and acoustic processing technology.

[1088] Masking sound generation

[1089] The server's AI generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, it may determine that natural sounds (such as a flowing river or white noise) are optimal. The AI ​​selects the optimal masking sound from a large sound database and generates new masking sounds as needed.

[1090] Sending and playing masking sounds

[1091] The generated masking sound is sent from the server to the device, which then plays it back through its built-in speaker. By mixing this masking sound with the noise, the noise is effectively camouflaged, providing a comfortable sound environment for the user.

[1092] Specific examples

[1093] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. The generated waterfall sound data is sent to the device, and the waterfall sound is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly.

[1094] In this way, by closely coordinating each step, it is possible to respond to noise environments that change in real time and provide users with an optimal and comfortable sound environment.

[1095] The processing flow will be explained below.

[1096] Step 1:

[1097] The user places the terminal in a location where noise is a concern, such as a conference room or office.

[1098] Step 2:

[1099] The device detects noise. The device's built-in microphone monitors the surrounding sounds in real time and detects noise that exceeds a certain threshold.

[1100] Step 3:

[1101] The device records the noise data. The device records the detected noise data, which includes sound waveform information, frequency components, volume, etc.

[1102] Step 4:

[1103] The device transmits the noise data to the server. The recorded noise data is transmitted from the device to the server.

[1104] Step 5:

[1105] The server receives the noise data. The server receives the noise data transmitted from the terminal.

[1106] Step 6:

[1107] The server analyzes the noise data and identifies the type and characteristics of the noise (e.g., construction noise, vehicle noise, talking, etc.).

[1108] Step 7:

[1109] The server generates the optimal masking sound. Based on the analysis results, the server's AI generates the optimal masking sound (e.g., white noise, natural sounds).

[1110] Step 8:

[1111] The server sends the masking sound to the terminal. The server then sends the generated masking sound to the terminal.

[1112] Step 9:

[1113] The terminal receives the masking sound. The terminal receives the masking sound transmitted from the server.

[1114] Step 10:

[1115] The device will play the masking sound, which will then be played through the built-in speaker to effectively camouflage the noise.

[1116] Step 11:

[1117] The user enjoys a comfortable sound environment. The user enjoys a comfortable environment in which noise is reduced by the masking sound.

[1118] Example 1

[1119] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1120] Conventional noise masking systems have the problem of being unable to generate and play appropriate masking sounds in real time even when they detect noise. High analytical accuracy is required to select and generate appropriate masking sounds, and adjustments are also necessary to ensure that the masking sounds are comfortable for the user, but current technology has difficulty meeting these requirements.

[1121] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1122] In this invention, the server includes means for detecting noise, means for recording noise data and transmitting it to the server, means for analyzing the noise data received by the server, means for generating an optimal masking sound based on the analysis results using a generative AI model, means for transmitting the generated masking sound to the terminal, and means for playing the masking sound received by the terminal. This makes it possible to accurately analyze noise in real time and provide the user with a comfortable masking sound.

[1123] The "noise detection means" is a function that uses a built-in microphone to monitor surrounding sounds in real time and detect noise that exceeds a certain threshold.

[1124] The "means for recording noise data and transmitting it to a server" is a function that records the waveform information, frequency components, sound intensity, etc. of the detected noise and transmits the data to a server using a secure communication protocol.

[1125] The "means for analyzing noise data received by the server" refers to a function that analyzes the received noise data using an acoustic processing library or machine learning algorithm to identify the type and characteristics of the noise.

[1126] "Means for generating optimal masking sounds based on analysis results using a generative AI model" refers to a function that uses a generative AI model based on analysis results to select or generate optimal masking sounds.

[1127] The "means for transmitting the generated masking sound to the terminal" is a communication function for transferring the generated masking sound data to the terminal.

[1128] The "means for reproducing the masking sound received by the terminal" is a function for reproducing the masking sound data received by the terminal through the built-in speaker.

[1129] A "generative AI model" is an artificial intelligence algorithm that generates or selects the optimal sound in response to a specific prompt.

[1130] A "prompt sentence" is an input sentence used to generate a specific output for a generative AI model.

[1131] This invention relates to a system that detects and analyzes noise, and generates and reproduces an optimal masking sound based on the noise. Specific embodiments of this system will be described below.

[1132] System Configuration

[1133] The system consists of the following main components:

[1134] 1. Terminal: A device that detects noise and communicates with the server.

[1135] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[1136] 3. User: An individual or group that uses the system and seeks a comfortable sound environment.

[1137] Noise Monitoring

[1138] The user places the device in a location where noise is a problem. The device uses a built-in, highly sensitive microphone to monitor the surrounding sounds in real time. For monitoring, an acoustic library (e.g., Librosa) is used to acquire sound waveform information, frequency components, and sound intensity.

[1139] Noise recording and transmission

[1140] When the device detects noise above a certain threshold, it records the data. The recording includes high-resolution waveform data, frequency content, and sound intensity. The recorded noise data is then sent from the device to a server via a secure communication protocol (e.g., HTTPS).

[1141] Noise data analysis

[1142] The server analyzes the received noise data. Machine learning algorithms (e.g., CNN, RNN) are used to identify the type of sound and its frequency components. The server then provides a prompt to the generative AI model to ask it for the optimal type of masking sound.

[1143] Masking sound generation

[1144] The generative AI model generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, the AI ​​model will select an appropriate natural sound (e.g., a waterfall or a flowing river) or generate a new one. The final masking sound data is stored on a server.

[1145] Sending and playing masking sounds

[1146] The server transmits the generated masking sound to the device, which then plays it back using its built-in speaker. This blends the noise and masking sound, providing a comfortable sound environment for the user.

[1147] Specific examples

[1148] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. The generated waterfall sound data is sent to the device, and the waterfall sound is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly.

[1149] Prompt Sentence Examples

[1150] Examples of prompts you can give to a generative AI model include:

[1151] "Construction noise detected. Please generate suitable masking sound."

[1152] "Based on the frequency content and waveform data, please suggest a sound that will effectively camouflage this noise."

[1153] By using such prompt sentences to have the generative AI model perform advanced sound generation, it is possible to provide optimal masking sounds in real time.

[1154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1155] Step 1: Monitoring the noise

[1156] The user places the device in a location where noise is a problem. The device uses a built-in, highly sensitive microphone to monitor the surrounding sounds in real time. For monitoring, an acoustic library (e.g., Librosa) is used to acquire sound waveform information, frequency components, and sound intensity.

[1157] Input: Real-time sound environment data

[1158] Output: Sound waveform information, frequency components, sound intensity data

[1159] Specific operation: The device's microphone captures surrounding sounds, and the acoustic library analyzes the captured sound data to obtain waveform information and frequency components in real time.

[1160] Step 2: Record and transmit the noise

[1161] When the device detects noise above a certain threshold, it records the data. The recording includes high-resolution waveform data, frequency content, and sound intensity. The recorded noise data is then sent to a server using a secure communication protocol (e.g., HTTPS).

[1162] Input: Monitored sound data

[1163] Output: Noise data sent to the server

[1164] Specific operation: When the device exceeds the noise threshold, it buffers the sound data and then transmits the saved noise data to the server through a secure communication protocol.

[1165] Step 3: Analyze the noise data

[1166] The server analyzes the received noise data. Machine learning algorithms (e.g., CNN or RNN) and acoustic processing libraries are used to identify the type and characteristics of the sound. Based on the analysis results, the server provides a prompt to the generative AI model.

[1167] Input: Transmitted noise data

[1168] Output: type and characteristics of noise, prompt

[1169] Specific operation: The server preprocesses the received sound data using an acoustic processing library, then inputs it into a machine learning model to identify the type and characteristics of the noise. Based on the identification results, it generates and sends a prompt to the generative AI model.

[1170] Step 4: Generate masking sounds

[1171] The generative AI model generates the optimal masking sound based on the prompt. For example, if the sound of construction work is detected, the AI ​​model will select an appropriate natural sound (e.g., a waterfall or a flowing river) or generate a new one. The generated masking sound data is stored on a server.

[1172] Input: prompt sentence, sound database

[1173] Output: Optimal masking sound data

[1174] Specific operation: The generative AI model searches for appropriate masking sounds from a sound database according to the prompt sentence, and generates new sounds as needed. The generated masking sound data is stored on the server.

[1175] Step 5: Send and play the masking sound

[1176] The server transmits the generated masking sound to the device, which then plays it back using its built-in speaker. This blends the noise and the masking sound, providing a comfortable sound environment for the user.

[1177] Input: Generated masking sound data

[1178] Output: Played masking sound

[1179] Specific operation: The server sends the generated masking sound data to the device via a secure communication protocol, and the device decodes the received masking sound and plays it on the built-in speaker. By playing the masking sound, the noise is effectively camouflaged.

[1180] (Application example 1)

[1181] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1182] Modern brick-and-mortar stores are required to provide a comfortable shopping environment for customers, but noise in the store can cause stress and reduce comfort. Noise can also make it difficult to accurately convey product explanations and guidance, reducing the quality of the customer experience. Therefore, there is a need for a system that can suppress noise in real time and provide a comfortable sound environment for customers.

[1183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1184] In this invention, the server includes means for analyzing noise data and generating an optimal masking sound, means for generating the masking sound using a generation AI model, and means for transmitting the generated masking sound to a terminal or a wearable device. This makes it possible to generate an optimal masking sound adapted to a noisy environment in real time and play it on a terminal or a device held by the customer.

[1185] - "Means for detecting noise" means a device for recognizing sounds occurring in the environment and measuring and recording such sounds.

[1186] "Means for analyzing noise data" refers to algorithms or systems that process and analyze collected noise data and identify the type and characteristics of noise.

[1187] The "means for reproducing the generated masking sound" refers to an output device such as a speaker or earphones for providing the masking sound sent from the server to the user.

[1188] "Means for playing masking sounds generated to adapt to a noisy environment through an audio device worn by the user" refers to a mechanism by which the user plays the optimal masking sound through a mobile terminal or wearable device, effectively camouflaging the noise.

[1189] "Means for monitoring noise using mobile or wearable devices" refers to the ability to detect and record noise in real time using portable devices such as smartphones or smart glasses.

[1190] "Means for generating masking sounds using generative AI models" refers to technology that uses machine learning algorithms to generate optimal masking sounds based on the results of noise analysis.

[1191] The "prompt sentence output by the generative AI model" is an instruction sentence that the AI ​​uses to generate masking sounds based on the noise analysis results.

[1192] System configuration

[1193] A system embodying this invention mainly comprises the following components: means for detecting noise, means for analyzing noise data, means for playing back generated masking sounds, means for playing back masking sounds generated to adapt to the noisy environment through an acoustic device worn by the user, and means for monitoring noise using a mobile terminal or a wearable device.

[1194] Hardware and Software

[1195] Devices: Mobile and wearable devices such as smartphones and smart glasses that monitor and record noise in real time.

[1196] Server: A high-performance computer for analyzing noise data and generating masking sounds. It runs generative AI models using machine learning frameworks such as TensorFlow and PyTorch.

[1197] An acoustic device, such as earphones, headphones, or the built-in speakers of smart glasses, that plays the generated masking sound.

[1198] Noise detection and data transmission

[1199] 1. The device detects noise and uses the built-in microphone to monitor the surrounding sound in real time. When noise exceeding a certain threshold is detected, the device records the sound. This noise data includes the sound's waveform information, frequency components, and sound intensity.

[1200] 2. The device sends the recorded noise data to the server.

[1201] Noise data analysis and masking sound generation

[1202] 1. The server analyzes the received noise data using a generative AI model built with TensorFlow and PyTorch. The analysis identifies the type of noise and its frequency components.

[1203] 2. Based on the analysis results, the generative AI model generates the optimal masking sound using a prompt sentence. For example, if the noise is a person speaking, the sound of light rain or wind will be determined to be an appropriate masking sound.

[1204] Example prompt sentence:

[1205] "Perform frequency analysis of human speech and generate non-disturbing natural sounds (e.g., rain or rustling leaves)."

[1206] Playback of masking sound

[1207] 1. The server generates masking sounds and sends them to the user's terminal or wearable device.

[1208] 2. The terminal or wearable device plays the received masking sound through earphones, headphones, or speakers in smart glasses.

[1209] Specific examples

[1210] Consider a specific example of how to make a customer's time in a store more comfortable. For example, a customer is selecting a product in a noisy physical store when noise occurs. A smartphone detects the noise and sends the data to a server. The server analyzes the noise as talking and uses a generative AI model to generate the sound of rain. This rain sound is sent to the customer's smartphone and played through earphones. This allows the customer to enjoy shopping in a comfortable acoustic environment.

[1211] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1212] Step 1:

[1213] The device detects noise and uses the built-in microphone to monitor surrounding sounds in real time. When the noise level exceeds a certain threshold, the device records the sound. At this time, the sound's waveform information, frequency components, and sound intensity are acquired as recorded data. The input is the surrounding ambient sound, and the output is noise data obtained from the microphone.

[1214] Step 2:

[1215] The device sends the recorded noise data to the server. The data includes sound waveform information, frequency components, and sound intensity. The input is the noise data acquired in step 1, and the output is a data packet sent to the server.

[1216] Step 3:

[1217] The server receives the noise data and begins analysis based on the received data. This analysis uses machine learning frameworks such as TensorFlow and PyTorch. The input is the data sent from the device, and the output is the analysis results that identify the type and characteristics of the noise.

[1218] Step 4:

[1219] The server uses a generative AI model to generate optimal masking sounds based on the analysis results. In this process, masking sounds are generated using predefined prompts. The inputs are the analysis results and prompts, and the output is the generated masking sound data. Specifically, the prompt used is, "Perform frequency analysis of human speech and generate non-interfering natural sounds (e.g., the sound of rain or rustling leaves)."

[1220] Step 5:

[1221] The server transmits the generated masking sound to a terminal or wearable device. The input is the generated masking sound data, and the output is a data packet for transmission.

[1222] Step 6:

[1223] The terminal or wearable device receives the masking sound data sent from the server and plays the received data on an acoustic device (such as earphones, headphones, or speakers in smart glasses). The input is the transmitted masking sound data, and the output is the played sound.

[1224] Step 7:

[1225] The user enjoys a comfortable sound environment through the masking sound that is played back. Specifically, the user listens to the masking sound created by the generative AI model as sound that reaches the ears through earphones or headphones.

[1226] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1227] This invention combines a system that detects and analyzes noise, and generates and plays optimal masking sounds based on the noise, with the ability to recognize the user's emotions and optimize the sound environment. Specific embodiments of this system are described below.

[1228] System configuration

[1229] The system consists of the following main components:

[1230] 1. Terminal: A device that detects noise and communicates with the server.

[1231] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[1232] 3. Emotion engine: An engine that recognizes the user's emotions.

[1233] 4. User: An individual or group that uses the system and seeks a comfortable sound environment.

[1234] Program processing

[1235] The processing of the program by each component of the system will be described in detail below.

[1236] Noise detection and data transmission

[1237] The user places the device in a location where noise is a problem (e.g., a conference room or office). The device uses a built-in microphone to monitor the surrounding sounds in real time. When noise exceeding a certain threshold is detected, the device records the sound and transmits the noise data to a server. This data includes the sound's waveform information, frequency components, and sound intensity.

[1238] Noise data analysis

[1239] The server analyzes the received noise data, specifically analyzing the type of sound and frequency components to identify the characteristics of the noise. This analysis uses machine learning algorithms and acoustic processing technology.

[1240] Masking sound generation

[1241] The server's AI generates the optimal masking sound based on the analysis results. For example, if construction sounds are detected, it may determine that natural sounds (such as a flowing river or white noise) are optimal. The AI ​​selects the optimal masking sound from a large sound database and, in some cases, generates a new masking sound.

[1242] Acquiring and analyzing emotion data

[1243] To further optimize the user's emotional state, the emotion engine captures and analyzes the user's facial expression, tone of voice, or biometric information in real time. For example, if the user's facial expression is recognized as tired or stressed, the emotion engine will send this information to the server.

[1244] Adjusting the masking sound

[1245] Based on the user's emotional data obtained from the emotion engine, the server further fine-tunes the generated masking sound. For example, if the user is feeling stressed, it may consider changing the sound to a more relaxing one (e.g., the sound of calming waves).

[1246] Sending and playing masking sounds

[1247] The generated masking sound (sound optimized for the user's emotions) is sent from the server to the device, which then plays it back through its built-in speaker. By mixing this masking sound with the noise, the noise is effectively camouflaged, providing the user with a comfortable sound environment that takes into account their emotional state.

[1248] Specific examples

[1249] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. At the same time, if the server determines that a meeting participant is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a quieter, gentler waterfall. The generated waterfall sound data is sent to the device, and the quiet, gentle sound of a waterfall is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly and participants to meet in a comfortable and relaxed environment.

[1250] In this way, by closely coordinating each step, it is possible to respond to changing noise environments in real time and provide users with an optimal, comfortable sound environment that also takes into account their emotional state.

[1251] The processing flow will be explained below.

[1252] Step 1:

[1253] The user places the terminal. The user places the terminal in a place where noise is a concern, such as a conference room or office.

[1254] Step 2:

[1255] The device detects noise. The device's built-in microphone monitors the surrounding sounds in real time and detects noise that exceeds a certain threshold.

[1256] Step 3:

[1257] The device records the noise data. The device records the detected noise data, which includes sound waveform information, frequency components, volume, etc.

[1258] Step 4:

[1259] The device transmits the noise data to the server. The recorded noise data is transmitted from the device to the server.

[1260] Step 5:

[1261] The server receives the noise data. The server receives the noise data transmitted from the terminal.

[1262] Step 6:

[1263] The server analyzes the noise data and performs analysis to identify the type and characteristics of the sound (e.g., construction noise, vehicle noise, voices, etc.) from the received data.

[1264] Step 7:

[1265] The server generates the optimal masking sound. Based on the analysis results, the server's AI automatically generates the optimal masking sound (e.g., white noise, natural sounds).

[1266] Step 8:

[1267] To recognize the user's emotions, the emotion engine captures the user's facial expressions and tone of voice in real time. The emotion engine collects the user's emotion data using a camera and microphone.

[1268] Step 9:

[1269] The emotion engine analyzes the user's emotional data, determines the user's emotional state from their facial expressions and tone of voice, and sends that data to the server.

[1270] Step 10:

[1271] The server receives and analyzes the emotional data and adjusts the masking sound. Based on the data from the emotion engine, the server fine-tunes the type and volume of the masking sound according to the user's emotional state.

[1272] Step 11:

[1273] The server sends the adjusted masking sound to the terminal. The optimized masking sound data is sent from the server to the terminal.

[1274] Step 12:

[1275] The device receives and plays the masking sound. The device plays the adjusted masking sound through the built-in speaker, effectively camouflaging the noise.

[1276] Step 13:

[1277] Users can enjoy a comfortable sound environment. By reducing noise and providing a sound environment that takes emotional state into consideration, users can enjoy a comfortable sound environment.

[1278] Example 2

[1279] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1280] Existing noise control systems simply detect noise and generate and play masking sounds, making it difficult to provide a flexible sound environment that responds to the user's emotional state. Furthermore, optimizing a noisy environment in real time requires integrated processing of noise detection, masking sound generation, and user emotion recognition, but no system currently exists that can do this efficiently.

[1281] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1282] In this invention, the server includes means for detecting noise, means for analyzing noise data, means for playing back the generated masking sound, means for recognizing and analyzing the emotional state of the user, and means for adjusting the masking sound based on the emotional state, thereby making it possible to provide a comfortable sound environment that takes the emotional state of the user into consideration.

[1283] "Means for detecting noise" refers to a device that monitors sounds in the environment in real time and detects sounds that exceed a threshold.

[1284] "Means for analyzing noise data" refers to technology that uses received sound data to analyze its frequency components, volume, etc., and identify specific noises.

[1285] "Means for reproducing the generated masking sound" refers to a device that selects an appropriate masking sound based on the analysis results and outputs it.

[1286] "Means for recognizing and analyzing the user's emotional state" refers to technology that acquires and analyzes the user's facial expressions, tone of voice, biometric information, etc. to identify the user's emotions.

[1287] The "means for adjusting the masking sound based on the emotional state" refers to a technique for adjusting the type and volume of the generated masking sound depending on the emotional state of the user.

[1288] "Means for transmitting noise data to a server" refers to a technology for transmitting noise data collected by a terminal to a server via a network.

[1289] The "means by which the server analyzes the noise data and generates an optimal masking sound" refers to a function by which the server analyzes the received noise data and generates an optimal masking sound.

[1290] The "means for transmitting the generated masking sound to the terminal" refers to a technique for transmitting the masking sound generated by the server to the terminal via a network.

[1291] "Means for acquiring a user's emotional state using their facial expressions, tone of voice, and biometric information" refers to devices such as cameras, microphones, and biometric sensors for acquiring a user's emotional state in real time, as well as the analysis technology for those devices.

[1292] MODE FOR CARRYING OUT THE INVENTION

[1293] This invention is a system that combines a system that detects and analyzes noise, and generates and plays optimal masking sounds based on the noise, with a function that recognizes the user's emotions and optimizes the sound environment. Specific embodiments of this system are described in detail below.

[1294] System configuration

[1295] The system consists of the following main components:

[1296] 1. Terminal: A device that detects noise and communicates with the server.

[1297] 2. Server: A computer system that analyzes noise data and generates optimal masking sounds.

[1298] 3. Emotion engine: An engine that recognizes the user's emotions.

[1299] 4. User: An individual or group that uses the system and seeks a comfortable sound environment.

[1300] Hardware and Software

[1301] Terminal

[1302] The device has a built-in microphone and speaker, a camera to recognize the user's facial expressions, a microphone to analyze the tone of voice, and sensors to capture biometric information. These devices are used to detect noise and capture emotional data.

[1303] server

[1304] The server requires powerful computing resources to analyze the noise data and generate masking sounds, using machine learning frameworks such as TensorFlow and PyTorch, as well as the Librosa acoustic processing library.

[1305] Emotion Engine

[1306] The emotion engine uses OpenFace to analyze the user's facial expressions and PyAudio to analyze audio data, making it possible to analyze the user's emotional state in real time.

[1307] Data processing and calculation

[1308] The system processes the following data:

[1309] 1. Noise detection and data transmission

[1310] The user places the device in a location where there is a noise problem. The device's built-in microphone monitors the surrounding sounds in real time, and when noise exceeding a certain threshold is detected, the device records the sound and transmits the noise data to the server. This data includes the sound's waveform information, frequency components, and sound intensity.

[1311] 2. Analysis of noise data

[1312] The server analyzes the received noise data, specifically by analyzing the frequency components of the sound using FFT (Fast Fourier Transform) and classifying the type of sound using a machine learning model.

[1313] 3. Masking sound generation

[1314] Based on the analysis results, the server's AI generates the optimal masking sound. The AI ​​selects the optimal masking sound from a large sound database (e.g., FreeSound) and, in some cases, generates a new masking sound.

[1315] 4. Acquisition and Analysis of Emotion Data

[1316] To further optimize the user's emotional state, the emotion engine captures and analyzes the user's facial expressions, tone of voice, or biometric information in real time through cameras, microphones, and biometric sensors.

[1317] 5. Adjusting the masking sound

[1318] Based on the data from the emotion engine, the server further adjusts the generated masking sounds, for example, changing them to more relaxing sounds if the user is feeling stressed.

[1319] 6. Sending and playing masking sounds

[1320] The generated masking sound is sent from the server to the device, which then plays it back through its built-in speaker. This sound blends with the noise, providing a comfortable sound environment for the user.

[1321] Specific examples

[1322] For example, consider a situation where users are in a meeting in a conference room and hear the sounds of construction work. A device in the conference room detects the construction sounds and sends the data to a server. The server analyzes the construction sounds and generates an appropriate waterfall sound. At the same time, if the server determines that a meeting participant is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a quieter, gentler waterfall. The generated waterfall sound data is sent to the device, and the quiet, gentle sound of a waterfall is played from the speakers in the conference room. This effectively masks the construction sounds, allowing the meeting to proceed smoothly and participants to meet in a comfortable and relaxed environment.

[1323] Prompt Sentence Examples

[1324] "There's construction noise during a meeting. Generate optimal sounds to mask the noise and adjust the sound based on the emotional data of the meeting participants."

[1325] In this way, this system can respond to changing noise environments in real time and provide users with an optimal, comfortable sound environment that also takes into account their emotional state.

[1326] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1327] Step 1: Noise detection and data transmission

[1328] The user places the device in a location where there is a noise problem. The device's built-in microphone monitors the surrounding sounds in real time, and if noise exceeding a certain threshold is detected, the device records the sound. The recorded data includes the sound's waveform information, frequency components, and sound intensity. After recording, the device compresses the data and sends it to the server.

[1329] Input: Ambient sound

[1330] Output: Compressed noise data

[1331] Specific operation: The device samples the sound waveform data and compares it with a threshold. If the threshold is exceeded, the device starts recording, then compresses the data and sends it to the server.

[1332] Step 2: Analyze the noise data

[1333] The server receives the noise data sent from the device, analyzes the frequency components using FFT (Fast Fourier Transform), and classifies the type of sound using a machine learning model (e.g., random forest or neural network).The analysis results are stored in a database.

[1334] Input: Compressed noise data

[1335] Output: Analysis results (sound frequency components and types)

[1336] Specific operation: The server reads the received data, applies FFT to extract frequency components, and classifies the sound type using a machine learning model. The analysis results are saved in a database.

[1337] Step 3: Generate the masking sound

[1338] The server's generation AI generates the optimal masking sound based on the analysis results. The generation AI selects the optimal masking sound from many sound databases (e.g., FreeSound) and generates new masking sounds as needed. The generated sound data is saved as a temporary file.

[1339] Input: Analysis results (sound frequency components and types)

[1340] Output: Generated masking sound data

[1341] Specific operation: The server's generation AI selects the optimal masking sound from the database based on the analysis results. If necessary, it generates new sounds and saves them in a temporary file.

[1342] Step 4: Acquire and analyze emotion data

[1343] The emotion engine installed in the device captures and analyzes the user's facial expressions, tone of voice, and biometric information in real time. Specifically, the emotion engine analyzes the data using tools such as OpenFace and PyAudio, and transmits the obtained emotional state to the server.

[1344] Input: User's facial expression, tone of voice, biometric information

[1345] Output: Parsed emotion data

[1346] Specific operation: The device captures the user's facial expressions and voice using the camera and microphone, analyzes them using OpenFace and PyAudio, and sends the analysis results to the server.

[1347] Step 5: Adjust the masking sound

[1348] The server receives the emotion data and adjusts the generated masking sound. If the user is feeling stressed, it changes the sound to a relaxing one. The adjusted sound data is also saved in a temporary file.

[1349] Input: Analyzed emotion data, generated masking sound data

[1350] Output: Adjusted masking sound data

[1351] Specific operation: The server analyzes the received emotional data and adjusts the generated masking sound based on the user's emotional state. The adjusted sound is saved in a temporary file.

[1352] Step 6: Send and play masking sounds

[1353] The server sends the adjusted masking sound data to the device, which then decodes the received masking sound and plays it through its built-in speaker, effectively camouflaging the noise.

[1354] Input: Adjusted masking sound data

[1355] Output: Masking sound played through the speaker

[1356] Specific operation: The server sends the generated sound data to the device, which decodes it and plays it through the speaker, optimizing the sound environment.

[1357] Through these steps, it becomes possible to provide a comfortable sound environment for users and optimize the noise environment in real time.

[1358] (Application example 2)

[1359] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1360] In modern brick-and-mortar retail environments, noise is a major problem that disrupts customers' comfortable shopping experiences. Furthermore, there is a need to provide a sound environment that not only reduces noise but also takes into account the emotional state of the customer. However, conventional noise masking systems only detect noise data and generate masking sounds, but do not support optimizing the sound environment based on the user's emotions. This poses a challenge in improving customer satisfaction in stores.

[1361] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1362] In this invention, the server includes means for analyzing noise data, means for generating the generated masking sound, means for recognizing the user's emotion and optimizing the sound environment, and means for transmitting the user's emotion data to the server. This makes it possible to centrally manage everything from noise detection and analysis to the generation and playback of masking sound according to the emotion, and to provide the optimal sound environment for customers in a physical store.

[1363] The "means for detecting noise" is a function that monitors sounds in the environment in real time and detects noise that exceeds a certain threshold.

[1364] The "means for analyzing noise data" is a function for analyzing detected noise data based on the type of sound, frequency components, intensity, etc.

[1365] The "means for reproducing the generated masking sound" is a function for reproducing the masking sound generated based on the analysis results through an audio output device such as a speaker.

[1366] "Means to recognize the user's emotions and optimize the sound environment" refers to a function that analyzes the user's facial expressions, tone of voice, biometric information, etc. to recognize emotions in real time, and selects and adjusts the optimal masking sound based on that.

[1367] "Means for transmitting noise data to a server" refers to a function for transmitting detected and recorded noise data to a server via a communication means such as the Internet.

[1368] The "means for the server to analyze noise data and generate the most suitable masking sound" is a function that analyzes the noise data received by the server and generates the most suitable masking sound.

[1369] The "means for transmitting the generated masking sound to the terminal" is a function for transmitting the optimal masking sound generated by the server to the terminal via a communication means such as the Internet.

[1370] The "means for transmitting user emotion data to the server" is a function for transmitting data obtained by analyzing the user emotion to the server.

[1371] "Means for adjusting to provide a comfortable sound environment" refers to a function that further fine-tunes the masking sounds generated and selected based on the user's emotions and noise characteristics to further enhance user comfort.

[1372] The present invention provides a system that effectively masks noise in environments such as brick-and-mortar stores, while providing a sound environment that takes into consideration the emotional state of the user. Specific embodiments for carrying out the present invention are described below.

[1373] System Configuration

[1374] This system consists of a terminal that detects noise, a server that performs analysis and generation, an emotion engine that recognizes the user's emotions, and a user.

[1375] Noise detection

[1376] The device has a built-in microphone for real-time noise monitoring. It constantly monitors the sounds in the environment, and when it detects noise exceeding a certain threshold, it records the sound's waveform information, frequency components, sound intensity, etc., and recognizes it as noise data.

[1377] Noise data transmission and analysis

[1378] The device sends the detected noise data to a server, which then analyzes the received noise data to identify the type and characteristics of the sound. This analysis is performed using machine learning algorithms and acoustic processing technologies (e.g., TensorFlow, PyTorch, etc.).

[1379] Masking sound generation

[1380] The server generates optimal masking sounds based on the analysis results. For example, natural sounds, especially the sound of a waterfall or white noise, may be suitable for noise caused by construction work. Using a generative AI model, optimal masking sounds are selected from an existing database or newly generated.

[1381] Emotion Recognition and Data Transmission

[1382] The user's emotional data (e.g., facial expressions, tone of voice, biometric information, etc.) recognized on the device is analyzed through the emotion engine. The analysis is performed using a camera and microphone, and machine learning models (e.g., OpenCV, TensorFlow) are used for emotion recognition. The recognized emotional data is then sent to the server.

[1383] Adjusting and playing masking sounds

[1384] The server receives the emotional data and further adjusts the masking sound based on the user's emotional state (for example, if the user is stressed, the sound may be changed to a more relaxing sound). The adjusted masking sound is then sent to the device and played through the device's speaker.

[1385] Specific examples

[1386] For example, imagine a cafe in a shopping mall with many customers and different types of noises occurring in the background. The device detects the noise and sends the data to the server. The server analyzes the data and generates the sound of a flowing river as the optimal masking sound. At the same time, if the server determines that the customer is feeling stressed based on their facial expression or tone of voice, it adjusts the sound to a more gentle sound of a flowing river. The adjusted sound is sent to the device and played in the cafe, allowing the customer to enjoy a comfortable acoustic environment.

[1387] Prompt Sentence Examples

[1388] "Analyze current noise data and generate optimal masking sound."

[1389] "Recognize the emotions from the user's facial expressions and select sounds that have a relaxing effect."

[1390] This makes it possible to realize a system that can solve noise problems in brick-and-mortar stores and provide a comfortable sound environment for customers. This invention is an important technology for providing a relaxing environment for store users.

[1391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1392] Step 1:

[1393] The device monitors the noise in the environment in real time and detects noise exceeding a certain threshold. The input is the environmental sound picked up by the device's microphone. The output is the detected noise data, which includes sound waveform information, frequency components, sound intensity, etc.

[1394] Step 2:

[1395] The terminal transmits the detected noise data to a server via the Internet. This process uses a transmission protocol (e.g., HTTP). The input is the detected noise data, and the output is a confirmation of transmission to the server.

[1396] Step 3:

[1397] The server analyzes the received noise data. This analysis uses machine learning algorithms (e.g., TensorFlow, PyTorch) to identify the type of sound and its frequency components. The input is the transmitted noise data, and the output is the analysis results.

[1398] Step 4:

[1399] The server generates the optimal masking sound based on the analysis results. A generative AI model is used for generation. The input is the analysis results, and the output is the generated masking sound. The generated sound is selected from natural sounds (such as the sound of a waterfall or white noise).

[1400] Step 5:

[1401] The user's emotional data (e.g., facial expressions, tone of voice, biometric information, etc.) recognized by the device is analyzed through the emotion engine. Real-time data from the camera and microphone is input for this analysis. The analysis results are sent to the server. The input is the user's real-time data, and the output is emotional data.

[1402] Step 6:

[1403] The server receives the emotional data and adjusts the masking sound based on the user's emotional state. For example, if the user is feeling stressed, the sound is changed to one with a relaxing effect. The input is the emotional data and the generated masking sound, and the output is the adjusted masking sound.

[1404] Step 7:

[1405] The adjusted masking sound is sent to the device and played back from the device's speaker. The input is the adjusted masking sound, and the output is the sound to be played back. This sound is mixed with the noise to create a comfortable sound environment for the user.

[1406] Through this series of steps, noise detection, analysis, generation of masking sounds, adjustment based on the user's emotions, and playback are all seamlessly performed, providing a comfortable sound environment for users in physical stores. This system not only effectively solves noise problems, but also provides a new experience that takes into account the user's emotional state.

[1407] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1408] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1409] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1410] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1411] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1412] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1413] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1414] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1415] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1416] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1417] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1418] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1419] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1420] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1421] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1422] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1423] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1424] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1425] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1426] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1427] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1428] The following is further disclosed regarding the above embodiment.

[1429] (Claim 1)

[1430] a means for detecting noise;

[1431] a means for analyzing the noise data;

[1432] means for playing the generated masking sound;

[1433] A system including:

[1434] (Claim 2)

[1435] means for transmitting the noise data to a server;

[1436] means for the server to analyze the noise data and generate an optimal masking sound;

[1437] means for transmitting the generated masking sound to a terminal;

[1438] The system of claim 1 further comprising:

[1439] (Claim 3)

[1440] 10. The system of claim 1, further comprising means for adjusting the masking sound to provide a comfortable sound environment for the user.

[1441] "Example 1"

[1442] (Claim 1)

[1443] a means for detecting noise;

[1444] means for recording noise data and transmitting the data to a server;

[1445] means for analyzing the noise data received by the server;

[1446] A means for generating optimal masking sounds based on the analysis results using a generative AI model;

[1447] means for transmitting the generated masking sound to a terminal;

[1448] means for playing the masking sound received by the terminal;

[1449] A system including:

[1450] (Claim 2)

[1451] means for the generative AI model to select or generate a masking sound based on a prompt sentence;

[1452] a means for the server to use a machine learning algorithm to identify the type and frequency components of the noise data;

[1453] a means for the device to monitor noise using a built-in microphone and speaker and play a masking sound;

[1454] 10. The system of claim 1, comprising:

[1455] (Claim 3)

[1456] 10. The system of claim 1, further comprising means for adjusting the masking sound in real time to provide a comfortable sound environment for the user.

[1457] "Application Example 1"

[1458] (Claim 1)

[1459] a means for detecting noise;

[1460] a means for analyzing the noise data;

[1461] means for playing the generated masking sound;

[1462] a means for reproducing a masking sound generated in accordance with a noise environment through an acoustic device worn by a user;

[1463] A means for monitoring noise using a mobile terminal or a wearable device;

[1464] A system including:

[1465] (Claim 2)

[1466] means for transmitting the noise data to a server;

[1467] means for the server to analyze the noise data and generate an optimal masking sound;

[1468] means for transmitting the generated masking sound to a terminal or a wearable device;

[1469] a means for generating a masking sound using a generative AI model;

[1470] The system of claim 1 further comprising:

[1471] (Claim 3)

[1472] means for adjusting the masking sound to provide a comfortable sound environment for a user;

[1473] a means for generating a masking sound in response to a prompt sentence output by the generative AI model;

[1474] 10. The system of claim 1, comprising:

[1475] "Example 2: Combining Emotion Engines"

[1476] (Claim 1)

[1477] a means for detecting noise;

[1478] a means for analyzing the noise data;

[1479] means for playing the generated masking sound;

[1480] means for recognizing and analyzing the emotional state of a user;

[1481] means for adjusting a masking sound based on said emotional state;

[1482] A system including:

[1483] (Claim 2)

[1484] means for transmitting the noise data to a server;

[1485] means for the server to analyze the noise data and generate an optimal masking sound;

[1486] means for transmitting the generated masking sound to a terminal;

[1487] The system of claim 1 further comprising:

[1488] (Claim 3)

[1489] 10. The system of claim 1, further comprising means for capturing an emotional state using a user's facial expression, tone of voice, and biometric information.

[1490] "Application example 2 when combining emotion engines"

[1491] (Claim 1)

[1492] a means for detecting noise;

[1493] a means for analyzing the noise data;

[1494] means for playing the generated masking sound;

[1495] A means for recognizing a user's emotions and optimizing the sound environment;

[1496] A system including:

[1497] (Claim 2)

[1498] means for transmitting the noise data to a server;

[1499] means for the server to analyze the noise data and generate an optimal masking sound;

[1500] means for transmitting the generated masking sound to a terminal;

[1501] means for transmitting the user's emotion data to a server;

[1502] The system of claim 1 further comprising:

[1503] (Claim 3)

[1504] 10. The system of claim 1, further comprising means for adjusting the masking sound to provide a comfortable sound environment for the user. [Explanation of symbols]

[1505] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for detecting noise; a means for analyzing the noise data; means for playing the generated masking sound; A system including:

2. means for transmitting the noise data to a server; means for the server to analyze the noise data and generate an optimal masking sound; means for transmitting the generated masking sound to a terminal; The system of claim 1 further comprising:

3. 2. The system of claim 1, further comprising means for adjusting the masking sound to provide a comfortable sound environment for the user.

Images