system

Abstract

We provide the system. [Solution] Means for collecting audio signals, A means for analyzing the aforementioned audio signal to identify multiple people participating in the conversation, A means of determining rules to restrict the disclosure of private information based on identified individuals, A means of adjusting the audio output by applying the said rule, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern communication, when using voice recognition technology, there is a risk that the user's privacy information may be inadvertently disclosed to a third party. In particular, when using a voice assistant or an AI agent, in a situation where multiple people participate in a conversation, the risk of such information leakage increases. To address this issue, it is necessary to provide an environment in which users can use voice devices with confidence.

Means for Solving the Problems

[0005] The present invention provides a system equipped with means for collecting voice signals, which uses means for analyzing the voice signals to identify conversation participants, thereby determining rules to restrict the disclosure of private information when participants include someone other than the user themselves. Furthermore, by providing means for adjusting the voice output based on these rules, the system prevents the user's private information from being disclosed to third parties. This configuration provides a system that protects user privacy while not compromising the convenience of the voice assistant.

[0006] An "audio signal" is data obtained by converting fluctuations in sound pressure into an electrical signal, and is acquired through an acoustic device such as a microphone.

[0007] "Analysis" refers to the means of examining data and information in detail to grasp and understand specific elements and structures.

[0008] "Multiple individuals participating in a conversation" refers to different speakers who can be identified through the audio signal, and this information is identified based on the characteristics of the audio data.

[0009] "Identification" is the process of distinguishing different pieces of information or elements and recognizing their respective characteristics and attributes.

[0010] "Privacy information" refers to confidential information about an individual that should not be disclosed to a third party without the individual's permission.

[0011] A "rule" is a set of principles or standards established to regulate or restrict actions or processes under specific circumstances.

[0012] "Adjusting audio output" is the process of modifying or limiting the information output from an audio system to meet specific requirements or conditions. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] [[ID=二十九]]It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, a numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system that utilizes an AI agent while protecting user privacy using speech recognition technology. Specifically, the terminal uses a microphone to acquire ambient audio signals and transmits this data to a server. The server analyzes the audio signals and identifies multiple people participating in the conversation. Based on the identification results, the server determines rules to restrict the disclosure of private information and transmits them to the terminal.

[0035] The device adjusts voice output based on received rules to prevent the disclosure of private information to third parties other than the user. This control allows the user to prevent unintended disclosure of private information while using the AI ​​agent.

[0036] As a concrete example, consider a scenario where a user and a friend ask an AI assistant about their schedules. The device captures the conversation between the user and the friend and sends that data to a server. The server performs voice analysis to identify the user and the friend. The server then creates rules that restrict the disclosure of certain time information and place names, taking into account the presence of the friend. The device applies these rules and provides only abstract information about specific times and places that should not be disclosed in front of the friend.

[0037] This invention aims to provide users with an environment in which they can use voice devices with peace of mind and to reduce privacy risks in voice-based communication.

[0038] The following describes the processing flow.

[0039] Step 1:

[0040] The device acquires ambient sound in real time through its microphone. Because the acquired audio signal is too large in data format, it undergoes noise filtering and compression to make it efficiently transmitted to the server.

[0041] Step 2:

[0042] The terminal sends voice data to the server at pre-configured times. Since the voice data reaches the server via the network, an appropriate protocol is used to minimize transmission delays and data loss.

[0043] Step 3:

[0044] The server uses a speech recognition engine to convert the audio data received from the terminal into text data. Based on the converted text data, it identifies the multiple people participating in the conversation and creates a participant list based on their identification information.

[0045] Step 4:

[0046] The server checks the participant list to determine if it includes any third parties other than the user. If there are individuals other than the user, it generates filtering rules that define what information should be made public and what information should remain private, based on a set of rules in accordance with the privacy policy.

[0047] Step 5:

[0048] The server sends the generated filtering rules to the terminal. These rules specify how to control the user's audio output and are intended to be applied on the terminal side.

[0049] Step 6:

[0050] The terminal adjusts audio output based on filtering rules received from the server. Specifically, content containing private information is replaced with general information, silence, or another signal sound, thereby implementing information control to prevent it from being heard by third parties.

[0051] Step 7:

[0052] Users continue conversations with friends and other participants via their voice devices. Device-side adjustments allow users to naturally provide only the necessary information within the conversation while preventing unintended privacy leaks.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] The present invention aims to provide an environment in which users can use voice agents with peace of mind while reducing the risk of privacy infringement in voice-based communication. In particular, there is a challenge in preventing the leakage of private information to third parties other than those involved when collecting and analyzing voice data, even when multiple people are involved.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server is a device for acquiring audio data, the device including means for removing noise, means for securely transmitting the audio data to another computing device, and means for identifying a person using speech recognition and natural language processing techniques. This makes it possible to determine rules for abstracting or hiding specific information and to adjust the audio to provide a new output to the user.

[0058] "Audio data" refers to information that represents ambient sounds in digital format.

[0059] "Noise reduction" is a process that removes unwanted background noise and other sounds from audio data, making the target audio clearer.

[0060] A "secure method" refers to a means of using encryption technology and secure protocols to maintain the confidentiality and integrity of information during data transmission and reception.

[0061] "Other computing devices" refers to external computers or servers connected to process, analyze, or store audio data.

[0062] "Speech recognition" is a technology that analyzes audio data and converts what is being said into text.

[0063] "Natural language processing technology" refers to computer technologies used to understand, generate, and analyze natural language used by humans.

[0064] "Identifying individuals" means identifying the speakers contained in audio data and distinguishing them based on the characteristics of each individual voice.

[0065] "Specific information" refers to personal information or information that may affect privacy that can be identified within the audio data.

[0066] "Abstraction" is the process of omitting the details of specific information and converting it into a more general expression.

[0067] "To hide" refers to the process of removing or concealing specific recognized information so that it is not visible or audible to other listeners.

[0068] "Rules" refer to instructions or standards regarding the restrictions on the disclosure and output methods of specific information, which are determined based on the analysis of audio data.

[0069] "Adjusting audio" means changing, editing, or processing the output generated from audio data according to specific rules.

[0070] "New output" refers to the adjusted voice message or information provided to the user based on the voice data.

[0071] The system in this invention is a data processing system that utilizes speech recognition technology and is designed for using a voice agent while protecting user privacy. The terminal collects ambient sound using a digital microphone and performs noise reduction processing. The processed voice data is transmitted to the server via a secure communication protocol.

[0072] The server converts audio data into text via a speech recognition engine. This process utilizes speech recognition software such as Google® Cloud Speech-to-Text API. The converted text data is analyzed using natural language processing techniques to identify multiple individuals involved in the conversation. This identifies the linguistic characteristics of each individual involved, contributing to the identification of personal information.

[0073] Subsequently, the server develops rules to control the disclosure of private information based on the identified data. For example, rules may be constructed so that information is hidden or abstracted when a specific keyword is recognized. The terminal receives these rules and adjusts its audio output to prevent individual private information from being leaked to third parties other than the user.

[0074] As a concrete example, consider a user asking an AI assistant, "Do you have any plans next Friday?" The device receives this question and sends data to the server. If the analysis identifies next Friday as a private appointment, the server sets up rules to abstract that information and provide a general response such as, "You have an appointment coming up."

[0075] Examples of prompts to input into a generative AI model include questions like, "How can a user hide specific confidential information while conversing with someone?" Such a system would make it possible to maintain the convenience of voice communication while protecting privacy.

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The device uses a microphone to capture ambient sound. This audio data is then processed using noise reduction technology to remove unwanted noise, resulting in clean audio data. This clean audio data is then passed on to the next processing step.

[0079] Step 2:

[0080] The terminal encrypts clean audio data and sends it to the server. This encryption process uses the SSL / TLS protocol to ensure data confidentiality and integrity. After the server receives the data, it is prepared for analysis.

[0081] Step 3:

[0082] The server inputs the received audio data into a speech recognition engine and converts it into text data. This process uses speech recognition software to accurately transcribe the audio content into text. The text data is then passed on to subsequent natural language processing.

[0083] Step 4:

[0084] The server uses natural language processing techniques to analyze text data and identify multiple people in a conversation. This process uses contextual analysis and speaker identification algorithms to pinpoint each person's utterance. The identified person information is then stored in a database.

[0085] Step 5:

[0086] The server develops privacy protection rules based on the identified individuals and the content of their statements. If specific keywords or phrases are detected, it determines guidelines for abstracting or concealing that information. These rules are then passed on to the next stage.

[0087] Step 6:

[0088] The server sends the established rules to the terminal. The terminal adjusts the audio output according to these rules, providing the user with modified audio feedback. This allows the user to receive the voice agent's output in a privacy-protected manner.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] In modern homes and with devices used by multiple people, there is a risk of privacy violations if information provided by voice assistants is leaked to third parties. It is necessary to solve this problem and provide an environment where users can use voice assistants with peace of mind.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes means for collecting voice information, means for analyzing the voice information to identify multiple members participating in the conversation, and means for determining a policy to restrict the disclosure of confidential information based on the identified members. This makes it possible to selectively output individual information and prevent the leakage of confidential information.

[0094] "Audio information" refers to sound data acquired through devices such as microphones.

[0095] "Members" refers to the multiple individuals or characters participating in the conversation within the audio information.

[0096] "Confidential information" refers to important personal information that, if made public, could potentially infringe on privacy.

[0097] A "policy" refers to specific rules and regulations established to restrict the disclosure of confidential information.

[0098] "Audio output" refers to the sound output from speakers or other devices that is generated based on audio information.

[0099] "Selectively outputting information" means restricting or adjusting the information output based on specific conditions.

[0100] The system for carrying out the present invention consists of multiple components. Voice information is acquired by a microphone installed in a terminal. This terminal is integrated into a device such as a smart home assistant. The voice information is sent from the terminal to a server in the cloud and processed by the server. This processing utilizes the Google Cloud Speech-to-Text API, which leverages speech recognition technology. Using this API, the voice data is converted into text data.

[0101] The server uses the converted text data to identify multiple members, leveraging NLP (Natural Language Processing) techniques. This analysis on the server utilizes, for example, IBM Watson® natural language processing services. After members are identified, policies are generated to restrict information disclosure for privacy purposes. These policies place particular emphasis on keeping confidential information private.

[0102] The generated policy is sent to the terminal. The terminal adjusts the audio output based on this policy. Specifically, it selectively outputs individual information that only certain individuals need to know, for example, through the voice assistant function. This makes it possible to prevent the leakage of confidential information.

[0103] To give an example, consider a scenario where a robot is active in a living room at home. If the mother gives the robot a voice command to check the schedule, the robot will provide the children present with a general overview of the schedule and provide the mother with detailed information via smartphone.

[0104] An example of a prompt for a generative AI model is: "Explain how to identify a specific person from the acquired voice data, analyze the content of the conversation, and create and apply rules to protect privacy."

[0105] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0106] Step 1:

[0107] The device acquires audio information using its built-in microphone. This audio information is recorded in real time and temporarily stored as digital data within the device. In this case, the input is physical sound, and the output is digitized audio data.

[0108] Step 2:

[0109] The terminal sends the acquired audio data to a server in the cloud. This process involves data processing that packets the audio data. The input is digital audio data, and the output is packet data that is transmitted over the network.

[0110] Step 3:

[0111] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API. The input here is audio packet data, and the output is text data. Speech recognition technology analyzes the characteristics of the sound and generates information as a string of characters.

[0112] Step 4:

[0113] The server uses NLP technologies such as IBM Watson based on text data to identify multiple participants in a conversation. The input is text data, and the output is participant identification information. Natural language processing is performed here to understand the context.

[0114] Step 5:

[0115] The server determines policies to restrict the disclosure of confidential information based on identified member information. These policies are dynamically generated using a generative AI model. The input is identification information, and the output is a set of rules representing the policies.

[0116] Step 6:

[0117] The terminal adjusts the audio output based on policies received from the server. Filtering is performed to shield specific information or transmit it only to appropriate targets. The input is the policy rule set, and the output is the adjusted audio output.

[0118] Step 7:

[0119] The user receives voice output from the robot and confirms the necessary information. Here, the input is the adjusted voice output, and the output is the user's understanding and actions. The user can use the information from the robot to efficiently perform everyday tasks.

[0120] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0121] This invention is a system that combines speech recognition technology and emotion recognition technology to enable the use of a more personalized voice assistant while protecting user privacy. The terminal uses a microphone to acquire ambient audio signals and transmits this data to a server. The server analyzes the audio signals to identify multiple people participating in a conversation, and at the same time uses an emotion engine to determine the user's emotional state from the voice.

[0122] Based on emotion recognition, the server generates rules that consider the user's current emotional state to determine what information to disclose and how. These rules are adjusted according to the degree of privacy, meaning that the presentation of private information may vary depending on the user's emotions.

[0123] The terminal receives filtering rules sent from the server and adjusts the audio output. At this time, information is presented that matches the user's emotional state, ensuring a comfortable response for the user.

[0124] As a concrete example, consider a scenario where a user asks an AI assistant for schedule information with a friend. The device collects voice data and sends it to a server. The server uses voice analysis and an emotion engine to detect if the user is "anxious." Based on this, the server creates rules to avoid providing excessive details to the friend and to provide only the minimum necessary information. The device applies these rules and outputs information that is appropriately adjusted when the friend is present.

[0125] This invention enables the use of a flexible voice assistant that responds to the user's emotions while protecting the user's privacy.

[0126] The following describes the processing flow.

[0127] Step 1:

[0128] The device continuously acquires ambient audio signals using its built-in microphone. These audio signals are de-noised and compressed before being sent to the server.

[0129] Step 2:

[0130] The terminal sends the collected voice data to the server. At this time, it adjusts the packet delivery to ensure efficiency depending on the network conditions.

[0131] Step 3:

[0132] The server analyzes the received audio data and converts it into text data through speech recognition. In parallel, the emotion engine analyzes the user's voice characteristics from the same audio data and determines their emotional state in real time.

[0133] Step 4:

[0134] The server identifies the people participating in the conversation based on the speech recognition results. The emotion engine determines whether the user's emotions are in a specific state (e.g., excited, anxious, relaxed).

[0135] Step 5:

[0136] The server determines appropriate filtering rules to restrict the disclosure of private information based on the identified person's information and the user's emotional state. These rules apply stricter restrictions, such as when emotions are particularly heightened.

[0137] Step 6:

[0138] The server sends the determined filtering rules to the terminal and provides instructions for the terminal to adjust the audio output.

[0139] Step 7:

[0140] The device adjusts the audio output in real time according to the received filtering rules. This ensures that the privacy information of the conversation is properly protected while also enabling responses that are sensitive to the user's emotions.

[0141] Step 8:

[0142] Users can continue conversations with third parties with peace of mind by receiving appropriately filtered and adjusted responses. This system protects both user privacy and emotional security.

[0143] (Example 2)

[0144] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0145] Voice assistants often provide information without adequately considering user privacy. As a result, users risk privacy violations due to a lack of appropriate information disclosure control in specific situations or emotional states. Furthermore, current technology makes it difficult to control information disclosure based on the user's emotional state, resulting in the inability to provide flexible and personalized information as intended by the user.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0147] In this invention, the server includes means for analyzing voice data to identify individual people, means for determining emotional states based on voice characteristics, and means for generating information disclosure control rules according to the determined emotional states. This enables flexible information disclosure control tailored to the user's emotional state, allowing for personalized information provision while protecting user privacy.

[0148] "Audio data" refers to audio signals acquired using audio equipment such as microphones, which have been converted into an analyzable digital format.

[0149] "Terminal means" refers to a device or mechanism that performs tasks such as acquiring, transmitting, and receiving voice data, and applying filtering rules.

[0150] "Information processing unit" refers to a computing device or computer system used to receive, analyze, identify, and generate rules for audio data.

[0151] "Means of identifying individuals" refer to algorithms and systems that analyze audio data to identify the speaker.

[0152] "Vocal characteristics" refer to attributes such as tone, speed, and volume of voice, and these attributes are used to determine emotional states.

[0153] "Means for determining emotional state" refers to technologies and algorithms used to infer a user's emotions and psychological state based on voice data.

[0154] "Information disclosure control rules" are criteria or policies that determine which information to disclose or withhold, and how, depending on the user's specific emotional state.

[0155] "Terminal means for adjusting data output" refers to a device or program that adjusts the format and content of information provided externally based on the generated information disclosure control rules.

[0156] Modes for carrying out the invention

[0157] This invention is a voice support system that integrates speech recognition technology and emotion recognition technology to provide information tailored to individual emotional states while prioritizing user privacy.

[0158] Hardware and software to be used

[0159] Device: To acquire audio, an audio device such as a smartphone or smart speaker is used. The built-in microphone is used for audio recording, and the device's operating system (e.g., Android® OS, iOS) is used to temporarily store the audio data.

[0160] Server: A cloud data processing system is used to analyze audio data and recognize emotions. A speech recognition API (e.g., a general speech recognition API) is used for analyzing the audio signal, and an AI service for emotion analysis (e.g., a general emotion analysis service) is used to determine the emotional state.

[0161] Data processing flow

[0162] The terminal acquires audio and transmits the data to the server via a secure connection. The server uses the received audio data to first convert it into text using a speech recognition system, then analyzes the audio characteristics to identify individual participants and determine their emotional states. The generative AI model used in this system (e.g., a general generative model) generates information disclosure control rules based on the user's emotions based on this information. The terminal receives these rules and adjusts the audio information it outputs to the user and other participants.

[0163] Specific example

[0164] For example, consider a scenario where a user wants to check their schedule information with an AI assistant while with a friend. If the user is feeling a little nervous, the server recognizes this emotional state and generates a rule to provide only the minimum necessary information, taking the presence of the friend into account. The device applies this rule and provides the user with appropriately adjusted voice information, such as, "We'll let you know the details of the schedule later."

[0165] Example of a prompt

[0166] "I want to check my schedule while I'm with my friends. However, I'm secretly nervous, so please keep it as concise as possible."

[0167] "I want to receive important announcements during the meeting, but I don't want the details to be known to others."

[0168] This invention allows users to enjoy a flexible and personalized voice assistant experience that is tailored to their emotions, while also protecting their privacy.

[0169] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0170] Step 1:

[0171] The device acquires ambient sound through its microphone. When the user naturally speaks a request to the voice assistant, the audio signal is input to the device as digital data. This data undergoes data processing such as noise reduction and volume normalization before being prepared for transmission to the next processing step.

[0172] Step 2:

[0173] The terminal transmits pre-processed audio data to the server via a secure communication protocol. The input to the server is the audio data provided by the terminal, which is encrypted to ensure data security.

[0174] Step 3:

[0175] The server analyzes the received audio data using a speech recognition system. The input audio data is first converted to text by an acoustic model, from which prosody and vocal features are extracted. The output consists of text data and identified vocal characteristics.

[0176] Step 4:

[0177] The server performs emotion recognition based on identified speech characteristics. An emotion recognition algorithm analyzes these characteristics to determine the user's emotional state, such as "tension" or "calmness." The output of this process is an emotional state label.

[0178] Step 5:

[0179] The server generates information disclosure control rules based on the recognized emotional state. The generating AI model receives the emotional state as a prompt and determines the appropriate method of information disclosure for the specific situation. The output is a set of rules that provide guidance on disclosing or not disclosing information.

[0180] Step 6:

[0181] The terminal uses information disclosure control rules received from the server to adjust the information provided to the user. The input to this process is the control rules, and the audio output is optimized according to these rules. The content and presentation of the information are changed here, and the final output is presented to the user in either audio or text format.

[0182] Through this series of steps, users can use the voice assistant with peace of mind, while taking their own feelings and circumstances into consideration.

[0183] (Application Example 2)

[0184] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0185] Conventional voice assistant systems lacked the ability to detect emotions and were unable to provide appropriate responses based on the user's emotional state. Furthermore, they lacked sufficient privacy protection in situations involving multiple people, making it difficult to disclose or withhold information according to the user's wishes. Therefore, there is a need for the development of voice assistant technology that users can use with confidence.

[0186] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0187] In this invention, the server includes means for analyzing voice signals to identify multiple people participating in a conversation, means for determining the emotional state of a user from their voice using emotion recognition technology, and means for determining filtering rules that restrict the disclosure of private information based on the emotional state. This makes it possible to protect privacy while providing comfortable and appropriate voice responses according to the user's emotions.

[0188] "Audio signals" refer to sound captured as electrical signals, and are fundamental information used when collecting and analyzing audio data.

[0189] "Emotion recognition technology" is a technology that detects and analyzes a person's emotional state from data such as voice and facial expressions.

[0190] "Privacy information" refers to information about an individual whose disclosure should be restricted.

[0191] "Filtering rules" are criteria or policies used to select information based on specific conditions and decide whether to make it public or not.

[0192] "Audio output" refers to sound information produced by speech synthesis or other means and output from speakers or other devices.

[0193] A "server" is a computer system that performs data processing and provides services over a network.

[0194] "Household appliances" refer to automated devices used in a family's living space.

[0195] To implement this invention, the home device is equipped with a microphone for collecting voice signals. The voice signals are sent to a server and converted into text data using a speech recognition engine (e.g., Google Speech-to-Text API). Based on this data, an emotion engine such as IBM Watson Tone Analyzer is used as emotion recognition technology to determine the user's emotional state. On the server, filtering rules are generated to determine what information to disclose and to what extent, based on the determined emotional information. This includes setting conditions that take user privacy into consideration. The generated filtering rules are sent to the home device and used to adjust the voice output. The voice output is synthesized to provide a comfortable and appropriate response according to the user's emotional state. As a concrete example of operation, if the server determines that a family member is in a bad mood, the robot will speak in a subdued tone that takes the mood into consideration and offer suggestions to help them relax.

[0196] An example of a prompt used in the generation AI model would be: "Generate an appropriate conversation based on the emotional state of the family. For example, if the family is angry, generate a dialogue that incorporates gentle words to soothe their anger." Based on this prompt, the server will generate the conversation.

[0197] Through the processing flow described above, this invention can provide a customized voice assistant service that responds to the user's emotions, thereby facilitating smoother communication within the home.

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] The device collects sounds emitted within the home using a microphone and transmits the audio signal to a server. The input is an audio signal, and the output is the transfer of audio data to the server. Specifically, it collects family conversations and everyday sounds in real time.

[0201] Step 2:

[0202] The server converts the received audio signal into text data using a speech recognition engine. The input is audio data, and the output is text data. As part of the data processing, the audio signal is analyzed and converted into a string. At this stage, for example, a statement like "What are your plans for today?" is converted into text.

[0203] Step 3:

[0204] The server analyzes the converted text data using an emotion recognition engine to determine the user's emotional state. The input is text data, and the output is emotional information. As a data calculation, it performs analysis based on an emotion model to determine emotional states such as "stressed" or "relaxed." Specifically, the emotion recognition model estimates emotions from the words and context in the text.

[0205] Step 4:

[0206] The server generates appropriate filtering rules based on the determined emotional state. The input is emotional information, and the output is filtering rules. As part of the data processing, privacy levels are set and response content is adjusted based on the emotional state. For example, it considers family relationships to determine a policy for presenting information in a way that does not cause discomfort.

[0207] Step 5:

[0208] The server sends the generated filtering rules to the terminal and adjusts the voice output. The input is the filtering rules, and the output is the adjusted voice response. The terminal synthesizes voice according to these rules and provides an appropriate response to the user. Specifically, for example, it might say "What shall we do today?" in a gentle voice, performing an action to lighten the atmosphere in the home.

[0209] In this way, the system generates appropriate responses through conversations within the home, enabling comfortable and secure communication for the user.

[0210] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0211] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search)<url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0212] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0213] [Second Embodiment]

[0214] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0215] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0216] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0217] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0218] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0219] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0220] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0221] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0222] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0223] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0224] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0225] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0226] This invention is a system that utilizes an AI agent while protecting user privacy using speech recognition technology. Specifically, the terminal uses a microphone to acquire ambient audio signals and transmits this data to a server. The server analyzes the audio signals and identifies multiple people participating in the conversation. Based on the identification results, the server determines rules to restrict the disclosure of private information and transmits them to the terminal.

[0227] The device adjusts voice output based on received rules to prevent the disclosure of private information to third parties other than the user. This control allows the user to prevent unintended disclosure of private information while using the AI ​​agent.

[0228] As a concrete example, consider a scenario where a user and a friend ask an AI assistant about their schedules. The device captures the conversation between the user and the friend and sends that data to a server. The server performs voice analysis to identify the user and the friend. The server then creates rules that restrict the disclosure of certain time information and place names, taking into account the presence of the friend. The device applies these rules and provides only abstract information about specific times and places that should not be disclosed in front of the friend.

[0229] This invention aims to provide users with an environment in which they can use voice devices with peace of mind and to reduce privacy risks in voice-based communication.

[0230] The following describes the processing flow.

[0231] Step 1:

[0232] The device acquires ambient sound in real time through its microphone. Because the acquired audio signal is too large in data format, it undergoes noise filtering and compression to make it efficiently transmitted to the server.

[0233] Step 2:

[0234] The terminal sends voice data to the server at pre-configured times. Since the voice data reaches the server via the network, an appropriate protocol is used to minimize transmission delays and data loss.

[0235] Step 3:

[0236] The server uses a speech recognition engine to convert the audio data received from the terminal into text data. Based on the converted text data, it identifies the multiple people participating in the conversation and creates a participant list based on their identification information.

[0237] Step 4:

[0238] The server checks the participant list to determine if it includes any third parties other than the user. If there are individuals other than the user, it generates filtering rules that define what information should be made public and what information should remain private, based on a set of rules in accordance with the privacy policy.

[0239] Step 5:

[0240] The server sends the generated filtering rules to the terminal. These rules specify how to control the user's audio output and are intended to be applied on the terminal side.

[0241] Step 6:

[0242] The terminal adjusts audio output based on filtering rules received from the server. Specifically, content containing private information is replaced with general information, silence, or another signal sound, thereby implementing information control to prevent it from being heard by third parties.

[0243] Step 7:

[0244] Users continue conversations with friends and other participants via their voice devices. Device-side adjustments allow users to naturally provide only the necessary information within the conversation while preventing unintended privacy leaks.

[0245] (Example 1)

[0246] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0247] The present invention aims to provide an environment in which users can use voice agents with peace of mind while reducing the risk of privacy infringement in voice-based communication. In particular, there is a challenge in preventing the leakage of private information to third parties other than those involved when collecting and analyzing voice data, even when multiple people are involved.

[0248] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0249] In this invention, the server is a device for acquiring audio data, the device including means for removing noise, means for securely transmitting the audio data to another computing device, and means for identifying a person using speech recognition and natural language processing techniques. This makes it possible to determine rules for abstracting or hiding specific information and to adjust the audio to provide a new output to the user.

[0250] "Audio data" refers to information that represents ambient sounds in digital format.

[0251] "Noise reduction" is a process that removes unwanted background noise and other sounds from audio data, making the target audio clearer.

[0252] A "secure method" refers to a means of using encryption technology and secure protocols to maintain the confidentiality and integrity of information during data transmission and reception.

[0253] "Other computing devices" refers to external computers or servers connected to process, analyze, or store audio data.

[0254] "Speech recognition" is a technology that analyzes audio data and converts what is being said into text.

[0255] "Natural language processing technology" refers to computer technologies used to understand, generate, and analyze natural language used by humans.

[0256] "Identifying individuals" means identifying the speakers contained in audio data and distinguishing them based on the characteristics of each individual voice.

[0257] "Specific information" refers to personal information or information that may affect privacy that can be identified within the audio data.

[0258] "Abstraction" is the process of omitting the details of specific information and converting it into a more general expression.

[0259] "To hide" refers to the process of removing or concealing specific recognized information so that it is not visible or audible to other listeners.

[0260] "Rules" refer to instructions or standards regarding the restrictions on the disclosure and output methods of specific information, which are determined based on the analysis of audio data.

[0261] "Adjusting audio" means changing, editing, or processing the output generated from audio data according to specific rules.

[0262] "New output" refers to the adjusted voice message or information provided to the user based on the voice data.

[0263] The system in this invention is a data processing system that utilizes speech recognition technology and is designed for using a voice agent while protecting user privacy. The terminal collects ambient sound using a digital microphone and performs noise reduction processing. The processed voice data is transmitted to the server via a secure communication protocol.

[0264] The server converts audio data into text via a speech recognition engine. This process utilizes speech recognition software such as the Google Cloud Speech-to-Text API. The converted text data is then analyzed using natural language processing techniques to identify multiple individuals involved in the conversation. This identifies the linguistic characteristics of each individual involved, contributing to the identification of personal information.

[0265] Subsequently, the server develops rules to control the disclosure of private information based on the identified data. For example, rules may be constructed so that information is hidden or abstracted when a specific keyword is recognized. The terminal receives these rules and adjusts its audio output to prevent individual private information from being leaked to third parties other than the user.

[0266] As a concrete example, consider a user asking an AI assistant, "Do you have any plans next Friday?" The device receives this question and sends data to the server. If the analysis identifies next Friday as a private appointment, the server sets up rules to abstract that information and provide a general response such as, "You have an appointment coming up."

[0267] Examples of prompts to input into a generative AI model include questions like, "How can a user hide specific confidential information while conversing with someone?" Such a system would make it possible to maintain the convenience of voice communication while protecting privacy.

[0268] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0269] Step 1:

[0270] The device uses a microphone to capture ambient sound. This audio data is then processed using noise reduction technology to remove unwanted noise, resulting in clean audio data. This clean audio data is then passed on to the next processing step.

[0271] Step 2:

[0272] The terminal encrypts clean audio data and sends it to the server. This encryption process uses the SSL / TLS protocol to ensure data confidentiality and integrity. After the server receives the data, it is prepared for analysis.

[0273] Step 3:

[0274] The server inputs the received audio data into a speech recognition engine and converts it into text data. This process uses speech recognition software to accurately transcribe the audio content into text. The text data is then passed on to subsequent natural language processing.

[0275] Step 4:

[0276] The server uses natural language processing techniques to analyze text data and identify multiple people in a conversation. This process uses contextual analysis and speaker identification algorithms to pinpoint each person's utterance. The identified person information is then stored in a database.

[0277] Step 5:

[0278] The server develops privacy protection rules based on the identified individuals and the content of their statements. If specific keywords or phrases are detected, it determines guidelines for abstracting or concealing that information. These rules are then passed on to the next stage.

[0279] Step 6:

[0280] The server sends the formulated rules to the terminal. The terminal adjusts the voice output according to these rules and provides modified voice feedback to the user. As a result, the user can receive the output of the voice agent in a privacy-protected form.

[0281] (Application Example 1)

[0282] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0283] In modern households and devices used by multiple people, there is a possibility that privacy may be violated due to the leakage of information provided by voice assistants to third parties. It is necessary to solve this problem and provide an environment where users can use voice assistants with confidence.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0285] In this invention, the server includes means for collecting voice information, means for analyzing the voice information to identify a plurality of members participating in the conversation, and means for determining a policy for restricting the disclosure of confidential information based on the identified members. As a result, it is possible to selectively output individual information and prevent the leakage of confidential information.

[0286] "Voice information" is audio data obtained through devices such as microphones.

[0287] "Members" are a plurality of people or characters participating in the conversation within the voice information.

[0288] "Confidential information" is important personal information that may violate privacy if disclosed.

[0289] A "policy" refers to specific rules and regulations established to restrict the disclosure of confidential information.

[0290] "Audio output" refers to the sound output from speakers or other devices that is generated based on audio information.

[0291] "Selectively outputting information" means restricting or adjusting the information output based on specific conditions.

[0292] The system for carrying out the present invention consists of multiple components. Voice information is acquired by a microphone installed in a terminal. This terminal is integrated into a device such as a smart home assistant. The voice information is sent from the terminal to a server in the cloud and processed by the server. This processing utilizes the Google Cloud Speech-to-Text API, which leverages speech recognition technology. Using this API, the voice data is converted into text data.

[0293] The server uses the converted text data to identify multiple members, leveraging NLP (Natural Language Processing) techniques. This analysis on the server utilizes, for example, IBM Watson's natural language processing services. Once members are identified, policies are generated to restrict information disclosure for privacy purposes. These policies place particular emphasis on keeping confidential information private.

[0294] The generated policy is sent to the terminal. The terminal adjusts the audio output based on this policy. Specifically, it selectively outputs individual information that only certain individuals need to know, for example, through the voice assistant function. This makes it possible to prevent the leakage of confidential information.

[0295] To give an example, consider a scenario where a robot is active in a living room at home. If the mother gives the robot a voice command to check the schedule, the robot will provide the children present with a general overview of the schedule and provide the mother with detailed information via smartphone.

[0296] An example of a prompt for a generative AI model is: "Explain how to identify a specific person from the acquired voice data, analyze the content of the conversation, and create and apply rules to protect privacy."

[0297] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0298] Step 1:

[0299] The device acquires audio information using its built-in microphone. This audio information is recorded in real time and temporarily stored as digital data within the device. In this case, the input is physical sound, and the output is digitized audio data.

[0300] Step 2:

[0301] The terminal sends the acquired audio data to a server in the cloud. This process involves data processing that packets the audio data. The input is digital audio data, and the output is packet data that is transmitted over the network.

[0302] Step 3:

[0303] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API. The input here is audio packet data, and the output is text data. Speech recognition technology analyzes the characteristics of the sound and generates information as a string of characters.

[0304] Step 4:

[0305] The server uses NLP technologies such as IBM Watson based on text data to identify multiple participants in a conversation. The input is text data, and the output is participant identification information. Natural language processing is performed here to understand the context.

[0306] Step 5:

[0307] Based on the identified constituent member information, the server determines a policy for restricting the disclosure of confidential information. This policy is dynamically generated using a generative AI model. The input is the identification information, and the output is a set of rules as the policy.

[0308] Step 6:

[0309] The terminal adjusts the acoustic output based on the policy received from the server. Filtering is performed to mask specific information or transmit it only to appropriate targets. The input is the set of rules of the policy, and the output is the adjusted voice output.

[0310] Step 7:

[0311] The user receives the voice output obtained from the robot and checks the necessary information. The input here is the adjusted voice output, and the output is the user's understanding and actions. The user can use the information from the robot to efficiently perform daily tasks.

[0312] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0313] The present invention is a system that combines voice recognition technology and emotion recognition technology to enable the use of a more personalized voice assistant while protecting the user's privacy. The terminal uses a microphone to acquire ambient voice signals and transmits the data to the server. The server analyzes the voice signals, identifies multiple people participating in the conversation, and at the same time discriminates the user's emotional state from the voice using an emotion engine.

[0314] Based on emotion recognition, the server generates rules that consider the user's current emotional state to determine what information to disclose and how. These rules are adjusted according to the degree of privacy, meaning that the presentation of private information may vary depending on the user's emotions.

[0315] The terminal receives filtering rules sent from the server and adjusts the audio output. At this time, information is presented that matches the user's emotional state, ensuring a comfortable response for the user.

[0316] As a concrete example, consider a scenario where a user asks an AI assistant for schedule information with a friend. The device collects voice data and sends it to a server. The server uses voice analysis and an emotion engine to detect if the user is "anxious." Based on this, the server creates rules to avoid providing excessive details to the friend and to provide only the minimum necessary information. The device applies these rules and outputs information that is appropriately adjusted when the friend is present.

[0317] This invention enables the use of a flexible voice assistant that responds to the user's emotions while protecting the user's privacy.

[0318] The following describes the processing flow.

[0319] Step 1:

[0320] The device continuously acquires ambient audio signals using its built-in microphone. These audio signals are de-noised and compressed before being sent to the server.

[0321] Step 2:

[0322] The terminal sends the collected voice data to the server. At this time, it adjusts the packet delivery to ensure efficiency depending on the network conditions.

[0323] Step 3:

[0324] The server analyzes the received audio data and converts it into text data through speech recognition. In parallel, the emotion engine analyzes the user's voice characteristics from the same audio data and determines their emotional state in real time.

[0325] Step 4:

[0326] The server identifies the people participating in the conversation based on the speech recognition results. The emotion engine determines whether the user's emotions are in a specific state (e.g., excited, anxious, relaxed).

[0327] Step 5:

[0328] The server determines appropriate filtering rules to restrict the disclosure of private information based on the identified person's information and the user's emotional state. These rules apply stricter restrictions, such as when emotions are particularly heightened.

[0329] Step 6:

[0330] The server sends the determined filtering rules to the terminal and provides instructions for the terminal to adjust the audio output.

[0331] Step 7:

[0332] The device adjusts the audio output in real time according to the received filtering rules. This ensures that the privacy information of the conversation is properly protected while also enabling responses that are sensitive to the user's emotions.

[0333] Step 8:

[0334] Users can continue conversations with third parties with peace of mind by receiving appropriately filtered and adjusted responses. This system protects both user privacy and emotional security.

[0335] (Example 2)

[0336] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0337] Voice assistants often provide information without adequately considering user privacy. As a result, users risk privacy violations due to a lack of appropriate information disclosure control in specific situations or emotional states. Furthermore, current technology makes it difficult to control information disclosure based on the user's emotional state, resulting in the inability to provide flexible and personalized information as intended by the user.

[0338] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0339] In this invention, the server includes means for analyzing voice data to identify individual people, means for determining emotional states based on voice characteristics, and means for generating information disclosure control rules according to the determined emotional states. This enables flexible information disclosure control tailored to the user's emotional state, allowing for personalized information provision while protecting user privacy.

[0340] "Audio data" refers to audio signals acquired using audio equipment such as microphones, which have been converted into an analyzable digital format.

[0341] "Terminal means" refers to a device or mechanism that performs tasks such as acquiring, transmitting, and receiving voice data, and applying filtering rules.

[0342] "Information processing unit" refers to a computing device or computer system used to receive, analyze, identify, and generate rules for audio data.

[0343] "Means of identifying individuals" refer to algorithms and systems that analyze audio data to identify the speaker.

[0344] "Vocal characteristics" refer to attributes such as tone, speed, and volume of voice, and these attributes are used to determine emotional states.

[0345] "Means for determining emotional state" refers to technologies and algorithms used to infer a user's emotions and psychological state based on voice data.

[0346] "Information disclosure control rules" are criteria or policies that determine which information to disclose or withhold, and how, depending on the user's specific emotional state.

[0347] "Terminal means for adjusting data output" refers to a device or program that adjusts the format and content of information provided externally based on the generated information disclosure control rules.

[0348] Modes for carrying out the invention

[0349] This invention is a voice support system that integrates speech recognition technology and emotion recognition technology to provide information tailored to individual emotional states while prioritizing user privacy.

[0350] Hardware and software to be used

[0351] Device: To acquire audio, an audio device such as a smartphone or smart speaker is used. The built-in microphone is used for audio recording, and the device's operating system (e.g., Android OS, iOS) is used to temporarily store the audio data.

[0352] Server: A cloud data processing system is used to analyze audio data and recognize emotions. A speech recognition API (e.g., a general speech recognition API) is used for analyzing the audio signal, and an AI service for emotion analysis (e.g., a general emotion analysis service) is used to determine the emotional state.

[0353] Data processing flow

[0354] The terminal acquires audio and transmits the data to the server via a secure connection. The server uses the received audio data to first convert it into text using a speech recognition system, then analyzes the audio characteristics to identify individual participants and determine their emotional states. The generative AI model used in this system (e.g., a general generative model) generates information disclosure control rules based on the user's emotions based on this information. The terminal receives these rules and adjusts the audio information it outputs to the user and other participants.

[0355] Specific example

[0356] For example, consider a scenario where a user wants to check their schedule information with an AI assistant while with a friend. If the user is feeling a little nervous, the server recognizes this emotional state and generates a rule to provide only the minimum necessary information, taking the presence of the friend into account. The device applies this rule and provides the user with appropriately adjusted voice information, such as, "We'll let you know the details of the schedule later."

[0357] Example of a prompt

[0358] "I want to check my schedule while I'm with my friends. However, I'm secretly nervous, so please keep it as concise as possible."

[0359] "I want to receive important announcements during the meeting, but I don't want the details to be known to others."

[0360] This invention allows users to enjoy a flexible and personalized voice assistant experience that is tailored to their emotions, while also protecting their privacy.

[0361] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0362] Step 1:

[0363] The device acquires ambient sound through its microphone. When the user naturally speaks a request to the voice assistant, the audio signal is input to the device as digital data. This data undergoes data processing such as noise reduction and volume normalization before being prepared for transmission to the next processing step.

[0364] Step 2:

[0365] The terminal transmits pre-processed audio data to the server via a secure communication protocol. The input to the server is the audio data provided by the terminal, which is encrypted to ensure data security.

[0366] Step 3:

[0367] The server analyzes the received audio data using a speech recognition system. The input audio data is first converted to text by an acoustic model, from which prosody and vocal features are extracted. The output consists of text data and identified vocal characteristics.

[0368] Step 4:

[0369] The server performs emotion recognition based on identified speech characteristics. An emotion recognition algorithm analyzes these characteristics to determine the user's emotional state, such as "tension" or "calmness." The output of this process is an emotional state label.

[0370] Step 5:

[0371] The server generates information disclosure control rules based on the recognized emotional state. The generating AI model receives the emotional state as a prompt and determines the appropriate method of information disclosure for the specific situation. The output is a set of rules that provide guidance on disclosing or not disclosing information.

[0372] Step 6:

[0373] The terminal uses information disclosure control rules received from the server to adjust the information provided to the user. The input to this process is the control rules, and the audio output is optimized according to these rules. The content and presentation of the information are changed here, and the final output is presented to the user in either audio or text format.

[0374] Through this series of steps, users can use the voice assistant with peace of mind, while taking their own feelings and circumstances into consideration.

[0375] (Application Example 2)

[0376] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0377] Conventional voice assistant systems lacked the ability to detect emotions and were unable to provide appropriate responses based on the user's emotional state. Furthermore, they lacked sufficient privacy protection in situations involving multiple people, making it difficult to disclose or withhold information according to the user's wishes. Therefore, there is a need for the development of voice assistant technology that users can use with confidence.

[0378] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0379] In this invention, the server includes means for analyzing voice signals to identify multiple people participating in a conversation, means for determining the emotional state of a user from their voice using emotion recognition technology, and means for determining filtering rules that restrict the disclosure of private information based on the emotional state. This makes it possible to protect privacy while providing comfortable and appropriate voice responses according to the user's emotions.

[0380] "Audio signals" refer to sound captured as electrical signals, and are fundamental information used when collecting and analyzing audio data.

[0381] "Emotion recognition technology" is a technology that detects and analyzes a person's emotional state from data such as voice and facial expressions.

[0382] "Privacy information" refers to information about an individual whose disclosure should be restricted.

[0383] "Filtering rules" are criteria or policies used to select information based on specific conditions and decide whether to make it public or not.

[0384] "Audio output" refers to sound information produced by speech synthesis or other means and output from speakers or other devices.

[0385] A "server" is a computer system that performs data processing and provides services over a network.

[0386] "Household appliances" refer to automated devices used in a family's living space.

[0387] To implement this invention, the home device is equipped with a microphone for collecting voice signals. The voice signals are sent to a server and converted into text data using a speech recognition engine (e.g., Google Speech-to-Text API). Based on this data, an emotion engine such as IBM Watson Tone Analyzer is used as emotion recognition technology to determine the user's emotional state. On the server, filtering rules are generated to determine what information to disclose and to what extent, based on the determined emotional information. This includes setting conditions that take user privacy into consideration. The generated filtering rules are sent to the home device and used to adjust the voice output. The voice output is synthesized to provide a comfortable and appropriate response according to the user's emotional state. As a concrete example of operation, if the server determines that a family member is in a bad mood, the robot will speak in a subdued tone that takes the mood into consideration and offer suggestions to help them relax.

[0388] An example of a prompt used in the generation AI model would be: "Generate an appropriate conversation based on the emotional state of the family. For example, if the family is angry, generate a dialogue that incorporates gentle words to soothe their anger." Based on this prompt, the server will generate the conversation.

[0389] Through the processing flow described above, this invention can provide a customized voice assistant service that responds to the user's emotions, thereby facilitating smoother communication within the home.

[0390] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0391] Step 1:

[0392] The device collects sounds emitted within the home using a microphone and transmits the audio signal to a server. The input is an audio signal, and the output is the transfer of audio data to the server. Specifically, it collects family conversations and everyday sounds in real time.

[0393] Step 2:

[0394] The server converts the received audio signal into text data using a speech recognition engine. The input is audio data, and the output is text data. As part of the data processing, the audio signal is analyzed and converted into a string. At this stage, for example, a statement like "What are your plans for today?" is converted into text.

[0395] Step 3:

[0396] The server analyzes the converted text data using an emotion recognition engine to determine the user's emotional state. The input is text data, and the output is emotional information. As a data calculation, it performs analysis based on an emotion model to determine emotional states such as "stressed" or "relaxed." Specifically, the emotion recognition model estimates emotions from the words and context in the text.

[0397] Step 4:

[0398] The server generates appropriate filtering rules based on the determined emotional state. The input is emotional information, and the output is filtering rules. As part of the data processing, privacy levels are set and response content is adjusted based on the emotional state. For example, it considers family relationships to determine a policy for presenting information in a way that does not cause discomfort.

[0399] Step 5:

[0400] The server sends the generated filtering rules to the terminal and adjusts the voice output. The input is the filtering rules, and the output is the adjusted voice response. The terminal synthesizes voice according to these rules and provides an appropriate response to the user. Specifically, for example, it might say "What shall we do today?" in a gentle voice, performing an action to lighten the atmosphere in the home.

[0401] In this way, the system generates appropriate responses through conversations within the home, enabling comfortable and secure communication for the user.

[0402] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0403] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0404] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0405] [Third Embodiment]

[0406] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0407] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0408] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0409] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0410] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0412] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0413] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0414] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0415] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0416] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0417] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0418] This invention is a system that utilizes an AI agent while protecting user privacy using speech recognition technology. Specifically, the terminal uses a microphone to acquire ambient audio signals and transmits this data to a server. The server analyzes the audio signals and identifies multiple people participating in the conversation. Based on the identification results, the server determines rules to restrict the disclosure of private information and transmits them to the terminal.

[0419] The device adjusts voice output based on received rules to prevent the disclosure of private information to third parties other than the user. This control allows the user to prevent unintended disclosure of private information while using the AI ​​agent.

[0420] As a concrete example, consider a scenario where a user and a friend ask an AI assistant about their schedules. The device captures the conversation between the user and the friend and sends that data to a server. The server performs voice analysis to identify the user and the friend. The server then creates rules that restrict the disclosure of certain time information and place names, taking into account the presence of the friend. The device applies these rules and provides only abstract information about specific times and places that should not be disclosed in front of the friend.

[0421] This invention aims to provide users with an environment in which they can use voice devices with peace of mind and to reduce privacy risks in voice-based communication.

[0422] The following describes the processing flow.

[0423] Step 1:

[0424] The device acquires ambient sound in real time through its microphone. Because the acquired audio signal is too large in data format, it undergoes noise filtering and compression to make it efficiently transmitted to the server.

[0425] Step 2:

[0426] The terminal sends voice data to the server at pre-configured times. Since the voice data reaches the server via the network, an appropriate protocol is used to minimize transmission delays and data loss.

[0427] Step 3:

[0428] The server uses a speech recognition engine to convert the audio data received from the terminal into text data. Based on the converted text data, it identifies the multiple people participating in the conversation and creates a participant list based on their identification information.

[0429] Step 4:

[0430] The server checks the participant list to determine if it includes any third parties other than the user. If there are individuals other than the user, it generates filtering rules that define what information should be made public and what information should remain private, based on a set of rules in accordance with the privacy policy.

[0431] Step 5:

[0432] The server sends the generated filtering rules to the terminal. These rules specify how to control the user's audio output and are intended to be applied on the terminal side.

[0433] Step 6:

[0434] The terminal adjusts audio output based on filtering rules received from the server. Specifically, content containing private information is replaced with general information, silence, or another signal sound, thereby implementing information control to prevent it from being heard by third parties.

[0435] Step 7:

[0436] Users continue conversations with friends and other participants via their voice devices. Device-side adjustments allow users to naturally provide only the necessary information within the conversation while preventing unintended privacy leaks.

[0437] (Example 1)

[0438] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0439] The present invention aims to provide an environment in which users can use voice agents with peace of mind while reducing the risk of privacy infringement in voice-based communication. In particular, there is a challenge in preventing the leakage of private information to third parties other than those involved when collecting and analyzing voice data, even when multiple people are involved.

[0440] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0441] In this invention, the server is a device for acquiring audio data, the device including means for removing noise, means for securely transmitting the audio data to another computing device, and means for identifying a person using speech recognition and natural language processing techniques. This makes it possible to determine rules for abstracting or hiding specific information and to adjust the audio to provide a new output to the user.

[0442] "Audio data" refers to information that represents ambient sounds in digital format.

[0443] "Noise reduction" is a process that removes unwanted background noise and other sounds from audio data, making the target audio clearer.

[0444] A "secure method" refers to a means of using encryption technology and secure protocols to maintain the confidentiality and integrity of information during data transmission and reception.

[0445] "Other computing devices" refers to external computers or servers connected to process, analyze, or store audio data.

[0446] "Speech recognition" is a technology that analyzes audio data and converts what is being said into text.

[0447] "Natural language processing technology" refers to computer technologies used to understand, generate, and analyze natural language used by humans.

[0448] "Identifying individuals" means identifying the speakers contained in audio data and distinguishing them based on the characteristics of each individual voice.

[0449] "Specific information" refers to personal information or information that may affect privacy that can be identified within the audio data.

[0450] "Abstraction" is the process of omitting the details of specific information and converting it into a more general expression.

[0451] "To hide" refers to the process of removing or concealing specific recognized information so that it is not visible or audible to other listeners.

[0452] "Rules" refer to instructions or standards regarding the restrictions on the disclosure and output methods of specific information, which are determined based on the analysis of audio data.

[0453] "Adjusting audio" means changing, editing, or processing the output generated from audio data according to specific rules.

[0454] "New output" refers to the adjusted voice message or information provided to the user based on the voice data.

[0455] The system in this invention is a data processing system that utilizes speech recognition technology and is designed for using a voice agent while protecting user privacy. The terminal collects ambient sound using a digital microphone and performs noise reduction processing. The processed voice data is transmitted to the server via a secure communication protocol.

[0456] The server converts audio data into text via a speech recognition engine. This process utilizes speech recognition software such as the Google Cloud Speech-to-Text API. The converted text data is then analyzed using natural language processing techniques to identify multiple individuals involved in the conversation. This identifies the linguistic characteristics of each individual involved, contributing to the identification of personal information.

[0457] Subsequently, the server develops rules to control the disclosure of private information based on the identified data. For example, rules may be constructed so that information is hidden or abstracted when a specific keyword is recognized. The terminal receives these rules and adjusts its audio output to prevent individual private information from being leaked to third parties other than the user.

[0458] As a concrete example, consider a user asking an AI assistant, "Do you have any plans next Friday?" The device receives this question and sends data to the server. If the analysis identifies next Friday as a private appointment, the server sets up rules to abstract that information and provide a general response such as, "You have an appointment coming up."

[0459] Examples of prompts to input into a generative AI model include questions like, "How can a user hide specific confidential information while conversing with someone?" Such a system would make it possible to maintain the convenience of voice communication while protecting privacy.

[0460] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0461] Step 1:

[0462] The device uses a microphone to capture ambient sound. This audio data is then processed using noise reduction technology to remove unwanted noise, resulting in clean audio data. This clean audio data is then passed on to the next processing step.

[0463] Step 2:

[0464] The terminal encrypts clean audio data and sends it to the server. This encryption process uses the SSL / TLS protocol to ensure data confidentiality and integrity. After the server receives the data, it is prepared for analysis.

[0465] Step 3:

[0466] The server inputs the received audio data into a speech recognition engine and converts it into text data. This process uses speech recognition software to accurately transcribe the audio content into text. The text data is then passed on to subsequent natural language processing.

[0467] Step 4:

[0468] The server uses natural language processing techniques to analyze text data and identify multiple people in a conversation. This process uses contextual analysis and speaker identification algorithms to pinpoint each person's utterance. The identified person information is then stored in a database.

[0469] Step 5:

[0470] The server develops privacy protection rules based on the identified individuals and the content of their statements. If specific keywords or phrases are detected, it determines guidelines for abstracting or concealing that information. These rules are then passed on to the next stage.

[0471] Step 6:

[0472] The server sends the established rules to the terminal. The terminal adjusts the audio output according to these rules, providing the user with modified audio feedback. This allows the user to receive the voice agent's output in a privacy-protected manner.

[0473] (Application Example 1)

[0474] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0475] In modern homes and with devices used by multiple people, there is a risk of privacy violations if information provided by voice assistants is leaked to third parties. It is necessary to solve this problem and provide an environment where users can use voice assistants with peace of mind.

[0476] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0477] In this invention, the server includes means for collecting voice information, means for analyzing the voice information to identify multiple members participating in the conversation, and means for determining a policy to restrict the disclosure of confidential information based on the identified members. This makes it possible to selectively output individual information and prevent the leakage of confidential information.

[0478] "Audio information" refers to sound data acquired through devices such as microphones.

[0479] "Members" refers to the multiple individuals or characters participating in the conversation within the audio information.

[0480] "Confidential information" refers to important personal information that, if made public, could potentially infringe on privacy.

[0481] A "policy" refers to specific rules and regulations established to restrict the disclosure of confidential information.

[0482] "Audio output" refers to the sound output from speakers or other devices that is generated based on audio information.

[0483] "Selectively outputting information" means restricting or adjusting the information output based on specific conditions.

[0484] The system for carrying out the present invention consists of multiple components. Voice information is acquired by a microphone installed in a terminal. This terminal is integrated into a device such as a smart home assistant. The voice information is sent from the terminal to a server in the cloud and processed by the server. This processing utilizes the Google Cloud Speech-to-Text API, which leverages speech recognition technology. Using this API, the voice data is converted into text data.

[0485] The server uses the converted text data to identify multiple members, leveraging NLP (Natural Language Processing) techniques. This analysis on the server utilizes, for example, IBM Watson's natural language processing services. Once members are identified, policies are generated to restrict information disclosure for privacy purposes. These policies place particular emphasis on keeping confidential information private.

[0486] The generated policy is sent to the terminal. The terminal adjusts the audio output based on this policy. Specifically, it selectively outputs individual information that only certain individuals need to know, for example, through the voice assistant function. This makes it possible to prevent the leakage of confidential information.

[0487] To give an example, consider a scenario where a robot is active in a living room at home. If the mother gives the robot a voice command to check the schedule, the robot will provide the children present with a general overview of the schedule and provide the mother with detailed information via smartphone.

[0488] An example of a prompt for a generative AI model is: "Explain how to identify a specific person from the acquired voice data, analyze the content of the conversation, and create and apply rules to protect privacy."

[0489] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0490] Step 1:

[0491] The device acquires audio information using its built-in microphone. This audio information is recorded in real time and temporarily stored as digital data within the device. In this case, the input is physical sound, and the output is digitized audio data.

[0492] Step 2:

[0493] The terminal sends the acquired audio data to a server in the cloud. This process involves data processing that packets the audio data. The input is digital audio data, and the output is packet data that is transmitted over the network.

[0494] Step 3:

[0495] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API. The input here is audio packet data, and the output is text data. Speech recognition technology analyzes the characteristics of the sound and generates information as a string of characters.

[0496] Step 4:

[0497] The server uses NLP technologies such as IBM Watson based on text data to identify multiple participants in a conversation. The input is text data, and the output is participant identification information. Natural language processing is performed here to understand the context.

[0498] Step 5:

[0499] The server determines policies to restrict the disclosure of confidential information based on identified member information. These policies are dynamically generated using a generative AI model. The input is identification information, and the output is a set of rules representing the policies.

[0500] Step 6:

[0501] The terminal adjusts the audio output based on policies received from the server. Filtering is performed to shield specific information or transmit it only to appropriate targets. The input is the policy rule set, and the output is the adjusted audio output.

[0502] Step 7:

[0503] The user receives voice output from the robot and confirms the necessary information. Here, the input is the adjusted voice output, and the output is the user's understanding and actions. The user can use the information from the robot to efficiently perform everyday tasks.

[0504] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0505] This invention is a system that combines speech recognition technology and emotion recognition technology to enable the use of a more personalized voice assistant while protecting user privacy. The terminal uses a microphone to acquire ambient audio signals and transmits this data to a server. The server analyzes the audio signals to identify multiple people participating in a conversation, and at the same time uses an emotion engine to determine the user's emotional state from the voice.

[0506] Based on emotion recognition, the server generates rules that consider the user's current emotional state to determine what information to disclose and how. These rules are adjusted according to the degree of privacy, meaning that the presentation of private information may vary depending on the user's emotions.

[0507] The terminal receives filtering rules sent from the server and adjusts the audio output. At this time, information is presented that matches the user's emotional state, ensuring a comfortable response for the user.

[0508] As a concrete example, consider a scenario where a user asks an AI assistant for schedule information with a friend. The device collects voice data and sends it to a server. The server uses voice analysis and an emotion engine to detect if the user is "anxious." Based on this, the server creates rules to avoid providing excessive details to the friend and to provide only the minimum necessary information. The device applies these rules and outputs information that is appropriately adjusted when the friend is present.

[0509] This invention enables the use of a flexible voice assistant that responds to the user's emotions while protecting the user's privacy.

[0510] The following describes the processing flow.

[0511] Step 1:

[0512] The device continuously acquires ambient audio signals using its built-in microphone. These audio signals are de-noised and compressed before being sent to the server.

[0513] Step 2:

[0514] The terminal sends the collected voice data to the server. At this time, it adjusts the packet delivery to ensure efficiency depending on the network conditions.

[0515] Step 3:

[0516] The server analyzes the received audio data and converts it into text data through speech recognition. In parallel, the emotion engine analyzes the user's voice characteristics from the same audio data and determines their emotional state in real time.

[0517] Step 4:

[0518] The server identifies the people participating in the conversation based on the speech recognition results. The emotion engine determines whether the user's emotions are in a specific state (e.g., excited, anxious, relaxed).

[0519] Step 5:

[0520] The server determines appropriate filtering rules to restrict the disclosure of private information based on the identified person's information and the user's emotional state. These rules apply stricter restrictions, such as when emotions are particularly heightened.

[0521] Step 6:

[0522] The server sends the determined filtering rules to the terminal and provides instructions for the terminal to adjust the audio output.

[0523] Step 7:

[0524] The device adjusts the audio output in real time according to the received filtering rules. This ensures that the privacy information of the conversation is properly protected while also enabling responses that are sensitive to the user's emotions.

[0525] Step 8:

[0526] Users can continue conversations with third parties with peace of mind by receiving appropriately filtered and adjusted responses. This system protects both user privacy and emotional security.

[0527] (Example 2)

[0528] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0529] Voice assistants often provide information without adequately considering user privacy. As a result, users risk privacy violations due to a lack of appropriate information disclosure control in specific situations or emotional states. Furthermore, current technology makes it difficult to control information disclosure based on the user's emotional state, resulting in the inability to provide flexible and personalized information as intended by the user.

[0530] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0531] In this invention, the server includes means for analyzing voice data to identify individual people, means for determining emotional states based on voice characteristics, and means for generating information disclosure control rules according to the determined emotional states. This enables flexible information disclosure control tailored to the user's emotional state, allowing for personalized information provision while protecting user privacy.

[0532] "Audio data" refers to audio signals acquired using audio equipment such as microphones, which have been converted into an analyzable digital format.

[0533] "Terminal means" refers to a device or mechanism that performs tasks such as acquiring, transmitting, and receiving voice data, and applying filtering rules.

[0534] "Information processing unit" refers to a computing device or computer system used to receive, analyze, identify, and generate rules for audio data.

[0535] "Means of identifying individuals" refer to algorithms and systems that analyze audio data to identify the speaker.

[0536] "Vocal characteristics" refer to attributes such as tone, speed, and volume of voice, and these attributes are used to determine emotional states.

[0537] "Means for determining emotional state" refers to technologies and algorithms used to infer a user's emotions and psychological state based on voice data.

[0538] "Information disclosure control rules" are criteria or policies that determine which information to disclose or withhold, and how, depending on the user's specific emotional state.

[0539] "Terminal means for adjusting data output" refers to a device or program that adjusts the format and content of information provided externally based on the generated information disclosure control rules.

[0540] Modes for carrying out the invention

[0541] This invention is a voice support system that integrates speech recognition technology and emotion recognition technology to provide information tailored to individual emotional states while prioritizing user privacy.

[0542] Hardware and software to be used

[0543] Device: To acquire audio, an audio device such as a smartphone or smart speaker is used. The built-in microphone is used for audio recording, and the device's operating system (e.g., Android OS, iOS) is used to temporarily store the audio data.

[0544] Server: A cloud data processing system is used to analyze audio data and recognize emotions. A speech recognition API (e.g., a general speech recognition API) is used for analyzing the audio signal, and an AI service for emotion analysis (e.g., a general emotion analysis service) is used to determine the emotional state.

[0545] Data processing flow

[0546] The terminal acquires audio and transmits the data to the server via a secure connection. The server uses the received audio data to first convert it into text using a speech recognition system, then analyzes the audio characteristics to identify individual participants and determine their emotional states. The generative AI model used in this system (e.g., a general generative model) generates information disclosure control rules based on the user's emotions based on this information. The terminal receives these rules and adjusts the audio information it outputs to the user and other participants.

[0547] Specific example

[0548] For example, consider a scenario where a user wants to check their schedule information with an AI assistant while with a friend. If the user is feeling a little nervous, the server recognizes this emotional state and generates a rule to provide only the minimum necessary information, taking the presence of the friend into account. The device applies this rule and provides the user with appropriately adjusted voice information, such as, "We'll let you know the details of the schedule later."

[0549] Example of a prompt

[0550] "I want to check my schedule while I'm with my friends. However, I'm secretly nervous, so please keep it as concise as possible."

[0551] "I want to receive important announcements during the meeting, but I don't want the details to be known to others."

[0552] This invention allows users to enjoy a flexible and personalized voice assistant experience that is tailored to their emotions, while also protecting their privacy.

[0553] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0554] Step 1:

[0555] The device acquires ambient sound through its microphone. When the user naturally speaks a request to the voice assistant, the audio signal is input to the device as digital data. This data undergoes data processing such as noise reduction and volume normalization before being prepared for transmission to the next processing step.

[0556] Step 2:

[0557] The terminal transmits pre-processed audio data to the server via a secure communication protocol. The input to the server is the audio data provided by the terminal, which is encrypted to ensure data security.

[0558] Step 3:

[0559] The server analyzes the received audio data using a speech recognition system. The input audio data is first converted to text by an acoustic model, from which prosody and vocal features are extracted. The output consists of text data and identified vocal characteristics.

[0560] Step 4:

[0561] The server performs emotion recognition based on identified speech characteristics. An emotion recognition algorithm analyzes these characteristics to determine the user's emotional state, such as "tension" or "calmness." The output of this process is an emotional state label.

[0562] Step 5:

[0563] The server generates information disclosure control rules based on the recognized emotional state. The generating AI model receives the emotional state as a prompt and determines the appropriate method of information disclosure for the specific situation. The output is a set of rules that provide guidance on disclosing or not disclosing information.

[0564] Step 6:

[0565] The terminal uses information disclosure control rules received from the server to adjust the information provided to the user. The input to this process is the control rules, and the audio output is optimized according to these rules. The content and presentation of the information are changed here, and the final output is presented to the user in either audio or text format.

[0566] Through this series of steps, users can use the voice assistant with peace of mind, while taking their own feelings and circumstances into consideration.

[0567] (Application Example 2)

[0568] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0569] Conventional voice assistant systems lacked the ability to detect emotions and were unable to provide appropriate responses based on the user's emotional state. Furthermore, they lacked sufficient privacy protection in situations involving multiple people, making it difficult to disclose or withhold information according to the user's wishes. Therefore, there is a need for the development of voice assistant technology that users can use with confidence.

[0570] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0571] In this invention, the server includes means for analyzing voice signals to identify multiple people participating in a conversation, means for determining the emotional state of a user from their voice using emotion recognition technology, and means for determining filtering rules that restrict the disclosure of private information based on the emotional state. This makes it possible to protect privacy while providing comfortable and appropriate voice responses according to the user's emotions.

[0572] "Audio signals" refer to sound captured as electrical signals, and are fundamental information used when collecting and analyzing audio data.

[0573] "Emotion recognition technology" is a technology that detects and analyzes a person's emotional state from data such as voice and facial expressions.

[0574] "Privacy information" refers to information about an individual whose disclosure should be restricted.

[0575] "Filtering rules" are criteria or policies used to select information based on specific conditions and decide whether to make it public or not.

[0576] "Audio output" refers to sound information produced by speech synthesis or other means and output from speakers or other devices.

[0577] A "server" is a computer system that performs data processing and provides services over a network.

[0578] "Household appliances" refer to automated devices used in a family's living space.

[0579] To implement this invention, the home device is equipped with a microphone for collecting voice signals. The voice signals are sent to a server and converted into text data using a speech recognition engine (e.g., Google Speech-to-Text API). Based on this data, an emotion engine such as IBM Watson Tone Analyzer is used as emotion recognition technology to determine the user's emotional state. On the server, filtering rules are generated to determine what information to disclose and to what extent, based on the determined emotional information. This includes setting conditions that take user privacy into consideration. The generated filtering rules are sent to the home device and used to adjust the voice output. The voice output is synthesized to provide a comfortable and appropriate response according to the user's emotional state. As a concrete example of operation, if the server determines that a family member is in a bad mood, the robot will speak in a subdued tone that takes the mood into consideration and offer suggestions to help them relax.

[0580] An example of a prompt used in the generation AI model would be: "Generate an appropriate conversation based on the emotional state of the family. For example, if the family is angry, generate a dialogue that incorporates gentle words to soothe their anger." Based on this prompt, the server will generate the conversation.

[0581] Through the processing flow described above, this invention can provide a customized voice assistant service that responds to the user's emotions, thereby facilitating smoother communication within the home.

[0582] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0583] Step 1:

[0584] The device collects sounds emitted within the home using a microphone and transmits the audio signal to a server. The input is an audio signal, and the output is the transfer of audio data to the server. Specifically, it collects family conversations and everyday sounds in real time.

[0585] Step 2:

[0586] The server converts the received audio signal into text data using a speech recognition engine. The input is audio data, and the output is text data. As part of the data processing, the audio signal is analyzed and converted into a string. At this stage, for example, a statement like "What are your plans for today?" is converted into text.

[0587] Step 3:

[0588] The server analyzes the converted text data using an emotion recognition engine to determine the user's emotional state. The input is text data, and the output is emotional information. As a data calculation, it performs analysis based on an emotion model to determine emotional states such as "stressed" or "relaxed." Specifically, the emotion recognition model estimates emotions from the words and context in the text.

[0589] Step 4:

[0590] The server generates appropriate filtering rules based on the determined emotional state. The input is emotional information, and the output is filtering rules. As part of the data processing, privacy levels are set and response content is adjusted based on the emotional state. For example, it considers family relationships to determine a policy for presenting information in a way that does not cause discomfort.

[0591] Step 5:

[0592] The server sends the generated filtering rules to the terminal and adjusts the voice output. The input is the filtering rules, and the output is the adjusted voice response. The terminal synthesizes voice according to these rules and provides an appropriate response to the user. Specifically, for example, it might say "What shall we do today?" in a gentle voice, performing an action to lighten the atmosphere in the home.

[0593] In this way, the system generates appropriate responses through conversations within the home, enabling comfortable and secure communication for the user.

[0594] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0595] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0596] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0597] [Fourth Embodiment]

[0598] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0599] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0600] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0601] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0602] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0603] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0604] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0605] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0606] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0607] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0608] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0609] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0610] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0611] This invention is a system that utilizes an AI agent while protecting user privacy using speech recognition technology. Specifically, the terminal uses a microphone to acquire ambient audio signals and transmits this data to a server. The server analyzes the audio signals and identifies multiple people participating in the conversation. Based on the identification results, the server determines rules to restrict the disclosure of private information and transmits them to the terminal.

[0612] The device adjusts voice output based on received rules to prevent the disclosure of private information to third parties other than the user. This control allows the user to prevent unintended disclosure of private information while using the AI ​​agent.

[0613] As a concrete example, consider a scenario where a user and a friend ask an AI assistant about their schedules. The device captures the conversation between the user and the friend and sends that data to a server. The server performs voice analysis to identify the user and the friend. The server then creates rules that restrict the disclosure of certain time information and place names, taking into account the presence of the friend. The device applies these rules and provides only abstract information about specific times and places that should not be disclosed in front of the friend.

[0614] This invention aims to provide users with an environment in which they can use voice devices with peace of mind and to reduce privacy risks in voice-based communication.

[0615] The following describes the processing flow.

[0616] Step 1:

[0617] The device acquires ambient sound in real time through its microphone. Because the acquired audio signal is too large in data format, it undergoes noise filtering and compression to make it efficiently transmitted to the server.

[0618] Step 2:

[0619] The terminal sends voice data to the server at pre-configured times. Since the voice data reaches the server via the network, an appropriate protocol is used to minimize transmission delays and data loss.

[0620] Step 3:

[0621] The server uses a speech recognition engine to convert the audio data received from the terminal into text data. Based on the converted text data, it identifies the multiple people participating in the conversation and creates a participant list based on their identification information.

[0622] Step 4:

[0623] The server checks the participant list to determine if it includes any third parties other than the user. If there are individuals other than the user, it generates filtering rules that define what information should be made public and what information should remain private, based on a set of rules in accordance with the privacy policy.

[0624] Step 5:

[0625] The server sends the generated filtering rules to the terminal. These rules specify how to control the user's audio output and are intended to be applied on the terminal side.

[0626] Step 6:

[0627] The terminal adjusts audio output based on filtering rules received from the server. Specifically, content containing private information is replaced with general information, silence, or another signal sound, thereby implementing information control to prevent it from being heard by third parties.

[0628] Step 7:

[0629] Users continue conversations with friends and other participants via their voice devices. Device-side adjustments allow users to naturally provide only the necessary information within the conversation while preventing unintended privacy leaks.

[0630] (Example 1)

[0631] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0632] The present invention aims to provide an environment in which users can use voice agents with peace of mind while reducing the risk of privacy infringement in voice-based communication. In particular, there is a challenge in preventing the leakage of private information to third parties other than those involved when collecting and analyzing voice data, even when multiple people are involved.

[0633] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0634] In this invention, the server is a device for acquiring audio data, the device including means for removing noise, means for securely transmitting the audio data to another computing device, and means for identifying a person using speech recognition and natural language processing techniques. This makes it possible to determine rules for abstracting or hiding specific information and to adjust the audio to provide a new output to the user.

[0635] "Audio data" refers to information that represents ambient sounds in digital format.

[0636] "Noise reduction" is a process that removes unwanted background noise and other sounds from audio data, making the target audio clearer.

[0637] A "secure method" refers to a means of using encryption technology and secure protocols to maintain the confidentiality and integrity of information during data transmission and reception.

[0638] "Other computing devices" refers to external computers or servers connected to process, analyze, or store audio data.

[0639] "Speech recognition" is a technology that analyzes audio data and converts what is being said into text.

[0640] "Natural language processing technology" refers to computer technologies used to understand, generate, and analyze natural language used by humans.

[0641] "Identifying individuals" means identifying the speakers contained in audio data and distinguishing them based on the characteristics of each individual voice.

[0642] "Specific information" refers to personal information or information that may affect privacy that can be identified within the audio data.

[0643] "Abstraction" is the process of omitting the details of specific information and converting it into a more general expression.

[0644] "To hide" refers to the process of removing or concealing specific recognized information so that it is not visible or audible to other listeners.

[0645] "Rules" refer to instructions or standards regarding the restrictions on the disclosure and output methods of specific information, which are determined based on the analysis of audio data.

[0646] "Adjusting audio" means changing, editing, or processing the output generated from audio data according to specific rules.

[0647] "New output" refers to the adjusted voice message or information provided to the user based on the voice data.

[0648] The system in this invention is a data processing system that utilizes speech recognition technology and is designed for using a voice agent while protecting user privacy. The terminal collects ambient sound using a digital microphone and performs noise reduction processing. The processed voice data is transmitted to the server via a secure communication protocol.

[0649] The server converts audio data into text via a speech recognition engine. This process utilizes speech recognition software such as the Google Cloud Speech-to-Text API. The converted text data is then analyzed using natural language processing techniques to identify multiple individuals involved in the conversation. This identifies the linguistic characteristics of each individual involved, contributing to the identification of personal information.

[0650] Subsequently, the server develops rules to control the disclosure of private information based on the identified data. For example, rules may be constructed so that information is hidden or abstracted when a specific keyword is recognized. The terminal receives these rules and adjusts its audio output to prevent individual private information from being leaked to third parties other than the user.

[0651] As a concrete example, consider a user asking an AI assistant, "Do you have any plans next Friday?" The device receives this question and sends data to the server. If the analysis identifies next Friday as a private appointment, the server sets up rules to abstract that information and provide a general response such as, "You have an appointment coming up."

[0652] Examples of prompts to input into a generative AI model include questions like, "How can a user hide specific confidential information while conversing with someone?" Such a system would make it possible to maintain the convenience of voice communication while protecting privacy.

[0653] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0654] Step 1:

[0655] The device uses a microphone to capture ambient sound. This audio data is then processed using noise reduction technology to remove unwanted noise, resulting in clean audio data. This clean audio data is then passed on to the next processing step.

[0656] Step 2:

[0657] The terminal encrypts clean audio data and sends it to the server. This encryption process uses the SSL / TLS protocol to ensure data confidentiality and integrity. After the server receives the data, it is prepared for analysis.

[0658] Step 3:

[0659] The server inputs the received audio data into a speech recognition engine and converts it into text data. This process uses speech recognition software to accurately transcribe the audio content into text. The text data is then passed on to subsequent natural language processing.

[0660] Step 4:

[0661] The server uses natural language processing techniques to analyze text data and identify multiple people in a conversation. This process uses contextual analysis and speaker identification algorithms to pinpoint each person's utterance. The identified person information is then stored in a database.

[0662] Step 5:

[0663] The server develops privacy protection rules based on the identified individuals and the content of their statements. If specific keywords or phrases are detected, it determines guidelines for abstracting or concealing that information. These rules are then passed on to the next stage.

[0664] Step 6:

[0665] The server sends the established rules to the terminal. The terminal adjusts the audio output according to these rules, providing the user with modified audio feedback. This allows the user to receive the voice agent's output in a privacy-protected manner.

[0666] (Application Example 1)

[0667] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] In modern homes and with devices used by multiple people, there is a risk of privacy violations if information provided by voice assistants is leaked to third parties. It is necessary to solve this problem and provide an environment where users can use voice assistants with peace of mind.

[0669] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0670] In this invention, the server includes means for collecting voice information, means for analyzing the voice information to identify multiple members participating in the conversation, and means for determining a policy to restrict the disclosure of confidential information based on the identified members. This makes it possible to selectively output individual information and prevent the leakage of confidential information.

[0671] "Audio information" refers to sound data acquired through devices such as microphones.

[0672] "Members" refers to the multiple individuals or characters participating in the conversation within the audio information.

[0673] "Confidential information" refers to important personal information that, if made public, could potentially infringe on privacy.

[0674] A "policy" refers to specific rules and regulations established to restrict the disclosure of confidential information.

[0675] "Audio output" refers to the sound output from speakers or other devices that is generated based on audio information.

[0676] "Selectively outputting information" means restricting or adjusting the information output based on specific conditions.

[0677] The system for carrying out the present invention consists of multiple components. Voice information is acquired by a microphone installed in a terminal. This terminal is integrated into a device such as a smart home assistant. The voice information is sent from the terminal to a server in the cloud and processed by the server. This processing utilizes the Google Cloud Speech-to-Text API, which leverages speech recognition technology. Using this API, the voice data is converted into text data.

[0678] The server uses the converted text data to identify multiple members, leveraging NLP (Natural Language Processing) techniques. This analysis on the server utilizes, for example, IBM Watson's natural language processing services. Once members are identified, policies are generated to restrict information disclosure for privacy purposes. These policies place particular emphasis on keeping confidential information private.

[0679] The generated policy is sent to the terminal. The terminal adjusts the audio output based on this policy. Specifically, it selectively outputs individual information that only certain individuals need to know, for example, through the voice assistant function. This makes it possible to prevent the leakage of confidential information.

[0680] To give an example, consider a scenario where a robot is active in a living room at home. If the mother gives the robot a voice command to check the schedule, the robot will provide the children present with a general overview of the schedule and provide the mother with detailed information via smartphone.

[0681] An example of a prompt for a generative AI model is: "Explain how to identify a specific person from the acquired voice data, analyze the content of the conversation, and create and apply rules to protect privacy."

[0682] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0683] Step 1:

[0684] The device acquires audio information using its built-in microphone. This audio information is recorded in real time and temporarily stored as digital data within the device. In this case, the input is physical sound, and the output is digitized audio data.

[0685] Step 2:

[0686] The terminal sends the acquired audio data to a server in the cloud. This process involves data processing that packets the audio data. The input is digital audio data, and the output is packet data that is transmitted over the network.

[0687] Step 3:

[0688] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API. The input here is audio packet data, and the output is text data. Speech recognition technology analyzes the characteristics of the sound and generates information as a string of characters.

[0689] Step 4:

[0690] The server uses NLP technologies such as IBM Watson based on text data to identify multiple participants in a conversation. The input is text data, and the output is participant identification information. Natural language processing is performed here to understand the context.

[0691] Step 5:

[0692] The server determines policies to restrict the disclosure of confidential information based on identified member information. These policies are dynamically generated using a generative AI model. The input is identification information, and the output is a set of rules representing the policies.

[0693] Step 6:

[0694] The terminal adjusts the audio output based on policies received from the server. Filtering is performed to shield specific information or transmit it only to appropriate targets. The input is the policy rule set, and the output is the adjusted audio output.

[0695] Step 7:

[0696] The user receives voice output from the robot and confirms the necessary information. Here, the input is the adjusted voice output, and the output is the user's understanding and actions. The user can use the information from the robot to efficiently perform everyday tasks.

[0697] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0698] This invention is a system that combines speech recognition technology and emotion recognition technology to enable the use of a more personalized voice assistant while protecting user privacy. The terminal uses a microphone to acquire ambient audio signals and transmits this data to a server. The server analyzes the audio signals to identify multiple people participating in a conversation, and at the same time uses an emotion engine to determine the user's emotional state from the voice.

[0699] Based on emotion recognition, the server generates rules that consider the user's current emotional state to determine what information to disclose and how. These rules are adjusted according to the degree of privacy, meaning that the presentation of private information may vary depending on the user's emotions.

[0700] The terminal receives filtering rules sent from the server and adjusts the audio output. At this time, information is presented that matches the user's emotional state, ensuring a comfortable response for the user.

[0701] As a concrete example, consider a scenario where a user asks an AI assistant for schedule information with a friend. The device collects voice data and sends it to a server. The server uses voice analysis and an emotion engine to detect if the user is "anxious." Based on this, the server creates rules to avoid providing excessive details to the friend and to provide only the minimum necessary information. The device applies these rules and outputs information that is appropriately adjusted when the friend is present.

[0702] This invention enables the use of a flexible voice assistant that responds to the user's emotions while protecting the user's privacy.

[0703] The following describes the processing flow.

[0704] Step 1:

[0705] The device continuously acquires ambient audio signals using its built-in microphone. These audio signals are de-noised and compressed before being sent to the server.

[0706] Step 2:

[0707] The terminal sends the collected voice data to the server. At this time, it adjusts the packet delivery to ensure efficiency depending on the network conditions.

[0708] Step 3:

[0709] The server analyzes the received audio data and converts it into text data through speech recognition. In parallel, the emotion engine analyzes the user's voice characteristics from the same audio data and determines their emotional state in real time.

[0710] Step 4:

[0711] The server identifies the people participating in the conversation based on the speech recognition results. The emotion engine determines whether the user's emotions are in a specific state (e.g., excited, anxious, relaxed).

[0712] Step 5:

[0713] The server determines appropriate filtering rules to restrict the disclosure of private information based on the identified person's information and the user's emotional state. These rules apply stricter restrictions, such as when emotions are particularly heightened.

[0714] Step 6:

[0715] The server sends the determined filtering rules to the terminal and provides instructions for the terminal to adjust the audio output.

[0716] Step 7:

[0717] The device adjusts the audio output in real time according to the received filtering rules. This ensures that the privacy information of the conversation is properly protected while also enabling responses that are sensitive to the user's emotions.

[0718] Step 8:

[0719] Users can continue conversations with third parties with peace of mind by receiving appropriately filtered and adjusted responses. This system protects both user privacy and emotional security.

[0720] (Example 2)

[0721] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0722] Voice assistants often provide information without adequately considering user privacy. As a result, users risk privacy violations due to a lack of appropriate information disclosure control in specific situations or emotional states. Furthermore, current technology makes it difficult to control information disclosure based on the user's emotional state, resulting in the inability to provide flexible and personalized information as intended by the user.

[0723] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0724] In this invention, the server includes means for analyzing voice data to identify individual people, means for determining emotional states based on voice characteristics, and means for generating information disclosure control rules according to the determined emotional states. This enables flexible information disclosure control tailored to the user's emotional state, allowing for personalized information provision while protecting user privacy.

[0725] "Audio data" refers to audio signals acquired using audio equipment such as microphones, which have been converted into an analyzable digital format.

[0726] "Terminal means" refers to a device or mechanism that performs tasks such as acquiring, transmitting, and receiving voice data, and applying filtering rules.

[0727] "Information processing unit" refers to a computing device or computer system used to receive, analyze, identify, and generate rules for audio data.

[0728] "Means of identifying individuals" refer to algorithms and systems that analyze audio data to identify the speaker.

[0729] "Vocal characteristics" refer to attributes such as tone, speed, and volume of voice, and these attributes are used to determine emotional states.

[0730] "Means for determining emotional state" refers to technologies and algorithms used to infer a user's emotions and psychological state based on voice data.

[0731] "Information disclosure control rules" are criteria or policies that determine which information to disclose or withhold, and how, depending on the user's specific emotional state.

[0732] "Terminal means for adjusting data output" refers to a device or program that adjusts the format and content of information provided externally based on the generated information disclosure control rules.

[0733] Modes for carrying out the invention

[0734] This invention is a voice support system that integrates speech recognition technology and emotion recognition technology to provide information tailored to individual emotional states while prioritizing user privacy.

[0735] Hardware and software to be used

[0736] Device: To acquire audio, an audio device such as a smartphone or smart speaker is used. The built-in microphone is used for audio recording, and the device's operating system (e.g., Android OS, iOS) is used to temporarily store the audio data.

[0737] Server: A cloud data processing system is used to analyze audio data and recognize emotions. A speech recognition API (e.g., a general speech recognition API) is used for analyzing the audio signal, and an AI service for emotion analysis (e.g., a general emotion analysis service) is used to determine the emotional state.

[0738] Data processing flow

[0739] The terminal acquires audio and transmits the data to the server via a secure connection. The server uses the received audio data to first convert it into text using a speech recognition system, then analyzes the audio characteristics to identify individual participants and determine their emotional states. The generative AI model used in this system (e.g., a general generative model) generates information disclosure control rules based on the user's emotions based on this information. The terminal receives these rules and adjusts the audio information it outputs to the user and other participants.

[0740] Specific example

[0741] For example, consider a scenario where a user wants to check their schedule information with an AI assistant while with a friend. If the user is feeling a little nervous, the server recognizes this emotional state and generates a rule to provide only the minimum necessary information, taking the presence of the friend into account. The device applies this rule and provides the user with appropriately adjusted voice information, such as, "We'll let you know the details of the schedule later."

[0742] Example of a prompt

[0743] "I want to check my schedule while I'm with my friends. However, I'm secretly nervous, so please keep it as concise as possible."

[0744] "I want to receive important announcements during the meeting, but I don't want the details to be known to others."

[0745] This invention allows users to enjoy a flexible and personalized voice assistant experience that is tailored to their emotions, while also protecting their privacy.

[0746] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0747] Step 1:

[0748] The device acquires ambient sound through its microphone. When the user naturally speaks a request to the voice assistant, the audio signal is input to the device as digital data. This data undergoes data processing such as noise reduction and volume normalization before being prepared for transmission to the next processing step.

[0749] Step 2:

[0750] The terminal transmits pre-processed audio data to the server via a secure communication protocol. The input to the server is the audio data provided by the terminal, which is encrypted to ensure data security.

[0751] Step 3:

[0752] The server analyzes the received audio data using a speech recognition system. The input audio data is first converted to text by an acoustic model, from which prosody and vocal features are extracted. The output consists of text data and identified vocal characteristics.

[0753] Step 4:

[0754] The server performs emotion recognition based on identified speech characteristics. An emotion recognition algorithm analyzes these characteristics to determine the user's emotional state, such as "tension" or "calmness." The output of this process is an emotional state label.

[0755] Step 5:

[0756] The server generates information disclosure control rules based on the recognized emotional state. The generating AI model receives the emotional state as a prompt and determines the appropriate method of information disclosure for the specific situation. The output is a set of rules that provide guidance on disclosing or not disclosing information.

[0757] Step 6:

[0758] The terminal uses information disclosure control rules received from the server to adjust the information provided to the user. The input to this process is the control rules, and the audio output is optimized according to these rules. The content and presentation of the information are changed here, and the final output is presented to the user in either audio or text format.

[0759] Through this series of steps, users can use the voice assistant with peace of mind, while taking their own feelings and circumstances into consideration.

[0760] (Application Example 2)

[0761] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0762] Conventional voice assistant systems lacked the ability to detect emotions and were unable to provide appropriate responses based on the user's emotional state. Furthermore, they lacked sufficient privacy protection in situations involving multiple people, making it difficult to disclose or withhold information according to the user's wishes. Therefore, there is a need for the development of voice assistant technology that users can use with confidence.

[0763] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0764] In this invention, the server includes means for analyzing voice signals to identify multiple people participating in a conversation, means for determining the emotional state of a user from their voice using emotion recognition technology, and means for determining filtering rules that restrict the disclosure of private information based on the emotional state. This makes it possible to protect privacy while providing comfortable and appropriate voice responses according to the user's emotions.

[0765] "Audio signals" refer to sound captured as electrical signals, and are fundamental information used when collecting and analyzing audio data.

[0766] "Emotion recognition technology" is a technology that detects and analyzes a person's emotional state from data such as voice and facial expressions.

[0767] "Privacy information" refers to information about an individual whose disclosure should be restricted.

[0768] "Filtering rules" are criteria or policies used to select information based on specific conditions and decide whether to make it public or not.

[0769] "Audio output" refers to sound information produced by speech synthesis or other means and output from speakers or other devices.

[0770] A "server" is a computer system that performs data processing and provides services over a network.

[0771] "Household appliances" refer to automated devices used in a family's living space.

[0772] To implement this invention, the home device is equipped with a microphone for collecting voice signals. The voice signals are sent to a server and converted into text data using a speech recognition engine (e.g., Google Speech-to-Text API). Based on this data, an emotion engine such as IBM Watson Tone Analyzer is used as emotion recognition technology to determine the user's emotional state. On the server, filtering rules are generated to determine what information to disclose and to what extent, based on the determined emotional information. This includes setting conditions that take user privacy into consideration. The generated filtering rules are sent to the home device and used to adjust the voice output. The voice output is synthesized to provide a comfortable and appropriate response according to the user's emotional state. As a concrete example of operation, if the server determines that a family member is in a bad mood, the robot will speak in a subdued tone that takes the mood into consideration and offer suggestions to help them relax.

[0773] An example of a prompt used in the generation AI model would be: "Generate an appropriate conversation based on the emotional state of the family. For example, if the family is angry, generate a dialogue that incorporates gentle words to soothe their anger." Based on this prompt, the server will generate the conversation.

[0774] Through the processing flow described above, this invention can provide a customized voice assistant service that responds to the user's emotions, thereby facilitating smoother communication within the home.

[0775] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0776] Step 1:

[0777] The device collects sounds emitted within the home using a microphone and transmits the audio signal to a server. The input is an audio signal, and the output is the transfer of audio data to the server. Specifically, it collects family conversations and everyday sounds in real time.

[0778] Step 2:

[0779] The server converts the received audio signal into text data using a speech recognition engine. The input is audio data, and the output is text data. As part of the data processing, the audio signal is analyzed and converted into a string. At this stage, for example, a statement like "What are your plans for today?" is converted into text.

[0780] Step 3:

[0781] The server analyzes the converted text data using an emotion recognition engine to determine the user's emotional state. The input is text data, and the output is emotional information. As a data calculation, it performs analysis based on an emotion model to determine emotional states such as "stressed" or "relaxed." Specifically, the emotion recognition model estimates emotions from the words and context in the text.

[0782] Step 4:

[0783] The server generates appropriate filtering rules based on the determined emotional state. The input is emotional information, and the output is filtering rules. As part of the data processing, privacy levels are set and response content is adjusted based on the emotional state. For example, it considers family relationships to determine a policy for presenting information in a way that does not cause discomfort.

[0784] Step 5:

[0785] The server sends the generated filtering rules to the terminal and adjusts the voice output. The input is the filtering rules, and the output is the adjusted voice response. The terminal synthesizes voice according to these rules and provides an appropriate response to the user. Specifically, for example, it might say "What shall we do today?" in a gentle voice, performing an action to lighten the atmosphere in the home.

[0786] In this way, the system generates appropriate responses through conversations within the home, enabling comfortable and secure communication for the user.

[0787] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0788] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0789] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0790] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0791] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0792] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0793] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0794] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0795] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0796] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0797] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0798] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0799] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0800] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0801] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0802] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0803] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0804] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0805] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0806] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0807] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0808] The following is further disclosed regarding the embodiments described above.

[0809] (Claim 1)

[0810] Means for collecting audio signals,

[0811] A means for analyzing the aforementioned audio signal to identify multiple people participating in the conversation,

[0812] A means of determining rules to restrict the disclosure of private information based on identified individuals,

[0813] A means of adjusting the audio output by applying the said rule,

[0814] A system that includes this.

[0815] (Claim 2)

[0816] The system according to claim 1, wherein the aforementioned audio signal is transmitted by an external device.

[0817] (Claim 3)

[0818] The system according to claim 1, comprising means for detecting content containing private information and keeping such content confidential.

[0819] "Example 1"

[0820] (Claim 1)

[0821] A device for acquiring audio data, wherein the device has means for removing noise,

[0822] Means for transmitting the aforementioned audio data to another computing device in a secure manner,

[0823] A means of identifying a person using speech recognition and natural language processing technology,

[0824] A means for determining rules for abstracting or concealing specific information based on identified individuals,

[0825] A means for adjusting the audio based on the aforementioned rules to provide a new output to the user,

[0826] A system that includes this.

[0827] (Claim 2)

[0828] The system according to claim 1, wherein the aforementioned audio data is encrypted and transmitted by an external computing device.

[0829] (Claim 3)

[0830] The system according to claim 1, comprising means for abstracting or deleting specific information when such information is given.

[0831] "Application Example 1"

[0832] (Claim 1)

[0833] Means for collecting audio information,

[0834] Means for analyzing the aforementioned audio information to identify multiple members participating in the conversation,

[0835] A means of determining policies to restrict the disclosure of confidential information based on identified members,

[0836] A means of adjusting the sound output by applying the said policy,

[0837] A means of selectively outputting individual information in a situation where multiple people are present,

[0838] A system that includes this.

[0839] (Claim 2)

[0840] The system according to claim 1, wherein the aforementioned audio information is transmitted by an external device.

[0841] (Claim 3)

[0842] The system according to claim 1, comprising means for identifying content containing confidential information and concealing such content.

[0843] "Example 2 of combining an emotion engine"

[0844] (Claim 1)

[0845] A terminal means for acquiring audio data,

[0846] Means for transmitting the aforementioned audio data to an external information processing unit,

[0847] The aforementioned information processing unit analyzes the audio data and provides means for identifying individual persons,

[0848] A means of determining emotional states based on vocal characteristics,

[0849] A means for generating information disclosure control rules corresponding to the identified emotional state,

[0850] A system including terminal means for adjusting data output in accordance with the aforementioned rules.

[0851] (Claim 2)

[0852] The system according to claim 1, wherein information non-disclosure settings are made based on the emotional state generated in an external information processing unit.

[0853] (Claim 3)

[0854] The system according to claim 1, comprising means for using a generative model based on emotional states and flexibly adjusting the application rules for information output.

[0855] "Application example 2 when combining with an emotional engine"

[0856] (Claim 1)

[0857] Means for collecting audio signals,

[0858] A means for analyzing the aforementioned audio signal to identify multiple people participating in the conversation,

[0859] A means of determining a user's emotional state from their voice using emotion recognition technology,

[0860] A means of determining filtering rules that restrict the disclosure of private information based on emotional state,

[0861] A means of applying the said rule to provide an adjusted voice output that responds to the user's emotions,

[0862] A system that includes this.

[0863] (Claim 2)

[0864] The system according to claim 1, wherein the aforementioned audio signal is transmitted by an external device.

[0865] (Claim 3)

[0866] The system according to claim 1, comprising means for keeping confidential content, including private information detected by a household appliance. [Explanation of symbols]

[0867] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for collecting audio signals, A means for analyzing the aforementioned audio signal to identify multiple people participating in the conversation, A means of determining rules to restrict the disclosure of private information based on identified individuals, A means of adjusting the audio output by applying the said rule, A system that includes this.

2. The system according to claim 1, wherein the aforementioned audio signal is transmitted by an external device.

3. The system according to claim 1, comprising means for detecting content containing private information and keeping such content confidential.

Images