system

Abstract

This system improves the efficiency of data utilization by automatically extracting and deleting confidential information from data containing such information. [Solution] A means of obtaining data files from a database, A means of converting acquired audio data into text data, A means of identifying and extracting personal information from text data, Means for deleting or replacing identified personal information, A means of reconstructing audio data using revised text data, Means for storing and providing reconstructed audio data, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern enterprises, the utilization of voice data including personal information is extremely important. However, since it contains highly secure information, it must be properly managed. In the conventional method, there is a problem that the process of manually deleting top-secret information and personal information is complicated and time-consuming, and the efficiency of data utilization is significantly reduced. The present invention aims to solve such problems and improve the efficiency of data utilization by automatically extracting and deleting top-secret information from data including top-secret information.

Means for Solving the Problems

[0005] The present invention solves the above problems by the following means: a system including means for obtaining data files from a database, means for converting the obtained audio data into text data, means for identifying and extracting personal information from the text data, means for deleting or replacing the identified personal information, means for reconstructing the audio data using the revised text data, and means for storing and providing the reconstructed audio data. This enables the automatic extraction and deletion of highly confidential information and allows for the efficient and secure use of data.

[0006] A "data file" is a file used to store information, and may include audio, text, images, and other forms of data.

[0007] "Audio data" refers to digital data files containing recorded audio, including call recordings and voice messages.

[0008] "Text data" refers to digital data files composed of characters and symbols, and includes results such as transcriptions of audio data.

[0009] "Personal information" refers to information used to identify a specific individual, and includes names, addresses, telephone numbers, etc.

[0010] "Identification" refers to the act of finding specific patterns or information from data, and is the process of detecting personal information.

[0011] "Extraction" refers to the act of selecting necessary information from specific data, and specifically to the process of extracting highly confidential information from data.

[0012] "Deletion" refers to the act of completely removing specific information from data, and is the process of erasing unnecessary or highly confidential information.

[0013] "Replacement" refers to the act of changing specific information within data into other information, and is a technique used to mask personal information.

[0014] "Speech synthesis technology" refers to the technology used to convert text data into speech data, and specifically to the process of generating audio files that read text aloud.

[0015] "Speech recognition technology" refers to the technology used to convert speech data into text data, and the process of representing the content of speech as written characters. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Modes for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention provides a system for securely and efficiently processing data, including highly confidential and personal information, within a company. This system includes a series of processes for retrieving data files from a database, converting audio data to text data, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data. The processing of the system's program is described in detail below in natural language.

[0038] Data acquisition

[0039] server:

[0040] The server retrieves the target call data file from the company's database. For example, the server loads the "call_001.wav" file from the "customer_calls.db" database. This file contains the audio of a call with a customer stored in the database.

[0041] Converting audio data to text

[0042] server:

[0043] The server converts the acquired audio data into text data using a speech recognition API (e.g., Google Speech-to-Text API or Sphinx). The file "call_001.wav" is uploaded, and the transcription result is saved as a text file named "call_001_transcript.txt" in the API response. This text file contains a written record of the call.

[0044] Identification of Top Secret Information

[0045] AI models on the server:

[0046] An AI model, pre-trained on the server, analyzes the converted text data to identify personal information (such as name and address). The AI ​​model uses natural language processing techniques to tokenize the text, highlighting important identifying information based on specific patterns and context. For example, it identifies the name "Ichiro Tanaka" from the text "Hello, I'm Ichiro Tanaka. Today I'd like to talk about the contract details."

[0047] Extraction and deletion of top-secret information

[0048] server:

[0049] Identified personal information is removed from the text data and replaced appropriately. The server uses regular expressions to replace "Ichiro Tanaka" with "" or "anonymous". As a result, a revised text file "call_001_transcript_safe.txt" is generated, which may read something like "Hello, I'd like to talk about the contract details today."

[0050] Data reconstruction

[0051] server:

[0052] The audio data is reconstructed based on the revised text data. The server uses speech synthesis technology (for example, the Google Text-to-Speech API) to convert the revised text into audio data. This generates a new audio file called "call_001_safe.wav", which can play the corresponding portion of the original audio in a safe format. This file is saved for later use by the user.

[0053] Provision to users

[0054] User:

[0055] Users download the securely processed audio data "call_001_safe.wav" from the server and use it for business analysis or research purposes. This allows users to utilize a clean dataset that does not contain any personal information.

[0056] A concrete example is call center call data. By processing call data containing highly confidential information such as customer names and addresses using this system, relevant business analysis can be performed without the risk of personal information leakage. This significantly improves a company's data utilization capabilities and promotes the creation of business value.

[0057] The following describes the processing flow.

[0058] Step 1:

[0059] server:

[0060] The server retrieves call data files from the company's database. Specifically, it executes a query against the "database management system" to search for and retrieve the target call data (for example, "call_001.wav"). It searches using an SQL statement such as "SELECT FROM calls WHERE id='call_001'" and saves the retrieved file to a temporary working directory.

[0061] Step 2:

[0062] server:

[0063] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, the audio file is uploaded to a speech recognition service such as the Google Speech-to-Text API or Sphinx, and the audio data is converted into text data. The resulting transcript is saved as "call_001_transcript.txt".

[0064] Step 3:

[0065] AI models on the server:

[0066] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Specifically, it uses natural language processing techniques to tokenize the text and identify specific words and phrases (for example, names and addresses). This process reveals important identifying information such as the name "Ichiro Tanaka" and the address "Shinjuku-ku, Tokyo".

[0067] Step 4:

[0068] server:

[0069] Delete or replace identified personal information. Specifically, use regular expressions to replace personal information in the text with safe strings such as "" or "anonymous". For example, change "Hello, I'm Tanaka Ichiro." to "Hello, desu.". This will generate the revised text data "call_001_transcript_safe.txt".

[0070] Step 5:

[0071] server:

[0072] New audio data is generated based on the revised text data. Specifically, speech synthesis technology (e.g., Google Text-to-Speech API) is used to convert "call_001_transcript_safe.txt" into audio data. This generates a new audio file, "call_001_safe.wav," from which personal information has been removed. The generated audio file is saved in a temporary working directory.

[0073] Step 6:

[0074] server:

[0075] The generated secure audio file "call_001_safe.wav" will be saved and made available to the user. Specifically, the file will be moved to a designated folder, and a downloadable URL will be generated for the user. The user will then be able to use this URL to access the secure audio data for business analysis or research purposes.

[0076] (Example 1)

[0077] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0078] In processing data containing highly confidential and personal information within companies, there is a need to balance security and efficiency. Traditional methods often involve manual data processing, which is time-consuming and labor-intensive, and carries a high risk of information leaks due to human error. Furthermore, there has been a lack of methods to adequately protect personal information when handling audio data. Therefore, there is a need for technology that can automatically extract personal information from audio data, anonymize it, and make it reusable.

[0079] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0080] In this invention, the server includes means for obtaining data files from a database, means for converting the obtained audio data into text data using speech recognition technology, means including a generative artificial intelligence model for analyzing the converted text data and identifying and extracting personal information, means for deleting or replacing the identified personal information, means for reconstructing the audio data using speech synthesis technology with the revised text data, and means for storing and providing the reconstructed audio data. This enables the secure and efficient processing of data, including highly confidential information and personal information.

[0081] A "data file" is a file stored in a company or organization's database that contains information collected for a specific business or purpose.

[0082] "Audio data" refers to data that records human voices in digital format.

[0083] "Speech recognition technology" is a technology that converts speech data into text data.

[0084] "Text data" refers to data that records information written using characters and symbols in a digital format.

[0085] "Personal information" refers to information that can identify a specific individual, such as name, address, and telephone number.

[0086] A "generative artificial intelligence model" is an AI model that learns from large amounts of text data and analyzes the content of that text using natural language processing techniques.

[0087] "Deletion" refers to the operation of completely removing specific information from data.

[0088] "Replacement" refers to the operation of changing identified information into a different string or data.

[0089] "Speech synthesis technology" is a technology that artificially generates speech from text data.

[0090] "Reconstruction" refers to the process of creating new audio data based on revised text data.

[0091] "Saving" refers to the process of recording generated data on a digital medium so that it can be accessed later.

[0092] "Providing" refers to the operation of making generated data available for use by users.

[0093] This invention provides a system for securely and efficiently processing data, including highly confidential and personal information, within a company. This system includes a series of processes for acquiring audio data from a database, converting that audio data into text data using speech recognition technology, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data.

[0094] The server accesses the company's database and retrieves the target call data file. This operation uses database queries. For example, it executes the SQL query "SELECT FROM customer_calls WHERE call_id = '001'" to load the target call audio file "call_001.wav". This file contains the audio of the call with the customer.

[0095] The server converts the acquired audio data into text data using a speech recognition API. Specifically, it uses the Google Speech-to-Text API or Sphinx. The audio file "call_001.wav" is uploaded to the API, and the transcript result obtained as a response from the API is saved to "call_001_transcript.txt". This text file contains a record of the call content in text.

[0096] Next, an AI model on the server analyzes the converted text data to identify personal information (e.g., name and address). This AI model includes a pre-trained generative AI model and uses natural language processing techniques to tokenize the text, highlighting important identifying information based on specific patterns and contexts. For example, it identifies the name "Ichiro Tanaka" from the text "Hello, I'm Ichiro Tanaka. Today I'd like to talk about the contract details."

[0097] The server removes and appropriately replaces identified personal information from the text data. Specifically, it uses regular expressions to search for specific patterns and replaces corresponding parts with "" or "anonymous". As a result, it generates a revised text file "call_001_transcript_safe.txt" that reads something like "Hello, I'd like to talk about the contract details today."

[0098] The server then generates new audio data based on the revised text data. Specifically, it uses the Google Text-to-Speech API to generate audio data from the revised text. This generates a new audio file called "call_001_safe.wav" and plays the corresponding portion of the original audio in a secure format. This file is saved so that the user can use it later.

[0099] Users download the securely processed audio data "call_001_safe.wav" from the server and use it for business analysis and research purposes. This allows users to utilize a clean dataset that does not contain any personal information.

[0100] A concrete example is call center data. By processing call data containing highly confidential information such as customer names and addresses using this system, relevant business analysis can be performed without the risk of personal information leakage. This significantly improves a company's data utilization capabilities and promotes the creation of business value.

[0101] An example of a prompt for a generative AI model can be written as follows:

[0102] "Please extract the customer's name and address from this audio data, anonymize it, and create a reconstructed audio file."

[0103] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0104] Step 1:

[0105] The server accesses the company's database and retrieves the target call data file. Specifically, the server executes the SQL query "SELECT FROM customer_calls WHERE call_id = '001'" and loads the call audio file "call_001.wav" stored in the database. The input for this step is the database query, and the output is the retrieved audio file.

[0106] Step 2:

[0107] The server converts the acquired audio data into text data using a speech recognition API. Specifically, the server uploads the audio file "call_001.wav" to the Google Speech-to-Text API and receives the transcription result as a response from the API. This transcription result is saved to "call_001_transcript.txt". The input for this step is an audio file, and the output is a text file.

[0108] Step 3:

[0109] An AI model on the server analyzes the converted text data and identifies personal information. Specifically, the server inputs "call_001_transcript.txt" to the AI ​​model, and the model uses natural language processing techniques to extract personal information (e.g., name and address) from the text. The input for this step is a text file, and the output is a list of identified personal information.

[0110] Step 4:

[0111] The server removes identified personal information from the text data and replaces it appropriately. Specifically, the server uses regular expressions to search for specific patterns within "call_001_transcript.txt" and replaces the corresponding parts with "" or "anonymous". As a result of this process, a revised text file called "call_001_transcript_safe.txt" is generated. The input for this step is a list of identified personal information and a text file, and the output is a revised text file.

[0112] Step 5:

[0113] The server generates new audio data based on the revised text data. Specifically, the server uses the Google Text-to-Speech API to convert "call_001_transcript_safe.txt" into audio data and generates a new audio file "call_001_safe.wav". The input for this step is the revised text file, and the output is the new audio file.

[0114] Step 6:

[0115] The user downloads securely processed audio data from the server. Specifically, the user downloads "call_001_safe.wav" via the server's web interface or API and plays and analyzes it on their local device. The input for this step is a new audio file, and the output is the audio file saved on the user's local device.

[0116] (Application Example 1)

[0117] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0118] In recent years, with the increasing use of digital data, the secure processing of personal and confidential information, including voice data, has become a critical issue. In particular, real-time voice data processing requires the rapid and accurate detection of personal information and its appropriate processing while preventing data leaks. However, conventional systems have lacked sufficient means to protect personal information while processing voice data in real time, making it difficult to ensure user privacy.

[0119] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0120] In this invention, the server includes means for acquiring data files from data storage, means for converting acquired audio data into text data, means for identifying and extracting personal and confidential information from the text data, means for deleting or replacing the identified information, means for reconstructing audio data using the revised text data, means for storing and providing the reconstructed audio data, means for acquiring user audio in real time and detecting and replacing personal information, and means for generating and providing clean audio data in real time. This makes it possible to securely process and provide audio data containing personal and confidential information in real time.

[0121] "Data storage" refers to a storage device used to store and manage data.

[0122] A "data file" is a collection of information managed by a computer, and specifically includes audio data and text data.

[0123] "Audio data" refers to data obtained by digitizing sound signals, which is usually analyzed using speech recognition technology.

[0124] "Text data" is a collection of information represented by characters and symbols.

[0125] "Personal information" refers to information that can identify a specific individual, including names and addresses.

[0126] "Confidential information" refers to important information whose disclosure should be restricted, and includes internal company documents and contract details.

[0127] "Identification" is the process of distinguishing and verifying specific information.

[0128] "Extraction" is the process of extracting specific information from a large amount of data.

[0129] "Deletion" is the process of erasing existing data.

[0130] "Replacement" is the process of substituting original data with other data.

[0131] "Revised text data" refers to text data after personal or confidential information has been deleted or replaced.

[0132] "Reconstructing audio" is the process of generating new audio data based on text data.

[0133] "Real-time" refers to a state in which data is acquired, processed, and provided immediately.

[0134] "Clean audio data" refers to secure audio data from which personal and confidential information has been removed.

[0135] This invention provides a system for safely and efficiently processing voice data in a business. The system for carrying out this invention includes the following hardware and software configuration.

[0136] Hardware and software to be used

[0137] Data storage: A storage device that stores and manages data files. Examples include internal servers and cloud servers within a company.

[0138] Speech recognition technology: Technology for converting speech data into text data. Specific examples include "Google Speech-to-Text API" and "Sphinx".

[0139] AI Model: A model pre-trained to identify and extract personal and confidential information. Examples include implementations using Python's natural language processing libraries (such as NLTK and SpaCy).

[0140] Speech synthesis technology: A technology used to reconstruct text data into speech data. For example, the "Google Text-to-Speech API" is used.

[0141] Smart gadgets: Devices for processing audio data in real time during operation. Specifically, this includes smart glasses and head-mounted displays.

[0142] System Operation Overview

[0143] 1. Obtaining data files

[0144] The server retrieves an audio data file from data storage. For example, it loads the "call_001.wav" file from "customer_calls.db". This is an audio recording of a customer call stored in the company's database.

[0145] 2. Converting audio data to text data

[0146] The server converts the acquired audio data into text data using speech recognition technology. Specifically, it uploads "call_001.wav" to the API and saves the resulting transcription to a text file called "call_001_transcript.txt".

[0147] 3. Identification and extraction of personal information

[0148] Using an AI model pre-trained on the server, the converted text data is analyzed to identify personal and confidential information. Based on natural language processing technology, the text is tokenized and important identifying information is extracted.

[0149] 4. Deletion or replacement of information

[0150] Identified personal information is removed from the text data and replaced appropriately. The server uses regular expressions or similar methods to convert the data to "" or "anonymous". For example, from the text "Hello, I'm Tanaka Ichiro.", "Tanaka Ichiro" is replaced with "", generating "Hello, desu.".

[0151] 5. Reconstruct the audio data based on the revised text data.

[0152] The server uses speech synthesis technology to convert the revised text data into audio data. For example, it generates a new audio file named "call_001_safe.wav".

[0153] 6. Generate and provide clean audio data in real time.

[0154] This system acquires user voices in real time, uses an AI model to instantly detect and replace personal information, and generates clean voice data. This allows users to use secure voice data in real time while minimizing the risk of personal information leakage.

[0155] Examples of specific cases and prompt statements

[0156] Specific example: If a user says "Hello, I'm Tanaka Ichiro" during a meeting, the smart glasses instantly process the audio data and display "Hello, I am." on the screen.

[0157] Prompt example: Find and replace the name and address in this spoken text. Input: 'Hello, I'm Ichiro Tanaka. What time does the meeting start?'

[0158] This invention makes it possible to securely process and provide audio data containing personal and confidential information in real time, significantly improving security in corporate data operations.

[0159] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0160] Step 1:

[0161] Retrieving data files

[0162] The server retrieves the necessary audio data files from data storage. In this process, the server accesses the database and loads a specific audio file (e.g., "call_001.wav").

[0163] Input: Path to the audio data file in data storage

[0164] Output: Acquired audio data file

[0165] Specific action: The server loads "call_001.wav" from "customer_calls.db".

[0166] Step 2:

[0167] Converting audio data to text data

[0168] The server converts the acquired audio data into text data using speech recognition technology. For example, it can use the "Google Speech-to-Text API" to input audio data and generate a transcript.

[0169] Input: Audio data file (e.g., "call_001.wav")

[0170] Output: Text data file corresponding to the audio data (e.g., "call_001_transcript.txt")

[0171] Specific operation: Upload "call_001.wav" to the API and save the resulting transcription to "call_001_transcript.txt".

[0172] Step 3:

[0173] Identification and extraction of personal and confidential information

[0174] Using an AI model pre-trained on the server, the converted text data is analyzed to identify and extract personal and confidential information.

[0175] Input: Converted text data (e.g., "call_001_transcript.txt")

[0176] Output: List of identified personal and confidential information

[0177] Specific operation: Using an AI model (e.g., a Python natural language processing library), analyze specific patterns and contexts within text to identify information such as "Ichiro Tanaka".

[0178] Step 4:

[0179] Deletion or replacement of information

[0180] The server removes identified personal information from the text data and replaces it appropriately. For example, it uses regular expressions to convert "Ichiro Tanaka" to "".

[0181] Input: A list of identified personal and confidential information, a text data file (e.g., "call_001_transcript.txt")

[0182] Output: Revised text data (e.g., "call_001_transcript_safe.txt")

[0183] Specific action: Replace "Ichiro Tanaka" with quotation marks and generate revised text such as "Hello, desu."

[0184] Step 5:

[0185] Reconstruct audio data based on revised text data.

[0186] The server uses speech synthesis technology to convert the revised text data into speech data. For example, it generates a new audio file using the Google Text-to-Speech API.

[0187] Input: Revised text data file (e.g., "call_001_transcript_safe.txt")

[0188] Output: Reconstructed audio data file (e.g., "call_001_safe.wav")

[0189] Specific action: Save "call_001_transcript_safe.txt" as "call_001_safe.wav" using speech synthesis technology.

[0190] Step 6:

[0191] Generate and provide clean audio data in real time.

[0192] This system acquires user voices in real time, instantly detects and replaces personal information, and generates clean voice data.

[0193] Input: Audio data acquired in real time

[0194] Output: Clean audio data processed immediately.

[0195] Specific operation: Using an AI model, personal information is instantly identified, clean audio data is generated using speech synthesis technology, and displayed on the user's smart glasses.

[0196] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0197] This invention provides a system for securely and efficiently processing audio data containing highly confidential and personal information, and for recognizing and analyzing user emotions. This system includes specific processing steps such as retrieving data files from a database, converting audio data into text data, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data, as well as a process for recognizing user emotions using an emotion engine. The program processing of this system is described in detail below.

[0198] Data acquisition

[0199] server:

[0200] The server retrieves the target call data file from the company's database. For example, the server loads the "call_001.wav" file from the "database management system." This file contains the audio of a call with a customer.

[0201] Converting audio data to text

[0202] server:

[0203] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, a speech recognition service such as Google Speech-to-Text API or Sphinx is used to convert the audio data into text data. The resulting transcript is saved as "call_001_transcript.txt".

[0204] Identification of Top Secret Information

[0205] AI models on the server:

[0206] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Using natural language processing techniques, the text is tokenized to identify information such as names and addresses. Through this process, important identifying information such as "Ichiro Sato" and "Shibuya Ward, Tokyo" is extracted from the text "Hello, I'm Ichiro Sato. Today I'd like to talk about Shibuya Ward, Tokyo."

[0207] Extraction and deletion of personal information

[0208] server:

[0209] Identified personal information is removed or replaced from the text data. Specifically, regular expressions are used to replace personal information in the text with safe strings such as "" or "anonymous". As a result, a revised text file "call_001_transcript_safe.txt" is generated, which may contain text like "Hello, I'd like to talk about today."

[0210] Reconstructing audio data

[0211] server:

[0212] New audio data is generated based on the revised text data. Speech synthesis technology (e.g., Google Text-to-Speech API) is used to convert "call_001_transcript_safe.txt" into audio data. This generates a new audio file, "call_001_safe.wav," from which personal information has been removed.

[0213] Emotional analysis using an emotional engine

[0214] server:

[0215] The emotion engine on the server analyzes the acquired audio data "call_001.wav" to recognize the user's emotions. The emotion engine analyzes parameters such as voice tone, pitch, and speed to identify the user's emotions. For example, the server identifies the user's emotions as emotion tags such as "joy," "anger," and "sadness," and adds this information to the text data.

[0216] Addition of emotional information and response generation

[0217] server:

[0218] The emotion engine identifies emotional information and adds it to text data, automatically generating responses based on specific emotions. For example, if the user is angry, it might generate a response such as, "You seem angry. Let me explain calmly." This response is then reconstructed as corresponding audio data and provided to the user.

[0219] Data storage and provision to users

[0220] server:

[0221] The generated secure audio file "call_001_safe.wav" and text data with added emotional information are saved and made accessible to the user. Users can then download the data from the server and use it for business analysis or research purposes.

[0222] As a concrete example, consider customer call data from a call center. By processing call data containing highly confidential information such as customer names and addresses with this system, it becomes possible to perform sentiment analysis of call content while eliminating the risk of customer information leakage. This significantly improves the quality of customer service and operational efficiency.

[0223] The following describes the processing flow.

[0224] Step 1:

[0225] server:

[0226] The server retrieves the target call data file from the company's database. Specifically, the server uses a database query to search for and retrieve a call file such as "call_001.wav". For example, it executes the SQL command "SELECT FROM calls WHERE id='call_001'" and saves the result to a temporary working directory.

[0227] Step 2:

[0228] server:

[0229] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, a speech recognition service such as Google Speech-to-Text API or Sphinx is used to convert the audio data into text data. The audio file is uploaded using the API, and the converted text data is saved as "call_001_transcript.txt".

[0230] Step 3:

[0231] AI models on the server:

[0232] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Specifically, it uses natural language processing technology to tokenize the text and extract personal information such as names and addresses. Through this process, for example, "Hello, I'm Sato Ichiro. Today I'd like to talk about Shibuya Ward, Tokyo." can be interpreted as "Sato Ichiro" and "Shibuya Ward, Tokyo."

[0233] Step 4:

[0234] server:

[0235] Remove or replace identified personal information from the text data. Specifically, use regular expressions to replace personal information in the text with safe strings such as "" or "anonymous". For example, change "Hello, I'm Sato Ichiro." to "Hello, desu.". Generate the revised text file "call_001_transcript_safe.txt".

[0236] Step 5:

[0237] server:

[0238] New audio data will be generated based on the revised text data. Specifically, speech synthesis technology will be used to convert "call_001_transcript_safe.txt" into audio data. The revised text will be uploaded using the Google Text-to-Speech API or similar, the synthesized audio data "call_001_safe.wav" will be generated and saved to a temporary working directory.

[0239] Step 6:

[0240] server:

[0241] The emotion engine on the server analyzes the original audio data "call_001.wav" to recognize the user's emotions. Specifically, it analyzes parameters such as tone, pitch, and speed of the audio data to identify the user's emotions (e.g., joy, anger, sadness) and adds that information to the text data.

[0242] Step 7:

[0243] server:

[0244] Based on the emotion information identified by the emotion engine, emotion tags are added to the revised text data. Specifically, if the user is angry, a response such as "You seem angry. Let me explain calmly" is generated and added to the revised text. In some cases, audio data may be generated again based on this revised text data.

[0245] Step 8:

[0246] server:

[0247] The generated secure audio data "call_001_safe.wav" and revised text data will be saved and made available to users for access. Specifically, the server will place the files in a designated folder and generate a URL that users can download. Users will use this URL to access the secure data and use it for business analysis or research purposes.

[0248] This specific process not only automatically removes confidential information from call data, but also provides a system that recognizes user emotions and enables appropriate responses and analysis. This allows companies to significantly improve the quality of customer service and operational efficiency.

[0249] (Example 2)

[0250] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0251] Conventional voice data processing systems had security risks and efficiency issues when extracting and deleting personal information from voice data. Furthermore, they lacked the ability to identify user emotions and generate appropriate responses, in addition to simply deleting personal information. As a result, improvements in customer service quality and operational efficiency were not fully achieved.

[0252] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0253] In this invention, the server includes means for obtaining data files from a database, means for converting the obtained voice data into text data, means for identifying and extracting personal information from the text data, means for deleting or replacing the identified personal information, means for reconstructing voice data using the revised text data, means for storing and providing the reconstructed voice data, means for using an emotion engine for sentiment analysis of the voice data, and means for identifying the user's emotions and generating a response based on those emotions. This makes it possible to analyze the emotions of call content while eliminating the risk of leakage of personal information. Furthermore, it enables significant improvements in the quality of customer service and operational efficiency.

[0254] A "data file" refers to information stored in digital format, and includes various types of data such as audio, text, and images.

[0255] "Audio data" refers to digitized audio information, including various types of audio such as phone calls, recordings, and lectures.

[0256] "Text data" refers to data that stores characters and symbols in a digital format, in a format that is easily analyzable by software and systems.

[0257] "Personal information" refers to information used to identify a specific individual, and includes names, addresses, telephone numbers, and email addresses.

[0258] "Identification" means distinguishing specific information or data from other information or data.

[0259] "Extraction" refers to the process of extracting necessary information or features from a large amount of data.

[0260] "Deletion" means completely removing unnecessary or risky information from the data.

[0261] "Substitution" refers to replacing specific information with other information.

[0262] "Revised text data" refers to text data that has been modified by removing or replacing personal information from the original text data.

[0263] "Speech synthesis technology" is a technology that generates speech by imitating human voices based on text data.

[0264] An "emotion engine" is a general term for systems that analyze and identify emotions from voice and text data.

[0265] "Sentiment analysis" refers to the process of reading emotions from audio or text and identifying specific emotional states.

[0266] "Response" refers to the system's reply or reaction to user input or statements.

[0267] This invention provides a system for securely and efficiently processing voice data containing highly confidential and personal information, and further for recognizing and analyzing user emotions. This system includes the following main processing steps:

[0268] First, the server retrieves a data file from the database. Specifically, it uses a database management system to access the database and retrieve a designated audio data file (e.g., call_001.wav). Next, the server sends the retrieved audio data to a speech recognition API (e.g., Google Speech-to-Text API or Sphinx) and converts the retrieved audio data into text data. The resulting transcript is saved as a file (e.g., call_001_transcript.txt).

[0269] Subsequently, a pre-trained AI model on the server analyzes the text data and identifies personal information. Natural language processing (NLP) techniques are used to tokenize the text, and then identifying information such as names and addresses is extracted. The identified personal information is either deleted or replaced with a secure string by the server, thereby generating a revised text data file (e.g., call_001_transcript_safe.txt).

[0270] Furthermore, the server uses speech synthesis technology such as the Google Text-to-Speech API to generate new audio data based on this revised text data. This results in a new audio file (e.g., call_001_safe.wav) from which personal information has been removed.

[0271] Furthermore, an emotion engine within the server analyzes the voice data and recognizes the user's emotions. The emotion engine analyzes parameters such as voice tone, pitch, and speed, and identifies the user's emotions as emotion tags such as "joy," "anger," and "sadness." This information is added to the text data, and a response based on the specific emotion is automatically generated. The response is then reconstructed as voice data using speech synthesis technology and provided to the user.

[0272] For example, when processing customer call data in a call center, this system allows for the secure removal of highly confidential information such as customer names and addresses, while also enabling sentiment analysis of the call content. This is expected to improve the quality of customer service and enhance operational efficiency.

[0273] Examples of prompt statements include the following:

[0274] Please describe a system that processes customer call data, removes specific personal information, analyzes user sentiment, and generates appropriate responses. Also, please provide specific application scenarios in a call center.

[0275] The above describes a specific implementation of a system that can perform sentiment analysis of call content while eliminating the risk of personal information leakage.

[0276] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0277] Step 1:

[0278] Data acquisition

[0279] The server retrieves data files from the database.

[0280] Input: Database connection information and the ID of the audio data file to be retrieved (e.g., 001)

[0281] Process: The server executes a query on the database management system to obtain the specified audio data file (call_001.wav). Specifically, it executes the query "SELECT FROM calls WHERE id=001;".

[0282] Output: The obtained audio data file (call_001.wav)

[0283] Step 2:

[0284] Text conversion of audio data

[0285] The server sends the obtained audio data file to the speech recognition API.

[0286] Input: Audio data file (call_001.wav)

[0287] Process: The server sends the audio data to the endpoint of the Google Speech-to-Text API for conversion to text data. The text data returned from the API is saved as "call_001_transcript.txt".

[0288] Specific operation: The server sends the audio file to the speech recognition API as a POST request and receives the conversion result in JSON format. It saves it as a text file.

[0289] Output: Transcription text file (call_001_transcript.txt)

[0290] Step 3:

[0291] Identification of confidential information

[0292] The AI model in the server analyzes the text data to identify personal information.

[0293] Input: Transcription text (call_001_transcript.txt)

[0294] Processing: Text is tokenized using natural language processing (NLP) techniques, and personal information such as names and addresses is extracted.

[0295] Specific operation: The AI ​​model analyzes each word and phrase in the text to identify personal information such as "Ichiro Sato" and "Shibuya Ward, Tokyo."

[0296] Output: List of personal information (e.g., "Ichiro Sato", "Shibuya Ward, Tokyo")

[0297] Step 4:

[0298] Extraction and deletion of personal information

[0299] The server deletes or replaces identified personal information.

[0300] Input: Transcript text (call_001_transcript.txt) and a list of personal information

[0301] Processing: Regular expressions are used to replace identified personal information with "" or "anonymous". This generates the revised text data (call_001_transcript_safe.txt).

[0302] Specific operation: The server uses a list of identified personal information to replace the relevant parts of the text with regular expressions.

[0303] Output: Revised text data file (call_001_transcript_safe.txt)

[0304] Step 5:

[0305] Reconstructing audio data

[0306] The server generates new audio data based on the revised text data.

[0307] Input: Revised text data file (call_001_transcript_safe.txt)

[0308] Process: Implement the procedure of converting text data into audio data using the Google Text-to-Speech API.

[0309] Specific operation: The server sends the text to the endpoint of the Google Text-to-Speech API and saves the returned audio data as a file. Example: "call_001_safe.wav"

[0310] Output: New audio data file (call_001_safe.wav)

[0311] Step 6:

[0312] Sentiment analysis by the sentiment engine

[0313] The sentiment engine in the server analyzes the audio data and recognizes the user's sentiment.

[0314] Input: Original audio data file (call_001.wav)

[0315] Process: The sentiment engine analyzes parameters such as the tone, pitch, and speed of the voice to identify the user's sentiment.

[0316] Specific operation: The sentiment engine decomposes the audio file into segments, calculates the feature quantities of each segment, and assigns sentiment tags.

[0317] Output: Text data with sentiment information (sentiment tags such as "joy", "anger", "sadness", etc.)

[0318] Step 7:

[0319] Addition of sentiment information and response generation<00010​The server adds emotional information to text data and generates responses based on specific emotions.

[0321] Input: Text data with sentiment information and revised text data

[0322] Processing: Based on specific emotions, responses are automatically generated and then reconstructed using speech synthesis technology.

[0323] Specific operation: The server uses emotion-based response templates to generate appropriate response sentences, which are then converted into speech using a Text-to-Speech service.

[0324] Output: Response audio data file

[0325] Step 8:

[0326] Data storage and provision to users

[0327] The server stores all generated data and makes it accessible to users.

[0328] Input: Secure audio data (call_001_safe.wav), text data, and text data with sentiment information.

[0329] Processing: The server stores this data in a database or cloud storage and provides users with a URL or download link to access it.

[0330] Specific operation: The server uploads the file to the cloud storage service and generates a link.

[0331] Output: URLs for user access and download links

[0332] (Application Example 2)

[0333] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0334] In today's information society, the risk of leakage of personal and confidential information contained in voice data is increasing. In particular, call centers and security services need to understand and respond to the emotional states of customers and users, but the protection of personal information during this process is crucial. Furthermore, recognizing emotions from voice data and providing appropriate responses is a challenging task. To solve these problems, a system is needed that securely processes voice data, recognizes user emotions in real time, and provides appropriate responses.

[0335] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0336] In this invention, the server includes means for acquiring audio files from data storage, means for converting the acquired audio data into text data, means for identifying and extracting personal information from the text data, means for deleting or anonymizing the identified personal information, means for resynthesizing audio data using the revised text data, means for storing and providing the resynthesized audio data, means for recognizing and analyzing the user's emotions, means for generating a response based on the analysis results, and means for providing the generated response as resynthesized audio data. This enables the secure processing of personal information within the audio data and allows for appropriate responses according to the user's emotional state.

[0337] "Data storage" refers to devices or systems used to store digital data such as audio files.

[0338] An "audio file" is a digital file used to store audio data.

[0339] "Audio data" refers to data that records human voices and sounds in digital format.

[0340] "Text data" refers to digital data obtained by converting audio data into written text.

[0341] "Personal information" refers to information that can identify a specific individual, including names and addresses.

[0342] "Anonymization" is a process that makes it impossible to identify personal information.

[0343] "Speech recognition technology" is a technology that analyzes speech data and converts it into corresponding text data.

[0344] "Speech synthesis technology" is a technology for generating speech data from text data.

[0345] "Recognizing and analyzing emotions" means identifying a speaker's emotions from audio or text data and analyzing their emotional state.

[0346] "Generating responses" means automatically generating appropriate dialogue content based on recognized emotions.

[0347] A "system" is a collection of devices and software that combine multiple means to achieve a specific function.

[0348] To implement this invention, a system program is constructed that follows the steps below. First, the server acquires audio data and converts it into text data. This process uses speech recognition technologies such as the Google Speech-to-Text API or Sphinx. Personal information is identified from the acquired text data and deleted or anonymized using regular expressions or the like. The deleted or anonymized text data is converted back into audio data using speech synthesis technologies such as the Google Text-to-Speech API and stored.

[0349] The server further uses the acquired voice and text data to recognize and analyze the user's emotions. This is done using an emotion recognition engine. For example, it identifies the user's emotional state by analyzing parameters such as voice tone, pitch, and speed. Based on this analysis, it uses a generative AI model to generate an appropriate response and provides the generated response as voice data.

[0350] The following is a natural language explanation of the processes performed by the server.

[0351] The server retrieves audio files from data storage. These audio files may include, for example, audio data from security sites. The retrieved audio data is converted into text information using the Google Speech-to-Text API. This converts the audio data into text data. Next, the server uses natural language processing techniques to identify personal information from the text data and removes or anonymizes it using regular expressions. The text data from which personal information has been removed is converted back into speech using the Google Text-to-Speech API and stored as newly generated audio data.

[0352] Furthermore, the server uses an emotion recognition engine to perform emotion analysis on the acquired audio data. It identifies the user's emotional state from the tone and pitch of their voice and assigns emotion tags to it. Based on these emotion tags, a generative AI model generates an appropriate response, which is then resynthesized as audio data.

[0353] As a concrete example, if this system were implemented in smart glasses used by security guards, the microphones in the smart glasses would collect audio from the scene and send it to a server. The server would then convert this audio data into text, remove or anonymize personal information, reconstruct it as speech, and perform emotion recognition. Finally, it would generate an appropriate response and provide it as audio, enabling security guards to respond appropriately on-site.

[0354] An example of a prompt message input to a generative AI model might be: "Convert the audio data to text and anonymize personal information. Furthermore, analyze the emotional state of the text and generate an appropriate response based on that emotion." Based on such prompt messages, the system will automatically perform the necessary processing.

[0355] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0356] Step 1:

[0357] Audio data acquisition

[0358] The server retrieves an audio file from data storage. The input includes the path to the specified audio file. This data might be collected, for example, in a security setting. The retrieved audio file is then output.

[0359] Step 2:

[0360] Text conversion of audio data

[0361] The server sends the acquired audio file to the Google Speech-to-Text API. The audio file is used as input. Speech recognition technology is used to convert the audio data into text data. This process outputs the text data.

[0362] Step 3:

[0363] Identification of personal information

[0364] The server analyzes the converted text data using natural language processing techniques. The input is text data. The text is tokenized, and personal information (e.g., name, address) is identified. The identified personal information is then output.

[0365] Step 4:

[0366] Deletion or anonymization of personal information

[0367] The server removes or anonymizes identified personal information from the text. Text data and identified personal information are used as input. Regular expressions are used to replace personal information with "" or "anonymous". This process outputs the revised text data.

[0368] Step 5:

[0369] Re-synthesis of audio data

[0370] The server sends the revised text data to the Google Text-to-Speech API, which then resynthesizes the audio data. The revised text data is included as input. Using speech synthesis technology, new audio data is generated. This audio data is then output.

[0371] Step 6:

[0372] Emotion recognition and analysis

[0373] The server inputs the acquired and converted audio data into an emotion recognition engine to analyze the user's emotions. Audio data is used as input. Parameters such as voice tone and pitch are analyzed to identify the emotional state (e.g., joy, anger). The emotional state is tagged and output.

[0374] Step 7:

[0375] Response generation

[0376] The server generates a response using a generative AI model based on the recognized emotional state. The emotional state is included as input. The generative AI model automatically generates the corresponding response according to the prompt. This response is then output.

[0377] Step 8:

[0378] Re-combining the response into audio data

[0379] The server sends the generated response to the Google Text-to-Speech API for resynthesis into audio data. The generated response is used as input. New response audio data is generated using speech synthesis technology. This audio data is the final output.

[0380] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0381] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0382] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0383] [Second Embodiment]

[0384] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0385] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0386] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0387] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0388] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0389] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0390] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0391] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0392] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0393] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0394] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0395] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0396] This invention provides a system for securely and efficiently processing data, including highly confidential and personal information, within a company. This system includes a series of processes for retrieving data files from a database, converting audio data to text data, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data. The processing of the system's program is described in detail below in natural language.

[0397] Data acquisition

[0398] server:

[0399] The server retrieves the target call data file from the company's database. For example, the server loads the "call_001.wav" file from the "customer_calls.db" database. This file contains the audio of a call with a customer stored in the database.

[0400] Converting audio data to text

[0401] server:

[0402] The server converts the acquired audio data into text data using a speech recognition API (e.g., Google Speech-to-Text API or Sphinx). The file "call_001.wav" is uploaded, and the transcription result is saved as a text file named "call_001_transcript.txt" in the API response. This text file contains a written record of the call.

[0403] Identification of Top Secret Information

[0404] AI models on the server:

[0405] An AI model, pre-trained on the server, analyzes the converted text data to identify personal information (such as name and address). The AI ​​model uses natural language processing techniques to tokenize the text, highlighting important identifying information based on specific patterns and context. For example, it identifies the name "Ichiro Tanaka" from the text "Hello, I'm Ichiro Tanaka. Today I'd like to talk about the contract details."

[0406] Extraction and deletion of top-secret information

[0407] server:

[0408] Identified personal information is removed from the text data and replaced appropriately. The server uses regular expressions to replace "Ichiro Tanaka" with "" or "anonymous". As a result, a revised text file "call_001_transcript_safe.txt" is generated, which may read something like "Hello, I'd like to talk about the contract details today."

[0409] Data reconstruction

[0410] server:

[0411] The audio data is reconstructed based on the revised text data. The server uses speech synthesis technology (for example, the Google Text-to-Speech API) to convert the revised text into audio data. This generates a new audio file called "call_001_safe.wav", which can play the corresponding portion of the original audio in a safe format. This file is saved for later use by the user.

[0412] Provision to users

[0413] User:

[0414] Users download the securely processed audio data "call_001_safe.wav" from the server and use it for business analysis or research purposes. This allows users to utilize a clean dataset that does not contain any personal information.

[0415] A concrete example is call center call data. By processing call data containing highly confidential information such as customer names and addresses using this system, relevant business analysis can be performed without the risk of personal information leakage. This significantly improves a company's data utilization capabilities and promotes the creation of business value.

[0416] The following describes the processing flow.

[0417] Step 1:

[0418] server:

[0419] The server retrieves call data files from the company's database. Specifically, it executes a query against the "database management system" to search for and retrieve the target call data (for example, "call_001.wav"). It searches using an SQL statement such as "SELECT FROM calls WHERE id='call_001'" and saves the retrieved file to a temporary working directory.

[0420] Step 2:

[0421] server:

[0422] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, the audio file is uploaded to a speech recognition service such as the Google Speech-to-Text API or Sphinx, and the audio data is converted into text data. The resulting transcript is saved as "call_001_transcript.txt".

[0423] Step 3:

[0424] AI models on the server:

[0425] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Specifically, it uses natural language processing techniques to tokenize the text and identify specific words and phrases (for example, names and addresses). This process reveals important identifying information such as the name "Ichiro Tanaka" and the address "Shinjuku-ku, Tokyo".

[0426] Step 4:

[0427] server:

[0428] Delete or replace identified personal information. Specifically, use regular expressions to replace personal information in the text with safe strings such as "" or "anonymous". For example, change "Hello, I'm Tanaka Ichiro." to "Hello, desu.". This will generate the revised text data "call_001_transcript_safe.txt".

[0429] Step 5:

[0430] server:

[0431] New audio data is generated based on the revised text data. Specifically, speech synthesis technology (e.g., Google Text-to-Speech API) is used to convert "call_001_transcript_safe.txt" into audio data. This generates a new audio file, "call_001_safe.wav," from which personal information has been removed. The generated audio file is saved in a temporary working directory.

[0432] Step 6:

[0433] server:

[0434] The generated secure audio file "call_001_safe.wav" will be saved and made available to the user. Specifically, the file will be moved to a designated folder, and a downloadable URL will be generated for the user. The user will then be able to use this URL to access the secure audio data for business analysis or research purposes.

[0435] (Example 1)

[0436] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0437] In processing data containing highly confidential and personal information within companies, there is a need to balance security and efficiency. Traditional methods often involve manual data processing, which is time-consuming and labor-intensive, and carries a high risk of information leaks due to human error. Furthermore, there has been a lack of methods to adequately protect personal information when handling audio data. Therefore, there is a need for technology that can automatically extract personal information from audio data, anonymize it, and make it reusable.

[0438] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0439] In this invention, the server includes means for obtaining data files from a database, means for converting the obtained audio data into text data using speech recognition technology, means including a generative artificial intelligence model for analyzing the converted text data and identifying and extracting personal information, means for deleting or replacing the identified personal information, means for reconstructing the audio data using speech synthesis technology with the revised text data, and means for storing and providing the reconstructed audio data. This enables the secure and efficient processing of data, including highly confidential information and personal information.

[0440] A "data file" is a file stored in a company or organization's database that contains information collected for a specific business or purpose.

[0441] "Audio data" refers to data that records human voices in digital format.

[0442] "Speech recognition technology" is a technology that converts speech data into text data.

[0443] "Text data" refers to data that records information written using characters and symbols in a digital format.

[0444] "Personal information" refers to information that can identify a specific individual, such as name, address, and telephone number.

[0445] A "generative artificial intelligence model" is an AI model that learns from large amounts of text data and analyzes the content of that text using natural language processing techniques.

[0446] "Deletion" refers to the operation of completely removing specific information from data.

[0447] "Replacement" refers to the operation of changing identified information into a different string or data.

[0448] "Speech synthesis technology" is a technology that artificially generates speech from text data.

[0449] "Reconstruction" refers to the process of creating new audio data based on revised text data.

[0450] "Saving" refers to the process of recording generated data on a digital medium so that it can be accessed later.

[0451] "Providing" refers to the operation of making generated data available for use by users.

[0452] This invention provides a system for securely and efficiently processing data, including highly confidential and personal information, within a company. This system includes a series of processes for acquiring audio data from a database, converting that audio data into text data using speech recognition technology, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data.

[0453] The server accesses the company's database and retrieves the target call data file. This operation uses database queries. For example, it executes the SQL query "SELECT FROM customer_calls WHERE call_id = '001'" to load the target call audio file "call_001.wav". This file contains the audio of the call with the customer.

[0454] The server converts the acquired audio data into text data using a speech recognition API. Specifically, it uses the Google Speech-to-Text API or Sphinx. The audio file "call_001.wav" is uploaded to the API, and the transcript result obtained as a response from the API is saved to "call_001_transcript.txt". This text file contains a record of the call content in text.

[0455] Next, an AI model on the server analyzes the converted text data to identify personal information (e.g., name and address). This AI model includes a pre-trained generative AI model and uses natural language processing techniques to tokenize the text, highlighting important identifying information based on specific patterns and contexts. For example, it identifies the name "Ichiro Tanaka" from the text "Hello, I'm Ichiro Tanaka. Today I'd like to talk about the contract details."

[0456] The server removes and appropriately replaces identified personal information from the text data. Specifically, it uses regular expressions to search for specific patterns and replaces corresponding parts with "" or "anonymous". As a result, it generates a revised text file "call_001_transcript_safe.txt" that reads something like "Hello, I'd like to talk about the contract details today."

[0457] The server then generates new audio data based on the revised text data. Specifically, it uses the Google Text-to-Speech API to generate audio data from the revised text. This generates a new audio file called "call_001_safe.wav" and plays the corresponding portion of the original audio in a secure format. This file is saved so that the user can use it later.

[0458] Users download the securely processed audio data "call_001_safe.wav" from the server and use it for business analysis and research purposes. This allows users to utilize a clean dataset that does not contain any personal information.

[0459] A concrete example is call center data. By processing call data containing highly confidential information such as customer names and addresses using this system, relevant business analysis can be performed without the risk of personal information leakage. This significantly improves a company's data utilization capabilities and promotes the creation of business value.

[0460] An example of a prompt for a generative AI model can be written as follows:

[0461] "Please extract the customer's name and address from this audio data, anonymize it, and create a reconstructed audio file."

[0462] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0463] Step 1:

[0464] The server accesses the company's database and retrieves the target call data file. Specifically, the server executes the SQL query "SELECT FROM customer_calls WHERE call_id = '001'" and loads the call audio file "call_001.wav" stored in the database. The input for this step is the database query, and the output is the retrieved audio file.

[0465] Step 2:

[0466] The server converts the acquired audio data into text data using a speech recognition API. Specifically, the server uploads the audio file "call_001.wav" to the Google Speech-to-Text API and receives the transcription result as a response from the API. This transcription result is saved to "call_001_transcript.txt". The input for this step is an audio file, and the output is a text file.

[0467] Step 3:

[0468] An AI model on the server analyzes the converted text data and identifies personal information. Specifically, the server inputs "call_001_transcript.txt" to the AI ​​model, and the model uses natural language processing techniques to extract personal information (e.g., name and address) from the text. The input for this step is a text file, and the output is a list of identified personal information.

[0469] Step 4:

[0470] The server removes identified personal information from the text data and replaces it appropriately. Specifically, the server uses regular expressions to search for specific patterns within "call_001_transcript.txt" and replaces the corresponding parts with "" or "anonymous". As a result of this process, a revised text file called "call_001_transcript_safe.txt" is generated. The input for this step is a list of identified personal information and a text file, and the output is a revised text file.

[0471] Step 5:

[0472] The server generates new audio data based on the revised text data. Specifically, the server uses the Google Text-to-Speech API to convert "call_001_transcript_safe.txt" into audio data and generates a new audio file "call_001_safe.wav". The input for this step is the revised text file, and the output is the new audio file.

[0473] Step 6:

[0474] The user downloads securely processed audio data from the server. Specifically, the user downloads "call_001_safe.wav" via the server's web interface or API and plays and analyzes it on their local device. The input for this step is a new audio file, and the output is the audio file saved on the user's local device.

[0475] (Application Example 1)

[0476] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0477] In recent years, with the increasing use of digital data, the secure processing of personal and confidential information, including voice data, has become a critical issue. In particular, real-time voice data processing requires the rapid and accurate detection of personal information and its appropriate processing while preventing data leaks. However, conventional systems have lacked sufficient means to protect personal information while processing voice data in real time, making it difficult to ensure user privacy.

[0478] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0479] In this invention, the server includes means for acquiring data files from data storage, means for converting acquired audio data into text data, means for identifying and extracting personal and confidential information from the text data, means for deleting or replacing the identified information, means for reconstructing audio data using the revised text data, means for storing and providing the reconstructed audio data, means for acquiring user audio in real time and detecting and replacing personal information, and means for generating and providing clean audio data in real time. This makes it possible to securely process and provide audio data containing personal and confidential information in real time.

[0480] "Data storage" refers to a storage device used to store and manage data.

[0481] A "data file" is a collection of information managed by a computer, and specifically includes audio data and text data.

[0482] "Audio data" refers to data obtained by digitizing sound signals, which is usually analyzed using speech recognition technology.

[0483] "Text data" is a collection of information represented by characters and symbols.

[0484] "Personal information" refers to information that can identify a specific individual, including names and addresses.

[0485] "Confidential information" refers to important information whose disclosure should be restricted, and includes internal company documents and contract details.

[0486] "Identification" is the process of distinguishing and verifying specific information.

[0487] "Extraction" is the process of extracting specific information from a large amount of data.

[0488] "Deletion" is the process of erasing existing data.

[0489] "Replacement" is the process of substituting original data with other data.

[0490] "Revised text data" refers to text data after personal or confidential information has been deleted or replaced.

[0491] "Reconstructing audio" is the process of generating new audio data based on text data.

[0492] "Real-time" refers to a state in which data is acquired, processed, and provided immediately.

[0493] "Clean audio data" refers to secure audio data from which personal and confidential information has been removed.

[0494] This invention provides a system for safely and efficiently processing voice data in a business. The system for carrying out this invention includes the following hardware and software configuration.

[0495] Hardware and software to be used

[0496] Data storage: A storage device that stores and manages data files. Examples include internal servers and cloud servers within a company.

[0497] Speech recognition technology: Technology for converting speech data into text data. Specific examples include "Google Speech-to-Text API" and "Sphinx".

[0498] AI Model: A model pre-trained to identify and extract personal and confidential information. Examples include implementations using Python's natural language processing libraries (such as NLTK and SpaCy).

[0499] Speech synthesis technology: A technology used to reconstruct text data into speech data. For example, the "Google Text-to-Speech API" is used.

[0500] Smart gadgets: Devices for processing audio data in real time during operation. Specifically, this includes smart glasses and head-mounted displays.

[0501] System Operation Overview

[0502] 1. Obtaining data files

[0503] The server retrieves an audio data file from data storage. For example, it loads the "call_001.wav" file from "customer_calls.db". This is an audio recording of a customer call stored in the company's database.

[0504] 2. Converting audio data to text data

[0505] The server converts the acquired audio data into text data using speech recognition technology. Specifically, it uploads "call_001.wav" to the API and saves the resulting transcription to a text file called "call_001_transcript.txt".

[0506] 3. Identification and extraction of personal information

[0507] Using an AI model pre-trained on the server, the converted text data is analyzed to identify personal and confidential information. Based on natural language processing technology, the text is tokenized and important identifying information is extracted.

[0508] 4. Deletion or replacement of information

[0509] Identified personal information is removed from the text data and replaced appropriately. The server uses regular expressions or similar methods to convert the data to "" or "anonymous". For example, from the text "Hello, I'm Tanaka Ichiro.", "Tanaka Ichiro" is replaced with "", generating "Hello, desu.".

[0510] 5. Reconstruct the audio data based on the revised text data.

[0511] The server uses speech synthesis technology to convert the revised text data into audio data. For example, it generates a new audio file named "call_001_safe.wav".

[0512] 6. Generate and provide clean audio data in real time.

[0513] This system acquires user voices in real time, uses an AI model to instantly detect and replace personal information, and generates clean voice data. This allows users to use secure voice data in real time while minimizing the risk of personal information leakage.

[0514] Examples of specific cases and prompt statements

[0515] Specific example: If a user says "Hello, I'm Tanaka Ichiro" during a meeting, the smart glasses instantly process the audio data and display "Hello, I am." on the screen.

[0516] Prompt example: Find and replace the name and address in this spoken text. Input: 'Hello, I'm Ichiro Tanaka. What time does the meeting start?'

[0517] This invention makes it possible to securely process and provide audio data containing personal and confidential information in real time, significantly improving security in corporate data operations.

[0518] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0519] Step 1:

[0520] Retrieving data files

[0521] The server retrieves the necessary audio data files from data storage. In this process, the server accesses the database and loads a specific audio file (e.g., "call_001.wav").

[0522] Input: Path to the audio data file in data storage

[0523] Output: Acquired audio data file

[0524] Specific action: The server loads "call_001.wav" from "customer_calls.db".

[0525] Step 2:

[0526] Converting audio data to text data

[0527] The server converts the acquired audio data into text data using speech recognition technology. For example, it can use the "Google Speech-to-Text API" to input audio data and generate a transcript.

[0528] Input: Audio data file (e.g., "call_001.wav")

[0529] Output: Text data file corresponding to the audio data (e.g., "call_001_transcript.txt")

[0530] Specific operation: Upload "call_001.wav" to the API and save the resulting transcription to "call_001_transcript.txt".

[0531] Step 3:

[0532] Identification and extraction of personal and confidential information

[0533] Using an AI model pre-trained on the server, the converted text data is analyzed to identify and extract personal and confidential information.

[0534] Input: Converted text data (e.g., "call_001_transcript.txt")

[0535] Output: List of identified personal and confidential information

[0536] Specific operation: Using an AI model (e.g., a Python natural language processing library), analyze specific patterns and contexts within text to identify information such as "Ichiro Tanaka".

[0537] Step 4:

[0538] Deletion or replacement of information

[0539] The server removes identified personal information from the text data and replaces it appropriately. For example, it uses regular expressions to convert "Ichiro Tanaka" to "".

[0540] Input: A list of identified personal and confidential information, a text data file (e.g., "call_001_transcript.txt")

[0541] Output: Revised text data (e.g., "call_001_transcript_safe.txt")

[0542] Specific action: Replace "Ichiro Tanaka" with quotation marks and generate revised text such as "Hello, desu."

[0543] Step 5:

[0544] Reconstruct audio data based on revised text data.

[0545] The server uses speech synthesis technology to convert the revised text data into speech data. For example, it generates a new audio file using the Google Text-to-Speech API.

[0546] Input: Revised text data file (e.g., "call_001_transcript_safe.txt")

[0547] Output: Reconstructed audio data file (e.g., "call_001_safe.wav")

[0548] Specific action: Save "call_001_transcript_safe.txt" as "call_001_safe.wav" using speech synthesis technology.

[0549] Step 6:

[0550] Generate and provide clean audio data in real time.

[0551] This system acquires user voices in real time, instantly detects and replaces personal information, and generates clean voice data.

[0552] Input: Audio data acquired in real time

[0553] Output: Clean audio data processed immediately.

[0554] Specific operation: Using an AI model, personal information is instantly identified, clean audio data is generated using speech synthesis technology, and displayed on the user's smart glasses.

[0555] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0556] This invention provides a system for securely and efficiently processing audio data containing highly confidential and personal information, and for recognizing and analyzing user emotions. This system includes specific processing steps such as retrieving data files from a database, converting audio data into text data, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data, as well as a process for recognizing user emotions using an emotion engine. The program processing of this system is described in detail below.

[0557] Data acquisition

[0558] server:

[0559] The server retrieves the target call data file from the company's database. For example, the server loads the "call_001.wav" file from the "database management system." This file contains the audio of a call with a customer.

[0560] Converting audio data to text

[0561] server:

[0562] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, a speech recognition service such as Google Speech-to-Text API or Sphinx is used to convert the audio data into text data. The resulting transcript is saved as "call_001_transcript.txt".

[0563] Identification of Top Secret Information

[0564] AI models on the server:

[0565] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Using natural language processing techniques, the text is tokenized to identify information such as names and addresses. Through this process, important identifying information such as "Ichiro Sato" and "Shibuya Ward, Tokyo" is extracted from the text "Hello, I'm Ichiro Sato. Today I'd like to talk about Shibuya Ward, Tokyo."

[0566] Extraction and deletion of personal information

[0567] server:

[0568] Identified personal information is removed or replaced from the text data. Specifically, regular expressions are used to replace personal information in the text with safe strings such as "" or "anonymous". As a result, a revised text file "call_001_transcript_safe.txt" is generated, which may contain text like "Hello, I'd like to talk about today."

[0569] Reconstructing audio data

[0570] server:

[0571] New audio data is generated based on the revised text data. Speech synthesis technology (e.g., Google Text-to-Speech API) is used to convert "call_001_transcript_safe.txt" into audio data. This generates a new audio file, "call_001_safe.wav," from which personal information has been removed.

[0572] Emotional analysis using an emotional engine

[0573] server:

[0574] The emotion engine on the server analyzes the acquired audio data "call_001.wav" to recognize the user's emotions. The emotion engine analyzes parameters such as voice tone, pitch, and speed to identify the user's emotions. For example, the server identifies the user's emotions as emotion tags such as "joy," "anger," and "sadness," and adds this information to the text data.

[0575] Addition of emotional information and response generation

[0576] server:

[0577] The emotion engine identifies emotional information and adds it to text data, automatically generating responses based on specific emotions. For example, if the user is angry, it might generate a response such as, "You seem angry. Let me explain calmly." This response is then reconstructed as corresponding audio data and provided to the user.

[0578] Data storage and provision to users

[0579] server:

[0580] The generated secure audio file "call_001_safe.wav" and text data with added emotional information are saved and made accessible to the user. Users can then download the data from the server and use it for business analysis or research purposes.

[0581] As a concrete example, consider customer call data from a call center. By processing call data containing highly confidential information such as customer names and addresses with this system, it becomes possible to perform sentiment analysis of call content while eliminating the risk of customer information leakage. This significantly improves the quality of customer service and operational efficiency.

[0582] The following describes the processing flow.

[0583] Step 1:

[0584] server:

[0585] The server retrieves the target call data file from the company's database. Specifically, the server uses a database query to search for and retrieve a call file such as "call_001.wav". For example, it executes the SQL command "SELECT FROM calls WHERE id='call_001'" and saves the result to a temporary working directory.

[0586] Step 2:

[0587] server:

[0588] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, a speech recognition service such as Google Speech-to-Text API or Sphinx is used to convert the audio data into text data. The audio file is uploaded using the API, and the converted text data is saved as "call_001_transcript.txt".

[0589] Step 3:

[0590] AI models on the server:

[0591] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Specifically, it uses natural language processing technology to tokenize the text and extract personal information such as names and addresses. Through this process, for example, "Hello, I'm Sato Ichiro. Today I'd like to talk about Shibuya Ward, Tokyo." can be interpreted as "Sato Ichiro" and "Shibuya Ward, Tokyo."

[0592] Step 4:

[0593] server:

[0594] Remove or replace identified personal information from the text data. Specifically, use regular expressions to replace personal information in the text with safe strings such as "" or "anonymous". For example, change "Hello, I'm Sato Ichiro." to "Hello, desu.". Generate the revised text file "call_001_transcript_safe.txt".

[0595] Step 5:

[0596] server:

[0597] New audio data will be generated based on the revised text data. Specifically, speech synthesis technology will be used to convert "call_001_transcript_safe.txt" into audio data. The revised text will be uploaded using the Google Text-to-Speech API or similar, the synthesized audio data "call_001_safe.wav" will be generated and saved to a temporary working directory.

[0598] Step 6:

[0599] server:

[0600] The emotion engine on the server analyzes the original audio data "call_001.wav" to recognize the user's emotions. Specifically, it analyzes parameters such as tone, pitch, and speed of the audio data to identify the user's emotions (e.g., joy, anger, sadness) and adds that information to the text data.

[0601] Step 7:

[0602] server:

[0603] Based on the emotion information identified by the emotion engine, emotion tags are added to the revised text data. Specifically, if the user is angry, a response such as "You seem angry. Let me explain calmly" is generated and added to the revised text. In some cases, audio data may be generated again based on this revised text data.

[0604] Step 8:

[0605] server:

[0606] The generated secure audio data "call_001_safe.wav" and revised text data will be saved and made available to users for access. Specifically, the server will place the files in a designated folder and generate a URL that users can download. Users will use this URL to access the secure data and use it for business analysis or research purposes.

[0607] This specific process not only automatically removes confidential information from call data, but also provides a system that recognizes user emotions and enables appropriate responses and analysis. This allows companies to significantly improve the quality of customer service and operational efficiency.

[0608] (Example 2)

[0609] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0610] Conventional voice data processing systems had security risks and efficiency issues when extracting and deleting personal information from voice data. Furthermore, they lacked the ability to identify user emotions and generate appropriate responses, in addition to simply deleting personal information. As a result, improvements in customer service quality and operational efficiency were not fully achieved.

[0611] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0612] In this invention, the server includes means for obtaining data files from a database, means for converting the obtained voice data into text data, means for identifying and extracting personal information from the text data, means for deleting or replacing the identified personal information, means for reconstructing voice data using the revised text data, means for storing and providing the reconstructed voice data, means for using an emotion engine for sentiment analysis of the voice data, and means for identifying the user's emotions and generating a response based on those emotions. This makes it possible to analyze the emotions of call content while eliminating the risk of leakage of personal information. Furthermore, it enables significant improvements in the quality of customer service and operational efficiency.

[0613] A "data file" refers to information stored in digital format, and includes various types of data such as audio, text, and images.

[0614] "Audio data" refers to digitized audio information, including various types of audio such as phone calls, recordings, and lectures.

[0615] "Text data" refers to data that stores characters and symbols in a digital format, in a format that is easily analyzable by software and systems.

[0616] "Personal information" refers to information used to identify a specific individual, and includes names, addresses, telephone numbers, and email addresses.

[0617] "Identification" means distinguishing specific information or data from other information or data.

[0618] "Extraction" refers to the process of extracting necessary information or features from a large amount of data.

[0619] "Deletion" means completely removing unnecessary or risky information from the data.

[0620] "Substitution" refers to replacing specific information with other information.

[0621] "Revised text data" refers to text data that has been modified by removing or replacing personal information from the original text data.

[0622] "Speech synthesis technology" is a technology that generates speech by imitating human voices based on text data.

[0623] An "emotion engine" is a general term for systems that analyze and identify emotions from voice and text data.

[0624] "Sentiment analysis" refers to the process of reading emotions from audio or text and identifying specific emotional states.

[0625] "Response" refers to the system's reply or reaction to user input or statements.

[0626] This invention provides a system for securely and efficiently processing voice data containing highly confidential and personal information, and further for recognizing and analyzing user emotions. This system includes the following main processing steps:

[0627] First, the server retrieves a data file from the database. Specifically, it uses a database management system to access the database and retrieve a designated audio data file (e.g., call_001.wav). Next, the server sends the retrieved audio data to a speech recognition API (e.g., Google Speech-to-Text API or Sphinx) and converts the retrieved audio data into text data. The resulting transcript is saved as a file (e.g., call_001_transcript.txt).

[0628] Subsequently, a pre-trained AI model on the server analyzes the text data and identifies personal information. Natural language processing (NLP) techniques are used to tokenize the text, and then identifying information such as names and addresses is extracted. The identified personal information is either deleted or replaced with a secure string by the server, thereby generating a revised text data file (e.g., call_001_transcript_safe.txt).

[0629] Furthermore, the server uses speech synthesis technology such as the Google Text-to-Speech API to generate new audio data based on this revised text data. This results in a new audio file (e.g., call_001_safe.wav) from which personal information has been removed.

[0630] Furthermore, an emotion engine within the server analyzes the voice data and recognizes the user's emotions. The emotion engine analyzes parameters such as voice tone, pitch, and speed, and identifies the user's emotions as emotion tags such as "joy," "anger," and "sadness." This information is added to the text data, and a response based on the specific emotion is automatically generated. The response is then reconstructed as voice data using speech synthesis technology and provided to the user.

[0631] For example, when processing customer call data in a call center, this system allows for the secure removal of highly confidential information such as customer names and addresses, while also enabling sentiment analysis of the call content. This is expected to improve the quality of customer service and enhance operational efficiency.

[0632] Examples of prompt statements include the following:

[0633] Please describe a system that processes customer call data, removes specific personal information, analyzes user sentiment, and generates appropriate responses. Also, please provide specific application scenarios in a call center.

[0634] The above describes a specific implementation of a system that can perform sentiment analysis of call content while eliminating the risk of personal information leakage.

[0635] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0636] Step 1:

[0637] Data acquisition

[0638] The server retrieves data files from the database.

[0639] Input: Database connection information and the ID of the audio data file to be retrieved (e.g., 001)

[0640] Processing: The server executes a query on the database management system to retrieve the specified audio data file (call_001.wav). Specifically, it executes the query "SELECT FROM calls WHERE id=001;".

[0641] Output: Acquired audio data file (call_001.wav)

[0642] Step 2:

[0643] Converting audio data to text

[0644] The server sends the acquired audio data file to the speech recognition API.

[0645] Input: Audio data file (call_001.wav)

[0646] Processing: The server sends the audio data to the Google Speech-to-Text API endpoint and converts it to text data. The text data returned from the API is saved as "call_001_transcript.txt".

[0647] Specific operation: The server sends the audio file to the speech recognition API as a POST request and receives the conversion result in JSON format. It then saves this as a text file.

[0648] Output: Transcript text file (call_001_transcript.txt)

[0649] Step 3:

[0650] Identification of Top Secret Information

[0651] An AI model on the server analyzes text data and identifies personal information.

[0652] Input: Transcript text (call_001_transcript.txt)

[0653] Processing: Text is tokenized using natural language processing (NLP) techniques, and personal information such as names and addresses is extracted.

[0654] Specific operation: The AI ​​model analyzes each word and phrase in the text to identify personal information such as "Ichiro Sato" and "Shibuya Ward, Tokyo."

[0655] Output: List of personal information (e.g., "Ichiro Sato", "Shibuya Ward, Tokyo")

[0656] Step 4:

[0657] Extraction and deletion of personal information

[0658] The server deletes or replaces identified personal information.

[0659] Input: Transcript text (call_001_transcript.txt) and a list of personal information

[0660] Processing: Regular expressions are used to replace identified personal information with "" or "anonymous". This generates the revised text data (call_001_transcript_safe.txt).

[0661] Specific operation: The server uses a list of identified personal information to replace the relevant parts of the text with regular expressions.

[0662] Output: Revised text data file (call_001_transcript_safe.txt)

[0663] Step 5:

[0664] Reconstructing audio data

[0665] The server generates new audio data based on the revised text data.

[0666] Input: Revised text data file (call_001_transcript_safe.txt)

[0667] Process: Perform the steps to convert text data into speech data using the Google Text-to-Speech API.

[0668] Specific operation: The server sends text to the Google Text-to-Speech API endpoint and saves the returned audio data as a file. Example: "call_001_safe.wav"

[0669] Output: New audio data file (call_001_safe.wav)

[0670] Step 6:

[0671] Emotional analysis using an emotional engine

[0672] The emotion engine on the server analyzes the voice data and recognizes the user's emotions.

[0673] Input: Original audio data file (call_001.wav)

[0674] Processing: The emotion engine analyzes parameters such as voice tone, pitch, and speed to identify the user's emotions.

[0675] Specific operation: The emotion engine breaks down the audio file into segments, calculates the feature quantities of each segment, and assigns emotion tags to them.

[0676] Output: Text data with emotion information (e.g., emotion tags such as "joy", "anger", and "sadness")

[0677] Step 7:

[0678] Addition of emotional information and response generation

[0679] The server adds emotional information to text data and generates responses based on specific emotions.

[0680] Input: Text data with sentiment information and revised text data

[0681] Processing: Based on specific emotions, responses are automatically generated and then reconstructed using speech synthesis technology.

[0682] Specific operation: The server uses emotion-based response templates to generate appropriate response sentences, which are then converted into speech using a Text-to-Speech service.

[0683] Output: Response audio data file

[0684] Step 8:

[0685] Data storage and provision to users

[0686] The server stores all generated data and makes it accessible to users.

[0687] Input: Secure audio data (call_001_safe.wav), text data, and text data with sentiment information.

[0688] Processing: The server stores this data in a database or cloud storage and provides users with a URL or download link to access it.

[0689] Specific operation: The server uploads the file to the cloud storage service and generates a link.

[0690] Output: URLs for user access and download links

[0691] (Application Example 2)

[0692] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0693] In today's information society, the risk of leakage of personal and confidential information contained in voice data is increasing. In particular, call centers and security services need to understand and respond to the emotional states of customers and users, but the protection of personal information during this process is crucial. Furthermore, recognizing emotions from voice data and providing appropriate responses is a challenging task. To solve these problems, a system is needed that securely processes voice data, recognizes user emotions in real time, and provides appropriate responses.

[0694] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0695] In this invention, the server includes means for acquiring audio files from data storage, means for converting the acquired audio data into text data, means for identifying and extracting personal information from the text data, means for deleting or anonymizing the identified personal information, means for resynthesizing audio data using the revised text data, means for storing and providing the resynthesized audio data, means for recognizing and analyzing the user's emotions, means for generating a response based on the analysis results, and means for providing the generated response as resynthesized audio data. This enables the secure processing of personal information within the audio data and allows for appropriate responses according to the user's emotional state.

[0696] "Data storage" refers to devices or systems used to store digital data such as audio files.

[0697] An "audio file" is a digital file used to store audio data.

[0698] "Audio data" refers to data that records human voices and sounds in digital format.

[0699] "Text data" refers to digital data obtained by converting audio data into written text.

[0700] "Personal information" refers to information that can identify a specific individual, including names and addresses.

[0701] "Anonymization" is a process that makes it impossible to identify personal information.

[0702] "Speech recognition technology" is a technology that analyzes speech data and converts it into corresponding text data.

[0703] "Speech synthesis technology" is a technology for generating speech data based on text data.

[0704] "Recognizing and analyzing emotions" means identifying a speaker's emotions from audio or text data and analyzing their emotional state.

[0705] "Generating responses" means automatically generating appropriate dialogue content based on recognized emotions.

[0706] A "system" is a collection of devices and software that combine multiple means to achieve a specific function.

[0707] To implement this invention, a system program is constructed that follows the steps below. First, the server acquires audio data and converts it into text data. This process uses speech recognition technologies such as the Google Speech-to-Text API or Sphinx. Personal information is identified from the acquired text data and deleted or anonymized using regular expressions or the like. The deleted or anonymized text data is converted back into audio data using speech synthesis technologies such as the Google Text-to-Speech API and stored.

[0708] The server further uses the acquired voice and text data to recognize and analyze the user's emotions. This is done using an emotion recognition engine. For example, it identifies the user's emotional state by analyzing parameters such as voice tone, pitch, and speed. Based on this analysis, it uses a generative AI model to generate an appropriate response and provides the generated response as voice data.

[0709] The following is a natural language explanation of the processes performed by the server.

[0710] The server retrieves audio files from data storage. These audio files may include, for example, audio data from security sites. The retrieved audio data is converted into text information using the Google Speech-to-Text API. This converts the audio data into text data. Next, the server uses natural language processing techniques to identify personal information from the text data and removes or anonymizes it using regular expressions. The text data from which personal information has been removed is converted back into speech using the Google Text-to-Speech API and stored as newly generated audio data.

[0711] Furthermore, the server uses an emotion recognition engine to perform emotion analysis on the acquired audio data. It identifies the user's emotional state from the tone and pitch of their voice and assigns emotion tags to it. Based on these emotion tags, a generative AI model generates an appropriate response, which is then resynthesized as audio data.

[0712] As a concrete example, if this system were implemented in smart glasses used by security guards, the microphones in the smart glasses would collect audio from the scene and send it to a server. The server would then convert this audio data into text, remove or anonymize personal information, reconstruct it as speech, and perform emotion recognition. Finally, it would generate an appropriate response and provide it as audio, enabling security guards to respond appropriately on-site.

[0713] An example of a prompt message input to a generative AI model might be: "Convert the audio data to text and anonymize personal information. Furthermore, analyze the emotional state of the text and generate an appropriate response based on that emotion." Based on such prompt messages, the system will automatically perform the necessary processing.

[0714] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0715] Step 1:

[0716] Acquisition of audio data

[0717] The server retrieves an audio file from data storage. The input includes the path to the specified audio file. This data might be collected, for example, in a security setting. The retrieved audio file is then output.

[0718] Step 2:

[0719] Converting audio data to text

[0720] The server sends the acquired audio file to the Google Speech-to-Text API. The audio file is used as input. Speech recognition technology is used to convert the audio data into text data. This process outputs the text data.

[0721] Step 3:

[0722] Identification of personal information

[0723] The server analyzes the converted text data using natural language processing techniques. The input is text data. The text is tokenized, and personal information (e.g., name, address) is identified. The identified personal information is then output.

[0724] Step 4:

[0725] Deletion or anonymization of personal information

[0726] The server removes or anonymizes identified personal information from the text. Text data and identified personal information are used as input. Regular expressions are used to replace personal information with "" or "anonymous". This process outputs the revised text data.

[0727] Step 5:

[0728] Re-synthesis of audio data

[0729] The server sends the revised text data to the Google Text-to-Speech API, which then resynthesizes the audio data. The revised text data is included as input. Using speech synthesis technology, new audio data is generated. This audio data is then output.

[0730] Step 6:

[0731] Emotion recognition and analysis

[0732] The server inputs the acquired and converted audio data into an emotion recognition engine to analyze the user's emotions. Audio data is used as input. Parameters such as voice tone and pitch are analyzed to identify the emotional state (e.g., joy, anger). The emotional state is tagged and output.

[0733] Step 7:

[0734] Response generation

[0735] The server generates a response using a generative AI model based on the recognized emotional state. The emotional state is included as input. The generative AI model automatically generates the corresponding response according to the prompt. This response is then output.

[0736] Step 8:

[0737] Re-combining the response into audio data

[0738] The server sends the generated response to the Google Text-to-Speech API for resynthesis into audio data. The generated response is used as input. New response audio data is generated using speech synthesis technology. This audio data is the final output.

[0739] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0740] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0741] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0742] [Third Embodiment]

[0743] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0744] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0745] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0746] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0747] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0748] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0749] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0750] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0751] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0752] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0753] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0754] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0755] This invention provides a system for securely and efficiently processing data, including highly confidential and personal information, within a company. This system includes a series of processes for retrieving data files from a database, converting audio data to text data, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data. The processing of the system's program is described in detail below in natural language.

[0756] Data acquisition

[0757] server:

[0758] The server retrieves the target call data file from the company's database. For example, the server loads the "call_001.wav" file from the "customer_calls.db" database. This file contains the audio of a call with a customer stored in the database.

[0759] Converting audio data to text

[0760] server:

[0761] The server converts the acquired audio data into text data using a speech recognition API (e.g., Google Speech-to-Text API or Sphinx). The file "call_001.wav" is uploaded, and the transcription result is saved as a text file named "call_001_transcript.txt" in the API response. This text file contains a written record of the call.

[0762] Identification of Top Secret Information

[0763] AI models on the server:

[0764] An AI model, pre-trained on the server, analyzes the converted text data to identify personal information (such as name and address). The AI ​​model uses natural language processing techniques to tokenize the text, highlighting important identifying information based on specific patterns and context. For example, it identifies the name "Ichiro Tanaka" from the text "Hello, I'm Ichiro Tanaka. Today I'd like to talk about the contract details."

[0765] Extraction and deletion of top-secret information

[0766] server:

[0767] Identified personal information is removed from the text data and replaced appropriately. The server uses regular expressions to replace "Ichiro Tanaka" with "" or "anonymous". As a result, a revised text file "call_001_transcript_safe.txt" is generated, which may read something like "Hello, I'd like to talk about the contract details today."

[0768] Data reconstruction

[0769] server:

[0770] The audio data is reconstructed based on the revised text data. The server uses speech synthesis technology (for example, the Google Text-to-Speech API) to convert the revised text into audio data. This generates a new audio file called "call_001_safe.wav", which can play the corresponding portion of the original audio in a safe format. This file is saved for later use by the user.

[0771] Provision to users

[0772] User:

[0773] Users download the securely processed audio data "call_001_safe.wav" from the server and use it for business analysis or research purposes. This allows users to utilize a clean dataset that does not contain any personal information.

[0774] A concrete example is call center call data. By processing call data containing highly confidential information such as customer names and addresses using this system, relevant business analysis can be performed without the risk of personal information leakage. This significantly improves a company's data utilization capabilities and promotes the creation of business value.

[0775] The following describes the processing flow.

[0776] Step 1:

[0777] server:

[0778] The server retrieves call data files from the company's database. Specifically, it executes a query against the "database management system" to search for and retrieve the target call data (for example, "call_001.wav"). It searches using an SQL statement such as "SELECT FROM calls WHERE id='call_001'" and saves the retrieved file to a temporary working directory.

[0779] Step 2:

[0780] server:

[0781] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, the audio file is uploaded to a speech recognition service such as the Google Speech-to-Text API or Sphinx, and the audio data is converted into text data. The resulting transcript is saved as "call_001_transcript.txt".

[0782] Step 3:

[0783] AI models on the server:

[0784] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Specifically, it uses natural language processing techniques to tokenize the text and identify specific words and phrases (for example, names and addresses). This process reveals important identifying information such as the name "Ichiro Tanaka" and the address "Shinjuku-ku, Tokyo".

[0785] Step 4:

[0786] server:

[0787] Delete or replace identified personal information. Specifically, use regular expressions to replace personal information in the text with safe strings such as "" or "anonymous". For example, change "Hello, I'm Tanaka Ichiro." to "Hello, desu.". This will generate the revised text data "call_001_transcript_safe.txt".

[0788] Step 5:

[0789] server:

[0790] New audio data is generated based on the revised text data. Specifically, speech synthesis technology (e.g., Google Text-to-Speech API) is used to convert "call_001_transcript_safe.txt" into audio data. This generates a new audio file, "call_001_safe.wav," from which personal information has been removed. The generated audio file is saved in a temporary working directory.

[0791] Step 6:

[0792] server:

[0793] The generated secure audio file "call_001_safe.wav" will be saved and made available to the user. Specifically, the file will be moved to a designated folder, and a downloadable URL will be generated for the user. The user will then be able to use this URL to access the secure audio data for business analysis or research purposes.

[0794] (Example 1)

[0795] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0796] In processing data containing highly confidential and personal information within companies, there is a need to balance security and efficiency. Traditional methods often involve manual data processing, which is time-consuming and labor-intensive, and carries a high risk of information leaks due to human error. Furthermore, there has been a lack of methods to adequately protect personal information when handling audio data. Therefore, there is a need for technology that can automatically extract personal information from audio data, anonymize it, and make it reusable.

[0797] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0798] In this invention, the server includes means for obtaining data files from a database, means for converting the obtained audio data into text data using speech recognition technology, means including a generative artificial intelligence model for analyzing the converted text data and identifying and extracting personal information, means for deleting or replacing the identified personal information, means for reconstructing the audio data using speech synthesis technology with the revised text data, and means for storing and providing the reconstructed audio data. This enables the secure and efficient processing of data, including highly confidential information and personal information.

[0799] A "data file" is a file stored in a company or organization's database that contains information collected for a specific business or purpose.

[0800] "Audio data" refers to data that records human voices in digital format.

[0801] "Speech recognition technology" is a technology that converts speech data into text data.

[0802] "Text data" refers to data that records information written using characters and symbols in a digital format.

[0803] "Personal information" refers to information that can identify a specific individual, such as name, address, and telephone number.

[0804] A "generative artificial intelligence model" is an AI model that learns from large amounts of text data and analyzes the content of that text using natural language processing techniques.

[0805] "Deletion" refers to the operation of completely removing specific information from data.

[0806] "Replacement" refers to the operation of changing identified information into a different string or data.

[0807] "Speech synthesis technology" is a technology that artificially generates speech from text data.

[0808] "Reconstruction" refers to the process of creating new audio data based on revised text data.

[0809] "Saving" refers to the process of recording generated data on a digital medium so that it can be accessed later.

[0810] "Providing" refers to the operation of making generated data available for use by users.

[0811] This invention provides a system for securely and efficiently processing data, including highly confidential and personal information, within a company. This system includes a series of processes for acquiring audio data from a database, converting that audio data into text data using speech recognition technology, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data.

[0812] The server accesses the company's database and retrieves the target call data file. This operation uses database queries. For example, it executes the SQL query "SELECT FROM customer_calls WHERE call_id = '001'" to load the target call audio file "call_001.wav". This file contains the audio of the call with the customer.

[0813] The server converts the acquired audio data into text data using a speech recognition API. Specifically, it uses the Google Speech-to-Text API or Sphinx. The audio file "call_001.wav" is uploaded to the API, and the transcript result obtained as a response from the API is saved to "call_001_transcript.txt". This text file contains a record of the call content in text.

[0814] Next, an AI model on the server analyzes the converted text data to identify personal information (e.g., name and address). This AI model includes a pre-trained generative AI model and uses natural language processing techniques to tokenize the text, highlighting important identifying information based on specific patterns and contexts. For example, it identifies the name "Ichiro Tanaka" from the text "Hello, I'm Ichiro Tanaka. Today I'd like to talk about the contract details."

[0815] The server removes and appropriately replaces identified personal information from the text data. Specifically, it uses regular expressions to search for specific patterns and replaces corresponding parts with "" or "anonymous". As a result, it generates a revised text file "call_001_transcript_safe.txt" that reads something like "Hello, I'd like to talk about the contract details today."

[0816] The server then generates new audio data based on the revised text data. Specifically, it uses the Google Text-to-Speech API to generate audio data from the revised text. This generates a new audio file called "call_001_safe.wav" and plays the corresponding portion of the original audio in a secure format. This file is saved so that the user can use it later.

[0817] Users download the securely processed audio data "call_001_safe.wav" from the server and use it for business analysis and research purposes. This allows users to utilize a clean dataset that does not contain any personal information.

[0818] A concrete example is call center data. By processing call data containing highly confidential information such as customer names and addresses using this system, relevant business analysis can be performed without the risk of personal information leakage. This significantly improves a company's data utilization capabilities and promotes the creation of business value.

[0819] An example of a prompt for a generative AI model can be written as follows:

[0820] "Please extract the customer's name and address from this audio data, anonymize it, and create a reconstructed audio file."

[0821] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0822] Step 1:

[0823] The server accesses the company's database and retrieves the target call data file. Specifically, the server executes the SQL query "SELECT FROM customer_calls WHERE call_id = '001'" and loads the call audio file "call_001.wav" stored in the database. The input for this step is the database query, and the output is the retrieved audio file.

[0824] Step 2:

[0825] The server converts the acquired audio data into text data using a speech recognition API. Specifically, the server uploads the audio file "call_001.wav" to the Google Speech-to-Text API and receives the transcription result as a response from the API. This transcription result is saved to "call_001_transcript.txt". The input for this step is an audio file, and the output is a text file.

[0826] Step 3:

[0827] An AI model on the server analyzes the converted text data and identifies personal information. Specifically, the server inputs "call_001_transcript.txt" to the AI ​​model, and the model uses natural language processing techniques to extract personal information (e.g., name and address) from the text. The input for this step is a text file, and the output is a list of identified personal information.

[0828] Step 4:

[0829] The server removes identified personal information from the text data and replaces it appropriately. Specifically, the server uses regular expressions to search for specific patterns within "call_001_transcript.txt" and replaces the corresponding parts with "" or "anonymous". As a result of this process, a revised text file called "call_001_transcript_safe.txt" is generated. The input for this step is a list of identified personal information and a text file, and the output is a revised text file.

[0830] Step 5:

[0831] The server generates new audio data based on the revised text data. Specifically, the server uses the Google Text-to-Speech API to convert "call_001_transcript_safe.txt" into audio data and generates a new audio file "call_001_safe.wav". The input for this step is the revised text file, and the output is the new audio file.

[0832] Step 6:

[0833] The user downloads securely processed audio data from the server. Specifically, the user downloads "call_001_safe.wav" via the server's web interface or API and plays and analyzes it on their local device. The input for this step is a new audio file, and the output is the audio file saved on the user's local device.

[0834] (Application Example 1)

[0835] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0836] In recent years, with the increasing use of digital data, the secure processing of personal and confidential information, including voice data, has become a critical issue. In particular, real-time voice data processing requires the rapid and accurate detection of personal information and its appropriate processing while preventing data leaks. However, conventional systems have lacked sufficient means to protect personal information while processing voice data in real time, making it difficult to ensure user privacy.

[0837] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0838] In this invention, the server includes means for acquiring data files from data storage, means for converting acquired audio data into text data, means for identifying and extracting personal and confidential information from the text data, means for deleting or replacing the identified information, means for reconstructing audio data using the revised text data, means for storing and providing the reconstructed audio data, means for acquiring user audio in real time and detecting and replacing personal information, and means for generating and providing clean audio data in real time. This makes it possible to securely process and provide audio data containing personal and confidential information in real time.

[0839] "Data storage" refers to a storage device used to store and manage data.

[0840] A "data file" is a collection of information managed by a computer, and specifically includes audio data and text data.

[0841] "Audio data" refers to data obtained by digitizing sound signals, which is usually analyzed using speech recognition technology.

[0842] "Text data" is a collection of information represented by characters and symbols.

[0843] "Personal information" refers to information that can identify a specific individual, including names and addresses.

[0844] "Confidential information" refers to important information whose disclosure should be restricted, and includes internal company documents and contract details.

[0845] "Identification" is the process of distinguishing and verifying specific information.

[0846] "Extraction" is the process of extracting specific information from a large amount of data.

[0847] "Deletion" is the process of erasing existing data.

[0848] "Replacement" is the process of substituting original data with other data.

[0849] "Revised text data" refers to text data after personal or confidential information has been deleted or replaced.

[0850] "Reconstructing audio" is the process of generating new audio data based on text data.

[0851] "Real-time" refers to a state in which data is acquired, processed, and provided immediately.

[0852] "Clean audio data" refers to secure audio data from which personal and confidential information has been removed.

[0853] This invention provides a system for safely and efficiently processing voice data in a business. The system for carrying out this invention includes the following hardware and software configuration.

[0854] Hardware and software to be used

[0855] Data storage: A storage device that stores and manages data files. Examples include internal servers and cloud servers within a company.

[0856] Speech recognition technology: Technology for converting speech data into text data. Specific examples include "Google Speech-to-Text API" and "Sphinx".

[0857] AI Model: A model pre-trained to identify and extract personal and confidential information. Examples include implementations using Python's natural language processing libraries (such as NLTK and SpaCy).

[0858] Speech synthesis technology: A technology used to reconstruct text data into speech data. For example, the "Google Text-to-Speech API" is used.

[0859] Smart gadgets: Devices for processing audio data in real time during operation. Specifically, this includes smart glasses and head-mounted displays.

[0860] System Operation Overview

[0861] 1. Obtaining data files

[0862] The server retrieves an audio data file from data storage. For example, it loads the "call_001.wav" file from "customer_calls.db". This is an audio recording of a customer call stored in the company's database.

[0863] 2. Converting audio data to text data

[0864] The server converts the acquired audio data into text data using speech recognition technology. Specifically, it uploads "call_001.wav" to the API and saves the resulting transcription to a text file called "call_001_transcript.txt".

[0865] 3. Identification and extraction of personal information

[0866] Using an AI model pre-trained on the server, the converted text data is analyzed to identify personal and confidential information. Based on natural language processing technology, the text is tokenized and important identifying information is extracted.

[0867] 4. Deletion or replacement of information

[0868] Identified personal information is removed from the text data and replaced appropriately. The server uses regular expressions or similar methods to convert the data to "" or "anonymous". For example, from the text "Hello, I'm Tanaka Ichiro.", "Tanaka Ichiro" is replaced with "", generating "Hello, desu.".

[0869] 5. Reconstruct the audio data based on the revised text data.

[0870] The server uses speech synthesis technology to convert the revised text data into audio data. For example, it generates a new audio file named "call_001_safe.wav".

[0871] 6. Generate and provide clean audio data in real time.

[0872] This system acquires user voices in real time, uses an AI model to instantly detect and replace personal information, and generates clean voice data. This allows users to use secure voice data in real time while minimizing the risk of personal information leakage.

[0873] Examples of specific cases and prompt statements

[0874] Specific example: If a user says "Hello, I'm Tanaka Ichiro" during a meeting, the smart glasses instantly process the audio data and display "Hello, I am." on the screen.

[0875] Prompt example: Find and replace the name and address in this spoken text. Input: 'Hello, I'm Ichiro Tanaka. What time does the meeting start?'

[0876] This invention makes it possible to securely process and provide audio data containing personal and confidential information in real time, significantly improving security in corporate data operations.

[0877] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0878] Step 1:

[0879] Retrieving data files

[0880] The server retrieves the necessary audio data files from data storage. In this process, the server accesses the database and loads a specific audio file (e.g., "call_001.wav").

[0881] Input: Path to the audio data file in data storage

[0882] Output: Acquired audio data file

[0883] Specific action: The server loads "call_001.wav" from "customer_calls.db".

[0884] Step 2:

[0885] Converting audio data to text data

[0886] The server converts the acquired audio data into text data using speech recognition technology. For example, it can use the "Google Speech-to-Text API" to input audio data and generate a transcript.

[0887] Input: Audio data file (e.g., "call_001.wav")

[0888] Output: Text data file corresponding to the audio data (e.g., "call_001_transcript.txt")

[0889] Specific operation: Upload "call_001.wav" to the API and save the resulting transcription to "call_001_transcript.txt".

[0890] Step 3:

[0891] Identification and extraction of personal and confidential information

[0892] Using an AI model pre-trained on the server, the converted text data is analyzed to identify and extract personal and confidential information.

[0893] Input: Converted text data (e.g., "call_001_transcript.txt")

[0894] Output: List of identified personal and confidential information

[0895] Specific operation: Using an AI model (e.g., a Python natural language processing library), analyze specific patterns and contexts within text to identify information such as "Ichiro Tanaka".

[0896] Step 4:

[0897] Deletion or replacement of information

[0898] The server removes identified personal information from the text data and replaces it appropriately. For example, it uses regular expressions to convert "Ichiro Tanaka" to "".

[0899] Input: A list of identified personal and confidential information, a text data file (e.g., "call_001_transcript.txt")

[0900] Output: Revised text data (e.g., "call_001_transcript_safe.txt")

[0901] Specific action: Replace "Ichiro Tanaka" with quotation marks and generate revised text such as "Hello, desu."

[0902] Step 5:

[0903] Reconstruct audio data based on revised text data.

[0904] The server uses speech synthesis technology to convert the revised text data into speech data. For example, it generates a new audio file using the Google Text-to-Speech API.

[0905] Input: Revised text data file (e.g., "call_001_transcript_safe.txt")

[0906] Output: Reconstructed audio data file (e.g., "call_001_safe.wav")

[0907] Specific action: Save "call_001_transcript_safe.txt" as "call_001_safe.wav" using speech synthesis technology.

[0908] Step 6:

[0909] Generate and provide clean audio data in real time.

[0910] This system acquires user voices in real time, instantly detects and replaces personal information, and generates clean voice data.

[0911] Input: Audio data acquired in real time

[0912] Output: Clean audio data processed immediately.

[0913] Specific operation: Using an AI model, personal information is instantly identified, clean audio data is generated using speech synthesis technology, and displayed on the user's smart glasses.

[0914] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0915] This invention provides a system for securely and efficiently processing audio data containing highly confidential and personal information, and for recognizing and analyzing user emotions. This system includes specific processing steps such as retrieving data files from a database, converting audio data into text data, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data, as well as a process for recognizing user emotions using an emotion engine. The program processing of this system is described in detail below.

[0916] Data acquisition

[0917] server:

[0918] The server retrieves the target call data file from the company's database. For example, the server loads the "call_001.wav" file from the "database management system." This file contains the audio of a call with a customer.

[0919] Converting audio data to text

[0920] server:

[0921] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, a speech recognition service such as Google Speech-to-Text API or Sphinx is used to convert the audio data into text data. The resulting transcript is saved as "call_001_transcript.txt".

[0922] Identification of Top Secret Information

[0923] AI models on the server:

[0924] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Using natural language processing techniques, the text is tokenized to identify information such as names and addresses. Through this process, important identifying information such as "Ichiro Sato" and "Shibuya Ward, Tokyo" is extracted from the text "Hello, I'm Ichiro Sato. Today I'd like to talk about Shibuya Ward, Tokyo."

[0925] Extraction and deletion of personal information

[0926] server:

[0927] Identified personal information is removed or replaced from the text data. Specifically, regular expressions are used to replace personal information in the text with safe strings such as "" or "anonymous". As a result, a revised text file "call_001_transcript_safe.txt" is generated, which may contain text like "Hello, I'd like to talk about today."

[0928] Reconstructing audio data

[0929] server:

[0930] New audio data is generated based on the revised text data. Speech synthesis technology (e.g., Google Text-to-Speech API) is used to convert "call_001_transcript_safe.txt" into audio data. This generates a new audio file, "call_001_safe.wav," from which personal information has been removed.

[0931] Emotional analysis using an emotional engine

[0932] server:

[0933] The emotion engine on the server analyzes the acquired audio data "call_001.wav" to recognize the user's emotions. The emotion engine analyzes parameters such as voice tone, pitch, and speed to identify the user's emotions. For example, the server identifies the user's emotions as emotion tags such as "joy," "anger," and "sadness," and adds this information to the text data.

[0934] Addition of emotional information and response generation

[0935] server:

[0936] The emotion engine identifies emotional information and adds it to text data, automatically generating responses based on specific emotions. For example, if the user is angry, it might generate a response such as, "You seem angry. Let me explain calmly." This response is then reconstructed as corresponding audio data and provided to the user.

[0937] Data storage and provision to users

[0938] server:

[0939] The generated secure audio file "call_001_safe.wav" and text data with added emotional information are saved and made accessible to the user. Users can then download the data from the server and use it for business analysis or research purposes.

[0940] As a concrete example, consider customer call data from a call center. By processing call data containing highly confidential information such as customer names and addresses with this system, it becomes possible to perform sentiment analysis of call content while eliminating the risk of customer information leakage. This significantly improves the quality of customer service and operational efficiency.

[0941] The following describes the processing flow.

[0942] Step 1:

[0943] server:

[0944] The server retrieves the target call data file from the company's database. Specifically, the server uses a database query to search for and retrieve a call file such as "call_001.wav". For example, it executes the SQL command "SELECT FROM calls WHERE id='call_001'" and saves the result to a temporary working directory.

[0945] Step 2:

[0946] server:

[0947] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, a speech recognition service such as Google Speech-to-Text API or Sphinx is used to convert the audio data into text data. The audio file is uploaded using the API, and the converted text data is saved as "call_001_transcript.txt".

[0948] Step 3:

[0949] AI models on the server:

[0950] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Specifically, it uses natural language processing technology to tokenize the text and extract personal information such as names and addresses. Through this process, for example, "Hello, I'm Sato Ichiro. Today I'd like to talk about Shibuya Ward, Tokyo." can be interpreted as "Sato Ichiro" and "Shibuya Ward, Tokyo."

[0951] Step 4:

[0952] server:

[0953] Remove or replace identified personal information from the text data. Specifically, use regular expressions to replace personal information in the text with safe strings such as "" or "anonymous". For example, change "Hello, I'm Sato Ichiro." to "Hello, desu.". Generate the revised text file "call_001_transcript_safe.txt".

[0954] Step 5:

[0955] server:

[0956] New audio data will be generated based on the revised text data. Specifically, speech synthesis technology will be used to convert "call_001_transcript_safe.txt" into audio data. The revised text will be uploaded using the Google Text-to-Speech API or similar, the synthesized audio data "call_001_safe.wav" will be generated and saved to a temporary working directory.

[0957] Step 6:

[0958] server:

[0959] The emotion engine on the server analyzes the original audio data "call_001.wav" to recognize the user's emotions. Specifically, it analyzes parameters such as tone, pitch, and speed of the audio data to identify the user's emotions (e.g., joy, anger, sadness) and adds that information to the text data.

[0960] Step 7:

[0961] server:

[0962] Based on the emotion information identified by the emotion engine, emotion tags are added to the revised text data. Specifically, if the user is angry, a response such as "You seem angry. Let me explain calmly" is generated and added to the revised text. In some cases, audio data may be generated again based on this revised text data.

[0963] Step 8:

[0964] server:

[0965] The generated secure audio data "call_001_safe.wav" and revised text data will be saved and made available to users for access. Specifically, the server will place the files in a designated folder and generate a URL that users can download. Users will use this URL to access the secure data and use it for business analysis or research purposes.

[0966] This specific process not only automatically removes confidential information from call data, but also provides a system that recognizes user emotions and enables appropriate responses and analysis. This allows companies to significantly improve the quality of customer service and operational efficiency.

[0967] (Example 2)

[0968] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0969] Conventional voice data processing systems had security risks and efficiency issues when extracting and deleting personal information from voice data. Furthermore, they lacked the ability to identify user emotions and generate appropriate responses, in addition to simply deleting personal information. As a result, improvements in customer service quality and operational efficiency were not fully achieved.

[0970] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0971] In this invention, the server includes means for obtaining data files from a database, means for converting the obtained voice data into text data, means for identifying and extracting personal information from the text data, means for deleting or replacing the identified personal information, means for reconstructing voice data using the revised text data, means for storing and providing the reconstructed voice data, means for using an emotion engine for sentiment analysis of the voice data, and means for identifying the user's emotions and generating a response based on those emotions. This makes it possible to analyze the emotions of call content while eliminating the risk of leakage of personal information. Furthermore, it enables significant improvements in the quality of customer service and operational efficiency.

[0972] A "data file" refers to information stored in digital format, and includes various types of data such as audio, text, and images.

[0973] "Audio data" refers to digitized audio information, including various types of audio such as phone calls, recordings, and lectures.

[0974] "Text data" refers to data that stores characters and symbols in a digital format, in a format that is easily analyzable by software and systems.

[0975] "Personal information" refers to information used to identify a specific individual, and includes names, addresses, telephone numbers, and email addresses.

[0976] "Identification" means distinguishing specific information or data from other information or data.

[0977] "Extraction" refers to the process of extracting necessary information or features from a large amount of data.

[0978] "Deletion" means completely removing unnecessary or risky information from the data.

[0979] "Substitution" refers to replacing specific information with other information.

[0980] "Revised text data" refers to text data that has been modified by removing or replacing personal information from the original text data.

[0981] "Speech synthesis technology" is a technology that generates speech by imitating human voices based on text data.

[0982] An "emotion engine" is a general term for systems that analyze and identify emotions from voice and text data.

[0983] "Sentiment analysis" refers to the process of reading emotions from audio or text and identifying specific emotional states.

[0984] "Response" refers to the system's reply or reaction to user input or statements.

[0985] This invention provides a system for securely and efficiently processing voice data containing highly confidential and personal information, and further for recognizing and analyzing user emotions. This system includes the following main processing steps:

[0986] First, the server retrieves a data file from the database. Specifically, it uses a database management system to access the database and retrieve a designated audio data file (e.g., call_001.wav). Next, the server sends the retrieved audio data to a speech recognition API (e.g., Google Speech-to-Text API or Sphinx) and converts the retrieved audio data into text data. The resulting transcript is saved as a file (e.g., call_001_transcript.txt).

[0987] Subsequently, a pre-trained AI model on the server analyzes the text data and identifies personal information. Natural language processing (NLP) techniques are used to tokenize the text, and then identifying information such as names and addresses is extracted. The identified personal information is either deleted or replaced with a secure string by the server, thereby generating a revised text data file (e.g., call_001_transcript_safe.txt).

[0988] Furthermore, the server uses speech synthesis technology such as the Google Text-to-Speech API to generate new audio data based on this revised text data. This results in a new audio file (e.g., call_001_safe.wav) from which personal information has been removed.

[0989] Furthermore, an emotion engine within the server analyzes the voice data and recognizes the user's emotions. The emotion engine analyzes parameters such as voice tone, pitch, and speed, and identifies the user's emotions as emotion tags such as "joy," "anger," and "sadness." This information is added to the text data, and a response based on the specific emotion is automatically generated. The response is then reconstructed as voice data using speech synthesis technology and provided to the user.

[0990] For example, when processing customer call data in a call center, this system allows for the secure removal of highly confidential information such as customer names and addresses, while also enabling sentiment analysis of the call content. This is expected to improve the quality of customer service and enhance operational efficiency.

[0991] Examples of prompt statements include the following:

[0992] Please describe a system that processes customer call data, removes specific personal information, analyzes user sentiment, and generates appropriate responses. Also, please provide specific application scenarios in a call center.

[0993] The above describes a specific implementation of a system that can perform sentiment analysis of call content while eliminating the risk of personal information leakage.

[0994] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0995] Step 1:

[0996] Data acquisition

[0997] The server retrieves data files from the database.

[0998] Input: Database connection information and the ID of the audio data file to be retrieved (e.g., 001)

[0999] Processing: The server executes a query on the database management system to retrieve the specified audio data file (call_001.wav). Specifically, it executes the query "SELECT FROM calls WHERE id=001;".

[1000] Output: Acquired audio data file (call_001.wav)

[1001] Step 2:

[1002] Converting audio data to text

[1003] The server sends the acquired audio data file to the speech recognition API.

[1004] Input: Audio data file (call_001.wav)

[1005] Processing: The server sends the audio data to the Google Speech-to-Text API endpoint and converts it to text data. The text data returned from the API is saved as "call_001_transcript.txt".

[1006] Specific operation: The server sends the audio file to the speech recognition API as a POST request and receives the conversion result in JSON format. It then saves this as a text file.

[1007] Output: Transcript text file (call_001_transcript.txt)

[1008] Step 3:

[1009] Identification of Top Secret Information

[1010] An AI model on the server analyzes text data and identifies personal information.

[1011] Input: Transcript text (call_001_transcript.txt)

[1012] Processing: Text is tokenized using natural language processing (NLP) techniques, and personal information such as names and addresses is extracted.

[1013] Specific operation: The AI ​​model analyzes each word and phrase in the text to identify personal information such as "Ichiro Sato" and "Shibuya Ward, Tokyo."

[1014] Output: List of personal information (e.g., "Ichiro Sato", "Shibuya Ward, Tokyo")

[1015] Step 4:

[1016] Extraction and deletion of personal information

[1017] The server deletes or replaces identified personal information.

[1018] Input: Transcript text (call_001_transcript.txt) and a list of personal information

[1019] Processing: Regular expressions are used to replace identified personal information with "" or "anonymous". This generates the revised text data (call_001_transcript_safe.txt).

[1020] Specific operation: The server uses a list of identified personal information to replace the relevant parts of the text with regular expressions.

[1021] Output: Revised text data file (call_001_transcript_safe.txt)

[1022] Step 5:

[1023] Reconstructing audio data

[1024] The server generates new audio data based on the revised text data.

[1025] Input: Revised text data file (call_001_transcript_safe.txt)

[1026] Process: Perform the steps to convert text data into speech data using the Google Text-to-Speech API.

[1027] Specific operation: The server sends text to the Google Text-to-Speech API endpoint and saves the returned audio data as a file. Example: "call_001_safe.wav"

[1028] Output: New audio data file (call_001_safe.wav)

[1029] Step 6:

[1030] Emotional analysis using an emotional engine

[1031] The emotion engine on the server analyzes the voice data and recognizes the user's emotions.

[1032] Input: Original audio data file (call_001.wav)

[1033] Processing: The emotion engine analyzes parameters such as voice tone, pitch, and speed to identify the user's emotions.

[1034] Specific operation: The emotion engine breaks down the audio file into segments, calculates the feature quantities of each segment, and assigns emotion tags to them.

[1035] Output: Text data with emotion information (e.g., emotion tags such as "joy", "anger", and "sadness")

[1036] Step 7:

[1037] Addition of emotional information and response generation

[1038] The server adds emotional information to text data and generates responses based on specific emotions.

[1039] Input: Text data with sentiment information and revised text data

[1040] Processing: Based on specific emotions, responses are automatically generated and then reconstructed using speech synthesis technology.

[1041] Specific operation: The server uses emotion-based response templates to generate appropriate response sentences, which are then converted into speech using a Text-to-Speech service.

[1042] Output: Response audio data file

[1043] Step 8:

[1044] Data storage and provision to users

[1045] The server stores all generated data and makes it accessible to users.

[1046] Input: Secure audio data (call_001_safe.wav), text data, and text data with sentiment information.

[1047] Processing: The server stores this data in a database or cloud storage and provides users with a URL or download link to access it.

[1048] Specific operation: The server uploads the file to the cloud storage service and generates a link.

[1049] Output: URLs for user access and download links

[1050] (Application Example 2)

[1051] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1052] In today's information society, the risk of leakage of personal and confidential information contained in voice data is increasing. In particular, call centers and security services need to understand and respond to the emotional states of customers and users, but the protection of personal information during this process is crucial. Furthermore, recognizing emotions from voice data and providing appropriate responses is a challenging task. To solve these problems, a system is needed that securely processes voice data, recognizes user emotions in real time, and provides appropriate responses.

[1053] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1054] In this invention, the server includes means for acquiring audio files from data storage, means for converting the acquired audio data into text data, means for identifying and extracting personal information from the text data, means for deleting or anonymizing the identified personal information, means for resynthesizing audio data using the revised text data, means for storing and providing the resynthesized audio data, means for recognizing and analyzing the user's emotions, means for generating a response based on the analysis results, and means for providing the generated response as resynthesized audio data. This enables the secure processing of personal information within the audio data and allows for appropriate responses according to the user's emotional state.

[1055] "Data storage" refers to devices or systems used to store digital data such as audio files.

[1056] An "audio file" is a digital file used to store audio data.

[1057] "Audio data" refers to data that records human voices and sounds in digital format.

[1058] "Text data" refers to digital data obtained by converting audio data into written text.

[1059] "Personal information" refers to information that can identify a specific individual, including names and addresses.

[1060] "Anonymization" is a process that makes it impossible to identify personal information.

[1061] "Speech recognition technology" is a technology that analyzes speech data and converts it into corresponding text data.

[1062] "Speech synthesis technology" is a technology for generating speech data based on text data.

[1063] "Recognizing and analyzing emotions" means identifying a speaker's emotions from audio or text data and analyzing their emotional state.

[1064] "Generating responses" means automatically generating appropriate dialogue content based on recognized emotions.

[1065] A "system" is a collection of devices and software that combine multiple means to achieve a specific function.

[1066] To implement this invention, a system program is constructed that follows the steps below. First, the server acquires audio data and converts it into text data. This process uses speech recognition technologies such as the Google Speech-to-Text API or Sphinx. Personal information is identified from the acquired text data and deleted or anonymized using regular expressions or the like. The deleted or anonymized text data is converted back into audio data using speech synthesis technologies such as the Google Text-to-Speech API and stored.

[1067] The server further uses the acquired voice and text data to recognize and analyze the user's emotions. This is done using an emotion recognition engine. For example, it identifies the user's emotional state by analyzing parameters such as voice tone, pitch, and speed. Based on this analysis, it uses a generative AI model to generate an appropriate response and provides the generated response as voice data.

[1068] The following is a natural language explanation of the processes performed by the server.

[1069] The server retrieves audio files from data storage. These audio files may include, for example, audio data from security sites. The retrieved audio data is converted into text information using the Google Speech-to-Text API. This converts the audio data into text data. Next, the server uses natural language processing techniques to identify personal information from the text data and removes or anonymizes it using regular expressions. The text data from which personal information has been removed is converted back into speech using the Google Text-to-Speech API and stored as newly generated audio data.

[1070] Furthermore, the server uses an emotion recognition engine to perform emotion analysis on the acquired audio data. It identifies the user's emotional state from the tone and pitch of their voice and assigns emotion tags to it. Based on these emotion tags, a generative AI model generates an appropriate response, which is then resynthesized as audio data.

[1071] As a concrete example, if this system were implemented in smart glasses used by security guards, the microphones in the smart glasses would collect audio from the scene and send it to a server. The server would then convert this audio data into text, remove or anonymize personal information, reconstruct it as speech, and perform emotion recognition. Finally, it would generate an appropriate response and provide it as audio, enabling security guards to respond appropriately on-site.

[1072] An example of a prompt message input to a generative AI model might be: "Convert the audio data to text and anonymize personal information. Furthermore, analyze the emotional state of the text and generate an appropriate response based on that emotion." Based on such prompt messages, the system will automatically perform the necessary processing.

[1073] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1074] Step 1:

[1075] Acquisition of audio data

[1076] The server retrieves an audio file from data storage. The input includes the path to the specified audio file. This data might be collected, for example, in a security setting. The retrieved audio file is then output.

[1077] Step 2:

[1078] Converting audio data to text

[1079] The server sends the acquired audio file to the Google Speech-to-Text API. The audio file is used as input. Speech recognition technology is used to convert the audio data into text data. This process outputs the text data.

[1080] Step 3:

[1081] Identification of personal information

[1082] The server analyzes the converted text data using natural language processing techniques. The input is text data. The text is tokenized, and personal information (e.g., name, address) is identified. The identified personal information is then output.

[1083] Step 4:

[1084] Deletion or anonymization of personal information

[1085] The server removes or anonymizes identified personal information from the text. Text data and identified personal information are used as input. Regular expressions are used to replace personal information with "" or "anonymous". This process outputs the revised text data.

[1086] Step 5:

[1087] Re-synthesis of audio data

[1088] The server sends the revised text data to the Google Text-to-Speech API, which then resynthesizes the audio data. The revised text data is included as input. Using speech synthesis technology, new audio data is generated. This audio data is then output.

[1089] Step 6:

[1090] Emotion recognition and analysis

[1091] The server inputs the acquired and converted audio data into an emotion recognition engine to analyze the user's emotions. Audio data is used as input. Parameters such as voice tone and pitch are analyzed to identify the emotional state (e.g., joy, anger). The emotional state is tagged and output.

[1092] Step 7:

[1093] Response generation

[1094] The server generates a response using a generative AI model based on the recognized emotional state. The emotional state is included as input. The generative AI model automatically generates the corresponding response according to the prompt. This response is then output.

[1095] Step 8:

[1096] Re-combining the response into audio data

[1097] The server sends the generated response to the Google Text-to-Speech API for resynthesis into audio data. The generated response is used as input. New response audio data is generated using speech synthesis technology. This audio data is the final output.

[1098] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1099] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1100] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1101] [Fourth Embodiment]

[1102] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1103] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1104] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1105] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1106] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1107] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1108] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1109] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1110] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1111] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1112] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1113] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1114] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1115] This invention provides a system for securely and efficiently processing data, including highly confidential and personal information, within a company. This system includes a series of processes for retrieving data files from a database, converting audio data to text data, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data. The processing of the system's program is described in detail below in natural language.

[1116] Data acquisition

[1117] server:

[1118] The server retrieves the target call data file from the company's database. For example, the server loads the "call_001.wav" file from the "customer_calls.db" database. This file contains the audio of a call with a customer stored in the database.

[1119] Converting audio data to text

[1120] server:

[1121] The server converts the acquired audio data into text data using a speech recognition API (e.g., Google Speech-to-Text API or Sphinx). The file "call_001.wav" is uploaded, and the transcription result is saved as a text file named "call_001_transcript.txt" in the API response. This text file contains a written record of the call.

[1122] Identification of Top Secret Information

[1123] AI models on the server:

[1124] An AI model, pre-trained on the server, analyzes the converted text data to identify personal information (such as name and address). The AI ​​model uses natural language processing techniques to tokenize the text, highlighting important identifying information based on specific patterns and context. For example, it identifies the name "Ichiro Tanaka" from the text "Hello, I'm Ichiro Tanaka. Today I'd like to talk about the contract details."

[1125] Extraction and deletion of top-secret information

[1126] server:

[1127] Identified personal information is removed from the text data and replaced appropriately. The server uses regular expressions to replace "Ichiro Tanaka" with "" or "anonymous". As a result, a revised text file "call_001_transcript_safe.txt" is generated, which may read something like "Hello, I'd like to talk about the contract details today."

[1128] Data reconstruction

[1129] server:

[1130] The audio data is reconstructed based on the revised text data. The server uses speech synthesis technology (for example, the Google Text-to-Speech API) to convert the revised text into audio data. This generates a new audio file called "call_001_safe.wav", which can play the corresponding portion of the original audio in a safe format. This file is saved for later use by the user.

[1131] Provision to users

[1132] User:

[1133] Users download the securely processed audio data "call_001_safe.wav" from the server and use it for business analysis or research purposes. This allows users to utilize a clean dataset that does not contain any personal information.

[1134] A concrete example is call center call data. By processing call data containing highly confidential information such as customer names and addresses using this system, relevant business analysis can be performed without the risk of personal information leakage. This significantly improves a company's data utilization capabilities and promotes the creation of business value.

[1135] The following describes the processing flow.

[1136] Step 1:

[1137] server:

[1138] The server retrieves call data files from the company's database. Specifically, it executes a query against the "database management system" to search for and retrieve the target call data (for example, "call_001.wav"). It searches using an SQL statement such as "SELECT FROM calls WHERE id='call_001'" and saves the retrieved file to a temporary working directory.

[1139] Step 2:

[1140] server:

[1141] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, the audio file is uploaded to a speech recognition service such as the Google Speech-to-Text API or Sphinx, and the audio data is converted into text data. The resulting transcript is saved as "call_001_transcript.txt".

[1142] Step 3:

[1143] AI models on the server:

[1144] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Specifically, it uses natural language processing techniques to tokenize the text and identify specific words and phrases (for example, names and addresses). This process reveals important identifying information such as the name "Ichiro Tanaka" and the address "Shinjuku-ku, Tokyo".

[1145] Step 4:

[1146] server:

[1147] Delete or replace identified personal information. Specifically, use regular expressions to replace personal information in the text with safe strings such as "" or "anonymous". For example, change "Hello, I'm Tanaka Ichiro." to "Hello, desu.". This will generate the revised text data "call_001_transcript_safe.txt".

[1148] Step 5:

[1149] server:

[1150] New audio data is generated based on the revised text data. Specifically, speech synthesis technology (e.g., Google Text-to-Speech API) is used to convert "call_001_transcript_safe.txt" into audio data. This generates a new audio file, "call_001_safe.wav," from which personal information has been removed. The generated audio file is saved in a temporary working directory.

[1151] Step 6:

[1152] server:

[1153] The generated secure audio file "call_001_safe.wav" will be saved and made available to the user. Specifically, the file will be moved to a designated folder, and a downloadable URL will be generated for the user. The user will then be able to use this URL to access the secure audio data for business analysis or research purposes.

[1154] (Example 1)

[1155] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1156] In processing data containing highly confidential and personal information within companies, there is a need to balance security and efficiency. Traditional methods often involve manual data processing, which is time-consuming and labor-intensive, and carries a high risk of information leaks due to human error. Furthermore, there has been a lack of methods to adequately protect personal information when handling audio data. Therefore, there is a need for technology that can automatically extract personal information from audio data, anonymize it, and make it reusable.

[1157] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1158] In this invention, the server includes means for obtaining data files from a database, means for converting the obtained audio data into text data using speech recognition technology, means including a generative artificial intelligence model for analyzing the converted text data and identifying and extracting personal information, means for deleting or replacing the identified personal information, means for reconstructing the audio data using speech synthesis technology with the revised text data, and means for storing and providing the reconstructed audio data. This enables the secure and efficient processing of data, including highly confidential information and personal information.

[1159] A "data file" is a file stored in a company or organization's database that contains information collected for a specific business or purpose.

[1160] "Audio data" refers to data that records human voices in digital format.

[1161] "Speech recognition technology" is a technology that converts speech data into text data.

[1162] "Text data" refers to data that records information written using characters and symbols in a digital format.

[1163] "Personal information" refers to information that can identify a specific individual, such as name, address, and telephone number.

[1164] A "generative artificial intelligence model" is an AI model that learns from large amounts of text data and analyzes the content of that text using natural language processing techniques.

[1165] "Deletion" refers to the operation of completely removing specific information from data.

[1166] "Replacement" refers to the operation of changing identified information into a different string or data.

[1167] "Speech synthesis technology" is a technology that artificially generates speech from text data.

[1168] "Reconstruction" refers to the process of creating new audio data based on revised text data.

[1169] "Saving" refers to the process of recording generated data on a digital medium so that it can be accessed later.

[1170] "Providing" refers to the operation of making generated data available for use by users.

[1171] This invention provides a system for securely and efficiently processing data, including highly confidential and personal information, within a company. This system includes a series of processes for acquiring audio data from a database, converting that audio data into text data using speech recognition technology, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data.

[1172] The server accesses the company's database and retrieves the target call data file. This operation uses database queries. For example, it executes the SQL query "SELECT FROM customer_calls WHERE call_id = '001'" to load the target call audio file "call_001.wav". This file contains the audio of the call with the customer.

[1173] The server converts the acquired audio data into text data using a speech recognition API. Specifically, it uses the Google Speech-to-Text API or Sphinx. The audio file "call_001.wav" is uploaded to the API, and the transcript result obtained as a response from the API is saved to "call_001_transcript.txt". This text file contains a record of the call content in text.

[1174] Next, an AI model on the server analyzes the converted text data to identify personal information (e.g., name and address). This AI model includes a pre-trained generative AI model and uses natural language processing techniques to tokenize the text, highlighting important identifying information based on specific patterns and contexts. For example, it identifies the name "Ichiro Tanaka" from the text "Hello, I'm Ichiro Tanaka. Today I'd like to talk about the contract details."

[1175] The server removes and appropriately replaces identified personal information from the text data. Specifically, it uses regular expressions to search for specific patterns and replaces corresponding parts with "" or "anonymous". As a result, it generates a revised text file "call_001_transcript_safe.txt" that reads something like "Hello, I'd like to talk about the contract details today."

[1176] The server then generates new audio data based on the revised text data. Specifically, it uses the Google Text-to-Speech API to generate audio data from the revised text. This generates a new audio file called "call_001_safe.wav" and plays the corresponding portion of the original audio in a secure format. This file is saved so that the user can use it later.

[1177] Users download the securely processed audio data "call_001_safe.wav" from the server and use it for business analysis and research purposes. This allows users to utilize a clean dataset that does not contain any personal information.

[1178] A concrete example is call center data. By processing call data containing highly confidential information such as customer names and addresses using this system, relevant business analysis can be performed without the risk of personal information leakage. This significantly improves a company's data utilization capabilities and promotes the creation of business value.

[1179] An example of a prompt for a generative AI model can be written as follows:

[1180] "Please extract the customer's name and address from this audio data, anonymize it, and create a reconstructed audio file."

[1181] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1182] Step 1:

[1183] The server accesses the company's database and retrieves the target call data file. Specifically, the server executes the SQL query "SELECT FROM customer_calls WHERE call_id = '001'" and loads the call audio file "call_001.wav" stored in the database. The input for this step is the database query, and the output is the retrieved audio file.

[1184] Step 2:

[1185] The server converts the acquired audio data into text data using a speech recognition API. Specifically, the server uploads the audio file "call_001.wav" to the Google Speech-to-Text API and receives the transcription result as a response from the API. This transcription result is saved to "call_001_transcript.txt". The input for this step is an audio file, and the output is a text file.

[1186] Step 3:

[1187] An AI model on the server analyzes the converted text data and identifies personal information. Specifically, the server inputs "call_001_transcript.txt" to the AI ​​model, and the model uses natural language processing techniques to extract personal information (e.g., name and address) from the text. The input for this step is a text file, and the output is a list of identified personal information.

[1188] Step 4:

[1189] The server removes identified personal information from the text data and replaces it appropriately. Specifically, the server uses regular expressions to search for specific patterns within "call_001_transcript.txt" and replaces the corresponding parts with "" or "anonymous". As a result of this process, a revised text file called "call_001_transcript_safe.txt" is generated. The input for this step is a list of identified personal information and a text file, and the output is a revised text file.

[1190] Step 5:

[1191] The server generates new audio data based on the revised text data. Specifically, the server uses the Google Text-to-Speech API to convert "call_001_transcript_safe.txt" into audio data and generates a new audio file "call_001_safe.wav". The input for this step is the revised text file, and the output is the new audio file.

[1192] Step 6:

[1193] The user downloads securely processed audio data from the server. Specifically, the user downloads "call_001_safe.wav" via the server's web interface or API and plays and analyzes it on their local device. The input for this step is a new audio file, and the output is the audio file saved on the user's local device.

[1194] (Application Example 1)

[1195] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1196] In recent years, with the increasing use of digital data, the secure processing of personal and confidential information, including voice data, has become a critical issue. In particular, real-time voice data processing requires the rapid and accurate detection of personal information and its appropriate processing while preventing data leaks. However, conventional systems have lacked sufficient means to protect personal information while processing voice data in real time, making it difficult to ensure user privacy.

[1197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1198] In this invention, the server includes means for acquiring data files from data storage, means for converting acquired audio data into text data, means for identifying and extracting personal and confidential information from the text data, means for deleting or replacing the identified information, means for reconstructing audio data using the revised text data, means for storing and providing the reconstructed audio data, means for acquiring user audio in real time and detecting and replacing personal information, and means for generating and providing clean audio data in real time. This makes it possible to securely process and provide audio data containing personal and confidential information in real time.

[1199] "Data storage" refers to a storage device used to store and manage data.

[1200] A "data file" is a collection of information managed by a computer, and specifically includes audio data and text data.

[1201] "Audio data" refers to data obtained by digitizing sound signals, which is usually analyzed using speech recognition technology.

[1202] "Text data" is a collection of information represented by characters and symbols.

[1203] "Personal information" refers to information that can identify a specific individual, including names and addresses.

[1204] "Confidential information" refers to important information whose disclosure should be restricted, and includes internal company documents and contract details.

[1205] "Identification" is the process of distinguishing and verifying specific information.

[1206] "Extraction" is the process of extracting specific information from a large amount of data.

[1207] "Deletion" is the process of erasing existing data.

[1208] "Replacement" is the process of substituting original data with other data.

[1209] "Revised text data" refers to text data after personal or confidential information has been deleted or replaced.

[1210] "Reconstructing audio" is the process of generating new audio data based on text data.

[1211] "Real-time" refers to a state in which data is acquired, processed, and provided immediately.

[1212] "Clean audio data" refers to secure audio data from which personal and confidential information has been removed.

[1213] This invention provides a system for safely and efficiently processing voice data in a business. The system for carrying out this invention includes the following hardware and software configuration.

[1214] Hardware and software to be used

[1215] Data storage: A storage device that stores and manages data files. Examples include internal servers and cloud servers within a company.

[1216] Speech recognition technology: Technology for converting speech data into text data. Specific examples include "Google Speech-to-Text API" and "Sphinx".

[1217] AI Model: A model pre-trained to identify and extract personal and confidential information. Examples include implementations using Python's natural language processing libraries (such as NLTK and SpaCy).

[1218] Speech synthesis technology: A technology used to reconstruct text data into speech data. For example, the "Google Text-to-Speech API" is used.

[1219] Smart gadgets: Devices for processing audio data in real time during operation. Specifically, this includes smart glasses and head-mounted displays.

[1220] System Operation Overview

[1221] 1. Obtaining data files

[1222] The server retrieves an audio data file from data storage. For example, it loads the "call_001.wav" file from "customer_calls.db". This is an audio recording of a customer call stored in the company's database.

[1223] 2. Converting audio data to text data

[1224] The server converts the acquired audio data into text data using speech recognition technology. Specifically, it uploads "call_001.wav" to the API and saves the resulting transcription to a text file called "call_001_transcript.txt".

[1225] 3. Identification and extraction of personal information

[1226] Using an AI model pre-trained on the server, the converted text data is analyzed to identify personal and confidential information. Based on natural language processing technology, the text is tokenized and important identifying information is extracted.

[1227] 4. Deletion or replacement of information

[1228] Identified personal information is removed from the text data and replaced appropriately. The server uses regular expressions or similar methods to convert the data to "" or "anonymous". For example, from the text "Hello, I'm Tanaka Ichiro.", "Tanaka Ichiro" is replaced with "", generating "Hello, desu.".

[1229] 5. Reconstruct the audio data based on the revised text data.

[1230] The server uses speech synthesis technology to convert the revised text data into audio data. For example, it generates a new audio file named "call_001_safe.wav".

[1231] 6. Generate and provide clean audio data in real time.

[1232] This system acquires user voices in real time, uses an AI model to instantly detect and replace personal information, and generates clean voice data. This allows users to use secure voice data in real time while minimizing the risk of personal information leakage.

[1233] Examples of specific cases and prompt statements

[1234] Specific example: If a user says "Hello, I'm Tanaka Ichiro" during a meeting, the smart glasses instantly process the audio data and display "Hello, I am." on the screen.

[1235] Prompt example: Find and replace the name and address in this spoken text. Input: 'Hello, I'm Ichiro Tanaka. What time does the meeting start?'

[1236] This invention makes it possible to securely process and provide audio data containing personal and confidential information in real time, significantly improving security in corporate data operations.

[1237] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1238] Step 1:

[1239] Retrieving data files

[1240] The server retrieves the necessary audio data files from data storage. In this process, the server accesses the database and loads a specific audio file (e.g., "call_001.wav").

[1241] Input: Path to the audio data file in data storage

[1242] Output: Acquired audio data file

[1243] Specific action: The server loads "call_001.wav" from "customer_calls.db".

[1244] Step 2:

[1245] Converting audio data to text data

[1246] The server converts the acquired audio data into text data using speech recognition technology. For example, it can use the "Google Speech-to-Text API" to input audio data and generate a transcript.

[1247] Input: Audio data file (e.g., "call_001.wav")

[1248] Output: Text data file corresponding to the audio data (e.g., "call_001_transcript.txt")

[1249] Specific operation: Upload "call_001.wav" to the API and save the resulting transcription to "call_001_transcript.txt".

[1250] Step 3:

[1251] Identification and extraction of personal and confidential information

[1252] Using an AI model pre-trained on the server, the converted text data is analyzed to identify and extract personal and confidential information.

[1253] Input: Converted text data (e.g., "call_001_transcript.txt")

[1254] Output: List of identified personal and confidential information

[1255] Specific operation: Using an AI model (e.g., a Python natural language processing library), analyze specific patterns and contexts within text to identify information such as "Ichiro Tanaka".

[1256] Step 4:

[1257] Deletion or replacement of information

[1258] The server removes identified personal information from the text data and replaces it appropriately. For example, it uses regular expressions to convert "Ichiro Tanaka" to "".

[1259] Input: A list of identified personal and confidential information, a text data file (e.g., "call_001_transcript.txt")

[1260] Output: Revised text data (e.g., "call_001_transcript_safe.txt")

[1261] Specific action: Replace "Ichiro Tanaka" with quotation marks and generate revised text such as "Hello, desu."

[1262] Step 5:

[1263] Reconstruct audio data based on revised text data.

[1264] The server uses speech synthesis technology to convert the revised text data into speech data. For example, it generates a new audio file using the Google Text-to-Speech API.

[1265] Input: Revised text data file (e.g., "call_001_transcript_safe.txt")

[1266] Output: Reconstructed audio data file (e.g., "call_001_safe.wav")

[1267] Specific action: Save "call_001_transcript_safe.txt" as "call_001_safe.wav" using speech synthesis technology.

[1268] Step 6:

[1269] Generate and provide clean audio data in real time.

[1270] This system acquires user voices in real time, instantly detects and replaces personal information, and generates clean voice data.

[1271] Input: Audio data acquired in real time

[1272] Output: Clean audio data processed immediately.

[1273] Specific operation: Using an AI model, personal information is instantly identified, clean audio data is generated using speech synthesis technology, and displayed on the user's smart glasses.

[1274] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1275] This invention provides a system for securely and efficiently processing audio data containing highly confidential and personal information, and for recognizing and analyzing user emotions. This system includes specific processing steps such as retrieving data files from a database, converting audio data into text data, identifying and extracting personal information, deleting or replacing that information, and reconstructing the revised data, as well as a process for recognizing user emotions using an emotion engine. The program processing of this system is described in detail below.

[1276] Data acquisition

[1277] server:

[1278] The server retrieves the target call data file from the company's database. For example, the server loads the "call_001.wav" file from the "database management system." This file contains the audio of a call with a customer.

[1279] Converting audio data to text

[1280] server:

[1281] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, a speech recognition service such as Google Speech-to-Text API or Sphinx is used to convert the audio data into text data. The resulting transcript is saved as "call_001_transcript.txt".

[1282] Identification of Top Secret Information

[1283] AI models on the server:

[1284] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Using natural language processing techniques, the text is tokenized to identify information such as names and addresses. Through this process, important identifying information such as "Ichiro Sato" and "Shibuya Ward, Tokyo" is extracted from the text "Hello, I'm Ichiro Sato. Today I'd like to talk about Shibuya Ward, Tokyo."

[1285] Extraction and deletion of personal information

[1286] server:

[1287] Identified personal information is removed or replaced from the text data. Specifically, regular expressions are used to replace personal information in the text with safe strings such as "" or "anonymous". As a result, a revised text file "call_001_transcript_safe.txt" is generated, which may contain text like "Hello, I'd like to talk about today."

[1288] Reconstructing audio data

[1289] server:

[1290] New audio data is generated based on the revised text data. Speech synthesis technology (e.g., Google Text-to-Speech API) is used to convert "call_001_transcript_safe.txt" into audio data. This generates a new audio file, "call_001_safe.wav," from which personal information has been removed.

[1291] Emotional analysis using an emotional engine

[1292] server:

[1293] The emotion engine on the server analyzes the acquired audio data "call_001.wav" to recognize the user's emotions. The emotion engine analyzes parameters such as voice tone, pitch, and speed to identify the user's emotions. For example, the server identifies the user's emotions as emotion tags such as "joy," "anger," and "sadness," and adds this information to the text data.

[1294] Addition of emotional information and response generation

[1295] server:

[1296] The emotion engine identifies emotional information and adds it to text data, automatically generating responses based on specific emotions. For example, if the user is angry, it might generate a response such as, "You seem angry. Let me explain calmly." This response is then reconstructed as corresponding audio data and provided to the user.

[1297] Data storage and provision to users

[1298] server:

[1299] The generated secure audio file "call_001_safe.wav" and text data with added emotional information are saved and made accessible to the user. Users can then download the data from the server and use it for business analysis or research purposes.

[1300] As a concrete example, consider customer call data from a call center. By processing call data containing highly confidential information such as customer names and addresses with this system, it becomes possible to perform sentiment analysis of call content while eliminating the risk of customer information leakage. This significantly improves the quality of customer service and operational efficiency.

[1301] The following describes the processing flow.

[1302] Step 1:

[1303] server:

[1304] The server retrieves the target call data file from the company's database. Specifically, the server uses a database query to search for and retrieve a call file such as "call_001.wav". For example, it executes the SQL command "SELECT FROM calls WHERE id='call_001'" and saves the result to a temporary working directory.

[1305] Step 2:

[1306] server:

[1307] The acquired call data file "call_001.wav" is sent to a speech recognition API. Specifically, a speech recognition service such as Google Speech-to-Text API or Sphinx is used to convert the audio data into text data. The audio file is uploaded using the API, and the converted text data is saved as "call_001_transcript.txt".

[1308] Step 3:

[1309] AI models on the server:

[1310] A pre-trained AI model on the server analyzes the text data "call_001_transcript.txt" to identify personal information. Specifically, it uses natural language processing technology to tokenize the text and extract personal information such as names and addresses. Through this process, for example, "Hello, I'm Sato Ichiro. Today I'd like to talk about Shibuya Ward, Tokyo." can be interpreted as "Sato Ichiro" and "Shibuya Ward, Tokyo."

[1311] Step 4:

[1312] server:

[1313] Remove or replace identified personal information from the text data. Specifically, use regular expressions to replace personal information in the text with safe strings such as "" or "anonymous". For example, change "Hello, I'm Sato Ichiro." to "Hello, desu.". Generate the revised text file "call_001_transcript_safe.txt".

[1314] Step 5:

[1315] server:

[1316] New audio data will be generated based on the revised text data. Specifically, speech synthesis technology will be used to convert "call_001_transcript_safe.txt" into audio data. The revised text will be uploaded using the Google Text-to-Speech API or similar, the synthesized audio data "call_001_safe.wav" will be generated and saved to a temporary working directory.

[1317] Step 6:

[1318] server:

[1319] The emotion engine on the server analyzes the original audio data "call_001.wav" to recognize the user's emotions. Specifically, it analyzes parameters such as tone, pitch, and speed of the audio data to identify the user's emotions (e.g., joy, anger, sadness) and adds that information to the text data.

[1320] Step 7:

[1321] server:

[1322] Based on the emotion information identified by the emotion engine, emotion tags are added to the revised text data. Specifically, if the user is angry, a response such as "You seem angry. Let me explain calmly" is generated and added to the revised text. In some cases, audio data may be generated again based on this revised text data.

[1323] Step 8:

[1324] server:

[1325] The generated secure audio data "call_001_safe.wav" and revised text data will be saved and made available to users for access. Specifically, the server will place the files in a designated folder and generate a URL that users can download. Users will use this URL to access the secure data and use it for business analysis or research purposes.

[1326] This specific process not only automatically removes confidential information from call data, but also provides a system that recognizes user emotions and enables appropriate responses and analysis. This allows companies to significantly improve the quality of customer service and operational efficiency.

[1327] (Example 2)

[1328] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1329] Conventional voice data processing systems had security risks and efficiency issues when extracting and deleting personal information from voice data. Furthermore, they lacked the ability to identify user emotions and generate appropriate responses, in addition to simply deleting personal information. As a result, improvements in customer service quality and operational efficiency were not fully achieved.

[1330] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1331] In this invention, the server includes means for obtaining data files from a database, means for converting the obtained voice data into text data, means for identifying and extracting personal information from the text data, means for deleting or replacing the identified personal information, means for reconstructing voice data using the revised text data, means for storing and providing the reconstructed voice data, means for using an emotion engine for sentiment analysis of the voice data, and means for identifying the user's emotions and generating a response based on those emotions. This makes it possible to analyze the emotions of call content while eliminating the risk of leakage of personal information. Furthermore, it enables significant improvements in the quality of customer service and operational efficiency.

[1332] A "data file" refers to information stored in digital format, and includes various types of data such as audio, text, and images.

[1333] "Audio data" refers to digitized audio information, including various types of audio such as phone calls, recordings, and lectures.

[1334] "Text data" refers to data that stores characters and symbols in a digital format, in a format that is easily analyzable by software and systems.

[1335] "Personal information" refers to information used to identify a specific individual, and includes names, addresses, telephone numbers, and email addresses.

[1336] "Identification" means distinguishing specific information or data from other information or data.

[1337] "Extraction" refers to the process of extracting necessary information or features from a large amount of data.

[1338] "Deletion" means completely removing unnecessary or risky information from the data.

[1339] "Substitution" refers to replacing specific information with other information.

[1340] "Revised text data" refers to text data that has been modified by removing or replacing personal information from the original text data.

[1341] "Speech synthesis technology" is a technology that generates speech by imitating human voices based on text data.

[1342] An "emotion engine" is a general term for systems that analyze and identify emotions from voice and text data.

[1343] "Sentiment analysis" refers to the process of reading emotions from audio or text and identifying specific emotional states.

[1344] "Response" refers to the system's reply or reaction to user input or statements.

[1345] This invention provides a system for securely and efficiently processing voice data containing highly confidential and personal information, and further for recognizing and analyzing user emotions. This system includes the following main processing steps:

[1346] First, the server retrieves a data file from the database. Specifically, it uses a database management system to access the database and retrieve a designated audio data file (e.g., call_001.wav). Next, the server sends the retrieved audio data to a speech recognition API (e.g., Google Speech-to-Text API or Sphinx) and converts the retrieved audio data into text data. The resulting transcript is saved as a file (e.g., call_001_transcript.txt).

[1347] Subsequently, a pre-trained AI model on the server analyzes the text data and identifies personal information. Natural language processing (NLP) techniques are used to tokenize the text, and then identifying information such as names and addresses is extracted. The identified personal information is either deleted or replaced with a secure string by the server, thereby generating a revised text data file (e.g., call_001_transcript_safe.txt).

[1348] Furthermore, the server uses speech synthesis technology such as the Google Text-to-Speech API to generate new audio data based on this revised text data. This results in a new audio file (e.g., call_001_safe.wav) from which personal information has been removed.

[1349] Furthermore, an emotion engine within the server analyzes the voice data and recognizes the user's emotions. The emotion engine analyzes parameters such as voice tone, pitch, and speed, and identifies the user's emotions as emotion tags such as "joy," "anger," and "sadness." This information is added to the text data, and a response based on the specific emotion is automatically generated. The response is then reconstructed as voice data using speech synthesis technology and provided to the user.

[1350] For example, when processing customer call data in a call center, this system allows for the secure removal of highly confidential information such as customer names and addresses, while also enabling sentiment analysis of the call content. This is expected to improve the quality of customer service and enhance operational efficiency.

[1351] Examples of prompt statements include the following:

[1352] Please describe a system that processes customer call data, removes specific personal information, analyzes user sentiment, and generates appropriate responses. Also, please provide specific application scenarios in a call center.

[1353] The above describes a specific implementation of a system that can perform sentiment analysis of call content while eliminating the risk of personal information leakage.

[1354] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1355] Step 1:

[1356] Data acquisition

[1357] The server retrieves data files from the database.

[1358] Input: Database connection information and the ID of the audio data file to be retrieved (e.g., 001)

[1359] Processing: The server executes a query on the database management system to retrieve the specified audio data file (call_001.wav). Specifically, it executes the query "SELECT FROM calls WHERE id=001;".

[1360] Output: Acquired audio data file (call_001.wav)

[1361] Step 2:

[1362] Converting audio data to text

[1363] The server sends the acquired audio data file to the speech recognition API.

[1364] Input: Audio data file (call_001.wav)

[1365] Processing: The server sends the audio data to the Google Speech-to-Text API endpoint and converts it to text data. The text data returned from the API is saved as "call_001_transcript.txt".

[1366] Specific operation: The server sends the audio file to the speech recognition API as a POST request and receives the conversion result in JSON format. It then saves this as a text file.

[1367] Output: Transcript text file (call_001_transcript.txt)

[1368] Step 3:

[1369] Identification of Top Secret Information

[1370] An AI model on the server analyzes text data and identifies personal information.

[1371] Input: Transcript text (call_001_transcript.txt)

[1372] Processing: Text is tokenized using natural language processing (NLP) techniques, and personal information such as names and addresses is extracted.

[1373] Specific operation: The AI ​​model analyzes each word and phrase in the text to identify personal information such as "Ichiro Sato" and "Shibuya Ward, Tokyo."

[1374] Output: List of personal information (e.g., "Ichiro Sato", "Shibuya Ward, Tokyo")

[1375] Step 4:

[1376] Extraction and deletion of personal information

[1377] The server deletes or replaces identified personal information.

[1378] Input: Transcript text (call_001_transcript.txt) and a list of personal information

[1379] Processing: Regular expressions are used to replace identified personal information with "" or "anonymous". This generates the revised text data (call_001_transcript_safe.txt).

[1380] Specific operation: The server uses a list of identified personal information to replace the relevant parts of the text with regular expressions.

[1381] Output: Revised text data file (call_001_transcript_safe.txt)

[1382] Step 5:

[1383] Reconstructing audio data

[1384] The server generates new audio data based on the revised text data.

[1385] Input: Revised text data file (call_001_transcript_safe.txt)

[1386] Process: Perform the steps to convert text data into speech data using the Google Text-to-Speech API.

[1387] Specific operation: The server sends text to the Google Text-to-Speech API endpoint and saves the returned audio data as a file. Example: "call_001_safe.wav"

[1388] Output: New audio data file (call_001_safe.wav)

[1389] Step 6:

[1390] Emotional analysis using an emotional engine

[1391] The emotion engine on the server analyzes the voice data and recognizes the user's emotions.

[1392] Input: Original audio data file (call_001.wav)

[1393] Processing: The emotion engine analyzes parameters such as voice tone, pitch, and speed to identify the user's emotions.

[1394] Specific operation: The emotion engine breaks down the audio file into segments, calculates the feature quantities of each segment, and assigns emotion tags to them.

[1395] Output: Text data with emotion information (e.g., emotion tags such as "joy", "anger", and "sadness")

[1396] Step 7:

[1397] Addition of emotional information and response generation

[1398] The server adds emotional information to text data and generates responses based on specific emotions.

[1399] Input: Text data with sentiment information and revised text data

[1400] Processing: Based on specific emotions, responses are automatically generated and then reconstructed using speech synthesis technology.

[1401] Specific operation: The server uses emotion-based response templates to generate appropriate response sentences, which are then converted into speech using a Text-to-Speech service.

[1402] Output: Response audio data file

[1403] Step 8:

[1404] Data storage and provision to users

[1405] The server stores all generated data and makes it accessible to users.

[1406] Input: Secure audio data (call_001_safe.wav), text data, and text data with sentiment information.

[1407] Processing: The server stores this data in a database or cloud storage and provides users with a URL or download link to access it.

[1408] Specific operation: The server uploads the file to the cloud storage service and generates a link.

[1409] Output: URLs for user access and download links

[1410] (Application Example 2)

[1411] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1412] In today's information society, the risk of leakage of personal and confidential information contained in voice data is increasing. In particular, call centers and security services need to understand and respond to the emotional states of customers and users, but the protection of personal information during this process is crucial. Furthermore, recognizing emotions from voice data and providing appropriate responses is a challenging task. To solve these problems, a system is needed that securely processes voice data, recognizes user emotions in real time, and provides appropriate responses.

[1413] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1414] In this invention, the server includes means for acquiring audio files from data storage, means for converting the acquired audio data into text data, means for identifying and extracting personal information from the text data, means for deleting or anonymizing the identified personal information, means for resynthesizing audio data using the revised text data, means for storing and providing the resynthesized audio data, means for recognizing and analyzing the user's emotions, means for generating a response based on the analysis results, and means for providing the generated response as resynthesized audio data. This enables the secure processing of personal information within the audio data and allows for appropriate responses according to the user's emotional state.

[1415] "Data storage" refers to devices or systems used to store digital data such as audio files.

[1416] An "audio file" is a digital file used to store audio data.

[1417] "Audio data" refers to data that records human voices and sounds in digital format.

[1418] "Text data" refers to digital data obtained by converting audio data into written text.

[1419] "Personal information" refers to information that can identify a specific individual, including names and addresses.

[1420] "Anonymization" is a process that makes it impossible to identify personal information.

[1421] "Speech recognition technology" is a technology that analyzes speech data and converts it into corresponding text data.

[1422] "Speech synthesis technology" is a technology for generating speech data based on text data.

[1423] "Recognizing and analyzing emotions" means identifying a speaker's emotions from audio or text data and analyzing their emotional state.

[1424] "Generating responses" means automatically generating appropriate dialogue content based on recognized emotions.

[1425] A "system" is a collection of devices and software that combine multiple means to achieve a specific function.

[1426] To implement this invention, a system program is constructed that follows the steps below. First, the server acquires audio data and converts it into text data. This process uses speech recognition technologies such as the Google Speech-to-Text API or Sphinx. Personal information is identified from the acquired text data and deleted or anonymized using regular expressions or the like. The deleted or anonymized text data is converted back into audio data using speech synthesis technologies such as the Google Text-to-Speech API and stored.

[1427] The server further uses the acquired voice and text data to recognize and analyze the user's emotions. This is done using an emotion recognition engine. For example, it identifies the user's emotional state by analyzing parameters such as voice tone, pitch, and speed. Based on this analysis, it uses a generative AI model to generate an appropriate response and provides the generated response as voice data.

[1428] The following is a natural language explanation of the processes performed by the server.

[1429] The server retrieves audio files from data storage. These audio files may include, for example, audio data from security sites. The retrieved audio data is converted into text information using the Google Speech-to-Text API. This converts the audio data into text data. Next, the server uses natural language processing techniques to identify personal information from the text data and removes or anonymizes it using regular expressions. The text data from which personal information has been removed is converted back into speech using the Google Text-to-Speech API and stored as newly generated audio data.

[1430] Furthermore, the server uses an emotion recognition engine to perform emotion analysis on the acquired audio data. It identifies the user's emotional state from the tone and pitch of their voice and assigns emotion tags to it. Based on these emotion tags, a generative AI model generates an appropriate response, which is then resynthesized as audio data.

[1431] As a concrete example, if this system were implemented in smart glasses used by security guards, the microphones in the smart glasses would collect audio from the scene and send it to a server. The server would then convert this audio data into text, remove or anonymize personal information, reconstruct it as speech, and perform emotion recognition. Finally, it would generate an appropriate response and provide it as audio, enabling security guards to respond appropriately on-site.

[1432] An example of a prompt message input to a generative AI model might be: "Convert the audio data to text and anonymize personal information. Furthermore, analyze the emotional state of the text and generate an appropriate response based on that emotion." Based on such prompt messages, the system will automatically perform the necessary processing.

[1433] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1434] Step 1:

[1435] Acquisition of audio data

[1436] The server retrieves an audio file from data storage. The input includes the path to the specified audio file. This data might be collected, for example, in a security setting. The retrieved audio file is then output.

[1437] Step 2:

[1438] Converting audio data to text

[1439] The server sends the acquired audio file to the Google Speech-to-Text API. The audio file is used as input. Speech recognition technology is used to convert the audio data into text data. This process outputs the text data.

[1440] Step 3:

[1441] Identification of personal information

[1442] The server analyzes the converted text data using natural language processing techniques. The input is text data. The text is tokenized, and personal information (e.g., name, address) is identified. The identified personal information is then output.

[1443] Step 4:

[1444] Deletion or anonymization of personal information

[1445] The server removes or anonymizes identified personal information from the text. Text data and identified personal information are used as input. Regular expressions are used to replace personal information with "" or "anonymous". This process outputs the revised text data.

[1446] Step 5:

[1447] Re-synthesis of audio data

[1448] The server sends the revised text data to the Google Text-to-Speech API, which then resynthesizes the audio data. The revised text data is included as input. Using speech synthesis technology, new audio data is generated. This audio data is then output.

[1449] Step 6:

[1450] Emotion recognition and analysis

[1451] The server inputs the acquired and converted audio data into an emotion recognition engine to analyze the user's emotions. Audio data is used as input. Parameters such as voice tone and pitch are analyzed to identify the emotional state (e.g., joy, anger). The emotional state is tagged and output.

[1452] Step 7:

[1453] Response generation

[1454] The server generates a response using a generative AI model based on the recognized emotional state. The emotional state is included as input. The generative AI model automatically generates the corresponding response according to the prompt. This response is then output.

[1455] Step 8:

[1456] Re-combining the response into audio data

[1457] The server sends the generated response to the Google Text-to-Speech API for resynthesis into audio data. The generated response is used as input. New response audio data is generated using speech synthesis technology. This audio data is the final output.

[1458] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1459] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1460] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1461] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1462] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1463] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1464] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1465] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1466] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1467] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1468] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1469] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1470] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1471] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1472] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1473] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1474] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1475] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1476] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1477] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1478] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1479] The following is further disclosed regarding the embodiments described above.

[1480] (Claim 1)

[1481] A means of retrieving data files from a database,

[1482] A means of converting acquired audio data into text data,

[1483] A means of identifying and extracting personal information from text data,

[1484] Means for deleting or replacing identified personal information,

[1485] A means of reconstructing audio data using revised text data,

[1486] Means for storing and providing reconstructed audio data,

[1487] A system that includes this.

[1488] (Claim 2)

[1489] The system according to claim 1, which converts text data after the identified personal information has been removed into speech data using speech synthesis technology.

[1490] (Claim 3)

[1491] The system according to claim 1, which converts acquired audio data into text data using speech recognition technology.

[1492] "Example 1"

[1493] (Claim 1)

[1494] A means of retrieving data files from a database,

[1495] A means of converting acquired audio data into text data using speech recognition technology,

[1496] A means including a generative artificial intelligence model that analyzes converted text data and identifies and extracts personal information,

[1497] Means for deleting or replacing identified personal information,

[1498] A means of reconstructing speech data using speech synthesis technology with revised text data,

[1499] Means for storing and providing reconstructed audio data,

[1500] A system that includes this.

[1501] (Claim 2)

[1502] The system according to claim 1, which converts text data after the identified personal information has been removed into speech data using speech synthesis technology.

[1503] (Claim 3)

[1504] The system according to claim 1, which converts acquired audio data into text data using speech recognition technology.

[1505] "Application Example 1"

[1506] (Claim 1)

[1507] A means of retrieving data files from data storage,

[1508] A means of converting acquired audio data into text data,

[1509] A means for identifying and extracting personal and confidential information from text data,

[1510] Means for deleting or replacing identified information,

[1511] A means of reconstructing audio data using revised text data,

[1512] Means for storing and providing reconstructed audio data,

[1513] A means of acquiring user voice in real time and detecting and replacing personal information,

[1514] A means of generating and providing clean audio data in real time,

[1515] A system that includes this.

[1516] (Claim 2)

[1517] The system according to claim 1, which converts text data after the identified personal information has been removed into speech data using speech synthesis technology.

[1518] (Claim 3)

[1519] The system according to claim 1, which converts acquired audio data into text data using speech recognition technology.

[1520] "Example 2 of combining an emotion engine"

[1521] (Claim 1)

[1522] A means of retrieving data files from a database,

[1523] A means of converting acquired audio data into text data,

[1524] A means of identifying and extracting personal information from text data,

[1525] Means for deleting or replacing identified personal information,

[1526] A means of reconstructing audio data using revised text data,

[1527] Means for storing and providing reconstructed audio data,

[1528] A method using an emotion engine to analyze the emotions of voice data,

[1529] A means for identifying a user's emotions and generating a response based on those emotions,

[1530] A system that includes this.

[1531] (Claim 2)

[1532] The system according to claim 1, which converts text data after the identified personal information has been removed into speech data using speech synthesis technology.

[1533] (Claim 3)

[1534] The system according to claim 1, which converts acquired audio data into text data using speech recognition technology.

[1535] "Application example 2 when combining with an emotional engine"

[1536] (Claim 1)

[1537] A means of obtaining audio files from data storage,

[1538] A means of converting acquired audio data into text data,

[1539] A means of identifying and extracting personal information from text data,

[1540] Means for deleting or anonymizing identified personal information,

[1541] A means for resynthesizing audio data using revised text data,

[1542] Means for storing and providing the resynthesized audio data,

[1543] A means of recognizing and analyzing user emotions,

[1544] A means for generating a response based on the analysis results,

[1545] A means of providing the generated response as resynthesized audio data,

[1546] A system that includes this.

[1547] (Claim 2)

[1548] The system according to claim 1, which converts text data, after the identified personal information has been deleted or anonymized, into speech data using speech synthesis technology.

[1549] (Claim 3)

[1550] The system according to claim 1, which converts collected audio data into text data using speech recognition technology. [Explanation of Symbols]

[1551] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of retrieving data files from a database, A means of converting acquired audio data into text data, A means of identifying and extracting personal information from text data, Means for deleting or replacing identified personal information, A means of reconstructing audio data using revised text data, Means for storing and providing reconstructed audio data, A system that includes this.

2. The system according to claim 1, wherein text data after the identified personal information has been deleted is converted into speech data using speech synthesis technology.

3. The system according to claim 1, which converts acquired audio data into text data using speech recognition technology.

Images