system

Abstract

Provide a system. 【Solution means】 Means for automatically responding when a communication device receives voice transmission, Means for converting the received voice into character information using voice recognition technology, Means for analyzing the content of the received voice using natural language processing technology and generating a response based on it, Means for detecting the possibility of improper behavior and issuing a warning, Means for notifying a designated external organization of the detected suspicious transaction, Means for confirming the content of a transaction using voice input and identifying the presence or absence of abnormalities, Means for executing a warning to the user based on the identified abnormality and providing information for reconfirming the transaction, A system including means for storing voice data and text data in an information storage area to enable later analysis and reference.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, in many households and small and medium-sized enterprises, telephone communication is still used as the main means. However, there is an increasing number of frauds against this communication means, and efficient responses are required. Especially in the case of the elderly and busy enterprises, there is a need for a system that can prevent fraud victims and effectively handle telephone calls. For such problems, existing solutions are limited, and many are expensive or lack effective detection and response technologies. Therefore, there is a need to provide a low-cost and high-precision telephone response system.

Means for Solving the Problems

[0005] To address the above challenges, this invention introduces a means for a communication device to automatically respond when it receives voice communication and convert the voice to text using speech recognition technology. This makes it possible to analyze the requirements of the received voice using natural language processing technology and generate a response. It also includes a means for detecting potential fraudulent activity, and if a suspicious call is detected, it sends a warning to a designated external organization. This system reduces the risk of fraud for the elderly and businesses, and enables efficient information sharing and improved operational efficiency. Furthermore, by transmitting the converted text data to an external device via the network, it is possible to promote the visualization and understanding of information and improve user convenience.

[0006] "Communication equipment" refers to equipment that includes hardware and software for receiving and processing voice communications.

[0007] "Speech recognition technology" is a technology that analyzes speech signals and converts them into text data.

[0008] "Natural language processing technology" is a technology that enables computers to understand, analyze, and process human language.

[0009] "Requirements" refer to the intentions or purposes that a user intends to convey within a specific voice communication.

[0010] "Potential fraudulent activity" refers to a situation that carries a risk of being judged as fraudulent based on the content and patterns of communications.

[0011] "External organizations" refer to public institutions such as the police, designated families, or corporate security departments.

[0012] A "warning" is a notice or message issued to alert someone to danger.

[0013] "Text data" refers to character information generated by speech recognition technology.

[0014] A "network" is a collection of connected computers and devices for exchanging data and enabling communication.

[0015] An "external device" is a device that is connected separately from a communication device and can receive or display information.

Brief Description of the Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. <000008 or more lines of text]]It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0020] In the following embodiments, the labeled RAM (Random Access Memory) is a memory where information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disk (e.g., hard disk), or magnetic tape, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is an AI-based communication system that streamlines telephone communication and enables the detection of fraudulent activity and warning of users. Applicable to home and small businesses, it consists of multiple components including a communication device, a speech recognition system, a natural language processing engine, and an alerting system.

[0038] Specific examples of implementation

[0039] For home use

[0040] When the server receives a voice call, it automatically detects the incoming call and plays an initial response message such as, "This is an automated answering system." This prepares the user to begin the conversation. When the user provides voice input, the terminal sends it to the speech recognition system in real time, converting the voice data into text. The server then uses natural language processing technology to analyze the text and determine the user's requirements. For example, if the requirements include "I would like to inquire about delivery," the system generates an appropriate response and provides the user with voice guidance.

[0041] Next, the server sends the converted text to the family's LINE app or a designated email address to share the content and make the information visible. Furthermore, if there are signs of fraud in the communication content, the server detects the possibility of fraudulent activity in real time and notifies the family or a designated external organization.

[0042] For small and medium-sized enterprises

[0043] In a corporate setting, when a server detects an incoming call, it plays a customized IVR menu and guides the user through the department selection process. Once the user makes a selection, the terminal routes the call to the appropriate department or contact person. Speech recognition technology is used to transcribe the conversation into text, which the server then sends to the communication platform or email system. Additionally, the call data is recorded by the server and stored in the cloud or database in a format that allows for later analysis and review.

[0044] This system not only streamlines communication in home and business environments, but also reduces the risk of fraud, allowing users to handle phone calls with peace of mind. Furthermore, voice-to-text conversion facilitates information sharing and understanding, promoting increased work efficiency.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The server automatically detects incoming voice calls and controls the communication device to answer. It plays a message saying, "This is an automated answering system. Please leave a message," and then initiates the call.

[0048] Step 2:

[0049] The user initiates a phone conversation and states their requirements and questions.

[0050] Step 3:

[0051] The device captures the user's voice and transmits the data to the speech recognition system in real time.

[0052] Step 4:

[0053] The server acquires the text data received from the speech recognition system and begins analysis using natural language processing techniques. This analysis identifies the requirements and intent of the received speech.

[0054] Step 5:

[0055] Based on the analysis results, the server generates a corresponding standard response and sends it to the user via speech synthesis. For example, in the case of a delivery confirmation request, it would return a response such as "The shipment is on track."

[0056] Step 6:

[0057] The device continuously transcribes all call content into text and sends the necessary information via the communication network to a designated external device (such as LINE or email).

[0058] Step 7:

[0059] The server monitors call content and detects patterns that may indicate fraud in real time. If a suspicious call is detected, it automatically notifies family members or external organizations.

[0060] Step 8:

[0061] The server records specific calls as needed and stores them in a database for analysis and review. In a business environment, it shares and routes communications to relevant departments and individuals.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] In modern society, despite the diversification and evolution of communication methods, balancing user convenience and security in voice communication remains a challenge. In particular, effective communication and prevention of fraudulent activities in automated responses are crucial. However, current technologies have been insufficient to achieve these goals in an integrated and efficient manner. Therefore, a system is needed to solve this problem and enable users to communicate with peace of mind.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for a communication device to detect an audio signal and automatically initiate a response; means for converting the received audio into a string using audio analysis technology; means for analyzing the content of the received audio using language understanding technology and determining a response based thereon; means for identifying signs of fraudulent activity and providing notification; and means for warning a designated external mechanism about the identified suspicious call. This enables efficient automatic response and improved security in voice communications.

[0067] "Communication equipment" refers to devices used for sending and receiving voice, and has the function of exchanging voice data via telephone or network.

[0068] An "audio signal" is a representation of sound as an electrical signal, and is a format for transmitting audio information.

[0069] "Speech analysis" refers to the technology of converting speech signals into text or numerical data, and is achieved using speech recognition technology.

[0070] A "string" is a data format consisting of a sequence of characters, and it is generated through speech analysis.

[0071] "Language understanding" refers to the technology of using natural language processing to understand the content of string data and analyze its intent and requirements.

[0072] "Fraudulent activity" refers to any act that deviates from the normally expected and legitimate communication behavior, such as fraud or unauthorized access.

[0073] "Notification" is the act of informing a specific person of information, and is done through voice or message.

[0074] An "external mechanism" refers to an organization or system located outside the system that is responsible for notification and information processing.

[0075] This invention is an AI-based communication system that efficiently processes voice communications and detects fraudulent activity. The system consists of various components and is suitable for both home and small business use.

[0076] The server first detects the audio signal received through the communication device. The server responds in real time and plays an initial message to the user stating, "This is an automated answering system." An audio playback device is used at this stage, and VoIP technology can be utilized.

[0077] The device converts voice input from the user into text using speech analysis technology. For example, it uses a speech recognition API to obtain what the user says as text data. This text data is then sent to the server.

[0078] The server uses a natural language processing engine to apply language understanding technology to the received string. This process allows it to analyze the user's request. Using a generative AI model, it automatically recognizes specific requests such as "I want to inquire about delivery" and generates an appropriate response.

[0079] Furthermore, the server implements algorithms to detect signs of fraudulent activity in real time, determining the possibility of fraud by identifying specific keywords and phrases. If a suspicious call is detected, it is notified to a designated external mechanism, which may include messaging services or email systems.

[0080] In this system, converted text is sent via LINE or email for information sharing within families and companies, and call content is stored in a database, thereby improving work efficiency and security.

[0081] As a concrete example of a prompt, you can input a question like, "I've been getting some strange phone calls lately; are they safe?" into the generating AI model. The system will then analyze the content of the call and provide feedback regarding its safety.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The server receives audio signals from communication devices and automatically detects incoming calls. Input can be analog or digital audio data. This audio data triggers the server to prepare to play a voice message stating, "This call is an automated response system." The output is the playback of an automated response voice message utilizing VoIP technology.

[0085] Step 2:

[0086] The device collects voice input from the user and sends it to a speech recognition system. The input is raw voice waveform data. This data is passed to a speech recognition API, where signal processing converts it into string data. The output is text data representing the content of the voice.

[0087] Step 3:

[0088] The server receives text data generated by speech recognition and analyzes it using a natural language processing engine. The input is the text data from step 2. Based on the keywords and context contained in this data, a generative AI model is used to understand the user's request and determine an appropriate response. The output is a response text that is based on the user's intent.

[0089] Step 4:

[0090] The server passes the response text generated in the previous step to the speech synthesis engine, where it is converted into audio data. The input is the generated response text. This is processed by the speech synthesis system and output as an audio file. Specifically, it is played back to the user through their speakers.

[0091] Step 5:

[0092] The server detects in real time whether there are signs of fraud in the communication content. The input is text data of the call content. This data is analyzed by an algorithm to identify keywords and patterns that indicate fraudulent activity. The output is flag information regarding the possibility of fraud, and a warning is sent to a designated external mechanism as needed.

[0093] Step 6:

[0094] The server sends the converted text data to family or corporate devices via a specified communication network. In this process, the input is the analyzed text data. The data is transmitted via services such as LINE or email using a transmission protocol. The output is message data that can be viewed by the receiver.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In recent years, fraudulent activity in electronic payments has been increasing, making it difficult for users to conduct secure transactions. Current technology presents a significant challenge in detecting fraudulent activity via voice communication in real time and providing appropriate warnings to users. Furthermore, there is a lack of systems that can efficiently verify transaction details via voice and quickly identify fraudulent activity.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for confirming the details of a transaction using voice input and identifying whether or not there is an anomaly, means for detecting the possibility of fraudulent activity and issuing a warning, and means for storing voice data and text data in an information storage area to enable later analysis and reference. This provides an electronic payment system with improved security, enabling users to conduct transactions with peace of mind.

[0100] "Communication equipment" refers to electronic devices that have the function of receiving and transmitting voice signals.

[0101] "Speech recognition technology" is a technology that analyzes speech information and converts it into text information.

[0102] "Textual information" refers to text data converted using speech recognition technology.

[0103] "Natural language processing technology" is a technology that analyzes speech and text composed of natural language to understand their meaning and intent.

[0104] "Fraudulent activity" refers to any activity or act that attempts to gain profit through improper means.

[0105] A "warning" is a notification that informs the user of a potential danger or problem.

[0106] An "external organization" is an organization or institution located outside the system.

[0107] "Voice input" is the process of receiving the voice spoken by the user as data.

[0108] "Identification" is the act of recognizing the properties and characteristics of an object and classifying or distinguishing it.

[0109] "Information storage area" refers to storage devices and databases used to securely record and store data.

[0110] A "transaction" refers to an activity related to the exchange or purchase of goods or services.

[0111] To implement this invention, a server is configured as part of an electronic payment system, including communication equipment, speech recognition technology, natural language processing technology, and a warning system. First, the server uses a smartphone or other suitable voice-enabled device as communication equipment to receive voice input. The voice received by this device is converted into text information by an API incorporating speech recognition technology (for example, Google® Speech-to-Text API).

[0112] The text information converted in real time is semantically analyzed using natural language processing technology to identify potential fraudulent activity. Ideally, a natural language processing engine such as the Google Cloud Natural Language API should be used for this analysis. If signs of fraud are detected, an alert system will send an alert to the user. This alert can be communicated to the user via a screen display or audio message.

[0113] The server also stores all audio data and its analysis results in an information storage area. This ensures that the data is securely stored for later analysis and can be referenced by users and administrators at a later date.

[0114] As a concrete example, when a user attempts to make an electronic payment, they may be asked to confirm via voice, "Is this payment trustworthy?" This system converts the voice into text, analyzes it, and determines the security of the transaction. Based on the results, it sends a warning to the user, protecting them from fraudulent transactions.

[0115] An example of a prompt to input into the generating AI model is, "Create a program that analyzes the possibility of fraudulent activity and generates a warning based on this voice instruction." This prompt guides the AI ​​model to correctly analyze the user's voice input and take appropriate action.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The server receives voice input from the terminal. During this process, the user speaks to confirm the transaction details. This voice data is then used as input for the speech recognition system.

[0119] Step 2:

[0120] The server uses speech recognition technology to convert the received audio data into text. At this stage, the audio data is analyzed using the Google Speech-to-Text API and output as text. As a result, the content of the audio is converted into text format.

[0121] Step 3:

[0122] The server uses natural language processing technology to analyze text information. Here, the Google Cloud Natural Language API is used to process the data in order to understand the meaning and intent of the text. The analysis results output data that evaluates whether or not there are signs of fraudulent activity.

[0123] Step 4:

[0124] Based on the analysis results, the server will warn the user if there is a possibility of fraudulent activity. This warning will be communicated to the user via audio or on-screen display. The user can then re-evaluate the transaction based on this information and cancel it if necessary.

[0125] Step 5:

[0126] The server stores all audio data and analysis results in an information storage area. This data is recorded in a database in a secure and efficient manner. The stored data will later be available for analysis and reporting.

[0127] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0128] This invention provides a system that, in voice communication with a user, not only converts voice content into text and understands its content, but also analyzes the user's emotions to provide more appropriate responses and warnings. This system consists of a communication device, a voice recognition system, a natural language processing engine, an emotion engine, and a warning and notification system.

[0129] Specific examples of implementation

[0130] For home use

[0131] The server detects an incoming voice call and plays a message such as, "This is an automated answering system. Please tell us what you need to do." When the user begins speaking, the terminal inputs the user's voice into the speech recognition system in real time and converts it to text. Simultaneously, the emotion engine analyzes the voice data to identify the user's emotions.

[0132] The server analyzes the content of text data using natural language processing to recognize the requirements. For example, if a complaint about customer service is expressed, the emotion engine detects emotions such as dissatisfaction or anger. Based on this information, the server generates a faster and more courteous response than usual and takes appropriate action.

[0133] Furthermore, the server can manage user history by saving emotion recognition results to a database, which can then be used for future analysis and service improvements. If the emotion engine detects anger or anxiety exceeding a certain level, the server immediately notifies designated family members and, if necessary, sets up a three-way call to ensure peace of mind within the family.

[0134] For small and medium-sized enterprises

[0135] In a corporate setting, a server answers incoming calls and plays a customized IVR menu. Once the user has made their selection, the terminal converts the voice to text and simultaneously performs sentiment analysis using an emotion engine. The analysis results determine the quality of the response generated by the server, providing thoughtful customer service based on emotions.

[0136] In addition to the normal workflow, the server can store the results of emotion analysis as call data, which can be used to improve security and satisfaction. In the event that high-risk emotions are detected, an alert will be sent to the designated security department or administrator to ensure smooth operations within the company.

[0137] In this way, the present invention can improve the quality of service and enhance user safety and satisfaction by comprehensively analyzing and responding to user emotions and requirements.

[0138] The following describes the processing flow.

[0139] Step 1:

[0140] The server detects an incoming voice call, plays an automated response message, and initiates the call.

[0141] Step 2:

[0142] The user initiates a phone conversation and states their requirements and questions.

[0143] Step 3:

[0144] The device captures voice data from the user and sends it to a speech recognition system in real time to be converted into text data.

[0145] Step 4:

[0146] The server receives text data from the speech recognition system, analyzes this text using natural language processing technology, and identifies the user's requirements.

[0147] Step 5:

[0148] The device activates its emotion engine and simultaneously analyzes the received audio data to identify the user's emotions.

[0149] Step 6:

[0150] The server generates an appropriate response based on the analyzed requirements and sentiment information. For example, if the sentiment engine detects user dissatisfaction, it will provide a more courteous response.

[0151] Step 7:

[0152] The terminal generates the response using speech synthesis and plays it back to the user.

[0153] Step 8:

[0154] The server stores the transcribed call content and emotion recognition results in a database, managing them in a format that allows for later searching and analysis.

[0155] Step 9:

[0156] If the emotion engine detects severe stress or anger, the server will send an alert notification to designated external devices or family members and, if necessary, set up a three-way call with those parties.

[0157] (Example 2)

[0158] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0159] In conventional systems receiving voice communications on communication networks, the system simply converted the voice to text and analyzed the requirements, making it difficult to understand the user's emotions and provide an appropriate response. This has created a need to improve service quality and increase user satisfaction. Furthermore, in situations where emotional changes are a crucial indicator, there was a lack of a mechanism to respond quickly and notify the appropriate person of the necessary information.

[0160] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0161] In this invention, the server includes means for automatically responding when a communication device receives voice communication, means for converting the received voice into text using voice recognition technology, means for analyzing the voice data to identify the user's emotions, means for adjusting the quality of the response based on the results of the emotion analysis, and means for sending notifications to designated contacts according to the identified emotions. This enables flexible and high-quality responses that take into account the user's emotions, and allows for prompt responses by immediately sending notifications to the appropriate person as needed.

[0162] A "communication device" is a device that receives voice communications and enables voice communication with the user.

[0163] "Speech recognition technology" is a technology that converts speech data into text data.

[0164] "Natural language processing technology" is a technology that analyzes meaning from text data and identifies requirements.

[0165] "Emotion analysis" is the process of identifying a user's emotions based on voice data and text data.

[0166] "Response generation" is the process of creating an appropriate response based on analyzed requirements and emotions.

[0167] "Notification transmission" is the process of sending information to a designated contact point when certain conditions are met.

[0168] This invention is a system for analyzing user emotions in voice communication and providing appropriate responses. When a server receives voice communication via a communication device, it activates an automated response system and plays a predetermined message to the user. When the user begins to respond, the terminal converts the voice into text in real time using speech recognition technology. Specifically, it converts the voice data into text using an existing service such as Google Cloud Speech-to-Text.

[0169] Simultaneously, the terminal inputs voice data into a dedicated emotion engine to perform emotion analysis and identify the user's emotions. This engine analyzes and identifies emotions based on voice tone, speed, and word choice. The server uses natural language processing technology to analyze text data and process it to understand the user's intentions. Generative AI models such as OpenAI® GPT-3® are used to extract requirements from the text data and generate responses.

[0170] The server generates responses that are sensitive to the user's emotions, based on sentiment analysis and content analysis. If a user expresses dissatisfaction with the service, it provides a polite and prompt response according to a pre-configured response policy. Furthermore, by saving the analysis results to a database, the server can manage past communication history and use it for future service improvements and analysis.

[0171] As a concrete example, consider its use within a home. The server has a function that sends notifications to designated relatives or friends if it detects anger or anxiety in the user. This helps to maintain a sense of security within the home. An example of a prompt would be, "Please describe in natural language how the sentiment analysis system works in customer phone calls. Please provide specific examples for home and small businesses."

[0172] In this way, the invention can accurately grasp the user's emotions and enable high-quality communication.

[0173] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0174] Step 1:

[0175] The server receives an incoming voice call via a communication device. The automated response system activates and plays the message, "This is an automated response system. Please leave your message," to the user. The input is the incoming voice call, and the output is the played automated response message.

[0176] Step 2:

[0177] When a user begins to respond by voice, the device collects the audio through its microphone. This audio data is then input into a speech recognition system, which converts the speech to text. The input is the user's voice, and the output is the converted text data. This conversion uses speech recognition technology such as Google Cloud Speech-to-Text.

[0178] Step 3:

[0179] The device simultaneously inputs the collected audio data into an emotion engine to analyze the user's emotions. The input is the user's audio data, and the output is the identified emotion. Emotion analysis is performed based on voice tone, speed, and word choice.

[0180] Step 4:

[0181] The server uses natural language processing techniques to analyze text data and extract user requirements. The input is text data, and the output is information indicating the user's requirements. Generative AI models such as OpenAI GPT-3 are used for the analysis.

[0182] Step 5:

[0183] The server integrates sentiment analysis results and content analysis results to generate a response that takes the user's emotions into consideration. The input consists of identified emotions and requirement information, while the output is the generated response text. A pre-configured response policy may also be applied to the response.

[0184] Step 6:

[0185] The server plays the generated response as audio to the user. Furthermore, it saves the analysis results to a database. The input is the generated response text, and the output is the played audio and the saved data. This data will be used for future service improvements and history management.

[0186] Step 7:

[0187] The server, when its emotion engine detects a high level of anger or anxiety, sends a notification to designated contacts to ensure safety within homes and businesses. The input is the detection of high-risk emotions, and the output is the sent notification. The notification provides instructions and information as needed.

[0188] (Application Example 2)

[0189] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0190] In recent years, advancements in voice communication technology have created a need for systems that not only understand the content of conversations with users but also analyze their emotions to provide more appropriate responses. However, current technologies have not adequately achieved flexible responses and warning systems that take emotions into account. Furthermore, there is a lack of mechanisms to immediately notify appropriate external organizations in the event of sudden changes in emotions or fraudulent activity.

[0191] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0192] In this invention, the server includes means for converting received speech into text information using speech recognition technology, means for analyzing the requirements of the received speech using natural language processing technology and generating a response based thereon, and means for identifying the user's emotions from the speech data using emotion analysis technology. This makes it possible to analyze the user's emotions in voice dialogue and provide appropriate responses or warnings. It also makes it possible to detect the possibility of fraudulent activity and quickly notify a designated external organization if the emotions exceed a certain threshold.

[0193] "Communication technology" refers to technology that receives voice information from users and automatically responds based on that information.

[0194] "Speech recognition technology" is a technology that converts received speech into text information and uses it for analysis.

[0195] "Natural language processing technology" is a technology that understands the requirements of the textual information analyzed from received speech and generates an appropriate response.

[0196] "Emotion analysis technology" is a technology that identifies and analyzes a user's emotions from voice data.

[0197] "Potential fraudulent activity" refers to a situation where there is a possibility of fraud or misconduct occurring in the content of a received communication.

[0198] An "external agency" is an external organization or facility established to notify information when emotional or risky communications exceeding specific criteria are detected.

[0199] One embodiment of this invention is a voice communication system that analyzes speech through interaction with a user and provides an emotionally appropriate response. The server uses speech recognition software to convert speech data into text information in real time. This is done using the Python speech_recognition library. The converted text information is analyzed using natural language processing techniques to generate a response.

[0200] The server simultaneously identifies emotions from the audio data using emotion analysis technology. This emotion analysis utilizes generative AI models to detect specific emotional states. For example, when it receives audio from a user expressing anxiety, it can quickly recognize that emotion and issue an alert to prompt appropriate action.

[0201] As a concrete example, if a user says, "I'm feeling a little anxious today," the server converts the audio into text data using speech recognition technology and understands the content through natural language processing. Simultaneously, it uses sentiment analysis technology to recognize the user's emotion of anxiety. Based on this, the server generates a response appropriate to the emotion and, for example, sends a notification to a caregiver. An example prompt might look like this: "Please analyze the emotions in the following conversation. If the user seems restless, please advise how the caregiver should respond. Conversation content: 'I didn't sleep well last night... I'm feeling a little anxious today.'"

[0202] This configuration allows voice communication systems to respond flexibly, taking into account the user's emotions. It enables rapid detection of anticipated risks and fraudulent activities, and allows for countermeasures such as notification to external organizations as needed.

[0203] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0204] Step 1:

[0205] The server receives voice information from the user using communication technology. The user's voice data is provided to the server as input. The server then passes this data directly to the next process.

[0206] Step 2:

[0207] The server uses speech recognition software to convert received audio information into text. The input is audio data received from the user, and the output is text. This process converts the audio data into parseable text. Specifically, the Python `speech_recognition` library is used to automatically convert speech to text.

[0208] Step 3:

[0209] The server applies natural language processing techniques to the converted text information to analyze the user's requirements. The input is the text information obtained in step 2, and the output is the analyzed requirements. This process prepares the server to understand the user's intent and generate a specific response. The response content is then designed based on the generated requirements.

[0210] Step 4:

[0211] The server uses emotion analysis technology to identify the user's emotions from the received audio data. The input is again the original audio data, and the output is the identified emotional state. In this step, a generative AI model is used to analyze the emotions and detect specific emotions such as anxiety or relief. Based on the analysis results, the system is used to implement flexible responses that correspond to the emotions.

[0212] Step 5:

[0213] The server generates an appropriate notification and sends it to an external organization or designated device if the analyzed emotion meets certain criteria. The input is the emotional state obtained in step 4, and the output is the generated notification message. This process ensures that information is quickly shared with relevant parties when a specific emotional state is detected. Specifically, if the criteria are exceeded, the system triggers an alert and sends a real-time notification to the caregiver or designated external organization.

[0214] By performing these steps sequentially, an advanced voice communication system is realized that incorporates emotion analysis into voice dialogue, providing appropriate responses and warnings.

[0215] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0216] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0217] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0218] [Second Embodiment]

[0219] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0220] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0221] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0222] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0223] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0224] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0225] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0226] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0227] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0228] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0229] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0230] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0231] This invention is an AI-based communication system that streamlines telephone communication and enables the detection of fraudulent activity and warning of users. Applicable to home and small businesses, it consists of multiple components including a communication device, a speech recognition system, a natural language processing engine, and an alerting system.

[0232] Specific examples of implementation

[0233] For home use

[0234] When the server receives a voice call, it automatically detects the incoming call and plays an initial response message such as, "This is an automated answering system." This prepares the user to begin the conversation. When the user provides voice input, the terminal sends it to the speech recognition system in real time, converting the voice data into text. The server then uses natural language processing technology to analyze the text and determine the user's requirements. For example, if the requirements include "I would like to inquire about delivery," the system generates an appropriate response and provides the user with voice guidance.

[0235] Next, the server sends the converted text to the family's LINE app or a designated email address to share the content and make the information visible. Furthermore, if there are signs of fraud in the communication content, the server detects the possibility of fraudulent activity in real time and notifies the family or a designated external organization.

[0236] For small and medium-sized enterprises

[0237] In a corporate setting, when a server detects an incoming call, it plays a customized IVR menu and guides the user through the department selection process. Once the user makes a selection, the terminal routes the call to the appropriate department or contact person. Speech recognition technology is used to transcribe the conversation into text, which the server then sends to the communication platform or email system. Additionally, the call data is recorded by the server and stored in the cloud or database in a format that allows for later analysis and review.

[0238] This system not only streamlines communication in home and business environments, but also reduces the risk of fraud, allowing users to handle phone calls with peace of mind. Furthermore, voice-to-text conversion facilitates information sharing and understanding, promoting increased work efficiency.

[0239] The following describes the processing flow.

[0240] Step 1:

[0241] The server automatically detects incoming voice calls and controls the communication device to answer. It plays a message saying, "This is an automated answering system. Please leave a message," and then initiates the call.

[0242] Step 2:

[0243] The user initiates a phone conversation and states their requirements and questions.

[0244] Step 3:

[0245] The device captures the user's voice and transmits the data to the speech recognition system in real time.

[0246] Step 4:

[0247] The server retrieves the text data received from the speech recognition system and begins analysis using natural language processing techniques. This analysis identifies the requirements and intent of the received speech.

[0248] Step 5:

[0249] Based on the analysis results, the server generates a corresponding standard response and sends it to the user via speech synthesis. For example, in the case of a delivery confirmation request, it would return a response such as "The shipment is on track."

[0250] Step 6:

[0251] The device continuously transcribes all call content into text and sends the necessary information via the communication network to a designated external device (such as LINE or email).

[0252] Step 7:

[0253] The server monitors call content and detects patterns that may indicate fraud in real time. If a suspicious call is detected, it automatically notifies family members or external organizations.

[0254] Step 8:

[0255] The server records specific calls as needed and stores them in a database for analysis and review. In a business environment, it shares and routes communications to relevant departments and individuals.

[0256] (Example 1)

[0257] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0258] In modern society, despite the diversification and evolution of communication methods, balancing user convenience and security in voice communication remains a challenge. In particular, effective communication and prevention of fraudulent activities in automated responses are crucial. However, current technologies have been insufficient to achieve these goals in an integrated and efficient manner. Therefore, a system is needed to solve this problem and enable users to communicate with peace of mind.

[0259] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0260] In this invention, the server includes means for a communication device to detect an audio signal and automatically initiate a response; means for converting the received audio into a string using audio analysis technology; means for analyzing the content of the received audio using language understanding technology and determining a response based thereon; means for identifying signs of fraudulent activity and providing notification; and means for warning a designated external mechanism about the identified suspicious call. This enables efficient automatic response and improved security in voice communications.

[0261] "Communication equipment" refers to devices used for sending and receiving voice, and has the function of exchanging voice data via telephone or network.

[0262] An "audio signal" is a representation of sound as an electrical signal, and is a format for transmitting audio information.

[0263] "Speech analysis" refers to the technology of converting speech signals into text or numerical data, and is achieved using speech recognition technology.

[0264] A "string" is a data format consisting of a sequence of characters, and it is generated through speech analysis.

[0265] "Language understanding" refers to the technology of using natural language processing to understand the content of string data and analyze its intent and requirements.

[0266] "Fraudulent activity" refers to any act that deviates from the normally expected and legitimate communication behavior, such as fraud or unauthorized access.

[0267] "Notification" is the act of informing a specific person of information, and is done through voice or message.

[0268] An "external mechanism" refers to an organization or system located outside the system that is responsible for notification and information processing.

[0269] This invention is an AI-based communication system that efficiently processes voice communications and detects fraudulent activity. The system consists of various components and is suitable for both home and small business use.

[0270] The server first detects the audio signal received through the communication device. The server responds in real time and plays an initial message to the user stating, "This is an automated answering system." An audio playback device is used at this stage, and VoIP technology can be utilized.

[0271] The device converts voice input from the user into text using speech analysis technology. For example, it uses a speech recognition API to obtain what the user says as text data. This text data is then sent to the server.

[0272] The server uses a natural language processing engine to apply language understanding technology to the received string. This process allows it to analyze the user's request. Using a generative AI model, it automatically recognizes specific requests such as "I want to inquire about delivery" and generates an appropriate response.

[0273] Furthermore, the server implements algorithms to detect signs of fraudulent activity in real time, determining the possibility of fraud by identifying specific keywords and phrases. If a suspicious call is detected, it is notified to a designated external mechanism, which may include messaging services or email systems.

[0274] In this system, converted text is sent via LINE or email for information sharing within families and companies, and call content is stored in a database, thereby improving work efficiency and security.

[0275] As a concrete example of a prompt, you can input a question like, "I've been getting some strange phone calls lately; are they safe?" into the generating AI model. The system will then analyze the content of the call and provide feedback regarding its safety.

[0276] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0277] Step 1:

[0278] The server receives an audio signal from the communication device and automatically detects an incoming call. The inputs include analog or digital audio data. Using this audio data as a trigger, the server prepares to play an audio message saying "This call is an automated response system". As an output, an automated response audio message is played using VoIP technology.

[0279] Step 2:

[0280] The terminal collects voice input from the user and sends it to the speech recognition system. The input includes raw audio waveform data. This data is passed to the speech recognition API and converted into string data by performing signal processing. As an output, text data representing the content of the voice is generated.

[0281] Step 3:

[0282] The server receives the text data generated by speech recognition and analyzes it using a natural language processing engine. The input is the text data from Step 2. Based on the keywords and context contained in this data, a generative AI model is used to understand the user's request and determine an appropriate response. As an output, response text based on the user's intention is generated.

[0283] Step 4:

[0284] The server passes the response text generated in the previous step to the text-to-speech engine and converts it into audio data. The input is the generated response text. This is processed by the text-to-speech system and output as an audio file. Specifically, it is played through a speaker so that it can be heard by the user.

[0285] Step 5:

[0286] The server detects in real time whether there are signs of fraud in the communication content. The input is the text data of the call content. This data is analyzed by an algorithm to identify keywords and patterns indicating fraudulent behavior. As output, flag information regarding the likelihood of fraud is generated, and a warning is sent to a specified external agency as necessary.

[0287] Step 6:

[0288] The server sends the converted text data to devices within the family or enterprise via a specified communication network. In this process, the input is the analyzed text data. The data is sent via services such as LINE or email using a transmission protocol. As output, message data that can be confirmed by the receiver is distributed.

[0289] (Application Example 1)

[0290] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0291] In recent years, illegal acts in electronic payments have been increasing, making it difficult for users to conduct safe transactions. With current technology, there is a problem that it is extremely difficult to detect illegal acts in voice communication in real time and give appropriate warnings to users. Also, there is a lack of a system that can efficiently confirm the content of transactions by voice and quickly identify fraud, which has become an issue.

[0292] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0293] In this invention, the server includes means for confirming the details of a transaction using voice input and identifying whether or not there is an anomaly, means for detecting the possibility of fraudulent activity and issuing a warning, and means for storing voice data and text data in an information storage area to enable later analysis and reference. This provides an electronic payment system with improved security, enabling users to conduct transactions with peace of mind.

[0294] "Communication equipment" refers to electronic devices that have the function of receiving and transmitting voice signals.

[0295] "Speech recognition technology" is a technology that analyzes speech information and converts it into text information.

[0296] "Textual information" refers to text data converted using speech recognition technology.

[0297] "Natural language processing technology" is a technology that analyzes speech and text composed of natural language to understand their meaning and intent.

[0298] "Fraudulent activity" refers to any activity or act that attempts to gain profit through improper means.

[0299] A "warning" is a notification that informs the user of a potential danger or problem.

[0300] An "external organization" is an organization or institution located outside the system.

[0301] "Voice input" is the process of receiving the voice spoken by the user as data.

[0302] "Identification" is the act of recognizing the properties and characteristics of an object and classifying or distinguishing it.

[0303] "Information storage area" refers to storage devices and databases used to securely record and store data.

[0304] "Transaction" refers to activities related to the exchange or purchase of goods and services.

[0305] To implement this invention, as part of an electronic payment system, a server is configured that includes communication devices, speech recognition technology, natural language processing technology, and a warning system. First, the server uses a smartphone or other suitable voice-enabled device as a communication device for receiving voice input. The voice received by this device is converted into character information by an API that incorporates speech recognition technology (e.g., Google Speech-to-Text API).

[0306] The character information converted in real-time is semantically analyzed using natural language processing technology to identify the possibility of fraudulent behavior. It is desirable to use a natural language processing engine such as Google Cloud Natural Language API for the analysis. If signs of fraud are detected, an alert is sent to the user using the warning system. This warning can be notified to the user as a screen display or a voice message.

[0307] The server also stores all voice data and its analysis results in an information storage area. This securely stores the data for later analysis and allows users and administrators to refer to it later.

[0308] As a specific example, when a user attempts to make an electronic payment, they can request a voice confirmation such as "Is this payment reliable?" This system converts the voice into text, analyzes it, and determines the security of the transaction. Based on the results, a warning is sent to the user to protect the user from fraudulent transactions.

[0309] An example of a prompt sentence to input into the generative AI model is "Based on this voice instruction, create a program that analyzes the possibility of fraud and generates a warning." This prompt sentence enables the AI model to correctly analyze the user's voice input and provide a guide for taking appropriate actions.

[0310] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0311] Step 1:

[0312] The server receives voice input from the terminal. During this process, the user speaks to confirm the transaction details. This voice data is then used as input for the speech recognition system.

[0313] Step 2:

[0314] The server uses speech recognition technology to convert the received audio data into text. At this stage, the audio data is analyzed using the Google Speech-to-Text API and output as text. As a result, the content of the audio is converted into text format.

[0315] Step 3:

[0316] The server uses natural language processing technology to analyze text information. Here, the Google Cloud Natural Language API is used to process the data in order to understand the meaning and intent of the text. The analysis results output data that evaluates whether or not there are signs of fraudulent activity.

[0317] Step 4:

[0318] Based on the analysis results, the server will warn the user if there is a possibility of fraudulent activity. This warning will be communicated to the user via audio or on-screen display. The user can then re-evaluate the transaction based on this information and cancel it if necessary.

[0319] Step 5:

[0320] The server stores all audio data and analysis results in an information storage area. This data is recorded in a database in a secure and efficient manner. The stored data will later be available for analysis and reporting.

[0321] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0322] This invention provides a system that, in voice communication with a user, not only converts voice content into text and understands its content, but also analyzes the user's emotions to provide more appropriate responses and warnings. This system consists of a communication device, a voice recognition system, a natural language processing engine, an emotion engine, and a warning and notification system.

[0323] Specific examples of implementation

[0324] For home use

[0325] The server detects an incoming voice call and plays a message such as, "This is an automated answering system. Please tell us what you need to do." When the user begins speaking, the terminal inputs the user's voice into the speech recognition system in real time and converts it to text. Simultaneously, the emotion engine analyzes the voice data to identify the user's emotions.

[0326] The server analyzes the content of text data using natural language processing to recognize the requirements. For example, if a complaint about customer service is expressed, the emotion engine detects emotions such as dissatisfaction or anger. Based on this information, the server generates a faster and more courteous response than usual and takes appropriate action.

[0327] Furthermore, the server can manage user history by saving emotion recognition results to a database, which can then be used for future analysis and service improvements. If the emotion engine detects anger or anxiety exceeding a certain level, the server immediately notifies designated family members and, if necessary, sets up a three-way call to ensure peace of mind within the family.

[0328] For small and medium-sized enterprises

[0329] In a corporate setting, a server answers incoming calls and plays a customized IVR menu. Once the user has made their selection, the terminal converts the voice to text and simultaneously performs sentiment analysis using an emotion engine. The analysis results determine the quality of the response generated by the server, providing thoughtful customer service based on emotions.

[0330] In addition to the normal workflow, the server can store the results of emotion analysis as call data, which can be used to improve security and satisfaction. In the event that high-risk emotions are detected, an alert will be sent to the designated security department or administrator to ensure smooth operations within the company.

[0331] In this way, the present invention can improve the quality of service and enhance user safety and satisfaction by comprehensively analyzing and responding to user emotions and requirements.

[0332] The following describes the processing flow.

[0333] Step 1:

[0334] The server detects an incoming voice call, plays an automated response message, and initiates the call.

[0335] Step 2:

[0336] The user initiates a phone conversation and states their requirements and questions.

[0337] Step 3:

[0338] The device captures voice data from the user and sends it to a speech recognition system in real time to be converted into text data.

[0339] Step 4:

[0340] The server receives text data from the speech recognition system, analyzes this text using natural language processing technology, and identifies the user's requirements.

[0341] Step 5:

[0342] The device activates its emotion engine and simultaneously analyzes the received audio data to identify the user's emotions.

[0343] Step 6:

[0344] The server generates an appropriate response based on the analyzed requirements and sentiment information. For example, if the sentiment engine detects user dissatisfaction, it will provide a more courteous response.

[0345] Step 7:

[0346] The terminal generates the response using speech synthesis and plays it back to the user.

[0347] Step 8:

[0348] The server stores the transcribed call content and emotion recognition results in a database, managing them in a format that allows for later searching and analysis.

[0349] Step 9:

[0350] If the emotion engine detects severe stress or anger, the server will send an alert notification to designated external devices or family members and, if necessary, set up a three-way call with those parties.

[0351] (Example 2)

[0352] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0353] In conventional systems receiving voice communications on communication networks, the system simply converted the voice to text and analyzed the requirements, making it difficult to understand the user's emotions and provide an appropriate response. This has created a need to improve service quality and increase user satisfaction. Furthermore, in situations where emotional changes are a crucial indicator, there was a lack of a mechanism to respond quickly and notify the appropriate person of the necessary information.

[0354] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0355] In this invention, the server includes means for automatically responding when a communication device receives voice communication, means for converting the received voice into text using voice recognition technology, means for analyzing the voice data to identify the user's emotions, means for adjusting the quality of the response based on the results of the emotion analysis, and means for sending notifications to designated contacts according to the identified emotions. This enables flexible and high-quality responses that take into account the user's emotions, and allows for prompt responses by immediately sending notifications to the appropriate person as needed.

[0356] A "communication device" is a device that receives voice communications and enables voice communication with the user.

[0357] "Speech recognition technology" is a technology that converts speech data into text data.

[0358] "Natural language processing technology" is a technology that analyzes meaning from text data and identifies requirements.

[0359] "Emotion analysis" is the process of identifying a user's emotions based on voice data and text data.

[0360] "Response generation" is the process of creating an appropriate response based on analyzed requirements and emotions.

[0361] "Notification transmission" is the process of sending information to a designated contact point when certain conditions are met.

[0362] This invention is a system for analyzing user emotions in voice communication and providing appropriate responses. When a server receives voice communication via a communication device, it activates an automated response system and plays a predetermined message to the user. When the user begins to respond, the terminal converts the voice into text in real time using speech recognition technology. Specifically, it converts the voice data into text using an existing service such as Google Cloud Speech-to-Text.

[0363] Simultaneously, the terminal inputs voice data into a dedicated emotion engine to perform emotion analysis and identify the user's emotions. This engine analyzes and identifies emotions based on voice tone, speed, and word choice. The server uses natural language processing techniques to analyze text data and process it to understand the user's intentions. Generative AI models such as OpenAI GPT-3 are used to extract requirements from the text data and generate responses.

[0364] The server generates responses that are sensitive to the user's emotions, based on sentiment analysis and content analysis. If a user expresses dissatisfaction with the service, it provides a polite and prompt response according to a pre-configured response policy. Furthermore, by saving the analysis results to a database, the server can manage past communication history and use it for future service improvements and analysis.

[0365] As a concrete example, consider its use within a home. The server has a function that sends notifications to designated relatives or friends if it detects anger or anxiety in the user. This helps to maintain a sense of security within the home. An example of a prompt would be, "Please describe in natural language how the sentiment analysis system works in customer phone calls. Please provide specific examples for home and small businesses."

[0366] In this way, the invention can accurately grasp the user's emotions and enable high-quality communication.

[0367] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0368] Step 1:

[0369] The server receives an incoming voice call via a communication device. The automated response system activates and plays the message, "This is an automated response system. Please leave your message," to the user. The input is the incoming voice call, and the output is the played automated response message.

[0370] Step 2:

[0371] When a user begins to respond by voice, the device collects the audio through its microphone. This audio data is then input into a speech recognition system, which converts the speech to text. The input is the user's voice, and the output is the converted text data. This conversion uses speech recognition technology such as Google Cloud Speech-to-Text.

[0372] Step 3:

[0373] The device simultaneously inputs the collected audio data into an emotion engine to analyze the user's emotions. The input is the user's audio data, and the output is the identified emotion. Emotion analysis is performed based on voice tone, speed, and word choice.

[0374] Step 4:

[0375] The server uses natural language processing techniques to analyze text data and extract user requirements. The input is text data, and the output is information indicating the user's requirements. Generative AI models such as OpenAI GPT-3 are used for the analysis.

[0376] Step 5:

[0377] The server integrates sentiment analysis results and content analysis results to generate a response that takes the user's emotions into consideration. The input consists of identified emotions and requirement information, while the output is the generated response text. A pre-configured response policy may also be applied to the response.

[0378] Step 6:

[0379] The server plays the generated response as audio to the user. Furthermore, it saves the analysis results to a database. The input is the generated response text, and the output is the played audio and the saved data. This data will be used for future service improvements and history management.

[0380] Step 7:

[0381] The server, when its emotion engine detects a high level of anger or anxiety, sends a notification to designated contacts to ensure safety within homes and businesses. The input is the detection of high-risk emotions, and the output is the sent notification. The notification provides instructions and information as needed.

[0382] (Application Example 2)

[0383] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0384] In recent years, advancements in voice communication technology have created a need for systems that not only understand the content of conversations with users but also analyze their emotions to provide more appropriate responses. However, current technologies have not adequately achieved flexible responses and warning systems that take emotions into account. Furthermore, there is a lack of mechanisms to immediately notify appropriate external organizations in the event of sudden changes in emotions or fraudulent activity.

[0385] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0386] In this invention, the server includes means for converting received speech into text information using speech recognition technology, means for analyzing the requirements of the received speech using natural language processing technology and generating a response based thereon, and means for identifying the user's emotions from the speech data using emotion analysis technology. This makes it possible to analyze the user's emotions in voice dialogue and provide appropriate responses or warnings. It also makes it possible to detect the possibility of fraudulent activity and quickly notify a designated external organization if the emotions exceed a certain threshold.

[0387] "Communication technology" refers to technology that receives voice information from users and automatically responds based on that information.

[0388] "Speech recognition technology" is a technology that converts received speech into text information and uses it for analysis.

[0389] "Natural language processing technology" is a technology that understands the requirements of the textual information analyzed from received speech and generates an appropriate response.

[0390] "Emotion analysis technology" is a technology that identifies and analyzes a user's emotions from voice data.

[0391] "Potential fraudulent activity" refers to a situation where there is a possibility of fraud or misconduct occurring in the content of a received communication.

[0392] An "external agency" is an external organization or facility established to notify information when emotional or risky communications exceeding specific criteria are detected.

[0393] One embodiment of this invention is a voice communication system that analyzes speech through interaction with a user and provides an emotionally appropriate response. The server uses speech recognition software to convert speech data into text information in real time. This is done using the Python speech_recognition library. The converted text information is analyzed using natural language processing techniques to generate a response.

[0394] The server simultaneously identifies emotions from the audio data using emotion analysis technology. This emotion analysis utilizes generative AI models to detect specific emotional states. For example, when it receives audio from a user expressing anxiety, it can quickly recognize that emotion and issue an alert to prompt appropriate action.

[0395] As a concrete example, if a user says, "I'm feeling a little anxious today," the server converts the audio into text data using speech recognition technology and understands the content through natural language processing. Simultaneously, it uses sentiment analysis technology to recognize the user's emotion of anxiety. Based on this, the server generates a response appropriate to the emotion and, for example, sends a notification to a caregiver. An example prompt might look like this: "Please analyze the emotions in the following conversation. If the user seems restless, please advise how the caregiver should respond. Conversation content: 'I didn't sleep well last night... I'm feeling a little anxious today.'"

[0396] This configuration allows voice communication systems to respond flexibly, taking into account the user's emotions. It enables rapid detection of anticipated risks and fraudulent activities, and allows for countermeasures such as notification to external organizations as needed.

[0397] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0398] Step 1:

[0399] The server receives voice information from the user using communication technology. The user's voice data is provided to the server as input. The server then passes this data directly to the next process.

[0400] Step 2:

[0401] The server uses speech recognition software to convert received audio information into text. The input is audio data received from the user, and the output is text. This process converts the audio data into parseable text. Specifically, the Python `speech_recognition` library is used to automatically convert speech to text.

[0402] Step 3:

[0403] The server applies natural language processing techniques to the converted text information to analyze the user's requirements. The input is the text information obtained in step 2, and the output is the analyzed requirements. This process prepares the server to understand the user's intent and generate a specific response. The response content is then designed based on the generated requirements.

[0404] Step 4:

[0405] The server uses emotion analysis technology to identify the user's emotions from the received audio data. The input is again the original audio data, and the output is the identified emotional state. In this step, a generative AI model is used to analyze the emotions and detect specific emotions such as anxiety or relief. Based on the analysis results, the system is used to implement flexible responses that correspond to the emotions.

[0406] Step 5:

[0407] The server generates an appropriate notification and sends it to an external organization or designated device if the analyzed emotion meets certain criteria. The input is the emotional state obtained in step 4, and the output is the generated notification message. This process ensures that information is quickly shared with relevant parties when a specific emotional state is detected. Specifically, if the criteria are exceeded, the system triggers an alert and sends a real-time notification to the caregiver or designated external organization.

[0408] By performing these steps sequentially, an advanced voice communication system is realized that incorporates emotion analysis into voice dialogue, providing appropriate responses and warnings.

[0409] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0410] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0411] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0412] [Third Embodiment]

[0413] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0414] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0415] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0416] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0417] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0418] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0419] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0420] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0421] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0422] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0423] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0424] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0425] This invention is an AI-based communication system that streamlines telephone communication and enables the detection of fraudulent activity and warning of users. Applicable to home and small businesses, it consists of multiple components including a communication device, a speech recognition system, a natural language processing engine, and an alerting system.

[0426] Specific examples of implementation

[0427] For home use

[0428] When the server receives a voice call, it automatically detects the incoming call and plays an initial response message such as, "This is an automated answering system." This prepares the user to begin the conversation. When the user provides voice input, the terminal sends it to the speech recognition system in real time, converting the voice data into text. The server then uses natural language processing technology to analyze the text and determine the user's requirements. For example, if the requirements include "I would like to inquire about delivery," the system generates an appropriate response and provides the user with voice guidance.

[0429] Next, the server sends the converted text to the family's LINE app or a designated email address to share the content and make the information visible. Furthermore, if there are signs of fraud in the communication content, the server detects the possibility of fraudulent activity in real time and notifies the family or a designated external organization.

[0430] For small and medium-sized enterprises

[0431] In a corporate setting, when a server detects an incoming call, it plays a customized IVR menu and guides the user through the department selection process. Once the user makes a selection, the terminal routes the call to the appropriate department or contact person. Speech recognition technology is used to transcribe the conversation into text, which the server then sends to the communication platform or email system. Additionally, the call data is recorded by the server and stored in the cloud or database in a format that allows for later analysis and review.

[0432] This system not only streamlines communication in home and business environments, but also reduces the risk of fraud, allowing users to handle phone calls with peace of mind. Furthermore, voice-to-text conversion facilitates information sharing and understanding, promoting increased work efficiency.

[0433] The following describes the processing flow.

[0434] Step 1:

[0435] The server automatically detects incoming voice calls and controls the communication device to answer. It plays a message saying, "This is an automated answering system. Please leave a message," and then initiates the call.

[0436] Step 2:

[0437] The user initiates a phone conversation and states their requirements and questions.

[0438] Step 3:

[0439] The device captures the user's voice and transmits the data to the speech recognition system in real time.

[0440] Step 4:

[0441] The server retrieves the text data received from the speech recognition system and begins analysis using natural language processing techniques. This analysis identifies the requirements and intent of the received speech.

[0442] Step 5:

[0443] Based on the analysis results, the server generates a corresponding standard response and sends it to the user via speech synthesis. For example, in the case of a delivery confirmation request, it would return a response such as "The shipment is on track."

[0444] Step 6:

[0445] The device continuously transcribes all call content into text and sends the necessary information via the communication network to a designated external device (such as LINE or email).

[0446] Step 7:

[0447] The server monitors call content and detects patterns that may indicate fraud in real time. If a suspicious call is detected, it automatically notifies family members or external organizations.

[0448] Step 8:

[0449] The server records specific calls as needed and stores them in a database for analysis and review. In a business environment, it shares and routes communications to relevant departments and individuals.

[0450] (Example 1)

[0451] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0452] In modern society, despite the diversification and evolution of communication methods, balancing user convenience and security in voice communication remains a challenge. In particular, effective communication and prevention of fraudulent activities in automated responses are crucial. However, current technologies have been insufficient to achieve these goals in an integrated and efficient manner. Therefore, a system is needed to solve this problem and enable users to communicate with peace of mind.

[0453] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0454] In this invention, the server includes means for a communication device to detect an audio signal and automatically initiate a response; means for converting the received audio into a string using audio analysis technology; means for analyzing the content of the received audio using language understanding technology and determining a response based thereon; means for identifying signs of fraudulent activity and providing notification; and means for warning a designated external mechanism about the identified suspicious call. This enables efficient automatic response and improved security in voice communications.

[0455] "Communication equipment" refers to devices used for sending and receiving voice, and has the function of exchanging voice data via telephone or network.

[0456] An "audio signal" is a representation of sound as an electrical signal, and is a format for transmitting audio information.

[0457] "Speech analysis" refers to the technology of converting speech signals into text or numerical data, and is achieved using speech recognition technology.

[0458] A "string" is a data format consisting of a sequence of characters, and it is generated through speech analysis.

[0459] "Language understanding" refers to the technology of using natural language processing to understand the content of string data and analyze its intent and requirements.

[0460] "Fraudulent activity" refers to any act that deviates from the normally expected and legitimate communication behavior, such as fraud or unauthorized access.

[0461] "Notification" is the act of informing a specific person of information, and is done through voice or message.

[0462] An "external mechanism" refers to an organization or system located outside the system that is responsible for notification and information processing.

[0463] This invention is an AI-based communication system that efficiently processes voice communications and detects fraudulent activity. The system consists of various components and is suitable for both home and small business use.

[0464] The server first detects the audio signal received through the communication device. The server responds in real time and plays an initial message to the user stating, "This is an automated answering system." An audio playback device is used at this stage, and VoIP technology can be utilized.

[0465] The device converts voice input from the user into text using speech analysis technology. For example, it uses a speech recognition API to obtain what the user says as text data. This text data is then sent to the server.

[0466] The server uses a natural language processing engine to apply language understanding technology to the received string. This process allows it to analyze the user's request. Using a generative AI model, it automatically recognizes specific requests such as "I want to inquire about delivery" and generates an appropriate response.

[0467] Furthermore, the server implements algorithms to detect signs of fraudulent activity in real time, determining the possibility of fraud by identifying specific keywords and phrases. If a suspicious call is detected, it is notified to a designated external mechanism, which may include messaging services or email systems.

[0468] In this system, converted text is sent via LINE or email for information sharing within families and companies, and call content is stored in a database, thereby improving work efficiency and security.

[0469] As a concrete example of a prompt, you can input a question like, "I've been getting some strange phone calls lately; are they safe?" into the generating AI model. The system will then analyze the content of the call and provide feedback regarding its safety.

[0470] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0471] Step 1:

[0472] The server receives audio signals from communication devices and automatically detects incoming calls. Input can be analog or digital audio data. This audio data triggers the server to prepare to play a voice message stating, "This call is an automated response system." The output is the playback of an automated response voice message utilizing VoIP technology.

[0473] Step 2:

[0474] The device collects voice input from the user and sends it to a speech recognition system. The input is raw voice waveform data. This data is passed to a speech recognition API, where signal processing converts it into string data. The output is text data representing the content of the voice.

[0475] Step 3:

[0476] The server receives text data generated by speech recognition and analyzes it using a natural language processing engine. The input is the text data from step 2. Based on the keywords and context contained in this data, a generative AI model is used to understand the user's request and determine an appropriate response. The output is a response text that is based on the user's intent.

[0477] Step 4:

[0478] The server passes the response text generated in the previous step to the speech synthesis engine, where it is converted into audio data. The input is the generated response text. This is processed by the speech synthesis system and output as an audio file. Specifically, it is played back to the user through their speakers.

[0479] Step 5:

[0480] The server detects in real time whether there are signs of fraud in the communication content. The input is text data of the call content. This data is analyzed by an algorithm to identify keywords and patterns that indicate fraudulent activity. The output is flag information regarding the possibility of fraud, and a warning is sent to a designated external mechanism as needed.

[0481] Step 6:

[0482] The server sends the converted text data to family or corporate devices via a specified communication network. In this process, the input is the analyzed text data. The data is transmitted via services such as LINE or email using a transmission protocol. The output is message data that can be viewed by the receiver.

[0483] (Application Example 1)

[0484] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0485] In recent years, fraudulent activity in electronic payments has been increasing, making it difficult for users to conduct secure transactions. Current technology presents a significant challenge in detecting fraudulent activity via voice communication in real time and providing appropriate warnings to users. Furthermore, there is a lack of systems that can efficiently verify transaction details via voice and quickly identify fraudulent activity.

[0486] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0487] In this invention, the server includes means for confirming the details of a transaction using voice input and identifying whether or not there is an anomaly, means for detecting the possibility of fraudulent activity and issuing a warning, and means for storing voice data and text data in an information storage area to enable later analysis and reference. This provides an electronic payment system with improved security, enabling users to conduct transactions with peace of mind.

[0488] "Communication equipment" refers to electronic devices that have the function of receiving and transmitting voice signals.

[0489] "Speech recognition technology" is a technology that analyzes speech information and converts it into text information.

[0490] "Textual information" refers to text data converted using speech recognition technology.

[0491] "Natural language processing technology" is a technology that analyzes speech and text composed of natural language to understand their meaning and intent.

[0492] "Fraudulent activity" refers to any activity or act that attempts to gain profit through improper means.

[0493] A "warning" is a notification that informs the user of a potential danger or problem.

[0494] An "external organization" is an organization or institution located outside the system.

[0495] "Voice input" is the process of receiving the voice spoken by the user as data.

[0496] "Identification" is the act of recognizing the properties and characteristics of an object and classifying or distinguishing it.

[0497] "Information storage area" refers to storage devices and databases used to securely record and store data.

[0498] A "transaction" refers to an activity related to the exchange or purchase of goods or services.

[0499] To implement this invention, a server is configured as part of an electronic payment system, including communication equipment, speech recognition technology, natural language processing technology, and a warning system. First, the server uses a smartphone or other suitable voice-enabled device as communication equipment to receive voice input. The voice received by this device is converted into text information by an API incorporating speech recognition technology (for example, the Google Speech-to-Text API).

[0500] The text information converted in real time is semantically analyzed using natural language processing technology to identify potential fraudulent activity. Ideally, a natural language processing engine such as the Google Cloud Natural Language API should be used for this analysis. If signs of fraud are detected, an alert system will send an alert to the user. This alert can be communicated to the user via a screen display or audio message.

[0501] The server also stores all audio data and its analysis results in an information storage area. This ensures that the data is securely stored for later analysis and can be referenced by users and administrators at a later date.

[0502] As a concrete example, when a user attempts to make an electronic payment, they may be asked to confirm via voice, "Is this payment trustworthy?" This system converts the voice into text, analyzes it, and determines the security of the transaction. Based on the results, it sends a warning to the user, protecting them from fraudulent transactions.

[0503] An example of a prompt to input into the generating AI model is, "Create a program that analyzes the possibility of fraudulent activity and generates a warning based on this voice instruction." This prompt guides the AI ​​model to correctly analyze the user's voice input and take appropriate action.

[0504] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0505] Step 1:

[0506] The server receives voice input from the terminal. During this process, the user speaks to confirm the transaction details. This voice data is then used as input for the speech recognition system.

[0507] Step 2:

[0508] The server uses speech recognition technology to convert the received audio data into text. At this stage, the audio data is analyzed using the Google Speech-to-Text API and output as text. As a result, the content of the audio is converted into text format.

[0509] Step 3:

[0510] The server uses natural language processing technology to analyze text information. Here, the Google Cloud Natural Language API is used to process the data in order to understand the meaning and intent of the text. The analysis results output data that evaluates whether or not there are signs of fraudulent activity.

[0511] Step 4:

[0512] Based on the analysis results, the server will warn the user if there is a possibility of fraudulent activity. This warning will be communicated to the user via audio or on-screen display. The user can then re-evaluate the transaction based on this information and cancel it if necessary.

[0513] Step 5:

[0514] The server stores all audio data and analysis results in an information storage area. This data is recorded in a database in a secure and efficient manner. The stored data will later be available for analysis and reporting.

[0515] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0516] This invention provides a system that, in voice communication with a user, not only converts voice content into text and understands its content, but also analyzes the user's emotions to provide more appropriate responses and warnings. This system consists of a communication device, a voice recognition system, a natural language processing engine, an emotion engine, and a warning and notification system.

[0517] Specific examples of implementation

[0518] For home use

[0519] The server detects an incoming voice call and plays a message such as, "This is an automated answering system. Please tell us what you need to do." When the user begins speaking, the terminal inputs the user's voice into the speech recognition system in real time and converts it to text. Simultaneously, the emotion engine analyzes the voice data to identify the user's emotions.

[0520] The server analyzes the content of text data using natural language processing to recognize the requirements. For example, if a complaint about customer service is expressed, the emotion engine detects emotions such as dissatisfaction or anger. Based on this information, the server generates a faster and more courteous response than usual and takes appropriate action.

[0521] Furthermore, the server can manage user history by saving emotion recognition results to a database, which can then be used for future analysis and service improvements. If the emotion engine detects anger or anxiety exceeding a certain level, the server immediately notifies designated family members and, if necessary, sets up a three-way call to ensure peace of mind within the family.

[0522] For small and medium-sized enterprises

[0523] In a corporate setting, a server answers incoming calls and plays a customized IVR menu. Once the user has made their selection, the terminal converts the voice to text and simultaneously performs sentiment analysis using an emotion engine. The analysis results determine the quality of the response generated by the server, providing thoughtful customer service based on emotions.

[0524] In addition to the normal workflow, the server can store the results of emotion analysis as call data, which can be used to improve security and satisfaction. In the event that high-risk emotions are detected, an alert will be sent to the designated security department or administrator to ensure smooth operations within the company.

[0525] In this way, the present invention can improve the quality of service and enhance user safety and satisfaction by comprehensively analyzing and responding to user emotions and requirements.

[0526] The following describes the processing flow.

[0527] Step 1:

[0528] The server detects an incoming voice call, plays an automated response message, and initiates the call.

[0529] Step 2:

[0530] The user initiates a phone conversation and states their requirements and questions.

[0531] Step 3:

[0532] The device captures voice data from the user and sends it to a speech recognition system in real time to be converted into text data.

[0533] Step 4:

[0534] The server receives text data from the speech recognition system, analyzes this text using natural language processing technology, and identifies the user's requirements.

[0535] Step 5:

[0536] The device activates its emotion engine and simultaneously analyzes the received audio data to identify the user's emotions.

[0537] Step 6:

[0538] The server generates an appropriate response based on the analyzed requirements and sentiment information. For example, if the sentiment engine detects user dissatisfaction, it will provide a more courteous response.

[0539] Step 7:

[0540] The terminal generates the response using speech synthesis and plays it back to the user.

[0541] Step 8:

[0542] The server stores the transcribed call content and emotion recognition results in a database, managing them in a format that allows for later searching and analysis.

[0543] Step 9:

[0544] If the emotion engine detects severe stress or anger, the server will send a warning notification to designated external devices or family members and, if necessary, set up a three-way call with those parties.

[0545] (Example 2)

[0546] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0547] In conventional systems receiving voice communications on communication networks, the system simply converted the voice to text and analyzed the requirements, making it difficult to understand the user's emotions and provide an appropriate response. This has created a need to improve service quality and increase user satisfaction. Furthermore, in situations where emotional changes are a crucial indicator, there was a lack of a mechanism to respond quickly and notify the appropriate person of the necessary information.

[0548] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0549] In this invention, the server includes means for automatically responding when a communication device receives voice communication, means for converting the received voice into text using voice recognition technology, means for analyzing the voice data to identify the user's emotions, means for adjusting the quality of the response based on the results of the emotion analysis, and means for sending notifications to designated contacts according to the identified emotions. This enables flexible and high-quality responses that take into account the user's emotions, and allows for prompt responses by immediately sending notifications to the appropriate person as needed.

[0550] A "communication device" is a device that receives voice communications and enables voice communication with the user.

[0551] "Speech recognition technology" is a technology that converts speech data into text data.

[0552] "Natural language processing technology" is a technology that analyzes meaning from text data and identifies requirements.

[0553] "Emotion analysis" is the process of identifying a user's emotions based on voice data and text data.

[0554] "Response generation" is the process of creating an appropriate response based on analyzed requirements and emotions.

[0555] "Notification transmission" is the process of sending information to a designated contact point when certain conditions are met.

[0556] This invention is a system for analyzing user emotions in voice communication and providing appropriate responses. When a server receives voice communication via a communication device, it activates an automated response system and plays a predetermined message to the user. When the user begins to respond, the terminal converts the voice into text in real time using speech recognition technology. Specifically, it converts the voice data into text using an existing service such as Google Cloud Speech-to-Text.

[0557] Simultaneously, the terminal inputs voice data into a dedicated emotion engine to perform emotion analysis and identify the user's emotions. This engine analyzes and identifies emotions based on voice tone, speed, and word choice. The server uses natural language processing techniques to analyze text data and process it to understand the user's intentions. Generative AI models such as OpenAI GPT-3 are used to extract requirements from the text data and generate responses.

[0558] The server generates responses that are sensitive to the user's emotions, based on sentiment analysis and content analysis. If a user expresses dissatisfaction with the service, it provides a polite and prompt response according to a pre-configured response policy. Furthermore, by saving the analysis results to a database, the server can manage past communication history and use it for future service improvements and analysis.

[0559] As a concrete example, consider its use within a home. The server has a function that sends notifications to designated relatives or friends if it detects anger or anxiety in the user. This helps to maintain a sense of security within the home. An example of a prompt would be, "Please describe in natural language how the sentiment analysis system works in customer phone calls. Please provide specific examples for home and small businesses."

[0560] In this way, the invention can accurately grasp the user's emotions and enable high-quality communication.

[0561] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0562] Step 1:

[0563] The server receives an incoming voice call via a communication device. The automated response system activates and plays the message, "This is an automated response system. Please leave your message," to the user. The input is the incoming voice call, and the output is the played automated response message.

[0564] Step 2:

[0565] When a user begins to respond by voice, the device collects the audio through its microphone. This audio data is then input into a speech recognition system, which converts the speech to text. The input is the user's voice, and the output is the converted text data. This conversion uses speech recognition technology such as Google Cloud Speech-to-Text.

[0566] Step 3:

[0567] The device simultaneously inputs the collected audio data into an emotion engine to analyze the user's emotions. The input is the user's audio data, and the output is the identified emotion. Emotion analysis is performed based on voice tone, speed, and word choice.

[0568] Step 4:

[0569] The server uses natural language processing techniques to analyze text data and extract user requirements. The input is text data, and the output is information indicating the user's requirements. Generative AI models such as OpenAI GPT-3 are used for the analysis.

[0570] Step 5:

[0571] The server integrates sentiment analysis results and content analysis results to generate a response that takes the user's emotions into consideration. The input consists of identified emotions and requirement information, while the output is the generated response text. A pre-configured response policy may also be applied to the response.

[0572] Step 6:

[0573] The server plays the generated response as audio to the user. Furthermore, it saves the analysis results to a database. The input is the generated response text, and the output is the played audio and the saved data. This data will be used for future service improvements and history management.

[0574] Step 7:

[0575] The server, when its emotion engine detects a high level of anger or anxiety, sends a notification to designated contacts to ensure safety within homes and businesses. The input is the detection of high-risk emotions, and the output is the sent notification. The notification provides instructions and information as needed.

[0576] (Application Example 2)

[0577] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0578] In recent years, advancements in voice communication technology have created a need for systems that not only understand the content of conversations with users but also analyze their emotions to provide more appropriate responses. However, current technologies have not adequately achieved flexible responses and warning systems that take emotions into account. Furthermore, there is a lack of mechanisms to immediately notify appropriate external organizations in the event of sudden changes in emotions or fraudulent activity.

[0579] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0580] In this invention, the server includes means for converting received speech into text information using speech recognition technology, means for analyzing the requirements of the received speech using natural language processing technology and generating a response based thereon, and means for identifying the user's emotions from the speech data using emotion analysis technology. This makes it possible to analyze the user's emotions in voice dialogue and provide appropriate responses or warnings. It also makes it possible to detect the possibility of fraudulent activity and quickly notify a designated external organization if the emotions exceed a certain threshold.

[0581] "Communication technology" refers to technology that receives voice information from users and automatically responds based on that information.

[0582] "Speech recognition technology" is a technology that converts received speech into text information and uses it for analysis.

[0583] "Natural language processing technology" is a technology that understands the requirements of the textual information analyzed from received speech and generates an appropriate response.

[0584] "Emotion analysis technology" is a technology that identifies and analyzes a user's emotions from voice data.

[0585] "Potential fraudulent activity" refers to a situation where there is a possibility of fraud or misconduct occurring in the content of a received communication.

[0586] An "external agency" is an external organization or facility established to notify information when emotional or risky communications exceeding specific criteria are detected.

[0587] One embodiment of this invention is a voice communication system that analyzes speech through interaction with a user and provides an emotionally appropriate response. The server uses speech recognition software to convert speech data into text information in real time. This is done using the Python speech_recognition library. The converted text information is analyzed using natural language processing techniques to generate a response.

[0588] The server simultaneously identifies emotions from the audio data using emotion analysis technology. This emotion analysis utilizes generative AI models to detect specific emotional states. For example, when it receives audio from a user expressing anxiety, it can quickly recognize that emotion and issue an alert to prompt appropriate action.

[0589] As a concrete example, if a user says, "I'm feeling a little anxious today," the server converts the audio into text data using speech recognition technology and understands the content through natural language processing. Simultaneously, it uses sentiment analysis technology to recognize the user's emotion of anxiety. Based on this, the server generates a response appropriate to the emotion and, for example, sends a notification to a caregiver. An example prompt might look like this: "Please analyze the emotions in the following conversation. If the user seems restless, please advise how the caregiver should respond. Conversation content: 'I didn't sleep well last night... I'm feeling a little anxious today.'"

[0590] This configuration allows voice communication systems to respond flexibly, taking into account the user's emotions. It enables rapid detection of anticipated risks and fraudulent activities, and allows for countermeasures such as notification to external organizations as needed.

[0591] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0592] Step 1:

[0593] The server receives voice information from the user using communication technology. The user's voice data is provided to the server as input. The server then passes this data directly to the next process.

[0594] Step 2:

[0595] The server uses speech recognition software to convert received audio information into text. The input is audio data received from the user, and the output is text. This process converts the audio data into parseable text. Specifically, the Python `speech_recognition` library is used to automatically convert speech to text.

[0596] Step 3:

[0597] The server applies natural language processing techniques to the converted text information to analyze the user's requirements. The input is the text information obtained in step 2, and the output is the analyzed requirements. This process prepares the server to understand the user's intent and generate a specific response. The response content is then designed based on the generated requirements.

[0598] Step 4:

[0599] The server uses emotion analysis technology to identify the user's emotions from the received audio data. The input is again the original audio data, and the output is the identified emotional state. In this step, a generative AI model is used to analyze the emotions and detect specific emotions such as anxiety or relief. Based on the analysis results, the system is used to implement flexible responses that correspond to the emotions.

[0600] Step 5:

[0601] The server generates an appropriate notification and sends it to an external organization or designated device if the analyzed emotion meets certain criteria. The input is the emotional state obtained in step 4, and the output is the generated notification message. This process ensures that information is quickly shared with relevant parties when a specific emotional state is detected. Specifically, if the criteria are exceeded, the system triggers an alert and sends a real-time notification to the caregiver or designated external organization.

[0602] By performing these steps sequentially, an advanced voice communication system is realized that incorporates emotion analysis into voice dialogue, providing appropriate responses and warnings.

[0603] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0604] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0605] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0606] [Fourth Embodiment]

[0607] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0608] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0609] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0610] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0611] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0612] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0613] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0614] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0615] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0616] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0617] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0618] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0619] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0620] This invention is an AI-based communication system that streamlines telephone communication and enables the detection of fraudulent activity and warning of users. Applicable to home and small businesses, it consists of multiple components including a communication device, a speech recognition system, a natural language processing engine, and an alerting system.

[0621] Specific examples of implementation

[0622] For home use

[0623] When the server receives a voice call, it automatically detects the incoming call and plays an initial response message such as, "This is an automated answering system." This prepares the user to begin the conversation. When the user provides voice input, the terminal sends it to the speech recognition system in real time, converting the voice data into text. The server then uses natural language processing technology to analyze the text and determine the user's requirements. For example, if the requirements include "I would like to inquire about delivery," the system generates an appropriate response and provides the user with voice guidance.

[0624] Next, the server sends the converted text to the family's LINE app or a designated email address to share the content and make the information visible. Furthermore, if there are signs of fraud in the communication content, the server detects the possibility of fraudulent activity in real time and notifies the family or a designated external organization.

[0625] For small and medium-sized enterprises

[0626] In a corporate setting, when a server detects an incoming call, it plays a customized IVR menu and guides the user through the department selection process. Once the user makes a selection, the terminal routes the call to the appropriate department or contact person. Speech recognition technology is used to transcribe the conversation into text, which the server then sends to the communication platform or email system. Additionally, the call data is recorded by the server and stored in the cloud or database in a format that allows for later analysis and review.

[0627] This system not only streamlines communication in home and business environments, but also reduces the risk of fraud, allowing users to handle phone calls with peace of mind. Furthermore, voice-to-text conversion facilitates information sharing and understanding, promoting increased work efficiency.

[0628] The following describes the processing flow.

[0629] Step 1:

[0630] The server automatically detects incoming voice calls and controls the communication device to answer. It plays a message saying, "This is an automated answering system. Please leave a message," and then initiates the call.

[0631] Step 2:

[0632] The user initiates a phone conversation and states their requirements and questions.

[0633] Step 3:

[0634] The device captures the user's voice and transmits the data to the speech recognition system in real time.

[0635] Step 4:

[0636] The server retrieves the text data received from the speech recognition system and begins analysis using natural language processing techniques. This analysis identifies the requirements and intent of the received speech.

[0637] Step 5:

[0638] Based on the analysis results, the server generates a corresponding standard response and sends it to the user via speech synthesis. For example, in the case of a delivery confirmation request, it would return a response such as "The shipment is on track."

[0639] Step 6:

[0640] The device continuously transcribes all call content into text and sends the necessary information via the communication network to a designated external device (such as LINE or email).

[0641] Step 7:

[0642] The server monitors call content and detects patterns that may indicate fraud in real time. If a suspicious call is detected, it automatically notifies family members or external organizations.

[0643] Step 8:

[0644] The server records specific calls as needed and stores them in a database for analysis and review. In a business environment, it shares and routes communications to relevant departments and individuals.

[0645] (Example 1)

[0646] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0647] In modern society, despite the diversification and evolution of communication methods, balancing user convenience and security in voice communication remains a challenge. In particular, effective communication and prevention of fraudulent activities in automated responses are crucial. However, current technologies have been insufficient to achieve these goals in an integrated and efficient manner. Therefore, a system is needed to solve this problem and enable users to communicate with peace of mind.

[0648] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0649] In this invention, the server includes means for a communication device to detect an audio signal and automatically initiate a response; means for converting the received audio into a string using audio analysis technology; means for analyzing the content of the received audio using language understanding technology and determining a response based thereon; means for identifying signs of fraudulent activity and providing notification; and means for warning a designated external mechanism about the identified suspicious call. This enables efficient automatic response and improved security in voice communications.

[0650] "Communication equipment" refers to devices used for sending and receiving voice, and has the function of exchanging voice data via telephone or network.

[0651] An "audio signal" is a representation of sound as an electrical signal, and is a format for transmitting audio information.

[0652] "Speech analysis" refers to the technology of converting speech signals into text or numerical data, and is achieved using speech recognition technology.

[0653] A "string" is a data format consisting of a sequence of characters, and it is generated through speech analysis.

[0654] "Language understanding" refers to the technology of using natural language processing to understand the content of string data and analyze its intent and requirements.

[0655] "Fraudulent activity" refers to any act that deviates from the normally expected and legitimate communication behavior, such as fraud or unauthorized access.

[0656] "Notification" is the act of informing a specific person of information, and is done through voice or message.

[0657] An "external mechanism" refers to an organization or system located outside the system that is responsible for notification and information processing.

[0658] This invention is an AI-based communication system that efficiently processes voice communications and detects fraudulent activity. The system consists of various components and is suitable for both home and small business use.

[0659] The server first detects the audio signal received through the communication device. The server responds in real time and plays an initial message to the user stating, "This is an automated answering system." An audio playback device is used at this stage, and VoIP technology can be utilized.

[0660] The device converts voice input from the user into text using speech analysis technology. For example, it uses a speech recognition API to obtain what the user says as text data. This text data is then sent to the server.

[0661] The server uses a natural language processing engine to apply language understanding technology to the received string. This process allows it to analyze the user's request. Using a generative AI model, it automatically recognizes specific requests such as "I want to inquire about delivery" and generates an appropriate response.

[0662] Furthermore, the server implements algorithms to detect signs of fraudulent activity in real time, determining the possibility of fraud by identifying specific keywords and phrases. If a suspicious call is detected, it is notified to a designated external mechanism, which may include messaging services or email systems.

[0663] In this system, converted text is sent via LINE or email for information sharing within families and companies, and call content is stored in a database, thereby improving work efficiency and security.

[0664] As a concrete example of a prompt, you can input a question like, "I've been getting some strange phone calls lately; are they safe?" into the generating AI model. The system will then analyze the content of the call and provide feedback regarding its safety.

[0665] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0666] Step 1:

[0667] The server receives audio signals from communication devices and automatically detects incoming calls. Input can be analog or digital audio data. This audio data triggers the server to prepare to play a voice message stating, "This call is an automated response system." The output is the playback of an automated response voice message utilizing VoIP technology.

[0668] Step 2:

[0669] The device collects voice input from the user and sends it to a speech recognition system. The input is raw voice waveform data. This data is passed to a speech recognition API, where signal processing converts it into string data. The output is text data representing the content of the voice.

[0670] Step 3:

[0671] The server receives text data generated by speech recognition and analyzes it using a natural language processing engine. The input is the text data from step 2. Based on the keywords and context contained in this data, a generative AI model is used to understand the user's request and determine an appropriate response. The output is a response text that is based on the user's intent.

[0672] Step 4:

[0673] The server passes the response text generated in the previous step to the speech synthesis engine, where it is converted into audio data. The input is the generated response text. This is processed by the speech synthesis system and output as an audio file. Specifically, it is played back to the user through their speakers.

[0674] Step 5:

[0675] The server detects in real time whether there are signs of fraud in the communication content. The input is text data of the call content. This data is analyzed by an algorithm to identify keywords and patterns that indicate fraudulent activity. The output is flag information regarding the possibility of fraud, and a warning is sent to a designated external mechanism as needed.

[0676] Step 6:

[0677] The server sends the converted text data to family or corporate devices via a specified communication network. In this process, the input is the analyzed text data. The data is transmitted via services such as LINE or email using a transmission protocol. The output is message data that can be viewed by the receiver.

[0678] (Application Example 1)

[0679] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0680] In recent years, fraudulent activity in electronic payments has been increasing, making it difficult for users to conduct secure transactions. Current technology presents a significant challenge in detecting fraudulent activity via voice communication in real time and providing appropriate warnings to users. Furthermore, there is a lack of systems that can efficiently verify transaction details via voice and quickly identify fraudulent activity.

[0681] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0682] In this invention, the server includes means for confirming the details of a transaction using voice input and identifying whether or not there is an anomaly, means for detecting the possibility of fraudulent activity and issuing a warning, and means for storing voice data and text data in an information storage area to enable later analysis and reference. This provides an electronic payment system with improved security, enabling users to conduct transactions with peace of mind.

[0683] "Communication equipment" refers to electronic devices that have the function of receiving and transmitting voice signals.

[0684] "Speech recognition technology" is a technology that analyzes speech information and converts it into text information.

[0685] "Textual information" refers to text data converted using speech recognition technology.

[0686] "Natural language processing technology" is a technology that analyzes speech and text composed of natural language to understand their meaning and intent.

[0687] "Fraudulent activity" refers to any activity or act that attempts to gain profit through improper means.

[0688] A "warning" is a notification that informs the user of a potential danger or problem.

[0689] An "external organization" is an organization or institution located outside the system.

[0690] "Voice input" is the process of receiving the voice spoken by the user as data.

[0691] "Identification" is the act of recognizing the properties and characteristics of an object and classifying or distinguishing it.

[0692] "Information storage area" refers to storage devices and databases used to securely record and store data.

[0693] A "transaction" refers to an activity related to the exchange or purchase of goods or services.

[0694] To implement this invention, a server is configured as part of an electronic payment system, including communication equipment, speech recognition technology, natural language processing technology, and a warning system. First, the server uses a smartphone or other suitable voice-enabled device as communication equipment to receive voice input. The voice received by this device is converted into text information by an API incorporating speech recognition technology (for example, the Google Speech-to-Text API).

[0695] The text information converted in real time is semantically analyzed using natural language processing technology to identify potential fraudulent activity. Ideally, a natural language processing engine such as the Google Cloud Natural Language API should be used for this analysis. If signs of fraud are detected, an alert system will send an alert to the user. This alert can be communicated to the user via a screen display or audio message.

[0696] The server also stores all audio data and its analysis results in an information storage area. This ensures that the data is securely stored for later analysis and can be referenced by users and administrators at a later date.

[0697] As a concrete example, when a user attempts to make an electronic payment, they may be asked to confirm via voice, "Is this payment trustworthy?" This system converts the voice into text, analyzes it, and determines the security of the transaction. Based on the results, it sends a warning to the user, protecting them from fraudulent transactions.

[0698] An example of a prompt to input into the generating AI model is, "Create a program that analyzes the possibility of fraudulent activity and generates a warning based on this voice instruction." This prompt guides the AI ​​model to correctly analyze the user's voice input and take appropriate action.

[0699] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0700] Step 1:

[0701] The server receives voice input from the terminal. During this process, the user speaks to confirm the transaction details. This voice data is then used as input for the speech recognition system.

[0702] Step 2:

[0703] The server uses speech recognition technology to convert the received audio data into text. At this stage, the audio data is analyzed using the Google Speech-to-Text API and output as text. As a result, the content of the audio is converted into text format.

[0704] Step 3:

[0705] The server uses natural language processing technology to analyze text information. Here, the Google Cloud Natural Language API is used to process the data in order to understand the meaning and intent of the text. The analysis results output data that evaluates whether or not there are signs of fraudulent activity.

[0706] Step 4:

[0707] Based on the analysis results, the server will warn the user if there is a possibility of fraudulent activity. This warning will be communicated to the user via audio or on-screen display. The user can then re-evaluate the transaction based on this information and cancel it if necessary.

[0708] Step 5:

[0709] The server stores all audio data and analysis results in an information storage area. This data is recorded in a database in a secure and efficient manner. The stored data will later be available for analysis and reporting.

[0710] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0711] This invention provides a system that, in voice communication with a user, not only converts voice content into text and understands its content, but also analyzes the user's emotions to provide more appropriate responses and warnings. This system consists of a communication device, a voice recognition system, a natural language processing engine, an emotion engine, and a warning and notification system.

[0712] Specific examples of implementation

[0713] For home use

[0714] The server detects an incoming voice call and plays a message such as, "This is an automated answering system. Please tell us what you need to do." When the user begins speaking, the terminal inputs the user's voice into the speech recognition system in real time and converts it to text. Simultaneously, the emotion engine analyzes the voice data to identify the user's emotions.

[0715] The server analyzes the content of text data using natural language processing to recognize the requirements. For example, if a complaint about customer service is expressed, the emotion engine detects emotions such as dissatisfaction or anger. Based on this information, the server generates a faster and more courteous response than usual and takes appropriate action.

[0716] Furthermore, the server can manage user history by saving emotion recognition results to a database, which can then be used for future analysis and service improvements. If the emotion engine detects anger or anxiety exceeding a certain level, the server immediately notifies designated family members and, if necessary, sets up a three-way call to ensure peace of mind within the family.

[0717] For small and medium-sized enterprises

[0718] In a corporate setting, a server answers incoming calls and plays a customized IVR menu. Once the user has made their selection, the terminal converts the voice to text and simultaneously performs sentiment analysis using an emotion engine. The analysis results determine the quality of the response generated by the server, providing thoughtful customer service based on emotions.

[0719] In addition to the normal workflow, the server can store the results of emotion analysis as call data, which can be used to improve security and satisfaction. In the event that high-risk emotions are detected, an alert will be sent to the designated security department or administrator to ensure smooth operations within the company.

[0720] In this way, the present invention can improve the quality of service and enhance user safety and satisfaction by comprehensively analyzing and responding to user emotions and requirements.

[0721] The following describes the processing flow.

[0722] Step 1:

[0723] The server detects an incoming voice call, plays an automated response message, and initiates the call.

[0724] Step 2:

[0725] The user initiates a phone conversation and states their requirements and questions.

[0726] Step 3:

[0727] The device captures voice data from the user and sends it to a speech recognition system in real time to be converted into text data.

[0728] Step 4:

[0729] The server receives text data from the speech recognition system, analyzes this text using natural language processing technology, and identifies the user's requirements.

[0730] Step 5:

[0731] The device activates its emotion engine and simultaneously analyzes the received audio data to identify the user's emotions.

[0732] Step 6:

[0733] The server generates an appropriate response based on the analyzed requirements and sentiment information. For example, if the sentiment engine detects user dissatisfaction, it will provide a more courteous response.

[0734] Step 7:

[0735] The terminal generates the response using speech synthesis and plays it back to the user.

[0736] Step 8:

[0737] The server stores the transcribed call content and emotion recognition results in a database, managing them in a format that allows for later searching and analysis.

[0738] Step 9:

[0739] If the emotion engine detects severe stress or anger, the server will send a warning notification to designated external devices or family members and, if necessary, set up a three-way call with those parties.

[0740] (Example 2)

[0741] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0742] In conventional systems receiving voice communications on communication networks, the system simply converted the voice to text and analyzed the requirements, making it difficult to understand the user's emotions and provide an appropriate response. This has created a need to improve service quality and increase user satisfaction. Furthermore, in situations where emotional changes are a crucial indicator, there was a lack of a mechanism to respond quickly and notify the appropriate person of the necessary information.

[0743] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0744] In this invention, the server includes means for automatically responding when a communication device receives voice communication, means for converting the received voice into text using voice recognition technology, means for analyzing the voice data to identify the user's emotions, means for adjusting the quality of the response based on the results of the emotion analysis, and means for sending notifications to designated contacts according to the identified emotions. This enables flexible and high-quality responses that take into account the user's emotions, and allows for prompt responses by immediately sending notifications to the appropriate person as needed.

[0745] A "communication device" is a device that receives voice communications and enables voice communication with the user.

[0746] "Speech recognition technology" is a technology that converts speech data into text data.

[0747] "Natural language processing technology" is a technology that analyzes meaning from text data and identifies requirements.

[0748] "Emotion analysis" is the process of identifying a user's emotions based on voice data and text data.

[0749] "Response generation" is the process of creating an appropriate response based on analyzed requirements and emotions.

[0750] "Notification transmission" is the process of sending information to a designated contact point when certain conditions are met.

[0751] This invention is a system for analyzing user emotions in voice communication and providing appropriate responses. When a server receives voice communication via a communication device, it activates an automated response system and plays a predetermined message to the user. When the user begins to respond, the terminal converts the voice into text in real time using speech recognition technology. Specifically, it converts the voice data into text using an existing service such as Google Cloud Speech-to-Text.

[0752] Simultaneously, the terminal inputs voice data into a dedicated emotion engine to perform emotion analysis and identify the user's emotions. This engine analyzes and identifies emotions based on voice tone, speed, and word choice. The server uses natural language processing techniques to analyze text data and process it to understand the user's intentions. Generative AI models such as OpenAI GPT-3 are used to extract requirements from the text data and generate responses.

[0753] The server generates responses that are sensitive to the user's emotions, based on sentiment analysis and content analysis. If a user expresses dissatisfaction with the service, it provides a polite and prompt response according to a pre-configured response policy. Furthermore, by saving the analysis results to a database, the server can manage past communication history and use it for future service improvements and analysis.

[0754] As a concrete example, consider its use within a home. The server has a function that sends notifications to designated relatives or friends if it detects anger or anxiety in the user. This helps to maintain a sense of security within the home. An example of a prompt would be, "Please describe in natural language how the sentiment analysis system works in customer phone calls. Please provide specific examples for home and small businesses."

[0755] In this way, the invention can accurately grasp the user's emotions and enable high-quality communication.

[0756] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0757] Step 1:

[0758] The server receives an incoming voice call via a communication device. The automated response system activates and plays the message, "This is an automated response system. Please leave your message," to the user. The input is the incoming voice call, and the output is the played automated response message.

[0759] Step 2:

[0760] When a user begins to respond by voice, the device collects the audio through its microphone. This audio data is then input into a speech recognition system, which converts the speech to text. The input is the user's voice, and the output is the converted text data. This conversion uses speech recognition technology such as Google Cloud Speech-to-Text.

[0761] Step 3:

[0762] The device simultaneously inputs the collected audio data into an emotion engine to analyze the user's emotions. The input is the user's audio data, and the output is the identified emotion. Emotion analysis is performed based on voice tone, speed, and word choice.

[0763] Step 4:

[0764] The server uses natural language processing techniques to analyze text data and extract user requirements. The input is text data, and the output is information indicating the user's requirements. Generative AI models such as OpenAI GPT-3 are used for the analysis.

[0765] Step 5:

[0766] The server integrates sentiment analysis results and content analysis results to generate a response that takes the user's emotions into consideration. The input consists of identified emotions and requirement information, while the output is the generated response text. A pre-configured response policy may also be applied to the response.

[0767] Step 6:

[0768] The server plays the generated response as audio to the user. Furthermore, it saves the analysis results to a database. The input is the generated response text, and the output is the played audio and the saved data. This data will be used for future service improvements and history management.

[0769] Step 7:

[0770] The server, when its emotion engine detects a high level of anger or anxiety, sends a notification to designated contacts to ensure safety within homes and businesses. The input is the detection of high-risk emotions, and the output is the sent notification. The notification provides instructions and information as needed.

[0771] (Application Example 2)

[0772] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0773] In recent years, advancements in voice communication technology have created a need for systems that not only understand the content of conversations with users but also analyze their emotions to provide more appropriate responses. However, current technologies have not adequately achieved flexible responses and warning systems that take emotions into account. Furthermore, there is a lack of mechanisms to immediately notify appropriate external organizations in the event of sudden changes in emotions or fraudulent activity.

[0774] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0775] In this invention, the server includes means for converting received speech into text information using speech recognition technology, means for analyzing the requirements of the received speech using natural language processing technology and generating a response based thereon, and means for identifying the user's emotions from the speech data using emotion analysis technology. This makes it possible to analyze the user's emotions in voice dialogue and provide appropriate responses or warnings. It also makes it possible to detect the possibility of fraudulent activity and quickly notify a designated external organization if the emotions exceed a certain threshold.

[0776] "Communication technology" refers to technology that receives voice information from users and automatically responds based on that information.

[0777] "Speech recognition technology" is a technology that converts received speech into text information and uses it for analysis.

[0778] "Natural language processing technology" is a technology that understands the requirements of the textual information analyzed from received speech and generates an appropriate response.

[0779] "Emotion analysis technology" is a technology that identifies and analyzes a user's emotions from voice data.

[0780] "Potential fraudulent activity" refers to a situation where there is a possibility of fraud or misconduct occurring in the content of a received communication.

[0781] An "external agency" is an external organization or facility established to notify information when emotional or risky communications exceeding specific criteria are detected.

[0782] One embodiment of this invention is a voice communication system that analyzes speech through interaction with a user and provides an emotionally appropriate response. The server uses speech recognition software to convert speech data into text information in real time. This is done using the Python speech_recognition library. The converted text information is analyzed using natural language processing techniques to generate a response.

[0783] The server simultaneously identifies emotions from the audio data using emotion analysis technology. This emotion analysis utilizes generative AI models to detect specific emotional states. For example, when it receives audio from a user expressing anxiety, it can quickly recognize that emotion and issue an alert to prompt appropriate action.

[0784] As a concrete example, if a user says, "I'm feeling a little anxious today," the server converts the audio into text data using speech recognition technology and understands the content through natural language processing. Simultaneously, it uses sentiment analysis technology to recognize the user's emotion of anxiety. Based on this, the server generates a response appropriate to the emotion and, for example, sends a notification to a caregiver. An example prompt might look like this: "Please analyze the emotions in the following conversation. If the user seems restless, please advise how the caregiver should respond. Conversation content: 'I didn't sleep well last night... I'm feeling a little anxious today.'"

[0785] This configuration allows voice communication systems to respond flexibly, taking into account the user's emotions. It enables rapid detection of anticipated risks and fraudulent activities, and allows for countermeasures such as notification to external organizations as needed.

[0786] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0787] Step 1:

[0788] The server receives voice information from the user using communication technology. The user's voice data is provided to the server as input. The server then passes this data directly to the next process.

[0789] Step 2:

[0790] The server uses speech recognition software to convert received audio information into text. The input is audio data received from the user, and the output is text. This process converts the audio data into parseable text. Specifically, the Python `speech_recognition` library is used to automatically convert speech to text.

[0791] Step 3:

[0792] The server applies natural language processing techniques to the converted text information to analyze the user's requirements. The input is the text information obtained in step 2, and the output is the analyzed requirements. This process prepares the server to understand the user's intent and generate a specific response. The response content is then designed based on the generated requirements.

[0793] Step 4:

[0794] The server uses emotion analysis technology to identify the user's emotions from the received audio data. The input is again the original audio data, and the output is the identified emotional state. In this step, a generative AI model is used to analyze the emotions and detect specific emotions such as anxiety or relief. Based on the analysis results, the system is used to implement flexible responses that correspond to the emotions.

[0795] Step 5:

[0796] The server generates an appropriate notification and sends it to an external organization or designated device if the analyzed emotion meets certain criteria. The input is the emotional state obtained in step 4, and the output is the generated notification message. This process ensures that information is quickly shared with relevant parties when a specific emotional state is detected. Specifically, if the criteria are exceeded, the system triggers an alert and sends a real-time notification to the caregiver or designated external organization.

[0797] By performing these steps sequentially, an advanced voice communication system is realized that incorporates emotion analysis into voice dialogue, providing appropriate responses and warnings.

[0798] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0799] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0800] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0801] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0802] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0803] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0804] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0805] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0806] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0807] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0808] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0809] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0810] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0811] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0812] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0813] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0814] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0815] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0816] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0817] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0818] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0819] The following is further disclosed regarding the embodiments described above.

[0820] (Claim 1)

[0821] A means for a communication device to automatically respond when it receives a voice communication,

[0822] A means of converting received audio into text using speech recognition technology,

[0823] A means for analyzing the requirements of received speech using natural language processing technology and generating a response based on that analysis,

[0824] A means of detecting potential fraudulent activity and issuing warnings,

[0825] A system that includes means for notifying a designated external agency of detected suspicious calls.

[0826] (Claim 2)

[0827] The system according to claim 1, comprising means for transmitting the converted text to a designated device via a pre-configured communication network.

[0828] (Claim 3)

[0829] A means of routing calls to the appropriate department or person based on user input,

[0830] The system according to claim 1, comprising means for storing call data in a database and enabling subsequent searching and analysis.

[0831] "Example 1"

[0832] (Claim 1)

[0833] A means by which a communication device detects an audio signal and automatically initiates a response,

[0834] A means of converting received audio into text using speech analysis technology,

[0835] A means for analyzing the content of received audio using language understanding technology and determining a response based on that analysis,

[0836] Means for identifying and notifying signs of fraudulent activity,

[0837] A system that includes means for alerting a designated external mechanism to an identified suspicious call.

[0838] (Claim 2)

[0839] The system according to claim 1, comprising means for transferring a converted string to a designated device via a pre-configured transmission network.

[0840] (Claim 3)

[0841] A means of distributing calls to the appropriate department or person in charge based on user input,

[0842] The system according to claim 1, comprising means for storing call information in a recording device and enabling subsequent retrieval and analysis.

[0843] "Application Example 1"

[0844] (Claim 1)

[0845] A means for a communication device to automatically respond when it receives a voice transmission,

[0846] A means for converting received audio into text information using speech recognition technology,

[0847] A means for analyzing the requirements of received speech using natural language processing technology and generating a response based on that analysis,

[0848] A means of detecting potential fraudulent activity and issuing warnings,

[0849] A means of notifying a designated external organization of detected suspicious transactions,

[0850] A means of verifying transaction details using voice input and identifying whether or not there are any abnormalities,

[0851] A means of issuing warnings to users based on identified anomalies and providing information to reconfirm transactions,

[0852] A system that includes means for storing audio data and text data in an information storage area, enabling later analysis and reference.

[0853] (Claim 2)

[0854] The system according to claim 1, comprising means for transmitting converted character information to a designated device via a pre-configured communication medium.

[0855] (Claim 3)

[0856] A means of routing calls to the appropriate operations department or staff member based on user input,

[0857] The system according to claim 1, comprising means for storing call data in an information base and enabling subsequent inspection and analysis.

[0858] "Example 2 of combining an emotion engine"

[0859] (Claim 1)

[0860] A means for a communication device to automatically respond when it receives a voice communication,

[0861] A means of converting received audio into text using speech recognition technology,

[0862] A means for analyzing the requirements of received speech using natural language processing technology and generating a response based on that analysis,

[0863] A method for analyzing voice data to identify the user's emotions,

[0864] A means of adjusting the quality of responses based on the results of emotion analysis,

[0865] A means of sending notifications to designated contacts in response to identified emotions,

[0866] A system that includes means for notifying a designated external agency of detected suspicious calls.

[0867] (Claim 2)

[0868] The system according to claim 1, comprising means for transmitting the converted text to a designated device via a pre-configured communication network.

[0869] (Claim 3)

[0870] A means of routing calls to the appropriate department or person based on user input,

[0871] The system according to claim 1, comprising means for storing call data, including analyzed emotional data, in a database and enabling subsequent retrieval and analysis.

[0872] "Application example 2 when combining with an emotional engine"

[0873] (Claim 1)

[0874] A means of automatically responding when communication technology receives voice information,

[0875] A means for converting received audio into text information using speech recognition technology,

[0876] A means for analyzing the requirements of received speech using natural language processing technology and generating a response based on that analysis,

[0877] A means of identifying a user's emotions from voice data using emotion analysis technology,

[0878] Based on the results of emotion analysis, means of issuing appropriate responses or warnings,

[0879] A means of detecting potential fraudulent activity and issuing warnings,

[0880] A system that includes a mechanism for notifying a designated external organization if an emotion identified through emotion analysis exceeds a certain threshold.

[0881] (Claim 2)

[0882] The system according to claim 1, comprising means for transmitting converted character information to a designated information processing device through a pre-configured information network.

[0883] (Claim 3)

[0884] A means for routing voice communications to the appropriate department or person in charge based on user input,

[0885] The system according to claim 1, comprising means for storing voice communication data in an information recording device and enabling subsequent retrieval and analysis. [Explanation of symbols]

[0886] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a communication device to automatically respond when it receives a voice transmission, A means for converting received audio into text information using speech recognition technology, A means for analyzing the requirements of received speech using natural language processing technology and generating a response based on that analysis, A means of detecting potential fraudulent activity and issuing warnings, A means of notifying a designated external organization of detected suspicious transactions, A means of verifying transaction details using voice input and identifying whether or not there are any abnormalities, A means of issuing warnings to users based on identified anomalies and providing information to reconfirm transactions, A system that includes means for storing audio data and text data in an information storage area, enabling later analysis and reference.

2. The system according to claim 1, further comprising means for transmitting converted character information to a designated device via a pre-configured communication medium.

3. A means of routing calls to the appropriate operations department or staff member based on user input, The system according to claim 1, comprising means for storing call data in an information base and enabling subsequent inspection and analysis.

Images